Our approach to charity

Related research

How we think about charity evaluation

Summary

This post outlines the principles that guide our research and our research methodology. The extended version is available below.

Our goals

At Founders Pledge, our goal is to empower entrepreneurs to do immense good. The first step in a member's journey is to take the pledge, the second step is to decide what to do with your money. Our research team is focused on this second step. By producing high-quality research and offering tailored guidance, we enable entrepreneurs not only to give more, but to give more effectively.

We identify the world’s most pressing problems and some of the most impactful charities working on those problems. Guided by the best research, we analyze in depth the results that charities produce - how much they improve the world for your marginal donation.

Key principles of effective giving

Our research process is designed to enable philanthropists to maximize their impact. To achieve this, we are guided by three core principles.

1. Problem area choice is crucial

One of the most fundamental ways to increase your impact is flexibility about problem area selection. Our research has found that money can continue to flow into problems long after they have become saturated, while other highly pressing causes continue to receive little attention. Thus, donors can increase their impact enormously by carefully choosing which problem area to work on.

2. Impact varies dramatically between different organizations

There are also large differences in impact within different problem areas. Solutions to tackle social problems have varying success rates, and often the most intuitive solution is not the most effective one. Indeed, the gap between the best and the rest may be especially large for higher-risk "hits-based giving": a risk-tolerant approach to philanthropy similar to VC investing, on which a small number of enormous successes account for a large share of the total impact and compensate for a large number of failed projects.

The gap between the best and the rest is so large that you can have a huge impact multiplier by strategically targeting the best interventions: a $1 million philanthropist can, in effect, become a $100 million philanthropist by following the research and focusing accordingly.

3. Careful research is needed to find the best opportunities

In the charity sector, careful research is difficult because charities are ultimately answerable to their donors rather than to the people they are supposed to serve. Consequently, the organizations that thrive are not necessarily those doing the best work, but rather may be those that are good at marketing and fundraising.

Moreover, research suggests that donors put much less thought into their charitable giving than they do into their business or investment decisions. The crucial feedback loop from customer satisfaction to organizational success, present in well-functioning markets, is much weaker in the charity sector. This is why independent and careful charity research is such a valuable part of the charity ecosystem.

Our methodology

While many evaluators carry out basic due diligence on the financial accounts and overheads of charities, we go a step further and evaluate the impact of organizations, with the aim of identifying some of the most effective charities in the world. We do this using a problem-first approach, which has three stages, based on the principles covered above:

Identify the world’s most pressing problems.
Prioritize solutions to these problems.
Identify highly effective organizations implementing these solutions.

The problem-first approach allows us to filter down to highly impactful organizations without reviewing all charities working on a problem.

We spend longer on a project than most charity evaluators: from initial cause area scoping to final recommendations, each of our reports takes between two and six months to complete, depending on the breadth of the topic.

1. Identifying the world’s most pressing problems

At Founders Pledge, our aim is to empower entrepreneurs to do immense good by solving some of the world’s most pressing problems. We can help to choose between problems by evaluating what we call key impact indicators, including:

Scale of the problem
Neglectedness of the problem i.e. is it overlooked by other funders
Level of coordination required
Political barriers to success

Our research has shown that there are large differences between problems in terms of these factors.

2. Prioritizing Solutions

After selecting pressing problems, we then move to the next stage: choosing solutions. We saw above that intuition is a poor guide to impact: some intuitively promising interventions do not work, or have much lower impact than others. This is why we have to use careful research to select solutions to social problems. Our evaluation methods differ depending on whether it is a testable intervention (such as providing vaccines), or a more hits-based approach (such as research or policy advocacy). (Learn more).

3. Finding charities

In the final stage of our research process, we search for organizations that implement our priority solutions, and filter them down according to the following criteria.

Track record
Strength of team
Room for more funding
Strength of future projects
Evidence of cost-effectiveness
Due diligence concerns, including transparency

We usually spend around 20 days of research time evaluating organizations that we eventually recommend. Our evaluation process differs between charities carrying out testable interventions and those pursuing higher-risk hits-based approaches.

Member-Driven Research

In order to provide the most value to our members, we may carry out research into problem areas in response to demand from the Founders Pledge community. There are a range of global and local causes that may not fit within our global priorities work, perhaps because they affect a relatively smaller number of people than other issues. Member-driven research projects follow the same methodology described above: identify the most promising solutions and then search for charities working on those solutions.

Research Partners

We are fortunate to be able to rely on a number of research partners who allow us to devote our in-house expertise to filling gaps left by other evaluators, rather than duplicating work. At present, our research partners are GiveWell and Farmed Animal Funders.

GiveWell conducts highly rigorous and internationally respected research into global health and development.
Farmed Animal Funders is the world’s largest collaborative of major funders aiming to end factory farming. They share a similar view on the best way to make progress on factory farming to us and conduct respected research.

We also have more informal expertise-sharing relationships with other major philanthropists and funder collaboratives.

Limitations of our approach

We pride ourselves on investigating some of the big questions in philanthropy, both practical and philosophical, but we do not have all the answers. Research on philanthropic impact is a relatively new field, and is always evolving. We always strive to highlight uncertainties in our analysis, to continuously improve our research methodology, and to keep track of new methodologies and insights as they emerge.

Extended version

Our Goals

We identify the world’s most pressing problems and some of the most impactful charities working on those problems. Our analysis extends far beyond the basic due diligence of assessing a charity’s financial health. Instead, we analyze in depth the results that charities produce - how much they improve the world - guided by the best research.

Today, the “philanthropic market” is not efficient: some of the most pressing problems remain neglected, and some of the best organizations do not get funded. This means that strategic philanthropists can hugely increase their impact by carefully choosing which problems to focus on, and supporting the most effective organizations working to solve those problems.

In this post, we outline the core beliefs that guide our research and the methodology we use to identify the very best funding opportunities for our members.

Key principles of effective giving

At Founders Pledge, we believe that philanthropists have an incredible opportunity to do good through their charitable giving. We recognize that in a perfect world, societies would not need to rely on the goodwill of philanthropists.

Unfortunately, sometimes governments and the private sector lack the incentives or the capacity to provide basic services such as healthcare or a functioning education system. Other challenges, such as protecting animals or the long-term safety of our planet, involve beneficiaries with no political voice and no economic power, so their interests tend to be ignored by governments and markets. Careful philanthropy can fill these gaps.

Our research process is designed to enable philanthropists to maximize their impact. To achieve this, we are guided by three core principles.

1. Problem area choice is crucial

One of the most fundamental ways to increase your impact is flexibility about problem area selection. A common view in the charity sector is that it’s impossible to compare problem areas. However, our research has found that there are large differences between many different problem areas, which drive large differences in the impact of work aiming to solve them. Money can continue to flow into problems long after they have become saturated, while other highly pressing causes continue to receive little attention.

Indeed, many of the most pressing problems are off the radar of the typical philanthropist. For example, at the start of 2019 we released a report on biosecurity and pandemic preparedness that recommended two organizations working to reduce the risk of global pandemics. At that time, few philanthropists prioritized this area, and it received far fewer resources than other areas such as education and climate change. The COVID-19 pandemic has shown the huge importance of pandemic preparedness, and how the allocation of philanthropic and public resources was far from optimal. This illustrates the value of keeping an open mind about what problems you want to work on.

Ultimately, flexibility about problem selection allows you to better optimize for your core values. Whether your aim is to reduce poverty, to improve animal welfare or to safeguard the future of humanity, you can greatly increase your impact by choosing a problem area carefully. The burgeoning field of global priorities research, to which Founders Pledge contributes, enables our members to navigate these questions. We discuss this in more detail in our methodology section.

We have completed dozens of high-level reviews of problem areas, and we work with our members to identify problem areas to focus on, based on their values.

2. Impact varies dramatically between different interventions and organizations

We have seen that between problem areas, there are often huge differences in the impact of the available solutions. But there are also large differences in impact within different problem areas. Our research has shown that solutions to tackle social problems have varying success rates and often the most intuitive solution is not the most effective one. Indeed, charities and philanthropists may not always produce the outcome they’re hoping to, even when driven by the best of intentions. Because of this, it is important to consider all the available evidence and rigorously evaluate a charity and its activities, rather than following our intuition.

Take as an illustration education interventions. Before looking at the chart below, ask yourself which of the following programs you would expect to have the greatest impact on educational attainment:

Reducing class size
Providing textbooks
Building libraries
Minimum conditional cash transfers
Merit scholarships for girls
Remedial education
Teacher incentives
Providing earnings information
Elected school committees linked to local government

When these interventions have been tested using rigorous methods, the impact on educational attainment has been as follows (though the results vary by context):

Figure 1. Standard deviation improvement in test scores per $100 Education effect Source: JPAL, Conducting Cost-Effectiveness Analysis

Intuitively promising interventions such as reducing class size and providing textbooks had no effect on learning outcomes. Moreover, among interventions that did work, ex post the most successful was 1,000x more impactful than the least successful. If we adjust for noisy data and the fact that the impact of interventions varies across context, true ex ante differences in impact will be smaller, but we still think they're significant.

We have found this pattern repeated again and again across different problem areas, such as global health and climate change. Indeed, the gap between the best and the rest may be even greater for higher-risk “hits-based giving”: a risk-tolerant approach to philanthropy similar to VC investing, on which a small number of enormous successes account for a large share of the total impact — and compensate for a large number of failed projects.

The contraceptive pill is a good illustration of this. Governments and the pharmaceutical industry were too cautious to fund research for the contraceptive pill, so the philanthropist Katherine McCormick stepped in to provide all of the early stage research funding. This was a high-risk decision which ultimately produced an immense social return on investment.

Similarly, the Rockefeller Foundation invested in research on improving agricultural productivity in the developing world, which is now commonly believed to have been the catalyst for the Green Revolution, which has likely saved hundreds of millions of lives.

This illustrates the value of intense focus on the very best solutions. The gap between the best and the rest is so large that you can have a huge impact multiplier by strategically targeting the best interventions: a $1 million philanthropist can, in effect, become a $100 million philanthropist by following the research and focusing accordingly.

3. Careful research is needed to find the highest-leverage opportunities

In the charity sector, careful research is both difficult and valuable. It is difficult because charities are ultimately answerable to their donors rather than to the people they are supposed to serve. Consequently, the organizations that thrive are not necessarily those doing the best work, but rather may be those that are good at marketing and fundraising.

Moreover, research suggests that donors put much less thought into their charitable giving than they do into their business or investment decisions. As a result, money can continue to flow to organizations even if they fail to meet their mission year after year. Donors also often lack the time and resources to carry out in-depth evaluations of charities working on highly complex problems. Thus, the crucial feedback loop from customer satisfaction to organizational success, present in well-functioning markets, is much weaker in the charity sector.

This is why independent and careful charity research is such a valuable part of the charity ecosystem. At Founders Pledge we have a research team who specialize in exactly that. We spend many months reviewing particular cause areas and charities, and consult with experts so that our members can confidently make intelligent, informed decisions about how to spend their money.

Our methodology

Identify the world’s most pressing problems.
Prioritize solutions to these problems.
Identify highly effective organizations implementing these solutions.

The problem-first approach allows us to filter down to highly impactful organizations without reviewing all charities working on a problem.

For example, for our research on climate change, we selected promoting innovation in neglected low-carbon technologies as one of our priority solutions. We then moved on to find and evaluate nonprofits that were working on this solution. This allowed us to find outstanding funding opportunities without having to review hundreds of nonprofits. This means that we may sometimes miss out on some excellent organizations, but it nevertheless enables us to identify outstanding organizations in an efficient way.

1. Identifying the world’s most pressing problems

At Founders Pledge, our aim is to empower entrepreneurs to do immense good by solving some of the world’s most pressing problems. More technically, we want to find cost-effective funding opportunities: that is, we want to maximize social impact per dollar. This can be measured in terms of any impact metric that you care about - educational improvements, lives saved, tonnes of CO2 averted, and so on.

Sometimes, we may already have information on the cost-effectiveness of work on different problem areas. For example, our research partner GiveWell estimates that it costs on the order of $3,000 to save the life of a child via their top recommended malaria charities. To put this in context, the British National Health Service considers it cost-effective to save a year of healthy life for £20,000.

Such large differences in cost-effectiveness are in turn driven by differences between problems in terms of key impact indicators, such as:

Scale of the problem
Neglectedness of the problem i.e. is it overlooked by other funders
Level of coordination required
Political barriers to success
Problem complexity
Whether solving the problem frees up resources for other problems

When we are assessing problems, we look at key impact indicators and other factors to understand how promising work on a problem might be. Our research has shown that, as we might expect, there are large differences between problems in terms of these factors. To illustrate, consider the examples of scale and neglectedness.

Scale and neglectedness

It is important to consider scale when looking for the causes where our money can make the biggest difference. Just as entrepreneurs will consider the size of the addressable market that can be served by a product, we want to focus on larger problems, other things equal.

Suppose that our aim is to reduce the risks of a global catastrophe. Expert estimates suggest that there is around a 1 in 1 million chance of a civilization-threatening asteroid or comet hitting Earth this century. This is at least 1,000x lower than the risk posed by climate change or engineered pandemics.^¹ This suggests that donors can have much greater impact by working on these problems than on asteroid defense: the difference in scale, other things equal, will drive large differences in the impact of donations.

Due to the psychological bias of scope insensitivity, it is easy to miss these large differences. Our emotional reaction to finding out that a problem kills 1 million people or 100 million people is similar, and yet these tragedies call for very different social responses.

Philanthropists themselves are not immune to this and other biases, which can lead to huge problems being left neglected. This is important because there often comes a point at which further investment in a problem starts to produce diminishing marginal returns: it often pays to work on less crowded problems, where your marginal dollar will go further. Consider Figure 2:

Figure 2. The scale and neglectedness of HIV and mental health HIV mental health Source: Overseas Development Institute, ‘Mental Health Funding and the SDGs’ (2016)

Even though the disease burden of mental health issues is twice that of HIV, it receives 50x less funding in development assistance. One third of low- and middle-income countries do not have a designated budget for mental health, so cheap and effective mental health treatments that high-income countries routinely provide will be lacking in lower-income countries. This is an indication that there may be more “low-hanging fruit” for those who aim to improve mental health than for those who aim to tackle HIV.

Of course, scale and neglectedness are not the only determinants of impact, but they do indicate that the philanthropic market is far from efficient. This is one reason why there are such huge differences between problems in terms of cost-effectiveness.

2. Prioritizing Solutions

After selecting pressing problems, we then move to the next stage: choosing solutions. We saw above that intuition is a poor guide to impact: some intuitively promising interventions do not work, or have much lower impact than others. This is why we have to use careful research to select solutions to social problems. Our evaluation methods differ depending on whether it is a testable intervention (such as providing vaccines), or a hard-to-test intervention (such as research or policy advocacy).

Testable solutions

Some interventions, such as distributing vaccines, malaria nets or cash transfers, can be tested with more or less rigorous social scientific methods. Even so, evaluating testable solutions can be challenging.

Just because something has worked well in one setting does not mean it automatically translates well to other contexts. This makes it difficult to extrapolate from single studies, even when they have positive results. For instance, there are a myriad of social factors which may influence how well a poverty alleviation program might work, many of which are difficult to predict, even for experts in the field.

Because of this, we prefer systematic reviews and meta-analyses, which combine the results of several high-quality studies, over individual studies. These comprehensive reviews tell us whether a program is likely to work across contexts. Our research also involves extensive consultation with world-leading experts in order to make sure that we capture all the available knowledge.

A good example of our approach to evaluating direct and testable interventions is our research on education in low- and middle-income countries. After surveying the academic literature and consulting with experts, we prioritized two solutions. We found that teaching at the right level - teaching people according to their ability rather than to their age - has been shown to be effective in nine high-quality studies. More surprisingly, we found that fortifying salt with iodine is one of the best ways to improve educational performance. Around 246 million children are iodine deficient; iodine deficiency in children reduces IQ by around four points, and deficiency can be easily and cheaply treated by fortifying salt with iodine.

In this case - as in others - careful review of the academic literature led us to interventions that we would have been unlikely to support if we were guided by intuition or a cursory review of the evidence.

Hard-to-test solutions

When it comes to less direct interventions, such as scientific research or policy work, it is much more challenging to test different approaches with rigorous trials. We therefore have to rely on other types of evidence and arguments to make our decisions.

A good example of how to put this into practice is our research on climate change. There are dozens of promising ways of approaching climate change, none of which can be easily tested using rigorous studies. It turns out we can also use the key impact indicators we discussed above to prioritize solutions:

The scale of the benefits of different solutions
The neglectedness of solutions
The level of coordination required for success
Robustness to high future energy demand
Political barriers

Using these criteria, we prioritized innovation in neglected low-carbon technologies. Countries can unilaterally affect emissions in other countries by funding innovation in low-carbon technologies. For example, by funding solar power in the early 2000s, Germany unilaterally reduced the cost of solar for the world, which will likely have a huge effect on future emissions.^²

Innovation is unique among climate solutions in having leverage on global emissions while requiring limited coordination. Moreover, certain key technologies receive much more attention than others from philanthropists, governments and the private sector.

Figure 3. The scale (plausible contribution to emissions reduction up to 2050) and neglectedness of different climate solutions Climate interventions Source: Climate Report calculations sheet

Because some of these technologies are so neglected, the returns to working on them are likely to be higher on the margin.

In summary, because we could not rely on research trials, we instead used these and other key impact indicators, which led us to prioritize innovation in neglected low-carbon technologies. This kind of approach is generally useful when there is limited direct evidence to help guide prioritization.

3. Finding charities

In the final stage of our research process, we search for and filter organizations that implement our priority solutions. In education, we search for organizations implementing teaching at the right level or salt iodization; in climate, we search for organizations promoting innovation in neglected low-carbon technology, and so on.

We initially draw up a long-list of all charities we can find running the priority solutions recommendations from other experts, aligned philanthropists, and internet searches.

We then evaluate each long-listed charity according to six criteria:

Track record
Strength of team
Room for more funding
Strength of future projects
Evidence of cost-effectiveness
Due diligence concerns, including transparency

We shortlist the charities that score highly across all criteria (usually 3 to 10 organizations). These are then evaluated in more depth according to the same criteria. Because we go far beyond the limited due diligence carried out by many charity evaluators, we usually spend around 20 days of research time evaluating organizations that we eventually recommend. In all cases, we try to build models that roughly estimate the marginal impact that organizations have.

Our evaluation process differs between charities carrying out testable and less testable interventions.

Charities carrying out testable interventions

For charities running directly testable interventions, we will especially focus on the strength of their internal monitoring, their commitment to ongoing assessment, and how transparent they are about their findings. We also build more detailed cost-effectiveness models to compare charities.

We favor charities that have experience of delivering our preferred intervention, and publish impact evaluations of their previous work which show clear, measurable improvements on the desired measures. For instance, a charity carrying out interpersonal group therapy in Uganda would need to show that its participants are meaningfully better off as a result of the program. Organizations that we recommend typically have a demonstrable commitment to transparency and rigorous self-evaluation, identifying room for improvement whenever possible.

Charities carrying out less testable interventions

For charities carrying out less testable interventions, such as policy advocacy or scientific research, evaluation of their past success is much more difficult. For a hits-based approach to giving, charity evaluation is more like evaluating a portfolio of VC investments than (say) picking the correct medical treatment.

Firstly, with direct interventions, it is usually fairly easy to establish that a charity brought about a particular change. For example, we can know from monitoring data that a charity has distributed an antimalarial bednet. Establishing this for indirect work is much more challenging. Any change that is achieved will usually be the product of a broad coalition of actors, and it is difficult to disentangle who did what. Attributing impact in this context involves investigating the political context and gathering evidence and testimony from the key actors involved. To understand what difference the organization made, we need to assess some inherently uncertain counterfactuals - i.e. what would have happened if the organization had not existed.

Secondly, when evaluating the success of a campaign, we need more than a simple win/lose approach. Most advocacy campaigns bring policy change forwards, rather than create change that would otherwise never have happened. Moreover, policy could be repealed in the future. To reduce uncertainty about these counterfactuals, we need to understand the political landscape in-depth and the role that different actors play in it.

Having evaluated the past success of an organization, we then face the task of evaluating their future work. This raises unique challenges: success is sensitive to highly contextual windows of political opportunity, so past impact is not a very reliable guide to future performance. A good track record is therefore valuable largely as a signal of general organizational competence and ability to get things done. The way that an organization selects which projects to work on is especially important given fluid political contexts.

Due to the uncertainty inherent in this kind of assessment, identifying high-risk organizations is more akin to venture capitalism than it is to finding well-tested direct programs. Notwithstanding the challenges of evaluation, with hits-based giving the potential gains of a few big wins can compensate for other failures.

Member-Driven Research

In order to provide the most value to our members, we may carry out research into problem areas in response to demand from the Founders Pledge community. There are a range of global and local causes that may not fit within our global priorities work, because perhaps they affect a relatively smaller number of people than other issues, but are of great concern to our members. If there is a large number of members interested in working in an area, we may conduct research into it.

Member-driven research projects follow the same methodology described above: identify the most promising solutions and then search for charities working on those solutions.

Research Partners

GiveWell conducts highly rigorous and internationally respected research into global health and development
Farmed Animal Funders is the world’s largest collaborative of major funders aiming to end factory farming. They share a similar view on the best way to make progress on factory farming to us and conduct respected research

We also have more informal expertise-sharing relationships with other major philanthropists and funder collaboratives.

Limitations of our approach

Notes

Toby Ord, The Precipice: Existential Risk and the Future of Humanity (Bloomsbury Publishing, 2020), 167. ↩
Goksin Kavlak, James McNerney, and Jessika E. Trancik, “Evaluating the Causes of Cost Reduction in Photovoltaic Modules,” Energy Policy 123 (December 1, 2018): 700–710, https://doi.org/10.1016/j.enpol.2018.08.015. ↩