LEEP developed its first cost-effectiveness analysis (CEA) in 2021, in which we estimated the expected cost per disability-adjusted life year (DALY)-equivalent averted of implementing our first paint program. As LEEP has expanded its programming over the past few years, we have outgrown this original CEA.

Our updated model is designed to estimate the impact of LEEP’s programming across 13 countries. The 13 countries included are those that were established programs at the time we began creating this model. As well as adding data for 12 additional countries, we updated the broader estimation strategy in the model in hopes of bringing more transparency to our approach and improving the accuracy of our final cost-effectiveness estimate.

For LEEP, the benefit of developing a CEA is three-fold. First, CEAs guide our decision making towards where we can have the highest impact, by highlighting what factors drive our cost-effectiveness. Second, CEAs increase the transparency of LEEP’s activity and impact, holding us accountable to funders, government and industry partners. And third, we hope that by explaining the structure, evidence base, and limitations of our model we can inform the decision making of other stakeholders working to reduce the global burden of lead poisoning.

$43.2 billion saved in lost earnings

across 13 programs, by 2100, projected

Measuring costs and impact

We define total costs in this model as the government and LEEP costs required to implement LEEP’s intervention. We measure total impact in our CEA using the disability-adjusted life year (DALY) metric. One DALY represents the equivalent of one year of healthy life lost. Since DALYs measure the loss of healthy life, we estimate total impact in terms of the total DALYs averted through LEEP’s intervention.

Figure 1 provides an overview of the estimation strategy we use in our multi-country CEA. To estimate the total health benefits actualised through LEEP’s intervention, we estimate the health benefits (in DALYs averted) that the average individual experiences as a result of averting lead paint exposure during childhood. Then, we multiply this per-person figure by the total number of individuals who avoid lead paint exposure as a result of LEEP’s intervention. We use a parallel approach to estimate total income benefits.

In order to calculate total benefits across the channels of health and income, we convert income benefits (in dollars) into units of “DALY-equivalents” using a moral weight. Following GiveWell’s standard, we assume that 2.5 years of lost income is equivalent to one DALY (i.e. one year of healthy life lost). Our estimate of total benefits (i.e. total discounted DALY-equivalents averted through intervention) is thus equal to the sum of total health DALYs and total income DALY-equivalents averted through LEEP’s intervention, relative to the counterfactual scenario in which LEEP does not intervene in program countries.

Figure 1: Overview of LEEP’s 2024 multi-country cost-effectiveness analysis

How cost-effective are LEEP’s lead paint elimination programs?

According to our updated CEA, it costs an average of $4.49 to avert one DALY-equivalent through LEEP’s programs, in expectation. The 90% confidence interval for this estimate is [0.78, 12.93].

One useful question we can ask is: how does the cost-effectiveness of LEEP’s paint prevention program compare to the cost-effectiveness of giving money directly to people who are living in poverty? To answer this question, we can apply a unit conversion to a GiveWell CEA of GiveDirectly, an organisation that gives cash directly to people living in poverty. From this we calculate that it costs $746 to avert one DALY-equivalent through the programs of GiveDirectly. While a direct comparison may not be appropriate due to the fact that GiveDirectly’s program is much larger in its scale than LEEP’s, and has been studied much more extensively, we do find that this gives us reason to believe that LEEP’s program is relatively cost-effective in the broader context of global health and development interventions.

What else do these results tell us?

Our model also allows us to estimate absolute impacts and costs associated with LEEP’s programs. Overall, we predict that between 2021 and 2100, LEEP’s 13 programs will avert a total of 900,000 DALY-equivalents. The total income benefit of LEEP’s activity over this time period will be approximately $43.2 billion, which is equal to 756,000 DALY-equivalents. The model predicts 145,000 DALYs are averted through the health benefits that result from LEEP’s paint program. In total, we estimate that about 84% of the program’s impact is attributed to income benefits, whereas 16% is attributed to health benefits (see Figure 2).

We also find that in total, the 13 programs included in the analysis will prevent about 414 million homes from being painted with lead paint (Figure 3), potentially preventing 46.6 million people from being exposed to lead paint in their homes during childhood (Figure 4). Lastly, we project total costs (including both charity costs and government costs) to be around $4 million.

Where could our model go wrong?

Our estimate of cost-effectiveness is only valuable insofar as it enables LEEP and other stakeholders to make better decisions about how to allocate resources to improve lives. In order for our CEA to accurately inform decisionmaking, we need to be clear about the precise information that our cost-effectiveness estimate conveys to us and its limitations. 

Our CEA specifically estimates the cost per DALY-equivalent averted through LEEP’s 13 paint programs in particular. It does not factor in other impacts that LEEP might have through its non-paint-related efforts, such as LEEP’s research into other sources of lead exposure and LEEP’s advocacy for a global evidence-based agenda to address childhood lead poisoning. The costs, but not the potential impact of these initiatives, are accounted for within our CEA.

Our analysis is also limited in the scope of costs and benefits that we consider. In terms of benefits, we only consider the harms of lead that occur through the channels of health (specifically via cardiovascular disease, kidney disease, and intellectual disability) and income. However, there are other pathways through which lead negatively affects wellbeing. For example, lead exposure has also been linked to many other physical health problems, as well as higher rates of mental health disorders, behavioural issues, and violence. In terms of costs, we do not consider the counterfactual impact of LEEP time and funding within our model. It’s very possible that the funding allocated toward LEEP’s paint programs would have otherwise gone to other high-impact initiatives – yet this possibility is not accounted for within our model. 

CEAs are simplifications of reality that necessarily include value judgements, uncertain input values, and error (either human-related or due to poor information quality). Like any modelling exercise, they involve uncertainties induced by the adoption of input estimates whose true values are unknown. Some of these inputs, such as when enforcement/compliance will begin, are unknown because for most countries in our model they have not yet occurred. Importantly, this makes our model mostly predictive, rather than based on results that have already been achieved.  We conducted research in order to decide upon values and associated intuitive confidence intervals (CIs) for each estimate, but some level of uncertainty remains inevitable. For more information on how we decided on the values of these inputs, please see the model explainer. 

In order to identify the inputs that our model is most sensitive to, we conducted a formal sensitivity analysis. This is a procedure that addresses the question: how much does the value of each input impact the value of our final cost-effectiveness estimate? Our sensitivity analysis showed that the following inputs had the largest impacts on the model output:

More detailed information on how we estimated the increase in blood lead level (BLL) from living in a home with lead paint can be found here.

It is important to note that the confidence intervals and sensitivity analysis attempt to capture the uncertainty in the output as modelled by this CEA. However, there is additional uncertainty associated with the structure and logic of the model, and the extent to which it reflects reality, which is not captured in the confidence intervals of the inputs or the final output.

There are also limitations associated with the use of CEAs as a decision-making tool more broadly. We recognize that CEAs are just one of many methods that can be used to evaluate the effectiveness of an intervention, and may have broader limitations such as biassing the user towards interventions with more measurable results. We believe that it is important to consider the results of CEAs alongside assessments using other methods to allow for the most holistic evaluation of a given intervention.

Note: We use terminology “intuitive confidence interval” to reflect the fact that these confidence intervals are not formal confidence intervals as delineated by statistical models of parameter distributions. The parameters used to define these “intuitive” confidence intervals come from best guesses that are based on our team’s intuitive sense of uncertainty about the value of each particular input, and should not be used to draw conclusions about the statistical significance of a given estimate. 

Summary

This predictive modelling exercise is inherently quite uncertain, and several inputs to the model have wide confidence intervals. Nevertheless, the estimated cost-effectiveness, at less than $5 per DALY-equivalent averted, makes LEEP’s programs extremely promising among other global health and development interventions that have been similarly evaluated. This renews our confidence in our ability to achieve our goal as an organisation to improve people’s lives around the world as much as possible. Further, this analysis suggests LEEP’s programs may be a good fit for those donors who want to help others as much as possible with their resources. We will continue to revise this estimate as we gain new information, and remain open to modifying our activities in response to changes in the estimate. For now, we hope that through the ongoing pursuit of our mission to eliminate childhood lead poisoning, many more children will have the opportunity to live healthy, lead-free lives.