Analyzing Low Overlap in Lives Saved and Skewed Risk Ratios

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are our Top Charities saving the same lives each year?, published by GiveWell on June 18, 2024 on The Effective Altruism Forum.
Author: Adam Salisbury, Senior Research Associate
In a nutshell
We've had a longstanding concern that some of our top charity programs, including insecticide-treated nets, seasonal malaria chemoprevention (SMC), and vitamin A supplementation (VAS), may have less impact than we've estimated due to "repetitive saving." These programs provide health interventions to the same children under 5 years old annually or every 3 years. Our cost-effectiveness models currently assume that different lives are saved each year from these interventions.
We think it's possible the programs are actually saving the same, high-risk children over and over. In a worst-case scenario, this could mean the programs are saving 80% fewer cumulative lives than we thought.
Based on a shallow review of empirical evidence and talking to experts, our best guess is that we're only overstating the total lives saved by these programs by around 10%, because:
Under-5 deaths from malaria, pneumonia, diarrhea, etc., are concentrated in the first year or two of life. Even if the mortality risk were concentrated among the same children each year, the impact comes from the initial couple of years of treatment, which leaves less scope for repetitive saving in later years.
The highest-risk group of children appears to vary somewhat randomly each year, reducing overlap. For example, malaria hotspots can shift over time.
This broad conclusion also matches the opinions of two epidemiology experts we consulted.
Our main uncertainties are:
What about survival after age 5? We focus primarily on survival through age 5, but it's possible children whose deaths we avert in the under-5 window have lower life expectancy beyond age 5. Though we've thought less about this, we think there are both empirical and intuitive reasons to expect longer-term survival benefits from these programs.
We also think accounting for life expectancy post-treatment raises difficult moral questions: we'd worry about implicitly weighing lives saved in more deprived contexts as less valuable than others.
How persistent are at-risk populations? Ideally, we'd directly identify and track the children saved by the programs to see their future risk. In reality, we have to use proxies like children discharged from the hospital after severe malaria. These may not be perfect stand-ins if they developed some immunity or have lingering complications.
Summary
What's the issue?
Within GiveWell's research team, there's been a long standing question: are we overstating the impact of our top charities by failing to account for "repetitive saving", or averting the same deaths each year? (more)
3 of the programs our top charities support - insecticide-treated net campaigns, seasonal malaria chemoprevention (SMC), and vitamin A supplementation (VAS) - provide health commodities repetitively to children under 5 (e.g., they provide a round of SMC to children in one region of Burkina Faso in 2025, then provide another round to those same children in 2026, and so on).
Our cost-effectiveness models of these programs make the (implicit) assumption that we're saving a different set of lives each time we provide treatment. What our SMC model says, in effect, is: "a subset of children are saved in year 1 and returned to average life expectancy; in year 2, a different subset of children are saved and returned to average life expectancy".
This could be wrong if we're saving the same set of children each year - for instance, if SMC only delays deaths by a year, rather than returning children to average life expectancy (as we assume).
In a worst-case scenario, the 5-year SMC program we support might be saving 80% fewer cumulative lives than what we est...