Scaling Success: Theory of Change

If someone said a global health and development programme was sustainable, participatory, and holistic, you'd have to guess that they were saying something positive. But according to today's guest Karen Levy — deworming pioneer and veteran of Innovations for Poverty Action, Evidence Action, and Y Combinator — each of those three concepts has become so fashionable that they're at risk of being seriously overrated and applied where they don't belong.

Rebroadcast: this episode was originally released in March 2022.

Links to learn more, highlights, and full transcript.

Such concepts might even cause harm — trying to make a project embody all three is as likely to ruin it as help it flourish.

First, what do people mean by 'sustainability'? Usually they mean something like the programme will eventually be able to continue without needing further financial support from the donor. But how is that possible? Governments, nonprofits, and aid agencies aim to provide health services, education, infrastructure, financial services, and so on — and all of these require ongoing funding to pay for materials and staff to keep them running.

Given that someone needs to keep paying, Karen tells us that in practice, 'sustainability' is usually a euphemism for the programme at some point being passed on to someone else to fund — usually the national government. And while that can be fine, the national government of Kenya only spends $400 per person to provide each and every government service — just 2% of what the US spends on each resident. Incredibly tight budgets like that are typical of low-income countries.

'Participatory' also sounds nice, and inasmuch as it means leaders are accountable to the people they're trying to help, it probably is. But Karen tells us that in the field, ‘participatory’ usually means that recipients are expected to be involved in planning and delivering services themselves.

While that might be suitable in some situations, it's hardly something people in rich countries always want for themselves. Ideally we want government healthcare and education to be high quality without us having to attend meetings to keep it on track — and people in poor countries have as many or more pressures on their time. While accountability is desirable, an expectation of participation can be as much a burden as a blessing.

Finally, making a programme 'holistic' could be smart, but as Karen lays out, it also has some major downsides. For one, it means you're doing lots of things at once, which makes it hard to tell which parts of the project are making the biggest difference relative to their cost. For another, when you have a lot of goals at once, it's hard to tell whether you're making progress, or really put your mind to focusing on making one thing go extremely well. And finally, holistic programmes can be impractically expensive — Karen tells the story of a wonderful 'holistic school health' programme that, if continued, was going to cost 3.5 times the entire school's budget.

In this in-depth conversation, originally released in March 2022, Karen Levy and host Rob Wiblin chat about the above, as well as:

• Why it pays to figure out how you'll interpret the results of an experiment ahead of time
• The trouble with misaligned incentives within the development industry
• Projects that don't deliver value for money and should be scaled down
• How Karen accidentally became a leading figure in the push to deworm tens of millions of schoolchildren
• Logistical challenges in reaching huge numbers of people with essential services
• Lessons from Karen's many-decades career
• And much more

Chapters:

• Cold open (00:00:00)
• Rob's intro (00:01:33)
• The interview begins (00:02:21)
• Funding for effective altruist–mentality development projects (00:04:59)
• Pre-policy plans (00:08:36)
• ‘Sustainability’, and other myths in typical international development practice (00:21:37)
• ‘Participatoriness’ (00:36:20)
• ‘Holistic approaches’ (00:40:20)
• How the development industry sees evidence-based development (00:51:31)
• Initiatives in Africa that should be significantly curtailed (00:56:30)
• Misaligned incentives within the development industry (01:05:46)
• Deworming: the early days (01:21:09)
• The problem of deworming (01:34:27)
• Deworm the World (01:45:43)
• Where the majority of the work was happening (01:55:38)
• Logistical issues (02:20:41)
• The importance of a theory of change (02:31:46)
• Ways that things have changed since 2006 (02:36:07)
• Academic work vs policy work (02:38:33)
• Fit for Purpose (02:43:40)
• Living in Kenya (03:00:32)
• Underrated life advice (03:05:29)
• Rob’s outro (03:09:18)

Producer: Keiran Harris
Audio mastering: Ben Cordell and Ryan Kessler
Transcriptions: Katy Moore