If we have a study on the impact of a social program in a particular place and time, how confident can we be that we’ll get a similar result if we study the same program again somewhere else?
Dr Eva Vivalt is a lecturer in the Research School of Economics at the Australian National University. She compiled a huge database of impact evaluations in global development - including 15,024 estimates from 635 papers across 20 types of intervention - to help answer this question.
Her finding: not confident at all.
The typical study result differs from the average effect found in similar studies so far by almost 100%. That is to say, if all existing studies of a particular education program find that it improves test scores by 10 points - the next result is as likely to be negative or greater than 20 points, as it is to be between 0-20 points.
She also observed that results from smaller studies done with an NGO - often pilot studies - were more likely to look promising. But when governments tried to implement scaled-up versions of those programs, their performance would drop considerably.
For researchers hoping to figure out what works and then take those programs global, these failures of generalizability and ‘external validity’ should be disconcerting.
Is ‘evidence-based development’ writing a cheque its methodology can’t cash? Should this make us invest less in empirical research, or more to get actually reliable results?
Or as some critics say, is interest in impact evaluation distracting us from more important issues, like national or macroeconomic reforms that can’t be easily trialled?
We discuss this as well as Eva’s other research, including Y Combinator’s basic income study where she is a principal investigator.
Full transcript, links to related papers, and highlights from the conversation.
Links mentioned at the start of the show:
* 80,000 Hours Job Board
* 2018 Effective Altruism Survey
**Get this episode by subscribing to our podcast on the world’s most pressing problems and how to solve them: type *80,000 Hours* into your podcasting app.**
Questions include:
* What is the YC basic income study looking at, and what motivates it?
* How do we get people to accept clean meat?
* How much can we generalize from impact evaluations?
* How much can we generalize from studies in development economics?
* Should we be running more or fewer studies?
* Do most social programs work or not?
* The academic incentives around data aggregation
* How much can impact evaluations inform policy decisions?
* How often do people change their minds?
* Do policy makers update too much or too little in the real world?
* How good or bad are the predictions of experts? How does that change when looking at individuals versus the average of a group?
* How often should we believe positive results?
* What’s the state of development economics?
* Eva’s thoughts on our article on social interventions
* How much can we really learn from being empirical?
* How much should we really value RCTs?
* Is an Economics PhD overrated or underrated?
Get this episode by subscribing to our podcast: search for '80,000 Hours' in your podcasting app.
The 80,000 Hours Podcast is produced by Keiran Harris.
