What's the Value of a B Testing in Text?

From the earliest days of economics as a formal science, economists have been trying to understand the causes of the wealth of nations. In the field of development economics, the introduction of the randomized controlled trial has become an important tool in the broader toolbox of economists for trying to understand the many things that may cause the improvement of human welfare in lower income countries, too. This work has been ground breaking and recognized by the Nobel committee for its lasting importance. Like many others, I have been inspired by the development economists relentless effort to address poverty through rigorous causal inference and program evaluation.

While most people know that randomization is an important ingredient in causal inference, what is not as widely known is the Stable Unit Treatment Value Assumption, or SUTVA for short. We know that randomized treatments can eliminate selection bias and thus allow us to use realized outcomes in place of potential outcome. SUTVA requires that the potential outcomes themselves be stable and unchanging when treatment assignments of other units change, which brings to mind complex problems with externalities between people and interference when designing experiments, but as Imbens and Rubin note in their 2015 book, SUTVA also requires that treatments not vary unknowingly across units. Such “hidden variation in treatment” can make causal interpretation difficult if not possible.

But SUTVA also brings to mind the problems of external validity. When someone reading of a study’s large gains in a field experiment, they might then decide to roll out such a program at large scale. Assume for simplicity constant treatment effects for a moment — can they expect the same thing to happen in their community what happened in this particular trial? They can insofar as they do not inadvertently change the treatment itself by unknowingly varying the inputs in important ways. If the RCT found large literacy gains using a particular type worker, but to bring it to scale, the policymaker foregoes using those same inputs, then it becomes a new empirical question as to whether the effect found in the lab will in fact scale, even with constant treatment effects. But we have known this for centuries. Concepts like production functions and cost functions directly speak to these very things.

Successful policy requires evidence but also a set of skills that maybe aren’t there when designing or evaluating an RCT. It requires both cognitive skill, and perhaps even moreso non cognitive skill, as often the economist then must wear both a scientist hat, a manager hat, and an entrepreneur hat. And not every economist has those skills, or maybe even is interested in venturing into the messy world of building socially impactful policy. But I have also been inspired by a small group of applied development economists like Noam Angrist, Co-founder and Executive Director of Youth Impact (formerly Young 1ove), and Paul Niehaus at GiveDirectly, who create organizations that try to bring effective programs to larger scale while simultaneously committing themselves to constant evaluation of themselves and their programs. Whether it’s a trend or not, I don’t know, but I have been intrigued.

This week on The Mixtape with Scott, I have the pleasure of interviewing one of these economist entrepreneurs, Noam Angrist. Noam is, as I said, the co-founder and executive director of Youth Impact, a non-profit focused on improving the welfare of young people around the world through education and health programs. I met him for the first time several years ago because I noticed what he was doing and had done and wanted to introduce him to my students. So I wrote and asked him if he would be willing to be the de facto keynote speaker at a conference I was helping organize on causal inference. He graciously agreed and spoke with us about the organization he had helped found and the work they were doing in developing countries scaling rigorously evaluated interventions to reach thousands of youth around the world. I was very intrigued because of the blend of causal inference and economics with such creative entrepreneurial work.

Given the explosive success of the credibility revolution at changing hearts and minds, I suppose it was only a matter of time before economist-entrepreneurs trained in that way of thinking would begin moving outside of academic departments and into other parts of the world. We have seen it with economists in tech. Noam Angrist is an example of the contemporary economist-entrepreneur who works closely with governments, academia and even commerce to bring the best of all things to help achieve the age-old quest of economics of improving the well being of humans alive today.

In this interview, Noam and I talked about growing up in Massachusetts and the looking back serendipitous injury in high school that put him on a path to studying economics at MIT. He’s since embarked on an original career as an economist where his competency as a leader, his creativity and curiosity as a scientist and his innate commitment to public service and community is shaping the type of work and the way that work is done at Youth Impact. I hope you enjoy this interview as much as I did. Please follow, subscribe and consider supporting The Mixtape with Scott!

Scott's Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.