Do policies built around social and behavioral science research actually work? That’s a big, and contentious, question. It’s also almost an existential question for the disciplines involved. It’s also a question that Megan Stevenson, a professor of law and of economics at the University of Virginia School of Law, grapples with as she explores how well randomized control trials can predict the real-world efficacy of interventions in criminal justice. What she’s found so far in that particular niche has echoed across the research establishment.

As she writes in the abstract of an article she saw published in the Boston University Law Review:

"This Essay is built around a central empirical claim: that most reforms and interventions in the criminal legal space are shown to have little lasting effect when evaluated with gold standard methods. While this might be disappointing from the perspective of someone hoping to learn what levers to pull to achieve change, I argue that this teaches us something valuable about the structure of the social world. When it comes to the type of limited-scope interventions that lend themselves to high-quality evaluation, social change is hard to engineer. Stabilizing forces push people back towards the path they would have been on absent the intervention. Cascades—small interventions that lead to large and lasting changes—are rare. And causal processes are complex and context-dependent, meaning that a success achieved in one setting may not port well to another."

In this Social Science Bites podcast, Stevenson tells interviewer David Edmonds that “the paper is not saying ‘nothing works ever.’ It’s saying nothing works among this subset of interventions, and interventions, as we talked about, are the type of interventions that get studied by randomized control trials tend to be pretty limited in scope. You can randomly allocate money, but you can’t randomly allocate class or socioeconomic status.”

Despite this cautionary finding in her research. Stevenson hasn’t despaired about her career choice or that of other social and behavioral scientists. “Many of us are in this line of work because we care about the world,” she notes. “We want to make the world a better place. We want to think about the best way to do it. And this is valuable information along that path. It’s valuable information in that it shuts some doors. … So keep trying other doors, keep experimenting.”