Don't Worry About the Vase Podcast On METR's AI Coding RCT
Jul 18, 2025
David Rhine from METR, who ran a randomized controlled trial on AI coding tools, joins Dwarkesh Patel to discuss the surprising results. They reveal that experienced developers were actually 19% slower with AI assistance. The conversation explores the paradox of AI coding tools, highlighting how factors like task complexity and personal coding styles affect productivity. They also address the need for more rigorous studies to understand the nuanced impacts and potential biases in evaluating AI's effectiveness in coding.
AI Snips
Chapters
Transcript
Episode notes
AI Coding Tools Slowdown Surprise
- A randomized controlled trial (RCT) showed that AI coding tools unexpectedly slowed down experienced developers on complex projects.
- This challenges assumptions about AI productivity gains and highlights the need for real-world testing.
Self-Reports Mislead on AI Speed
- Self-reports on AI's speed-up effects are unreliable; developers believed AI sped them up but actual data showed the opposite.
- Multiple analyses confirmed AI slowed tasks across different measures and subsets of data.
Why AI Struggled in This Setting
- Key factors in AI slowdown include high developer familiarity with repos, large complex codebases, low AI reliability, and implicit context AI can't capture.
- AI's underperformance in well-known projects highlights its struggle with tacit knowledge and custom standards.