This week on Complex Systems, Patrick McKenzie (patio11) is joined by Joel Becker from METR. They discuss groundbreaking research on AI coding assistants.
Joel et al’s randomized controlled trial of 16 expert developers working on major open source projects revealed a counterintuitive finding: despite predictions of 24-40% speed improvements, developers actually took 19% longer to complete tasks when using AI tools, even though they retrospectively believed they were 20% faster. The conversation explores why even sophisticated professionals struggle to accurately assess their own productivity with AI tools, the industrial organization of software development, and the implications for AI's recursive self-improvement in research and development. It also touches on other perspectives from software developers using these tools professionally, and where we can expect them to improve rapidly.
–
Full transcript available here: www.complexsystemspodcast.com/the-great-developer-speed-up-with-joel-becker/
–
Sponsor:
This episode is brought to you by Mercury, the fintech trusted by 200K+ companies — from first milestones to running complex systems. Mercury offers banking that truly understands startups and scales with them. Start today at Mercury.com
Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group, Column N.A., and Evolve Bank & Trust; Members FDIC.
–
Recommended in this episode:
–
Timestamps:
(00:00) Intro
(00:34) Understanding AI evaluation methods
(02:04) METR's unique approach to AI evaluation
(03:10) The evolution of AI capabilities
(06:44) AI as coding assistants
(09:15) Research on AI's impact on developer productivity
(13:55) Sponsor: Mercury
(15:07) Challenges in measuring developer productivity
(20:38) Insights from the research paper
(31:26) The formalities of software development
(32:07) Automated tools and human discussions
(32:47) AI and style transfer in software
(34:35) The role of comments in AI coding
(36:51) The future of AI in software engineering
(40:25) Economic implications of AI in software
(46:53) Challenges and risks of AI in software
(59:03) Security concerns with AI-generated code
(01:04:59) Wrap
