[Linkpost] “METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” by Yadav

This is a link post.

This seems like an important piece of work - an RCT on the use of AI capabilities for developers. The TL;DR of the paper is

The devil is in the details of course. This post by Steve Newman does a good job of working through some of them.

I have highlighted some considerations from it:

The methodology was as follows:

1. METR recruited 16 developers from major open-source projects.
2. Each developer selected a list of coding tasks from their todo list, breaking up large projects into tasks that they could complete in an hour or two. In all, 246 tasks were included in the study.
3. The developers estimated how long it would take them to complete each task (a) under normal conditions, and (b) without using any AI tools. The percentage difference between these figures yields the predicted speedup – the degree to [...]

---

First published:
July 11th, 2025

Source:
https://forum.effectivealtruism.org/posts/rgBSYGBFMpKRpgHca/linkpost-metr-measuring-the-impact-of-early-2025-ai-on

Linkpost URL:
https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf

---

Narrated by TYPE III AUDIO.

---