LessWrong (Curated & Popular)

“METR: Measuring AI Ability to Complete Long Tasks” by Zach Stein-Perlman

Apr 7, 2025

Zach Stein-Perlman, author of a thought-provoking post on measuring AI task performance, discusses a groundbreaking metric for evaluating AI capabilities based on the length of tasks they can complete. He reveals that AI’s ability to tackle complex tasks has been doubling approximately every seven months for the last six years. The conversation highlights the implications of this rapid progress, the challenges AI still faces with longer tasks, and the urgency of preparing for a future where AI could autonomously handle significant work typically done by humans.

11:09

Episode guests

Zach Stein-Perlman

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

AI agents are demonstrating exponential growth in task completion length, significantly affecting future automation potential across industries.

Despite advancements, the correlation between AI model performance and real-world task success remains complex and requires further exploration.

Deep dives

Measuring AI Task Completion

The measurement of AI performance through the length of tasks it can complete has shown significant trends over recent years. This approach indicates that generalist AI agents are evolving to handle increasingly lengthy tasks autonomously, with performance doubling roughly every seven months. For instance, while current models can reliably manage tasks lasting under four minutes, they struggle with longer tasks, achieving less than 10% success on challenges that take human professionals more than four hours. Understanding these metrics helps clarify the relationship between AI capabilities and real-world utility, emphasizing the need for quantifiable benchmarks.

Intro

2min

Analyzing AI Performance in Relation to Human Task Completion

2min

Analyzing Model Success Rates in Long Task Completion

2min

Exponential Growth in AI Task Capability

2min

Understanding AI Progress Through Long Task Completion Metrics

4min

Summary: We propose measuring AI performance in terms of the length of tasks AI agents can complete. We show that this metric has been consistently exponentially increasing over the past 6 years, with a doubling time of around 7 months. Extrapolating this trend predicts that, in under five years, we will see AI agents that can independently complete a large fraction of software tasks that currently take humans days or weeks.

The length of tasks (measured by how long they take human professionals) that generalist frontier model agents can complete autonomously with 50% reliability has been doubling approximately every 7 months for the last 6 years. The shaded region represents 95% CI calculated by hierarchical bootstrap over task families, tasks, and task attempts.

Full paper | Github repo

We think that forecasting the capabilities of future AI systems is important for understanding and preparing for the impact of [...]

---

Outline:

(08:58) Conclusion

(09:59) Want to contribute?

---

First published:
March 19th, 2025

Source:
https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measuring-ai-ability-to-complete-long-tasks

---

Narrated by TYPE III AUDIO.

---

Images from the article: