Held-out tasks and domain generalization

When METR says something like "Claude Opus 4.5 has a 50% time horizon of 4 hours and 50 minutes", what does that mean? In this episode David Rein, METR researcher and co-author of the paper "Measuring AI ability to complete long tasks", talks about METR's work on measuring time horizons, the methodology behind those numbers, and what work remains to be done in this domain.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

Transcript: https://axrp.net/episode/2026/01/03/episode-47-david-rein-metr-time-horizons.html

Topics we discuss, and timestamps:

0:00:32 Measuring AI Ability to Complete Long Tasks

0:10:54 The meaning of "task length"

0:19:27 Examples of intermediate and hard tasks

0:25:12 Why the software engineering focus

0:32:17 Why task length as difficulty measure

0:46:32 Is AI progress going superexponential?

0:50:58 Is AI progress due to increased cost to run models?

0:54:45 Why METR measures model capabilities

1:04:10 How time horizons relate to recursive self-improvement

1:12:58 Cost of estimating time horizons

1:16:23 Task realism vs mimicking important task features

1:19:50 Excursus on "Inventing Temperature"

1:25:46 Return to task realism discussion

1:33:53 Open questions on time horizons

Links for METR:

Main website: https://metr.org/

X/Twitter account: https://x.com/METR_Evals/

Research we discuss:

Measuring AI Ability to Complete Long Tasks: https://arxiv.org/abs/2503.14499

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts: https://arxiv.org/abs/2411.15114

HCAST: Human-Calibrated Autonomy Software Tasks: https://arxiv.org/abs/2503.17354

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity: https://arxiv.org/abs/2507.09089

Anthropic Economic Index: Tracking AI's role in the US and global economy: https://www.anthropic.com/research/anthropic-economic-index-september-2025-report

Bridging RL Theory and Practice with the Effective Horizon (i.e. the Cassidy Laidlaw paper): https://arxiv.org/abs/2304.09853

How Does Time Horizon Vary Across Domains?: https://metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains/

Inventing Temperature: https://global.oup.com/academic/product/inventing-temperature-9780195337389

Is there a Half-Life for the Success Rates of AI Agents? (by Toby Ord): https://www.tobyord.com/writing/half-life

Lawrence Chan's response to the above: https://nitter.net/justanotherlaw/status/1920254586771710009

AI Task Length Horizons in Offensive Cybersecurity: https://sean-peters-au.github.io/2025/07/02/ai-task-length-horizons-in-offensive-cybersecurity.html

Episode art by Hamish Doodles: hamishdoodles.com