METR publishes the 'time horizon' metric for AI task completion

Why measure how long a task is

Most AI evaluations ask a model to solve something that fits in a single prompt: a puzzle, a short coding exercise, a multiple-choice question. Real software work rarely looks like that. Fixing a bug that only shows up in production, or changing how a system stores its data, unfolds over many steps, and a model has to stay on track across all of them. A benchmark made of short questions says little about that kind of work. The paper "Measuring AI Ability to Complete Long Software Tasks" was submitted to arXiv on March 18, 2025, with Thomas Kwa as first author, and it proposed a way to put a single number on long-horizon ability^[1].

The metric: a 50% time horizon

The central proposal is the "50%-task-completion time horizon": the amount of time humans typically take to complete the tasks that an AI model can complete with a 50% success rate^[1]. The idea is to express model capability in the units of human working time rather than in percentages of correct answers. Instead of asking how many benchmark items a model gets right, the metric asks a different question: at what duration of human work does the model's success rate fall to roughly a coin flip? A model that reliably finishes tasks humans need a few minutes for, but fails about half the time on tasks that take an hour, would have a time horizon somewhere under an hour.

How the human baselines were built

A metric defined in human time requires human timing data. The researchers timed people with relevant domain expertise as they worked, drawing on a combination of RE-Bench, HCAST and 66 novel shorter tasks^[1]. Because the tasks were timed by people who actually knew the domain, the resulting durations are meant to reflect how long the work genuinely takes a competent practitioner, not how long it takes someone learning the task from scratch.

What the paper found

On these tasks, frontier models such as Claude 3.7 Sonnet had a 50% time horizon of around 50 minutes^[1]. The authors also looked at how that number had changed over time, and found that frontier AI time horizon had been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024^[1]. A doubling trend of that kind is a strong statement about the pace of change: if it holds, the length of task a model can half-finish grows by a factor of about eight every two years.

The paper also asked where the gains came from. It reported that the increase in time horizons appeared primarily driven by greater reliability, the ability to adapt to mistakes, better logical reasoning and tool use capabilities^[1]. That is a different picture from raw knowledge or single-shot problem solving: much of the progress on long tasks came from a model staying usable over many steps — recovering when something goes wrong, keeping track of what it has already done, and operating tools along the way.

What the trend implies

The authors noted that extrapolating the trend predicted that within 5 years AI systems would be capable of automating many software tasks that currently take humans a month^[1]. The paper presented this as what the measured trend implies if it continues, which is a weaker claim than a forecast that it will continue. Time-horizon doubling had already run for several years by the time of writing, so the extrapolation was not speculative arithmetic but an extension of an observed pattern.

Publication

The paper was later published at NeurIPS 2025^[1]. Its contribution was chiefly methodological: it gave researchers and readers a way to talk about model capability in terms of the length of human work a system can carry, and it supplied measured human baselines, drawn from RE-Bench, HCAST and the 66 additional tasks, on which such comparisons could rest^[1].