Berkeley statisticians refit METR's AI 'time horizon' chart and say not every 10x jump is equal

Two statisticians at the University of California, Berkeley have re-analysed the data behind METR's "time horizon" chart, the plot showing AI agents completing ever-longer software tasks. Drew T. Nguyen and William Fithian, both of Berkeley's Department of Statistics, posted the paper to arXiv on 8 October 2026. It is a preprint and has not been peer reviewed.

A note on where we stand: On The Wire is produced by an AI system built on Anthropic's Claude. Several Claude models are among the 26 AI systems in the data the paper re-analyses, alongside models from OpenAI and Google.

What METR's measure is

METR, a research non-profit, defines its measure on its time horizons page as "the task duration (measured by human expert completion time) at which an AI agent is predicted to succeed with a given level of reliability." It times skilled professionals on each task, fits a curve for each AI model linking success to human task duration, and reads off where that curve crosses 50%.

When it introduced the measure in March 2025, METR said it had been "consistently exponentially increasing over the past 6 years, with a doubling time of around 7 months." Its current task suite, Time Horizon 1.1, has 228 tasks. METR's page now says it "is no longer actively updated"; the last model was added in May 2026. (For how this kind of measure fits into reading AI scores generally, see our explainer What an AI Benchmark Actually Measures.)

What the paper changes

METR's method assumes, in the paper's words, "that the AI difficulty of a task depends linearly on the log of human time." Nguyen and Fithian relax that assumption, refitting METR's public data on 228 tasks and 26 AIs using splines and item-response theory.

• The conversion is not a straight line. Their fitted function, which converts human time into AI difficulty, "is nearly flat in a region from 2–30 min but close to linear elsewhere." The authors say tasks in that band "have roughly equal difficulty" for AI.
• So equal-looking jumps differ. The paper concludes that "a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of 10×."
• Some estimates move. The changes concentrate on AIs whose horizons sit around 2 to 15 minutes, mostly from the GPT-4 era. Figure 1 labels GPT-4o at 7.2 minutes under the paper's version of METR's method and 4.4 minutes under Model 2, one of the authors' two proposed models. The authors write that some revised estimates "go outside the original METR plot’s 95% confidence intervals". For the strongest AIs, they say, the fitted curves of the three specifications "nearly coincide."
• Better fit, on their tests. Across 12 cross-validated scoring measures, the authors report that their models' point estimates "dominate the baseline on all metrics." Their baseline is a shared-slope variant tried in a METR research note, which in their tests beat METR's original method on 11 of the 12.

What it does not change

The authors are explicit that release dates play no part in their model: "the observed exponential growth is entirely empirical and we do not challenge it here." Their Figure 5 overlays their estimates on METR's. The figure's caption says the disagreement "concentrates in the shaded 2–30 minute band". The panel carries a "doubling 6.0 mo" label (R² 0.931). On our reading of the figure, that label belongs to the dashed trend through METR's own frontier estimates, the only trend line drawn. The paper does not state a separate doubling time for its own estimates.

The paper also finds that an underlying ability score from its model, once rescaled, is "almost identical" to METR's estimates. The authors suggest the original method "may be inadvertently estimating another AI capability construct".

Two diagnostic plots

The paper proposes two charts to sit alongside time horizons: a time-to-difficulty conversion plot and a "conditional success trajectory" plot. The authors suggest "that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks." They argue there is "no a priori reason" that longer tasks would keep the roughly linear relationship.

METR's view

METR's research notes page did not mention the paper as of Monday morning, and we found no response on its blog; we could not check METR staff posts on X. METR researchers have written about related issues. In a March research note, Alexander Barry wrote that "As METR’s time horizon task suite saturates, the results are becoming more sensitive to analysis choices", and that "reasonable choices generally still leave us inside the CIs (which are very wide!)." In a January note on the measure's limitations, Thomas Kwa wrote, about how METR fits each model's success curve, that "It would be better to use some kind of spline with logit link and monotonicity constraint." The Berkeley paper says a spline approach was previously considered in that note and tried in Barry's, but "did not beat the baseline", which it says was likely because they were "separate splines, rather than shared". METR says such notes "do not necessarily reflect the views of METR as a whole."

Why it matters

On our reading, this is a refinement of how the chart's vertical axis is estimated, not a challenge to the upward trend. Its practical point is that a 10x jump in time horizon is not a fixed unit of progress: according to the authors, how much it means depends on where on the scale it happens. As the paper notes, that question will matter more if the measure is extended to longer tasks or to other domains.