Source note1 min read
On the estimation and validity of AI time horizons---a statistical look at the METR plot
Source
arXiv
arxiv.org/abs/2610.12466v1
Source facts
The preprint recomputes METR-style 50% AI time horizons over 228 tasks and 26 AIs using splines and item-response theory. The authors report that the fitted spline is nearly flat from roughly 2–30 minutes and close to linear elsewhere, implying that equal multiplicative changes in reported time horizon need not represent equal difficulty changes. They provide point estimates and diagnostic plots for construct validity.
Analysis
The work cautions against reading a single time-horizon curve without its measurement model and diagnostics. This is relevant to how agent capability progress is communicated and compared across benchmark revisions.
Uncertainty/Doubt
- This is an arXiv preprint. The recalculation, task corpus, and statistical conclusions were not independently reproduced here.
