Will METR's "50%-task-completion time horizon" SotA double in under 7 months?

Question

METR recently published an analysis called Measuring AI ability to complete long tasks.

The analysis suggests that the 50%-task-completion time horizon for the State of the Art AI agent is doubling every 7 months:

[image]Figure 1 from the METR paper illustrates the historical exponential growth trend and the ~7 month doubling time.

Market details

This market resolves based on published data from METR (Model Evaluation & Threat Research) or a successor organization explicitly continuing this line of research using a demonstrably equivalent methodology.

Metric: The "50%-task-completion time horizon" as defined in METR's paper.

Baseline date (T0):  Feb 24th 2025 (the public release of Claude 3.7 Sonnet)

Baseline State-of-the-Art (SotA) Horizon (H0): 40 minutes the value of the trend line at date T0.

Target Horizon (H1): 80 minutes (2 * H0, i.e. a single doubling).

Target Date (T1): The date on which a model is publicly accessible through for which METR (or another organization using equivalent methodology) reports a 50% task completion time horizon of at least H1 (80 minutes) using their established methodology. This date T1 is the model's public access date, not the date of the METR report itself.

Resolution Criterion

This market asks whether the next doubling of the SotA – reaching a horizon of at least 80 minutes (H1) – will occur faster than this historical average. Specifically, will the time elapsed between the effective date of the previous SotA (T0) and the date the new SotA horizon (H1) is achieved (T1) be less than 212 days (i.e. by 24th September 2025).

Market closing date

The market will close on the 24th of October 2025 to provide 30 days for METR (or another org) to run the METR analysis. It will close earlier if METR (or another org using equivalent methodology) confirm that the 80 minute 50%-task-completion time horizon is achieved before this date.

Update 2025-04-16 (PST) (AI summary of creator comment): Important Notes:

The resolution will accept a METR-evaluated model even if changes in the task set prevent a direct numerical comparison to the original baseline figures.

A model that exceeds the 80-minute threshold (for example, reaching 90 minutes) using METR’s current evaluation qualifies.

The public release date of the model is used to determine the timing criteria, so a release date that meets the requirement (as in the case of o3 on 16th April) confirms resolution.

Manifold Markets · Accepted Answer

Yes — resolved on Apr 16, 2025 by Manifold Markets prediction market.