The members of the AI futures project have given an update and they appear to now be relying on the 80% time horizon length graph from METR for their predictions rather than the 50% time horizon length graph. This implies that a 50% time horizon is not enough. While I think markets for 50% time horizons are useful, I now think that more attention needs to be paid to 80% time horizon lengths. I am planning to create markets for 80% time horizons either tonight or some other time this week unless someone beats me to it.

@MaxLennartson Here is my source: https://www.aifuturesmodel.com/#section-timehorizonandtheautomatedcodermilestone. Sorry for not posting this earlier.

Has METR said anything about how long their tasks even go up to? Or are they just arbitrarily adding tasks as the models improve

@HenryE when they measure time horizon via "pass if any agent solved the task across all their runs" they got 16 hours. This was after removing a few problematic tasks so it's higher than the "official" estimate would be, unless they formally relaunch their task suite with those tasks removed.

@JoshYou <2 hours can already be resolved to no

@LoweLundin can't resolve one option early on a single-resolution market like this

@JoshYou Ah, sorry!

jim order up for the ">=16 hours" option

@jacksonpolack , @Bayesian , @SemioticRivalry , @Velaris , @khang2009 , @Gen , @skibidist , @evan , @brod , @100Anonymous , @Ziddletwix ,, @Trazyn , @bagelfan , @geuber , @nikki , @ProjectVictory , @sahaj , @bohaska , @Odoacre

jim orders are large limit orders, generally at better than market prices.

Opt in / opt out thread: https://manifold.markets/post/jim-order-notification-optin-thread

@JoshYou 6-8 hours is the obvious bet to make based on current trends. I have a couple contrarian jim orders up for longer horizons

If a model is released by February, but is not evaluated, will this market resolve based on its eventual evaluation? Or does this market only deal with evaluations carried out prior to February?

@jim it resolves to the highest score as of April 30, 2026 of any model released by February

My guess is that the time horizon will be 24 hours.

I'm not sure exactly what value I would use for o3 for the purpose of this market, but it appears to be around 100 minutes. METR also says o3 has an 1.8x longer time horizon than Claude 3.7 on HCAST, which makes up the majority of their task suite.

@JoshYou I am also curious about these preliminary evaluations, see https://www.lesswrong.com/posts/ohgGyu78DXLMaASpE/why-does-metr-score-o3-as-effective-for-such-a-long-time