By which years will AI be shown to have a better log loss than the Metaculus community pred. on <= 1 year predictions?

1.2kṀ1493

2031

70%

2030

63%

2028

61%

2029

61%

2027

45%

2026

19%

2025

2024

By which years will AI + scaffolding be shown to have a better log loss than the Metaculus community prediction on <= 1 year predictions?

Does not necessarily need to be evaluated on every Metaculus question, just a representative sample of Metaculus questions not selected to favor the AI with at least 20 unique predictors. Resolves to years where AI+scaffolding has been shown to have a better log loss anytime before the end of that year (including previous years).

I will not trade on this market.

Get

1,000

to start trading!

People are also trading

Will AI image generating models score >= 90% on Winoground by June 1, 2025?

11% chance

Will pre-2026 AI out-forecast the Metaculus community?

58% chance

Will an AI model outperform 95% of Manifold users on accuracy before 2026?

49% chance

How well will I (@draaglom) forecast on Metaculus in 2025? (Peer accuracy leaderboard)

What year will the first AI exceed 80% on MLE-bench?

Will AI progress surprise Metaculus?

77% chance

Will we have better-than-human-aggregate forecasting AIs by the end of 2024?

4% chance

When will a real money prediction market about OpenAI announcing AGI first stay above 90% for one month?

Will Metaculus still exist and have active forecasting throughout 2025?

93% chance

Next year will I think that AI is better than me at math?

Sort by:

https://arxiv.org/abs/2402.18563

bought Ṁ12 NO

Seems like this cannot resolve YES in 2024 because a random sample of <= 1 year questions will include some that end in Feb 2025, so the earliest this could be shown is then.

If such questions are excluded then it isn't a representative sample.

Unless someone started this project in 2023 and didn't tell me about it. Which is 10%, sure.

@MartinRandall If someone can use a model trained on text before a date and uses no external information after that date, I'd accept the predictions of that model for a year after the date.

@NoaNabeshima Oof, then we could reresolve 2024 to yes in 2030 after we have a better understanding of AI??

@MartinRandall No because it would need to be *shown* before EOY 2024. The <=1 year predictions don't need to be related to the year this ability is demonstrated except that they can't be too jerrymandered

bought Ṁ10 YES

@NoaNabeshima I see. So if in 2030 I train an AI based only on data from before 2024, and then I have it make predictions on Metaculus questions in 2024, again using data only from before 2024, and it beats the human predictors from 2024, that resolves 2030 to YES, only.

bought Ṁ7 YES

Seems similar to https://manifold.markets/dp/will-we-have-betterthanhumanaggrega except easier since only on 1 year time horizon? Why was 2024 lower here before I traded?

@EliLifland it's harder because it's performance on Metaculus only. Performance on Manifold is easier, eg acc outperforms me substantially.

The AI isn’t allowed to see the community prediction?

@EliLifland I am not sure, am thinking ab this.

@EliLifland Yes, the AI isn't allowed to see the community prediction. LMK if this makes anyone unhappy.

Comment hidden

Forecasters will incorporate AI tools into their workflows, so this could occur much later than the point where AI becomes a better forecaster than most (unassisted) humans on the platform.