By which years will AI be shown to have a better log loss than the Metaculus community pred. on <= 1 year predictions?

162

Ṁ1.4K

Ṁ1.2K

2031

ALL

70%

2030

61%

2029

54%

2028

46%

2027

35%

2026

14%

2025

2024

By which years will AI + scaffolding be shown to have a better log loss than the Metaculus community prediction on <= 1 year predictions?

Does not necessarily need to be evaluated on every Metaculus question, just a representative sample of Metaculus questions not selected to favor the AI with at least 20 unique predictors. Resolves to years where AI+scaffolding has been shown to have a better log loss anytime before the end of that year (including previous years).

I will not trade on this market.

Get Ṁ200 play money

Related questions

[Metaculus] Will any OpenAI or Anthropic model be in the top-10 model with a non-proprietary license on May 31, 2024?

5% chance

In what year will an OpenAI natural language processing model be more competent than me in my area of expertise?

2034

When will a weakly general AI become publicly known?

2028

[Metaculus] Will OpenAI claim GPT-5 is AGI within 30 days after its release?

7% chance

When will AI be better than humans at AI research? (Basically AGI)

Will an AI solve any important mathematical conjecture before January 1st, 2030?

69% chance

Will an AI model outperform 95% of Manifold users on accuracy before 2026?

67% chance

In 2 years, what will the Metaculus prediction for AGI timelines be?

2031

When will self-improving AI outperform human-developed AI?

2030

Will pre-2026 AI out-forecast the Metaculus community?

Sort by:

https://arxiv.org/abs/2402.18563

bought Ṁ12 2028 NO

Seems like this cannot resolve YES in 2024 because a random sample of <= 1 year questions will include some that end in Feb 2025, so the earliest this could be shown is then.

If such questions are excluded then it isn't a representative sample.

Unless someone started this project in 2023 and didn't tell me about it. Which is 10%, sure.

@MartinRandall If someone can use a model trained on text before a date and uses no external information after that date, I'd accept the predictions of that model for a year after the date.

@NoaNabeshima Oof, then we could reresolve 2024 to yes in 2030 after we have a better understanding of AI??

@MartinRandall No because it would need to be *shown* before EOY 2024. The <=1 year predictions don't need to be related to the year this ability is demonstrated except that they can't be too jerrymandered

bought Ṁ10 2030 YES

@NoaNabeshima I see. So if in 2030 I train an AI based only on data from before 2024, and then I have it make predictions on Metaculus questions in 2024, again using data only from before 2024, and it beats the human predictors from 2024, that resolves 2030 to YES, only.

bought Ṁ7 2029 YES

Seems similar to https://manifold.markets/dp/will-we-have-betterthanhumanaggrega except easier since only on 1 year time horizon? Why was 2024 lower here before I traded?

@EliLifland it's harder because it's performance on Metaculus only. Performance on Manifold is easier, eg acc outperforms me substantially.

The AI isn’t allowed to see the community prediction?

@EliLifland I am not sure, am thinking ab this.

@EliLifland Yes, the AI isn't allowed to see the community prediction. LMK if this makes anyone unhappy.

Comment hidden

Forecasters will incorporate AI tools into their workflows, so this could occur much later than the point where AI becomes a better forecaster than most (unassisted) humans on the platform.

@SG the model could be private for the beginning of the forecasting period or the scaffolding could be complex

reposted

I love that this is a specific operationalization of the question "When will AI forecasters beat humans?"

Would love to hear thoughts, whether you're a human or AI~

Does "not selected to favor the AI" mean that AI is not allowed to choose which questions to answer?

@Nick6d8e yes