By which years will AI be shown to have a better log loss than the Metaculus community pred. on <= 1 year predictions?

By which years will AI + scaffolding be shown to have a better log loss than the Metaculus community prediction on <= 1 year predictions?

Does not necessarily need to be evaluated on every Metaculus question, just a representative sample of Metaculus questions not selected to favor the AI with at least 20 unique predictors. Resolves to years where AI+scaffolding has been shown to have a better log loss anytime before the end of that year (including previous years).

I will not trade on this market.

Get Ṁ600 play money
Sort by:
bought Ṁ12 2028 NO

Seems like this cannot resolve YES in 2024 because a random sample of <= 1 year questions will include some that end in Feb 2025, so the earliest this could be shown is then.

If such questions are excluded then it isn't a representative sample.

Unless someone started this project in 2023 and didn't tell me about it. Which is 10%, sure.

@MartinRandall If someone can use a model trained on text before a date and uses no external information after that date, I'd accept the predictions of that model for a year after the date.

@NoaNabeshima Oof, then we could reresolve 2024 to yes in 2030 after we have a better understanding of AI??

@MartinRandall No because it would need to be *shown* before EOY 2024. The <=1 year predictions don't need to be related to the year this ability is demonstrated except that they can't be too jerrymandered

bought Ṁ10 2030 YES

@NoaNabeshima I see. So if in 2030 I train an AI based only on data from before 2024, and then I have it make predictions on Metaculus questions in 2024, again using data only from before 2024, and it beats the human predictors from 2024, that resolves 2030 to YES, only.

bought Ṁ7 2029 YES

Seems similar to except easier since only on 1 year time horizon? Why was 2024 lower here before I traded?

@EliLifland it's harder because it's performance on Metaculus only. Performance on Manifold is easier, eg acc outperforms me substantially.

The AI isn’t allowed to see the community prediction?

@EliLifland I am not sure, am thinking ab this.

@EliLifland Yes, the AI isn't allowed to see the community prediction. LMK if this makes anyone unhappy.

Comment hidden
Comment hidden
Comment hidden
Comment hidden
Comment hidden
Comment hidden
Comment hidden

Forecasters will incorporate AI tools into their workflows, so this could occur much later than the point where AI becomes a better forecaster than most (unassisted) humans on the platform.

@SG the model could be private for the beginning of the forecasting period or the scaffolding could be complex


I love that this is a specific operationalization of the question "When will AI forecasters beat humans?"

Would love to hear thoughts, whether you're a human or AI~

Does "not selected to favor the AI" mean that AI is not allowed to choose which questions to answer?