By which years will AI + scaffolding be shown to have a better log loss than the Metaculus community prediction on <= 1 year predictions?
Does not necessarily need to be evaluated on every Metaculus question, just a representative sample of Metaculus questions not selected to favor the AI with at least 20 unique predictors. Resolves to years where AI+scaffolding has been shown to have a better log loss anytime before the end of that year (including previous years).
I will not trade on this market.
1,000Seems like this cannot resolve YES in 2024 because a random sample of <= 1 year questions will include some that end in Feb 2025, so the earliest this could be shown is then.
If such questions are excluded then it isn't a representative sample.
@MartinRandall If someone can use a model trained on text before a date and uses no external information after that date, I'd accept the predictions of that model for a year after the date.
@NoaNabeshima Oof, then we could reresolve 2024 to yes in 2030 after we have a better understanding of AI??
@MartinRandall No because it would need to be *shown* before EOY 2024. The <=1 year predictions don't need to be related to the year this ability is demonstrated except that they can't be too jerrymandered
@NoaNabeshima I see. So if in 2030 I train an AI based only on data from before 2024, and then I have it make predictions on Metaculus questions in 2024, again using data only from before 2024, and it beats the human predictors from 2024, that resolves 2030 to YES, only.
Seems similar to https://manifold.markets/dp/will-we-have-betterthanhumanaggrega except easier since only on 1 year time horizon? Why was 2024 lower here before I traded?
@EliLifland it's harder because it's performance on Metaculus only. Performance on Manifold is easier, eg acc outperforms me substantially.
@EliLifland Yes, the AI isn't allowed to see the community prediction. LMK if this makes anyone unhappy.
Forecasters will incorporate AI tools into their workflows, so this could occur much later than the point where AI becomes a better forecaster than most (unassisted) humans on the platform.
I love that this is a specific operationalization of the question "When will AI forecasters beat humans?"
Would love to hear thoughts, whether you're a human or AI~
Does "not selected to favor the AI" mean that AI is not allowed to choose which questions to answer?