See https://www.astralcodexten.com/p/2023-prediction-contest
I have scores already. About 3300 people entered Blind Mode. What percentile among these people do you think Metaculus (see https://www.metaculus.com/tournament/2023-contest/) got?
I will probably post results on Monday; this question will close then. It will resolve to the percentile, so if it got 50th percentile, it will resolve to 50.
See question about Manifold at https://manifold.markets/ScottAlexander/what-percentile-did-manifold-get-in
1,000🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ29 | |
| 2 | Ṁ25 | |
| 3 | Ṁ15 | |
| 4 | Ṁ4 | |
| 5 | Ṁ0 |
People are also trading
@BoltonBailey seems to have reproduced this result here: https://manifold.markets/market/will-manifold-get-a-better-brier-sc-2486658619a2
Why is Manifold so bad? What is going on here? Is it possible that market structure is just worse for forecasting than whatever Metaculus is doing?
@wadimiusz I don't think I "reproduced" anything here. Note that this outcome compares an aggregate forecast to a bunch of individual forecasts, whereas I was comparing different forms of aggregates.
@BoltonBailey To elaborate more on whether the outcomes of these contests on their own should really update us that much one way or the other: I made an additional page in my spreadsheet to look at what the outcomes would have been if the true probabilities of the events were all independent and exactly as probable as the blind-mode median said. You can see that even generously assuming that this aggregation was always right, there still would have been a 13% chance of it getting a worse Brier score by chance alone.
@BoltonBailey And for the Bayesians out there, the log score difference was only 0.82 bits between the medians and the manifold price, a likelihood ratio of about 4:7. And this doesn't even account for potential correlations between questions.
So while we evaluated worse, I am not particularly sure that we were actually predicting worse, or that even if we were objectively predicting worse, that we won't eventually improve to a point where we are better than this "median of ACX readers" baseline.