Right now, using the current trade sampling methodology on our calibration page, Manifold has a Brier score of ~0.172.
Criteria: On or about September 1st 2023, I will programmatically draw 10 samples. If 5 or more out of 10 samples have Brier scores less than 0.17, this market resolves YES. (Results only valid using the current methodology. If we switch to using something else or I have no easy way to calculate the Brier score using the same approach, this resolves N/A.)
Related questions
🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ612 | |
| 2 | Ṁ242 | |
| 3 | Ṁ157 | |
| 4 | Ṁ122 | |
| 5 | Ṁ90 |
Manifold's calibration score hasn't improved using this methodology...but it has if you filter out nonpredictive markets, which you can do from @wasabipesto's calibration site!
My take: the probability of this happening is less than 5%.
Why I believe this: The vast majority of times when I check the calibration page, the Brier score is more than 0.17. Today I did this more systematically, and all 5 out of 5 samples had Brier higher than 0.17 (in fact, higher than 0.171). I'd estimate the frequency of <0.17 samples is maybe 0.1, and would be quite surprised if it was more than 0.2. Even if the frequency was 0.2, the probability of the market resolving YES would be just 3.3%.
(Why I share this now: I'm already heavily invested in the market and am running out of liquidity - I wouldn't mind if the market started to agree with me and I could take some of my mana out.)
@JohnSmithb9be I’m not actually sure I can explain why I feel this way so if anyone who’s a stats expert can critique this opinion I would welcome it!
Reasons why calibration might improve:
We grow and have more users trading in markets.
We continue to unlist whalebait and non-predictive markets (which tend to be less calibrated).
More people create bots and bot trading algorithms get better (partially a function of having more training data)
@SG Increasing users might actually decrease reliability. Probably some growth is good, but growth that is too fast I suspect will lure in what we call "dumb money" in betting market circles.