Explanation of the analysis:
Consider all resolved, binary markets (binary just because I'm being lazy about handling free response markets). Now consider the market probability after the first unique trader, the second trader, and so on. When considering the n-th trader scenario, we exclude any markets with fewer than n traders entirely.
Manifold is, overall, well-calibrated, and it is probably the case that adding more unique traders increases accuracy. So: going through the procedure above, what's the smallest value of n where Manifold will be well-calibrated?
For the purposes of this market, "well-calibrated" means that when I bucket predictions around 2.5%, 10%, 20%, ..., 90%, 97.%, at least 7 of the 11 buckets will be within 5 percentage points of perfect calibration.
If you would like to see how everything is calculated my code is/will be here: https://github.com/vluzko/manifoldpy
5
3
2504
6110
If anyone would like to check the code it is here: https://github.com/vluzko/manifoldpy/blob/main/scripts/num_trader_calibration.py
Market should resolve soon: I am worried there is a bug in my script so I'm going to check it more before I actually do resolve.
After 0 unique traders, the probability is always 50%. If the fraction of markets which resolve YES was between 45% and 55%, that would make it well calibrated.
I don't think calibration is the right thing to measure here. It might be better to compare Brier score after k unique traders with Brier score at market close.
They're not well-calibrated at start: older markets could start with non 50/50 probabilities, and this is still available through the API.
Honestly I just don't really care about Brier score. Proper scoring rules are nice but they don't really give you any insight into how to update given a predictor's output.
@LivInTheLookingGlass Is there a way to make a pseudonumeric market besides the API?
@Yev Thank you! I will probably just use the API but it's good to spread this information around
After how many unique traders will Manifold Markets be well calibrated?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition