resolved Nov 1
After how many unique traders will Manifold Markets be well calibrated?

Explanation of the analysis:

Consider all resolved, binary markets (binary just because I'm being lazy about handling free response markets). Now consider the market probability after the first unique trader, the second trader, and so on. When considering the n-th trader scenario, we exclude any markets with fewer than n traders entirely.

Manifold is, overall, well-calibrated, and it is probably the case that adding more unique traders increases accuracy. So: going through the procedure above, what's the smallest value of n where Manifold will be well-calibrated?

For the purposes of this market, "well-calibrated" means that when I bucket predictions around 2.5%, 10%, 20%, ..., 90%, 97.%, at least 7 of the 11 buckets will be within 5 percentage points of perfect calibration.

If you would like to see how everything is calculated my code is/will be here: https://github.com/vluzko/manifoldpy

Chosen
39%
LivInTheLookingGlass avatar5
13%
Yev avatar3
8%
bravoechonovember avatar250
6%
Yev avatar4
5%
mukimameaddict avatar6
3%
Yev avatar1
1.9%
LivInTheLookingGlass avatar10
1.6%

🏅 Top traders

#NameTotal profit
1Ṁ137
2Ṁ1
Sort by:
vluzko avatar
Vincent Luczkow
vluzko avatar
Vincent Luczkow

Market should resolve soon: I am worried there is a bug in my script so I'm going to check it more before I actually do resolve.

Yev avatar
Yev

Please resolve this

Yev avatar
Yevsold Ṁ12 of 1
Yev answered
0
Yev avatar
Yevbought Ṁ1

After 0 unique traders, the probability is always 50%. If the fraction of markets which resolve YES was between 45% and 55%, that would make it well calibrated.

Yev avatar
Yev

I don't think calibration is the right thing to measure here. It might be better to compare Brier score after k unique traders with Brier score at market close.

vluzko avatar
Vincent Luczkow

They're not well-calibrated at start: older markets could start with non 50/50 probabilities, and this is still available through the API.

Honestly I just don't really care about Brier score. Proper scoring rules are nice but they don't really give you any insight into how to update given a predictor's output.

Yev avatar
Yevsold Ṁ2 of 0

@vluzko Point taken, the answer is not 0. But I expect it to be very small (possibly even 1). Calibration is necessary for a good market but not sufficient.

LivInTheLookingGlass avatar
Olivia

Let me make a pseudonumeric market for this. Can you suggest an upper bound?

LivInTheLookingGlass avatar
Olivia

Okay, the biggest market I can find is ~550 traders, so here is one for 600 at log scale:

vluzko avatar
Vincent Luczkow

@LivInTheLookingGlass Is there a way to make a pseudonumeric market besides the API?

LivInTheLookingGlass avatar
Olivia

@vluzko sadly no

Yev avatar
Yev

@vluzko You can duplicate an existing pseudonumeric market

vluzko avatar
Vincent Luczkow

@Yev Thank you! I will probably just use the API but it's good to spread this information around

ManifoldDream avatar
Manifold in the WildBot

After how many unique traders will Manifold Markets be well calibrated?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition