Explanation of the analysis:

Consider all resolved, binary markets (binary just because I'm being lazy about handling free response markets). Now consider the market probability after the first unique trader, the second trader, and so on. When considering the n-th trader scenario, we exclude any markets with fewer than n traders entirely.

Manifold is, overall, well-calibrated, and it is probably the case that adding more unique traders increases accuracy. So: going through the procedure above, what's the smallest value of n where Manifold will be well-calibrated?

For the purposes of this market, "well-calibrated" means that when I bucket predictions around 2.5%, 10%, 20%, ..., 90%, 97.%, at least 7 of the 11 buckets will be within 5 percentage points of perfect calibration.

If you would like to see how everything is calculated my code is/will be here: https://github.com/vluzko/manifoldpy