Calibration

Explore how Manifold's predictions compare to real-world outcomes. Our track record demonstrates the power of collective forecasting.

📊 Why are markets better than polls or experts?
One paper about predicting scientific paper replication compared these forecasting methods. It found that prediction markets outperformed surveys and
...could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.”
Either prediction markets are more accurate than experts, or experts should be able to make a lot of money on them, and in doing so correct the markets.
🤑 Are markets resistant to manipulation and hype?
As the market prices moves further from the true probability, the odd's pricing becomes better for traders to correct it in the right direction. Naturally, this increases the incentive to bet accurately as there is more money to be made once the market resolves.
Robin Hanson explores this further in his paper, A Manipulator Can Aid Prediction Market Accuracy. In it he examines how both historical and lab data fail to find substantial effects of manipulation on average price accuracy. Furthermore, he finds in his model that adding a manipulator may even increase accuracy as it increases noise trading which tends to have a positive effect in low liquidity markets.
🌱 Do markets with few traders and low liquidity work?
Yes! And very reliably! The paper Prediction Markets: Practical Experiments in Small Markets and Behaviours Observed, concluded,
“16 or more traders should be sufficient to obtain quality predictions. Smaller markets may be just as useful, though they may exhibit biases of under confidence toward market favourites.”
Our own data has shown that somewhere between 10 - 20 traders our calibration no longer improves with more traders. We still need to conduct analysis on the impact liquidity has on accuracy.

Overall Calibration

Predicted vs actual outcomes across all markets

0.17342Brier

This chart shows whether events happened as often as we predicted. The closer the blue dots are to the diagonal line, the better our calibration. A dot at 70% on the x-axis should appear at 70% on the y-axis if exactly 70% of those markets resolved yes.

Resolved Yes
Market Probability
  1. 1Every hour we sample 2% of all past trades on resolved binary questions with 15 or more traders. Current sample size: 89k trades.
  2. 2For each sampled trade, we find the average probability between the start and end.
  3. 3We group trades with similar probabilities together.
  4. 4Then, we check for trades that said there was e.g. a 60% chance, and how often those markets resolve yes. For perfect calibration, we expect 60% of them to have resolved yes.
  5. 5We repeat this at each probability interval to plot the calibration curve.

Note: This methodology uses trade-weighted rather than time-weighted calibration. Market accuracy may be better than reflected here, as large miscalibrated trades are usually corrected immediately.

Case Studies

Notable examples of prediction market accuracy

Predicting Trump's arrest

3k
Will Donald Trump be arrested before 2024?YES

On March 18th Trump posted on Truth Social that he believes he was about to be arrested, causing our market to spike to 88%. However, since December our market had already been hovering around 40% on average before anyone else was even discussing it as a true possibility.

Al-Ahli Arab hospital explosion

6.2k
Was an IDF Strike responsible for the Al-Ahli Hospital Explosion?1.0%

Just 3 hours after initial local reports, this market was created. Within 1 hour it had already been pushed down to 6%, before settling between 6-20% as more news emerged. Meanwhile, major outlets still presented conflicting headlines, which led to the BBC conceding that a reporter had been wrong to speculate.

Predicting SBF fraud

2.7k
SBF convicted of a felony before 2026YES

Manifold had a market stable between 5-10% that SBF would be convicted of a felony 1-month before there was any news about it. It then immediately reacted correctly to rumors before any official statements were made.

2022 US Midterm Elections

Manifold outperformed real money prediction markets and was almost as accurate as FiveThirtyEight when forecasting the 2022 US midterm elections.

Additional Resources

See more charts and analysis courtesy of @wasabipesto from our data in 2022.

© Manifold Markets, Inc.•Terms•Privacy