Calibration

See how Manifold's predictions compare to real-world outcomes and explore our track record of accuracy.

📊 Why are markets better than polls or experts?

One paper about predicting scientific paper replication compared these forecasting methods. It found that prediction markets outperformed surveys and

...could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.”

Either prediction markets are more accurate than experts, or experts should be able to make a lot of money on them, and in doing so correct the markets.

🤑 Are markets resistant to manipulation and hype?

As the market prices moves further from the true probability, the odd's pricing becomes better for traders to correct it in the right direction. Naturally, this increases the incentive to bet accurately as there is more money to be made once the market resolves.

Robin Hanson explores this further in his paper, A Manipulator Can Aid Prediction Market Accuracy. In it he examines how both historical and lab data fail to find substantial effects of manipulation on average price accuracy. Furthermore, he finds in his model that adding a manipulator may even increase accuracy as it increases noise trading which tends to have a positive effect in low liquidity markets.

See also, Scott Alexander's failed manipulation attempt on Manifold.

🌱 Do markets with few traders and low liquidity work?

Yes! And very reliably! The paper Prediction Markets: Practical Experiments in Small Markets and Behaviours Observed, concluded,

“16 or more traders should be sufficient to obtain quality predictions. Smaller markets may be just as useful, though they may exhibit biases of under confidence toward market favourites.”

Our own data has shown that somewhere between 10 - 20 traders our calibration no longer improves with more traders. We still need to conduct analysis on the impact liquidity has on accuracy.

Overall Calibration

This chart shows whether events happened as often as we predicted. We want the blue dots to be as close to the diagonal line as possible!

A dot with a question probability of 70% means we have a group of markets that were predicted to have a 70% chance of occurring. If our predictions are perfectly calibrated, then 70% of those markets should have resolved yes and it should appear on the y-axis at 70%.

Resolved Yes

Question probability

Brier score:

0.17448

A score between 0.1 and 0.2 is very good! Our data shows our markets are very accurate.

Methodology details

Case Studies

Predicting Trump's arrest

Will Donald Trump be arrested before 2024?YES

On March 18th Trump posted on Truth Social that he believes he was about to be arrested, this caused our market to spike to 88%. However, since December our market had already been hovering around 40% on average before anyone else was even discussing it as a true possibility of happening!

Al-Ahli Arab hospital explosion

6.2k

Was an IDF Strike responsible for the Al-Ahli Hospital Explosion?1.0%

Just 3 hours after the initial local reports of the explosion, we had this market made. Within 1 hour of creation it had already been pushed to down 6%, before eventually settling between 6-20% over the next few hours as more news came to light.

Meanwhile, major news outlets still presented conflicting headlines, which eventually led to the BBC conceding that a reporter had been wrong to speculate in his analysis.

Predicting SBF fraud

2.7k

SBF convicted of a felony before 2026YES

Manifold had a market stable between 5-10% that SBF would be convicted of a felony 1-month before there was any news about it. It then immediately reacted correctly to rumors before any official statements were made.

How we performed on the 2022 US midterms

Manifold outperformed real money prediction markets and was almost as accurate as FiveThiryEight when forecasting the 2022 US midterm elections.

Additional Resources

See more charts and analysis courtesy of @wasabipesto from our data in 2022.