Track Record and Accuracy
๐ Why are markets better than polls or experts?
One paper about predicting scientific paper replication compared these forecasting methods. It found that prediction markets outperformed surveys and
...could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.โ
Either prediction markets are more accurate than experts, or experts should be able to make a lot of money on them, and in doing so correct the markets.
๐ธ How does play-money compare to real money?
The paper, Prediction Markets: Does Money Matter?, concludes,
โWe found that neither type of market was systematically more accurate than the other across 208 games. In other words, prediction markets based on play money can be just as accurate as those based on real money... The essential ingredient seems to be a motivated and knowledgeable community of traders, and money is just one among many practical ways of attracting such traders.โ
This aligns with Manifold's high calibration thanks to our users being motivated by social prestige, league ranks, and the fear of losing mana.
๐ค Are markets resistant to manipulation and hype?
As the market prices moves further from the true probability, the odd's pricing becomes better for traders to correct it in the right direction. Naturally, this increases the incentive to bet accurately as there is more money to be made once the market resolves.
Robin Hanson explores this further in his paper, A Manipulator Can Aid Prediction Market Accuracy. In it he examines how both historical and lab data fail to find substantial effects of manipulation on average price accuracy. Furthermore, he finds in his model that adding a manipulator may even increase accuracy as it increases noise trading which tends to have a positive effect in low liquidity markets.
See also, Scott Alexander's failed manipulation attempt on Manifold.
๐ฑ Do markets with few traders and low liquidity work?
Yes! And very reliably! The paper Prediction Markets: Practical Experiments in Small Markets and Behaviours Observed, concluded,
โ16 or more traders should be sufficient to obtain quality predictions. Smaller markets may be just as useful, though they may exhibit biases of under confidence toward market favourites.โ
Our own data has shown that somewhere between 10 - 20 traders our calibration no longer improves with more traders. We still need to conduct analysis on the impact liquidity has on accuracy.
Case studies
Predicting Trump's arrest
On March 18th Trump posted on Truth Social that he believes he was about to be arrested, this caused our market to spike to 88%. However, since December our market had already been hovering around 40% on average before anyone else was even discussing it as a true possibility of happening!
Al-Ahli Arab hospital explosion
Just 3 hours after the initial local reports of the explosion, we had this market made. Within 1 hour of creation it had already been pushed to down 6%, before eventually settling between 6-20% over the next few hours as more news came to light.
Meanwhile, major news outlets still presented conflicting headlines, which eventually led to the BBC conceding that a reporter had been wrong to speculate in his analysis.
Meanwhile, major news outlets still presented conflicting headlines, which eventually led to the BBC conceding that a reporter had been wrong to speculate in his analysis.
Predicting SBF fraud
Manifold had a market stable between 5-10% that SBF would be convicted of a felony 1-month before there was any news about it. It then immediately reacted correctly to rumors before any official statements were made.
How we performed on the 2022 US midterms
Manifold outperformed real money prediction markets and was almost as accurate as FiveThiryEight when forecasting the 2022 US midterm elections.
Overall calibration
This chart show whether events happened as often as we predicted. We want to blue dots to be as close to the diagonal line as possible!
A dot with a question probability of 70% means we have a group of markets that were predicted to have a 70% chance of occurring. If our predictions are perfectly calibrated, then 70% of those markets should have resolved yes and it should appear on the y-axis at 70%.
Resolved Yes
Question probability
- Methodology and Brier scoreTL;DR Our data shows our markets are very accurate!ยLearn more
See more charts courtesy of @wasabipesto from our data in 2022.