https://asteriskmag.com/issues/05/prediction-markets-have-an-elections-problem-jeremiah-johnson

@CalebW Hm, this gives me an interesting idea: Maybe Manifold is more accurate because there are fewer traders. Sites with lots of traders are going to have lots of dumb money, but those with relatively fewer will be able to overwhelm it as long as they have a few well-informed traders making reasonable trades. We even saw this with the most traded market on Manifold (LK-99): During the time when it experienced a surge of traders, it made terrible predictions, which have been corrected now that activity has quite down.

@PlasmaBallin I don't think so. In LK99, the main thing was that it was a lot of new traders, who were worse as predictors than the average experienced Manifold trader. I believe what matters is the balance of smart and dumb money.

In fact, more predictors does (on average) give better predictions. https://www.metaculus.com/notebooks/14951/more-is-probably-more--forecasting-accuracy-and-number-of-forecasters-on-metaculus/. Since that's about metaculus and not a prediction market, it isn't direct evidence, but I would bet that if we ran similar analysis on prediction markets, more traders would on average = better predictive performance.

@CalebW Reposting my comment below... I disagree with Jeremiah. I used to believe his argument re politics but I no longer think it stands up to scrutiny. Real money wins! It's not the first time some smart people think they can easily outperform the market.

This is fundamentally a bet about whether you trust real money US politics betting. My contrarian view is they are unfairly criticized for being influenced by biased partisans. Read this review of the 2020 market. Using state betting as a proxy, markets predicted Biden winning 308 vs 306 actual, and Trump winning 230 vs 232 actual. There's no way to assess the singular WHO WINS market but it's connected to the state betting through arbitrage so we can basically assess them as a complete model.

Bookies have Biden at just 72% chance of winning the nom, and this website is simply divergent for me to be confident in the 93% figure. Football/soccer markets are known to be insanely well calibrated even with the fans and biased betters, why should the Dem Nom market be different? You're asking me to have faith in a small number of virtual accounts over the proven record of real money?

@PlainBG The problem with that comparison is that the bookies have tens of thousands of previous football games to compare against and learn from and train algorithms against. There are about ten past nominations to learn from that are at all relevant. Anything beyond that is just being really close to Americans politics and having good intuitions.

The bookies in Britain don’t have any real structural reasons to be better - the amount of money they’re expecting to make on American politics pays for what, a couple full time salaries? And if you’re one of the two full time American politics employees working for them, how strong are your incentives to be accurate? If you say something is 70% likely and then it comes true, and you only predict a few events per year, it takes a lot of time before your boss can really say that you’re doing a bad job.

@Gabrielle I would argue markets are even better relative to models for small sample size events. Algorithms trained on large sample and advanced stats are a big part of bookies toolkit but adjusting the odds based on money coming in is crucial too. In political betting, the latter plays a bigger role because predictive models are so weak relative to sports. It doesn't change the calculus that markets are better relative to the alternative (punditry in this case). The only question is whether real money is superior to virtual money and as much as I LOVE this website it's not as proven.

I'm late to this discussion, but as the trader with by far the biggest stake on the biggest Biden market (https://manifold.markets/NathanpmYoung/will-biden-be-the-2024-democratic-n), I obviously have strong views on this topic, so I'll chime in - better late than never!

I think the most likely explanations are a combination of the following factors:

1) ROI considerations. Price != prediction

There's a huge difference in ROI / discount rate / opportunity cost on Manifold vs Polymarket. There are two big reasons: because of loans (i.e. free leverage) on Manifold which decrease opportunity costs, and because money on Polymarket can always be withdrawn and invested elsewhere, unlike Manifold.

These ROI considerations mean that the market price should not be interpreted as the literal market prediction, but instead as part of a sort of prediction confidence interval. If the price is 80%, that doesn't mean that the (money-weighted) average prediction is 80%. It means the average prediction is likely in a range around 80%, with a wider range when discount rates are higher.

Additionally, this range is asymmetric - it's not centered at the market price - because buying the higher-priced YES shares requires putting up much more capital, so the market price needs to be farther from your own prediction to be worth the same ROI. See this comment for more detail.

2) Different distributions of traders

I claim that the distribution of capital on Manifold is much more closely tied to past prediction/trading success, and that a much larger portion of capital on Manifold is in the hands of "superforecasters". This is essentially because capital will end up distributed based on prediction/trading success absent any inflows and outflows, and real money can be more freely deposited and withdrawn than mana. An obvious example of the impact of this on the Biden market is myself, as the trader with by far the biggest stake.

Also, I think it's reasonably clear that Polymarket traders are, on average, more Republican-leaning than Manifold traders, and that this likely causes some amount of skew in the predictions.

---

Some explanations that I don't buy:

I reject the theory that real-money markets are fundamentally more accurate than play-money or non-monetary prediction mechanisms. The evidence I've seen is mixed, so I don't think there's a clear advantage based on real money alone. And note that we only have a very small number of data points in terms of platforms - there are so many important differences between various real-money markets (e.g. fees, position limits, liquidity mechanisms) that I think are much more important than the differences between real money vs play money. I think many people believe it because it makes sense as a story, not because there's much actual evidence for it. While I do believe Polymarket prices a bit more than Manifold on average, when digging into specific markets I think there's usually more relevant factors.

Fees aren't a good explanation either. Some real-money markets do have high fees and their accuracy does suffer from it (e.g. PredictIt), but Polymarket does not. Some people have proposed that it's difficult to get money into and out of Polymarket, but this doesn't make much sense to me as a rationale - if anything I'd argue that keeping more money in the prediction platform makes it more accurate (see point 2 above), and we're comparing to Manifold where you can't really withdraw at all.

@jack I would bet on predictit but I don't currently own any crypto and I don't really want to bother with it. Might seem like a stupid reason but I doubt I'm the only one. My understanding is that there will be processing fees inherent to buying into crypto and then cashing out. I don't know how big those fees would be. I also don't know how the value of the crypto would change. Learning about all of this so I can bet on polymarket hasn't been a priority for me.

@Tumbles Yes, betting on real-money markets is a huge hassle with many obstacles, and betting on a play-money market is much easier. But is your point just that Manifold might be more accurate because it gets more predictors because it is easier to participate? If so, we can just look at the participant counts directly. And we can indeed see that the Manifold Biden nomination market has 4k traders while the Polymarket one has about 1k (I think). But that doesn't really do much to explain the price discrepancy imo.

@jack I didn't have much of a thesis tbh, was just tacking that experience onto you pointing out polymarket doesn't have fees. I personally agree with your analysis. ROI, mana distribution, and demographics would be my top three explanations.

@jack Is there a version of this claim that could be grounded out into a concrete statistical claim?

> I claim that the distribution of capital on Manifold is much more closely tied to past prediction/trading success, and that a much larger portion of capital on Manifold is in the hands of "superforecasters". This is essentially because capital will end up distributed based on prediction/trading success absent any inflows and outflows, and real money can be more freely deposited and withdrawn than mana.

Something like:
- "[Manifold users weighted by M$ currently invested] have better past accuracy scores than [Manifold users weighted by # of markets bet in], and by a greater amount than [Polymarket users weighted by $ currently invested] have better scores [Polymarket users weighted by # of markets bet in]."
...which would be true if net-deposits were sufficiently uncorrelated with past accuracy, and would be false if net-deposits were sufficiently positively correlated with past accuracy. (The latter might happen if successful PM traders double down and unsuccessful PM traders net-withdraw and leave.)

Yeah, pretty much. You'd need to calculate accuracy scores that are comparable, which means identifying markets that are exactly the same or almost exactly the same (something I've looked at a bit). And then see how that correlates with user investments.

Trying something kinda crazy like this but for AI doom. Except this time, I'm promising to un-resolve the N/A resolution if we're all alive in the future:

Thanks to everyone for the discussion! Of all the answers, this one gave me the most additional insight.

As much as I consider betting on Biden to be guaranteed profit, it's even more guaranteed profit to bet against RFK Jr, Dean Phillips, etc and keep all the options arbitraged so that you literally profit no matter what.

It's not shocking to me that there are a handful of people out there who would each bet their one long-shot candidate up to 5% on the off-chance that Biden dies. But if four different people are willing to bet up four different long-shots to 5%, suddenly that's 20% of the probability space and Biden ends up at ~80% even if those four different people all think he should be at 90%+.

Sending 500 mana to @Gabrielle!

@Joshua By the way, it seems that in the last week Polymarket introduced a new version of their dem nom market that has a different format. I don't understand all the exact mechanics, but I think it's doing some amount of automatic arbitrage like our dependent multichoice markets?

The probabilities are already a bit more sane.

Old format Polymarket:

New format Polymarket:

Ignore Phillips at 50%, that just means no one is betting on him.

@Joshua In the new one, Biden is already 2 points higher. The difference is that there is something like the 'negative risk' that Insight Prediction and Predictit have, where you can convert no shares to the other yesses. So, you can buy Robert Kennedy at 98, and instead of holding that for six months, you can convert to yes shares, and then sell the others, and not be stuck with a 4% APR when risk-free Treasuries pay 5%. The diff in the Biden price is just that the sharks discovered the new market already and the people buying no aren't as clever. They could be shorting biden at 82 cents elsewhere even.

Actually, I think we can disprove this explanation by looking at the markets in detail. (I think it's a real effect, but a minor one, and doesn't really explain the discrepancy.)

If you look at a well-traded, high-liquidity market on just yes/no Biden, e.g. https://polymarket.com/event/who-will-win-the-us-2024-democratic-presidential-nomination/will-joe-biden-win-the-us-2024-democratic-presidential-nomination?tid=1706039928964 and look at the order book - you will see that plenty of people are willing to bet NO on Biden at the market price. So it's definitely not just people betting YES on other candidates.

And we can think about, maybe the people betting NO on Biden are arbing against people betting YES on other candidates, but I'm pretty sure this isn't what's happening - the order books and trade activity don't seem to support it, and the ROI would be terrible.

😂😂

🤔🤔🤔

I'm surprised nobody wrote this yet

@JonasVollmer Because it’s false lol

Do you have a link to the data/analysis on this historical outperformance? I haven't seen that to be consistently true. And especially not on US politics markets. I've done some analysis e.g. https://firstsigma.substack.com/p/midterm-elections-forecast-comparison-analysis and am working on more

@jack I think it's mostly true based on observing prices anecdotally (without making a detailed analysis). I'm not really aware of any exceptions apart from Biden Dem Primary + Trump POTUS, but I can list lots of examples in the other direction (NH primary, TIME PotY, LK-99, OpenAI).

The usual ranking seems to be: Kalshi > Polymarket >> Manifold >> PredictIt

@JonasVollmer Hmm. I agree that on average I usually trust Polymarket more than Manifold when both are well-traded, but in my also anecdotal experience it's more like a 60-40 advantage than 90-10. And once I'm informed about the market then I don't really factor that in much, the "base rate" of typical accuracy is easily overridden by specific knowledge about the market when deciding which of two different prices I trust more.

My own typical ranking is something like: Metaculus > Kalshi > Polymarket > Manifold > PredictIt (assuming high activity, of course).

Going through the examples you cite in favor of Polymarket, I only see one clear win?

NH primary - Manifold seems slightly ahead. Biggest markets were https://manifold.markets/AaronBailey/will-haley-beat-trump-in-new-hampsh vs https://polymarket.com/event/new-hampshire-republican-primary-winner?tid=1706060933337 and looks like Manifold typically was slightly lower.

TIME POTY - Polymarket big win. They had Swift priced much higher than Manifold

LK99 - tie. You have to remember to compare markets with the same end date, because the biggest LK99 market on Manifold was 2024 while the Polymarket one was 2023. If you compare

https://manifold.markets/StrayClimb/will-the-lk99-room-temp-ambient-pre-17fc7cb7a2a0 to https://polymarket.com/event/is-the-room-temp-superconductor-real?tid=1706060606277 they are basically the same.

OpenAI - looks like a tie to me, but not sure which market you had in mind - I'm guessing this one: https://polymarket.com/event/sam-back-as-ceo-of-openai?tid=1706061218208 vs https://manifold.markets/NealShrestha58d3/sam-altman-will-return-to-openai-by

@jack Good points, I don't really have the time to dig into the details here, but I think a thorough analysis would actually be pretty interesting.

I think Kalshi did better than Polymarket and Manifold on OpenAI (and yes, that question).