If someone does statistical analysis on manifold markets ability to predict things, will I be impressed by the accuracy?
8
166
แน€240
resolved Jan 3
Resolved
NO

there are aready good ways of measuring how good someone or something is at predicting things, for example Brier score
https://en.wikipedia.org/wiki/Brier_score

I have heard alot from rationalists about how accurate prediction markets will be/are and how one day soon everyone will use prediction markets to make every decision, I am wondering whether they actually are/will be impressive, I have also heard that manifold markets will not be as accurate since it doesn't use real money so I will give it some slack.

if someone calculates how accuarate manifold (or another prediction market is) with a sufficiently large sample size, in a way that they can convince me is a relatively random sample of markets, and where the people betting on the result couldn't influence the result to a large degree and with the predictions being analyzed are locked in sufficiently before the result happens (anyone can acurately predict something that already happened) and also finds or calculates similar metrics for individual people making similar predictions and shows me their analysis, then I will resolve based on how impressive I think the prediction markets results are.

I will resolve no if someone can show me that manifold markets (or another prediction market) is worse at predicting than experts. other than that resolution will be up to my own discretion of what i think is impressive.

I reserve the right to be impressed or unimpressed by anything.

Get แน€1,000 play money

๐Ÿ… Top traders

#NameTotal profit
1แน€29
2แน€8
Sort by:
predicted YES

Would you be able to leave your thoughts on why you marked no? I understand analysis like this has been done and has shown manifold to be well calibrated. I can see it's linked below but hasn't received a response from you?

@ElliotDavies i would guess that the analysis you are talking about that shows it's well calibrated uses the probablity at the time the market closes, which is in my experience is usually after the event has all ready happened which is why I added the requirement that it must be sufficiently before the event. if you play around with that tool on probabilty on close along with some other sensible requirements you can get the brier score down to .0077 which is very impressive, except that i don't believe that's at all related to prediction ability. if you use probability at 75% or time-weighted probability with other sensible requirements which i think should be enough to verify that theres enough time for people to make predictions but still before the event happens the brier scores I could get it to give are .14-.18, a lot less impressive, this tool didn't give me an option to compare it to individuals prediction ability so when i went looking for an individuals brier score I found the good judgement projects study which found that Doug Lorch got a brier score of .14 on his predictions, so an individual either beat or matched all of manifold, granted i didn't take into account how much luck played a part in Doug Lorch's score, and there's probably other arguments one could make, but i didn't want to spend too long doing analysis and this market is largely subjective and I did not find it impressive. if you have any other analysis that you think could change my mind I'd love to see it though it's too late for me to change the resolution.

@bluerat Any update?

@nikki has anyone done statistical analysis?

Probably going to resolve N/A, but ill give it like another few hours to see if anyone has any

bought แน€70 of YES

I imagine such an analysis would not include markets without x number of investors - which is what I think is the main issue with manifold markets

@ElliotDavies Agree. Accuracy of prediction markets should be roughly lognormal with the number of traders. Unfortunately the repeat banhammers and fines have almost certainly stifled the shit out of PMs from being what they should've been by now.