I'm rerunning / updating my [Manifold calibration analysis code](https://github.com/vluzko/manifold-markets-python).
Considering all resolved binary markets, will Manifold be correctly calibrated halfway between the start and close of the market?
Procedure: All markets get bucketed into 10% intervals (5% at the edges). This gives us 11 buckets: 2.5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 97.5%. Then we compute what fraction of each bucket actually resolved YES. Perfect calibration would be if 2.5% of the 2.5% bucket resolved YES, 10% of the 10% bucket, etc.
I will consider Manifold well-calibrated if 7 out of the 11 are within 5 percentage points of perfect calibration (so if 5% to 15% of 10% confidence markets resolve YES then the 10% bucket is well-calibrated, if 7 buckets are well-calibrated this market resolves YES). (This is not an ideal measure of calibration but I want something simple so the market resolution criteria is easily understandable).
🏅 Top traders
I had seen this previously but didn't really read it properly until now (doing a write-up for the newsletter).
And I must say, this is fantastic! Thanks for taking the time to create this!
There were a few bugs with my previous table. This one should be more accurate:
Looks like it's probably right but given how close 80% and 40% are to the edge I'm gonna take a couple M$ of NO in case we find another bug! mind posting your code/data?
@Yev Was this market midpoint tho?
Hmm this seems pretty consistently undershooting, I wonder if someone could bot everything up by 1% and be profitable (excluding 1-5% range ofc where we expect higher % due to ppl not bidding down as returns deminish. Botting everything at 4-5% down by a percent ot two might also be a reasonable idea in expectation)
@GeorgeVii This is genuinely very cool tho. Well done folks!
@GeorgeVii I think I accidentally measured halfway between start and resolve instead of halfway between start and close. I don't expect it to make much of a difference, but I should redo the calculations.
@SG I am certainly going to run that as a reference class but the purpose of this question isn't to ask about "good" Manifold markets or anything like that. It is just asking "if you see a Manifold market and all you know is that it's a real market intended to be bet on, how much information does that give you?"
Are you going to look at all markets, or only markets with at least a certain volume / #traders?
(if all, markets like https://manifold.markets/Marketeer1?tab=markets are going to distort the results)
@Yev I was originally planning on all markets (thinking the number of markets like this would be small), but this may actually be enough markets to distort the results, particularly for the 50% bucket. I will filter for markets with at least 5 trades. If that does not work / someone else points out other things that need to be filtered I will update the criteria further.
At market midpoint will Manifold be well calibrated?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition