Will manifold markets add a way to view calibration histograms for users, tags, or folds?
15
107
Ṁ534Ṁ102
resolved Dec 8
Resolved
NO1D
1W
1M
ALL
This market resolves to yes if there is some way to view a histogram of "% at closing" vs. "% resolved to yes in bucket" for either a user's created markets, a user's participated markets, a tag, a fold, a search query, or the site as a whole.
This does not apply if the histogram is generated by a third party, it must be on the manifold.markets site.
This market will resolve early (to yes) if said feature is announced or appears on the site before April 30th, 2022. The market will not resolve to no before the indicated closing time.
General thoughts:
I expect that for larger groups this would quickly lead to very boring histograms as calibration errors are arbitraged out, but for smaller groups or individuals it could serve as way to find persistent biases.
This could also serve as a useful mechanism to judge the quality of a user or community's judgement, especially if coupled with some indicator of bid volume (like a note showing how many markets or how much money is in each bucket.)
Also, question for the crowd: Would weighting the buckets by market volume or by a user's bid size be meaningful? show useful additional information not captured by a raw-market-count histogram?
Get Ṁ200 play money
Related questions
🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ46 | |
| 2 | Ṁ29 | |
| 3 | Ṁ23 | |
| 4 | Ṁ18 | |
| 5 | Ṁ9 |
Sort by:
Calibration graph was added long after this market closed. See here:
https://github.com/search?q=repo%3Amanifoldmarkets%2Fmanifold+calibration&type=commits&s=committer-date&o=asc
Thanks @Eliza for doing the research!
Lacking input from creator or traders, I think this resolves No.
@EvanDaniel Actually this is fairly old, possibly predating the current calibration graph. Does that count?
Basically, except that farther from the line is some combination of "more conservative" and "less liquid", either could cause you to move away from the center. In both cases you're not spending all the money needed to move the market to your actual belief.
Getting right up against the X=Y line requires that you're playing risky and have enough cash to basically move the market where you want it.
Crossing the X=Y line would just be wrong. I think it's reasonable to call that a form of calibration error but the exact causes would be up to the user to figure out (just like calibration errors in other contexts).
Closer to the line is riskier play, farther from the line is more conservative. Up to the individual which is better. On the wrong side of the line is just bad.
PS. A histogram of the implied probability Delta against the resolution % might be another interesting way to do it. See how accurate you are in correcting the market.
Hm interesting proposal! So something like https://i.imgur.com/cxwrIPh.png ?
I think it'd be pretty, but I'm not sure exactly how to interpret it -- are you trying to get the red and green lines closer to the X=Y line, or farther?
(props to @ohAitch, @jbeshir, @gwillen on the omnipolis discord for helping think this through.)
@austin: The fact that the implied probability is a bound is enough to make a basic version of calibration for trades.
For yes trades, you'd bucket trades by implied prob and check to see if the %-resolved in each bucket is greater than the implied prob.
For no trades, you'd do the same but a "calibrated" outcome is when all the buckets have an implied prob that's greater than the %-resolved.
It's a weaker guarantee than the standard definition but it's still a useful way judge trades. "Better calibrated" trading strategies would minimize the gap between the yes-trade-%-resolved and no-trade-%-resolved, while making sure the x=y line is between the two bounds.
(crosspost from Omnipolis discord)
I really want to implement something like a calibration chart! Scott's charts from his yearly ACX predictions are super inspirational. One problem though: it's not clear that a YES buy that moves implied prob from 20->25% should mean "your belief was 25 percent" - your belief was at least 25, but could have been up to 100
So basically I'm not sure how to calibrate trades. (beyond just profit/loss)
Calibrating markets you create is doable (just use the initial probability); though you might also think that markets on different timescales (resolving in a week vs one year) deserve to be counted separately somehow