🏅 Top traders

@JamesGrugett This can be resolved right?

Can resolve?

Will it happen for next season: https://manifold.markets/jack/will-leagues-profit-be-calculated-o-a60c3ba0eb2a

Will Leagues profit be calculated on bets made before the season by Season 5?

50% chance. This is the same as https://manifold.markets/JamesGrugett/will-leagues-profit-be-calculated-o but including Season 5. Resolves YES if League profits include any bets made before the season. Resolves NO if League profits only include bets made during the season. Also related: https://manifold.markets/jack/will-leagues-profit-count-only-real Here's James's description: I'm looking for further input and analysis from the community on this question! Originallly we concieved of leagues as a mini-game that restarts from scratch each month, so any older bets were discounted. But the idea of including profit on your entire portfolio (all shares that you own) is starting to grow on me! Pros Simpler: League profit equivalent to your profit this month Long term bets will pay off at some point for leagues Fluctuations always have consequences: if you gain at one point, you will lose at another point Fixes the unintuitive case where selling out of a position at a profit leads to a loss if prices keeps trending in same direction Cons Means a lot of your performance is determined from past bets, rather than specifically the work you put in this month. E.g. large positions from previous bets could ensure victory or doom you. Means the leagues computation is more complex in that it considers more bets. If you inspect a users' profit and loss, there could be like a thousand markets contributing to their profit. Technically, this is not so difficult. The UX might be a bit worse. What do you think? [link preview]

Can this resolve now?

Here's another related idea I was thinking about - counting only profits on resolved questions, or only realized profits:

@jack I was thinking about this too, the existing system only counts bets made during the month and counts the paper value, not the ultimate value. Locking in realized profits whether by selling high or by resolution, make sure that everything is "real" and not just some paper bubble, but it probably drastically reduces the size of qualifying events, unless they lift the restriction to only bets made in the subject month. Profit taking in a volatile market is not necessarily the same as predictive strength, and realized profits will further incentivize short term markets over long term markets.

`select user_id from user_embeddings

where user_id = any($2)

order by user_embeddings.interest_embedding <=> (

select interest_embedding from user_embeddings where user_id = $1

)`

@JamesGrugett correct me if I am wrong in my interpretation:
I have been curious about this, so I looked it up. Cohorts are not balanced by any performance factor other than your current tier. You are lined up by your interest vector and then the line is sliced into cohorts such that ostensibly the traders in your cohort are in your interest neighborhood. This might not be the case where you are on an interest neighborhood boundary and the cohort before you is full you could end up in a different pool. Not sure how continuous the interest metric is but I imagine there could be some discontinuity or sharp neighborhood boundaries.

So in many ways one tier is not equivalent to another tier, these are sudo-interest leagues. Platinum Tier Politics may out perform Diamond Tier Philosophy.

This interest grouping probably makes for a much more diffuse tier system. Ostensibly if you are grouped up by interests then you should expect to be trading in the same markets as the traders in your cohort so theoretically the cohort is more likely to actually be adversarial such as a sport like tennis rather than purely performance based such as a sport like bowling.

I don't know how interests are computed, but any attempt to calibrate performance metrics will be to a degree segregated by these interest neighborhoods.

@ShitakiIntaki Yes, this is all correct. You tend to be matched into cohorts of people with similar interests, which means people who bet in similar markets.

Not making this change seems prima facie terrible to me. If Leagues is arranged to be the primary thing people are aiming at doing well on, then if they don't track whether you are correctly predicting questions (but only whether you are correctly predicting the direction of change of opinion within a month) that seems to make every market on the site a kind of Keynsian beauty contest. And doing well on Manifold stops being a legible signal of general forecasting skill. (And even if you keep trying to maximize your profits, it's not clear what success means, because you are playing against people who are playing a different game, not other profit maximizers.)

I've used Manifold less lately, I think substantially because of Leagues—like, I don't know what I'm trying to do. I could try to do well in leagues, but that feels like not proper forecasting or what I want to do, or I could ignore leagues and try to maximize profit, but that seems like i'm just playing a different game from other people, so maybe it's easier etc, and who cares if I do well at it?

@KatjaGrace What are your thoughts on the calibration scores? The site in general, and mana leagues in particular, incentivize profit strategies, short term profits in the case of mana leagues. Suppose there were leagues ranked by calibration scores (which provided large enough calibration league bonus to offset forgoing profit taking) would that be more aligned with the values you hold?

@ShitakiIntaki Calibration scores are not a good measure of predictive ability: see my comment explaining why https://manifold.markets/jonsimon/will-manifold-add-a-best-calibrated#iOXpR0vr3gyCx8FgxinZ

Will Manifold add a Best Calibrated Leaderboard by May 2023?

Resolved NO. Manifold recently added a link to a per-user calibration plot on each user’s profile page. Will Manifold add a calibration score leaderboard to allow users to compete on this metric, as they currently do for Profit and Market creation? This market resolves Yes if they add such a leade…

@jack Calibration scores are not a great measure of predictive ability, but profit isn't any better, imo

@ShadowyZephyr I disagree, profit is generally better, but:

• you have to look at the appropriate set of markets (e.g. someone's profit-making on politics markets vs tech markets vs finance markets might be completely different) - the same is true for calibration of course

• you have to watch out for variance - profit can swing a lot on a few big markets, so it's high variance.

@jack I responded over at the original comment but will ask a short version here, because I am not familiar with how calibration is calculated.

• Is there a positive correlation between accuracy and calibration?

• Assuming there is a correlation, is the correlation between accuracy and calibration, stronger or otherwise less prone to manipulation or variance, than the correlation between accuracy and profits?

@ShitakiIntaki good accuracy generally requires good calibration, but good calibration is not sufficient to give good accuracy.

Also note, it is trivially easy to arrange all your predictions to get a perfect calibration - for example, if the only thing you do is randomly pick YES or NO at 50%, you will be perfectly calibrated. Making profit requires that you actually have an edge in the market.

@jack So my take away is that a low accuracy strategy can achieve a perfect calibration, perhaps constrained to discrete outcome systems because a continuous system could have upward and downward trends which would reflect as miscalibration to the lazy or low accuracy agent, which means that a high accuracy agent could have a strictly lower calibration score (because you cannot beat perfect), and therefore calibration should have no meaningful correlation to accuracy.

@ShitakiIntaki Well, if people optimized for calibration above all else, there wouldn't be a meaningful correlation. But in the real world, people do optimize for accuracy and not for calibration - and so in practice good accuracy tends to correlate with good correlation.

Which is why Manifold most definitely should not set up a reward system based on calibration alone - it's a metric that has its uses, but when people optimize for it it's horribly Goodhartable

@jack So then a hybrid system, with some weighting for both. Long term profits matter still but smooth out some of the variance of profit spikes by convolution with calibration, but not so much so that calibration dominates.

This is a tough one, but I'd lean towards saying that all profits from previous months should count for a few reasons:

• Accurate predictions in long term markets that take a while to come good seem like the most valuable/impressive predictions and we should probably be incentivising for them. One of the predictions I'm most proud of is my betting at this market - https://manifold.markets/Heaffey/will-the-bank-of-england-base-rate - I bet on YES at 4% back in May when everyone thought that it was looking very unlikely and by June it was trading at over 80%. I wouldn't have got the credit for this bet in May because the price hadn't gone up yet and I wouldn't have got the credit in later months because it's an old bet. But surely it's more impressive to make a prediction a few weeks before anyone else realises that I'm right?

• Selling out of a winning position in a later month can actually count against me. Let's say I want to profit-take from that market above and lock in some of my profits now that the market has reached 85%. I think that 85% is a fair level, but I'd like to get some of the profit from my 4% prediction even in the one in six chance that interest rates don't reach 5.5% so I sell half of my position. As it turns out that my original prediction was right, the percentage keeps on heading towards 100% and I'm happy to see my overall profit keeps ticking up. But the league is only counting the half of the bet that I sold off this month, so now I'm showing a negative return in the league for this month even as my profits keep going up!

• Including past months' bets would make it harder to manipulate markets without seeing losses in future months. You've bet the AI Doom market up from 30% to 45%? You'll see a profit this month, but you'll see a loss in future months unless you're going to try to keep it up there! Obviously manipulation is still possible, but it's harder.

@SimonGrayson Very good summary. Matches my thoughts on the topic.

I think it's weird that leagues only includes trades for the month, but for some things it makes a lot more sense - like, you probably don't want people with loads of markets insta-winning leagues from trader bonuses on old markets.

Profits are different because they are limited by being zero-sum (kinda). I'm sure there would be other ways to rebalance monthly activity without opening markets up to manipulation in relation to leagues. Ideas I thought of in the last 30 seconds:

• Daily bonus M25 actually giving M50 value in league points

• Traders from markets created within that month giving bonus league points

• Bonus league points for participating in "featured markets" (idk, like feature a market or something that you want people to play more, like showcase, but just give league points rather than juicing it with subsidies)

• Bonus league points for realised gains on trades held for longer period (incentivises trading longer term markets, with expectation of big payout if you're right in 2 years. Impacts risk-free rate and makes markets more efficient)

• Bonus league points for predicting something correctly that resolves regardless of bet size (would need to have minimum trader requirements, not be in a gambling group or whatever, etc.. probably not a great idea)

I'm sure if a few people dedicated some time to finding better points for leagues, there would be far better options than those, and better options than relying heavily on profit.

I like this change as part of keeping leagues fresh, and I think it has the benefits you describe. If I am predictably going to be punished each month for my past bets, maybe I should sell those shares?

An alternative: only count realized profits, but count all realized profits this month, regardless of when the bet was made. So sure, if you make a bunch of good predictions on an election, that might be a good league month for you. Sounds legit.

The market is pretty close to 50% at the moment, so maybe use one mechanism for a half of cohorts and other for another half and then poll whether the new idea worked? 😃

Side note: I would argue that if you are doing this change you need to adjust profits for total portfolio size. A nice side effect of counting only the last month is that whales cannot leverage their entire portfolios because you'd need to have a very big turnover rate every month.

@AlexbGoode Yeah, a competition with one month’s profits as a proportion of portfolio size at the start of the month could be very interesting.