paleink avatar
paleink
closes Dec 31
Will a winner of ACX prediction contest (blind mode) be a minmaxer?
10%
chance

minmaxer here is defined as anyone who has responded to >25 questions with either 99% or 1% probability
if i'm correct, there are 73 people (out of 3295) who have done that

Sort by:
StevenK avatar
Steven

From the contest form:

Scoring will be through some proper scoring rule, probably Brier score.

Does anyone know if there's been more info about this? Brier score is going to be a lot friendlier to min-maxing than log score, right?

Back of envelope calculation using https://stattrek.com/online-calculator/binomial:

Suppose a min-maxer puts 100% or 0% on all 50 questions, and gets it wrong 20% of the time. Then there's a 1.85% probability they'll make at most 4 mistakes, a 0.566% probability they'll make at most 3 mistakes, and a 0.129% probability they'll make at most 2 mistakes. With 73 players, there's a 61% chance at least one makes at most 4 mistakes, a 34% chance at least one makes at most 3 mistakes, and a 9% chance at least one makes at most 2 mistakes. These imply Brier scores of at most 4/50, 3/50, and 2/50, or 0.08, 0.06, and 0.04, respectively.

Then suppose everyone else puts either 0.2 or 0.8, and gets it wrong 20% of the time. Out of the 3222 normal players, there's a 98.4% chance at least one makes at most 2 mistakes, a 46% chance at least one makes at most 1 mistake, and a 4.5% chance at least one makes no mistakes. The Brier scores corresponding to this are (48*0.2^2 + 2*0.8^2)/50 = 0.064, (49*0.2^2 + 1*0.8^2)/50 = 0.052, and 0.2^2 = 0.04.

On these assumptions, if my calculations are right, the probability that the best minmaxer outperforms the best normal player is something like 22%, even though only 2.2% of players are minmaxers. So I do think minmaxing ends up increasing the probability of winning first place.

With log score, there's a 99.9% chance that every min-maxer gets at least one question wrong and dies an instant Bayesian death. So to the extent that minmaxing is bad, this is an argument for log score over Brier score.

I don't know how more realistic assumptions (differently skilled predictors, different question difficulty, minmaxing on only a subset of questions, less extreme minmaxing) would affect this.

paleink avatar
paleink

there are only 11 people who have gone all-in on this strategy, others have <50 questions minmaxed
and the limits are 1% and 99%, although you may have already knew that, so log score is not as punishing for them
also two serious minmaxers in a previous tournament (for 2022) got the worst (brier and log) scores, so take that as you wish
i personally think it won't happen, although i don't bet on my own markets