What Brier score will the best model achieve in the $125k Autocast warmup competition?
3
resolved Mar 8
Chosen
3%
apartresearch avatarAbove 20
27%
apartresearch avatarAbove 50
3%
apartresearch avatarAbove 60
3%
apartresearch avatarAbove 40
3%
apartresearch avatarAbove 30
3%
apartresearch avatarAbove 25
3%
apartresearch avatarAbove 85
3%

This question will resolve to the minimum Brier score achieved on the leaderboard by a qualified submission. The calibrated random baseline is 85. The lower the better. See an example of the Brier score in action.

See the competition page.

From the Evaluation page

For true/false and multiple-choice questions, we evaluate models using the Brier score, which is then divided by 2 to normalize between 0% and 100%. For numerical questions, we use L1 distance, bounded between 0% and 100%. We denote these question types as T/F, MCQ, and Numerical, respectively. To evaluate aggregate performance, we use a combined metric (T/F + MCQ + Numerical), which has a lower bound of 0%. A score of 0% indicates perfect prediction on all three question types. For more details, please check out the Autocast paper.

🏅 Top traders

#NameTotal profit
1Ṁ60
2Ṁ39
Sort by:
BTE avatar
Brian T. Edwards

Those don’t look like Brier Scores to me.

Related markets

Will @1941159478 finish first place in the Salem/CSPI Forecasting Tournament?54%
Will @zubbybadger finish first place in the Salem/CSPI Forecasting Tournament?45%
Will @1941159478 finish in the top 5 in the Salem/CSPI Forecasting Tournament?87%
Who Will be Top 5 of the Salem/CSPI Forecasting Tournament Leaderboards?
Will @HenriLemoine finish first place in the Salem/CSPI Forecasting Tournament?4%
Will a current Manifold Markets leaderboard user win the Salem/CSPI tournament?55%
Who Will be Top of the Salem/CSPI Forecasting Tournament Leaderboards?
In the ACX 2023 prediction contest, will there be controversy over whether someone playing on blind mode cheated?32%
Will @AdrianKelly finish in the top 5 in the Salem/CSPI Forecasting Tournament?8%
Will @HenriLemoine finish in the top 5 in the Salem/CSPI Forecasting Tournament?18%
Will a winner of ACX prediction contest (blind mode) be a minmaxer?11%
Will a winner of ACX prediction contest (full mode) be a minmaxer?9%
Will more than a third of the 2023 SSBMRank top 100 be Fox players?51%
Will @HenriLemoine finish in the top 20 in the Salem/CSPI Forecasting Tournament?76%
What will the total profit of the #1 trader be in the Salem/CSPI tournament?19K
Will the winner of the Salem/CSPI Forecasting Tournament be decided by profits from the first day being removed?41%
Who will have the highest overall profit in the Gravity Chess group at the end of the tournament?
Will anyone end the Manifold MTG Tournament with at least M$10,000 in profit?4%
Will anyone be disqualified from the CSPI tournament?32%
Will we run a Longevity Prize tournament?63%