Anything 30+ is honestly scary territory. FrontierMath is really impressive and all, and no doubt surprising, but kinda an oh that happened type thing. This is the kind of test that would make me seriously reconsider my beliefs about AGI, great market!

ooo maybe a market on which of those two is solved at 80% or above first

@Bayesian sounds like an incentive to finetune my deepseek-giga-overfitter-hle-memorized-v1 model by EOY

@Ziddletwix yeah but that would be CHEATING! and the leaderboard thing would CATCH IT

@Bayesian most likely, but maybe they'll just put an asterisk and scold it in a footnote for being sus & bad. unclear how enforcement is actually handled in practice

fkkkk they might put the footnote saying it's sus affffff then what are we gonna do

@copiumarc I don’t think HLE is harder than Frontier Math.

/Bayesian/which-of-frontiermath-and-humanitys

@mathvc @copiumarc may the person with the best model of reality win

Surely o3 will get >20%?

@qumeric if the benchmark is knowledge heavy it might not do that much better than 4o? prolly will tho. just some low chance that it doesn't

“The dataset consists of 3,000 challenging questions across over a hundred subjects. We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting.”

Well sorry but people are gonna overfit to this. Who is gonna judge whether the model is overfitted or not?

@mathvc yes i am confused by this point. so if some model near EOY is massively overfit to HLE, scores 90%+, and they chime in "yeah its performance wasn't so crazy strong on our few holdout problems, it probably overfit a bit", that still counts as 90%+ right? is the holdout set just used as a separate confirmation of overfitting, its not incorporated to the main score?

i agree this is troubling. What do you think would be the best way to proceed?

@Bayesian i found that scale.ai and safe.ai partnered to create this benchmark and it seems that they keep up-to-date evaluations of all frontier models:

https://scale.com/leaderboard/humanitys_last_exam

I guess we can trust their judgment? That is, they will not put a clearly overfitted model on the leaderboard since it makes the leaderboard useless

@Bayesian i dunno i think all benchmarks have caveats so i'd just pick some source for what each model has achieved on the benchmark & if their screener for overfitting is weak that's kinda priced in

@mathvc yeah i agree, that probably works well enough. will add to the description

upgraded to Premium