HumanEval 90% #2: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2025?
vluzko avatar
Vincent Luczkow
6
closes 2025
77%
chance

Benchmark link: https://paperswithcode.com/sota/code-generation-on-humaneval

pass@1 means the model gets a single attempt.

Technical AI Timelines
AI

Related markets

vluzko avatarVincent Luczkow
HumanEval 90% #1: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2024?57%
vluzko avatarVincent Luczkow
HumanEval 90% #3: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2026?79%
vluzko avatarVincent Luczkow
HumanEval 90% #5: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2028?87%
vluzko avatarVincent Luczkow
HumanEval 90% #4: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2027?81%
vluzko avatarVincent Luczkow
Will any AI be able to formalize >=90% of IMO problems by the start of 2025?40%
SneakySly avatarSneakySly
Will AI image generating models score >= 90% on Winoground by June 1, 2025?85%
dreev avatarDaniel Reeves
Will AI pass the Turing test by 2029 Jan 1?74%
vluzko avatarVincent Luczkow
Benchmark Gap #2: Once we have an algorithm with human level sample efficiency for major RL benchmarks, how many years will it be before there is an algorithm with human level sample efficiency on essentially all AAA video game tasks?1.6
MatthewBarnett avatarMatthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2028?40%
IsaacKing avatarIsaac
Will AI pass the Winograd schema challenge by the end of 2025?87%
MatthewBarnett avatarMatthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2030?59%
MatthewBarnett avatarMatthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2026?21%
VictorLi avatarVictor Li
Will an AI get a perfect SAT score before 2025?86%
PeterHroššo avatarPeter Hroššo
Will AI outcompete best humans in competitive programming before the end of 2023?11%
vluzko avatarVincent Luczkow
Will any AI solve >=50 IMO problems by the start of 2024?45%
vluzko avatarVincent Luczkow
Will any AI solve >=100 IMO problems by the start of 2024?19%
vluzko avatarVincent Luczkow
Will any AI be able to explain formal language proofs to >=50% of IMO problems by the start of 2025?63%
jack avatarJack
Will an AI outcompete the best humans on any one programming contest of IOI, ICPC, or CodeForces before 2025?31%
vluzko avatarVincent Luczkow
Will AI beat the best humans in competitive programming before the end of 2024?25%
BoltonBailey avatarBolton Bailey
Will AI get at least bronze on the IMO by 2025?71%