HumanEval 90% #2: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2025?
8
129
Ṁ444Ṁ190
2025
75%
chance
1D
1W
1M
ALL
Benchmark link: https://paperswithcode.com/sota/code-generation-on-humaneval
pass@1 means the model gets a single attempt.
Get Ṁ200 play money
Related questions
AI resolves at least X% on SWE-bench assistance, by 2025?
Will an AI agent system be able to score at least 40% on level 3 tasks in the GAIA benchmark before 2025.
49% chance
HumanEval 90% #3: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2026?
84% chance
Will OpenAI's next-gen math-focused model score at least 95% on the MATH benchmark?
62% chance
What will be the best score on the GPQA benchmark before 2025?
73% chance
Will an AI model outperform 95% of Manifold users on accuracy before 2026?
67% chance
By 2025, will most well-educated people expect AI to within 10 years be better at intellectual work than 99% of humans?
19% chance
Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?
62% chance
What will be the best score on the WebArena benchmark before 2025?
47% chance
Will any AI be able to formalize >=90% of IMO problems by the start of 2025?
32% chance