HumanEval 90% #2: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2025?
Basic
8
Ṁ443resolved Jun 10
Resolved
YES1D
1W
1M
ALL
Benchmark link: https://paperswithcode.com/sota/code-generation-on-humaneval
pass@1 means the model gets a single attempt.
This question is managed and resolved by Manifold.
Get
1,000and
3.00
1,0003.00Related questions
Related questions
Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?
68% chance
Will an AI achieve >85% performance on the FrontierMath benchmark before 2027?
68% chance
What will be the best normalized score achieved on the original 7 RE-Bench tasks by December 31st 2025?
By 2025, will most well-educated people expect AI to within 10 years be better at intellectual work than 99% of humans?
20% chance
Will any model get above human level on the Simple Bench benchmark before September 1st, 2025.
55% chance
What will be the best score on the WebArena benchmark before 2025?
64% chance
Will an AI SWE model score higher than 50% on SWE-bench in 2024?
20% chance
What will be the best score on the GPQA benchmark before 2025?
85% chance
AI resolves at least X% on SWE-bench WITH assistance, by 2028?
Will an autonomous agent resolve 90% of tasks on SWE-bench by 2025?
11% chance