HumanEval 90% #3: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2026?
Basic
8
Ṁ367resolved Jun 10
Resolved
YES1D
1W
1M
ALL
Benchmark link: https://paperswithcode.com/sota/code-generation-on-humaneval
pass@1 means the model gets a single attempt.
This question is managed and resolved by Manifold.
Get
1,000and
3.00
1,0003.00Related questions
Related questions
What will be the best normalized score achieved on the original 7 RE-Bench tasks by December 31st 2025?
Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?
61% chance
Will any model get above human level on the Simple Bench benchmark before September 1st, 2025.
55% chance
What will be the best score on the WebArena benchmark before 2025?
64% chance
Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?
62% chance
AI resolves at least X% on SWE-bench assistance, by 2025?
By 2025, will most well-educated people expect AI to within 10 years be better at intellectual work than 99% of humans?
20% chance
Will an AI achieve >85% performance on the FrontierMath benchmark before 2027?
29% chance
What will be the best score on the GPQA benchmark before 2025?
80% chance
AI resolves at least X% on SWE-bench WITH assistance, by 2028?