HumanEval 90% #1: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2024?

AI Technical AI Timelines

Mini

13

Ṁ2.0k

resolved Oct 22

Resolved

YES

1D

1W

1M

ALL

Benchmark link: https://paperswithcode.com/sota/code-generation-on-humaneval

pass@1 means the model gets a single attempt.

Get Ṁ600 play money

🏅 Top traders

#	Name	Total profit
1		Ṁ396
2		Ṁ268
3		Ṁ56
4		Ṁ30
5		Ṁ21

Sort by:

Unless I've misunderstood what this market is asking, it seems like the answer is yes. At the linked benchmark page, the top model is currently sitting at 94.4%

@js Thank you for pointing this out

Related questions

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?

Will a LLM beat human experts on GPQA by Jan 1, 2025?

Will an AI model outperform 95% of Manifold users on accuracy before 2026?

By 2025, will most well-educated people expect AI to within 10 years be better at intellectual work than 99% of humans?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

AI resolves at least X% on SWE-bench assistance, by 2025?

Human-machine intelligence parity achieved before 2028

Will AI pass Video Turing Test by 2030?

Human-machine intelligence parity achieved before 2030

Related questions

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?

AI resolves at least X% on SWE-bench assistance, by 2025?

Will a LLM beat human experts on GPQA by Jan 1, 2025?

Human-machine intelligence parity achieved before 2028

Will an AI model outperform 95% of Manifold users on accuracy before 2026?

Will AI pass Video Turing Test by 2030?

By 2025, will most well-educated people expect AI to within 10 years be better at intellectual work than 99% of humans?

Human-machine intelligence parity achieved before 2030