HumanEval 90% #1: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2024?
13
847
Ṁ2KṀ250
resolved Oct 22
Resolved
YES1D
1W
1M
ALL
Benchmark link: https://paperswithcode.com/sota/code-generation-on-humaneval
pass@1 means the model gets a single attempt.
Get Ṁ200 play money
🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ396 | |
| 2 | Ṁ268 | |
| 3 | Ṁ56 | |
| 4 | Ṁ30 | |
| 5 | Ṁ21 |
Sort by:
Technical AI Timelines questions
Will Llama-3 be (open sourced and) as good as GPT-4? [RESOLUTION IN QUESTION, TRADE WITH CAUTION]
59% chance
By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?
53% chance
Related questions
Will AI pass the Longbets version of the Turing test by the end of 2029?
50% chance
HumanEval 90% #2: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2025?
75% chance
Will any AI be able to formalize >=90% of IMO problems by the start of 2025?
32% chance
Will an opensource LLM on huggingface beat an average human at the most common LLM benchmarks by July 1, 2024?
79% chance
HumanEval 90% #3: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2026?
84% chance
Will an AI be capable of achieving a perfect score on the Putnam exam before 2030?
33% chance
Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?
62% chance
By 2025, will most well-educated people expect AI to within 10 years be better at intellectual work than 99% of humans?
19% chance
Will an AI model outperform 95% of Manifold users on accuracy before 2026?
67% chance
Will an AI agent system be able to score at least 40% on level 3 tasks in the GAIA benchmark before 2025.
49% chance