HumanEval 90% #3: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2026?
83%
chance
1D
1W
1M
ALL
Benchmark link: https://paperswithcode.com/sota/code-generation-on-humaneval
pass@1 means the model gets a single attempt.
Sort by:
Valery Cherepanovbought Ṁ30 of YES
GPT-4 is 82% zero shot https://arxiv.org/pdf/2303.12712.pdf#page=21
Sort by:
Related markets
Vincent Luczkow
HumanEval 90% #1: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2024?75%
Vincent Luczkow
HumanEval 90% #2: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2025?78%
Vincent Luczkow
HumanEval 90% #4: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2027?81%
Vincent Luczkow
HumanEval 90% #5: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2028?87%
Matthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2028?49%
Matthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2026?17%
Matthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2030?60%
Orpheus
Will an AI model outperform 95% of Manifold users on accuracy before 2026?40%
Vincent Luczkow
Will any AI be able to formalize >=90% of IMO problems by the start of 2025?41%
Related markets
Vincent Luczkow
HumanEval 90% #1: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2024?75%
Vincent Luczkow
HumanEval 90% #2: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2025?78%
Vincent Luczkow
HumanEval 90% #4: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2027?81%
Vincent Luczkow
HumanEval 90% #5: Will pass@1 performance on the HumanEval benchmark be >= 90% by 2028?87%
Matthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2028?49%
Matthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2026?17%
Matthew Barnett
Will an AI be capable of achieving a perfect score on the Putnam exam before 2030?60%
Orpheus
Will an AI model outperform 95% of Manifold users on accuracy before 2026?40%
Vincent Luczkow
Will any AI be able to formalize >=90% of IMO problems by the start of 2025?41%