When will no-calculator SOTA on the MATH dataset pass 90%?
9
123
Ṁ868Ṁ660
2040
1D
1W
1M
ALL
5%
68%
Before 2025
20%
2025-2027
4%
2027-2029
0.5%
2029-2031
0.2%
2031-2033
0.2%
2033-2035
1.6%
2035-2040
0.2%
2040-never
Get Ṁ600 play money
Sort by:
GPT-4 is at 42.5% pass@1 https://arxiv.org/pdf/2303.12712.pdf#page=36. It seems that it is possible to use majority voting (the same thing Minerva did). I would expect such a model to be around 65%.
More related questions
Related questions
Short Term AI 3.2: By June 2024 will SOTA on MATH be >= 90%?
77% chance
When will SOTA on MMLU STEM be >= 90%?
Will SOTA on MATH in Sep 2024 utilize a hard-coded search/amplification procedure?
56% chance
Will OpenAI's next-gen math-focused model score at least 95% on the MATH benchmark?
71% chance
MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?
44% chance
When will SOTA for Atari 100k pass human median and mean score on all 57 games?
BIG-bench accuracy 75% #5: Will SOTA for a single model on BIG-bench pass 75% by the start of 2028?
65% chance
BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?
60% chance
When will a single model first achieve 10@k solve rate >= 90% on the CodeContests dataset?
BIG-bench accuracy 75% #4: Will SOTA for a single model on BIG-bench pass 75% by the start of 2027?
67% chance