When will no-calculator SOTA on the MATH dataset pass 90%?
Basic
10
Ṁ1168resolved Nov 21
100%92%
Before 2025
0.2%Other
7%
2025-2027
0.3%
2027-2029
0.0%
2029-2031
0.0%
2031-2033
0.0%
2033-2035
0.0%
2035-2040
0.0%
2040-never
This question is managed and resolved by Manifold.
Get
1,000and
3.00
1,0003.00Sort by:
Math 91.6 released and publicly available: https://x.com/deepseek_ai/status/1859200145037869485?t=ljaJVfZgjJjTSKjRP6vf4g&s=19
GPT-4 is at 42.5% pass@1 https://arxiv.org/pdf/2303.12712.pdf#page=36. It seems that it is possible to use majority voting (the same thing Minerva did). I would expect such a model to be around 65%.
Related questions
Related questions
What will be true of the SOTA AI on the FrontierMath benchmark, before 2026?
What will be true of the SOTA AI on the FrontierMath benchmark, before 2027?
What will be true of the SOTA AI on the FrontierMath benchmark, before 2028?
BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?
65% chance
MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?
44% chance
When will SOTA for Atari 100k pass human median and mean score on all 57 games?
When will a single model first achieve 10@k solve rate >= 90% on the CodeContests dataset?
MMLU 99% #2: Will SOTA for MMLU (average) pass 99% by the start of 2025?
12% chance
MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?
16% chance
MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?
12% chance