When will no-calculator SOTA on the MATH dataset pass 90%?
Basic
9
Ṁ8682040
5%
68%
Before 2025
20%
2025-2027
4%
2027-2029
0.5%
2029-2031
0.2%
2031-2033
0.2%
2033-2035
1.6%
2035-2040
0.2%
2040-never
This question is managed and resolved by Manifold.
Get
1,000and
3.00
1,0003.00Sort by:
GPT-4 is at 42.5% pass@1 https://arxiv.org/pdf/2303.12712.pdf#page=36. It seems that it is possible to use majority voting (the same thing Minerva did). I would expect such a model to be around 65%.
Related questions
Related questions
BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?
65% chance
MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?
44% chance
When will SOTA for Atari 100k pass human median and mean score on all 57 games?
When will a single model first achieve 10@k solve rate >= 90% on the CodeContests dataset?
MMLU 99% #2: Will SOTA for MMLU (average) pass 99% by the start of 2025?
12% chance
MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?
16% chance
MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?
12% chance
BIG-bench accuracy 75% #5: Will SOTA for a single model on BIG-bench pass 75% by the start of 2028?
87% chance
BIG-bench accuracy 75% #3: Will SOTA for a single model on BIG-bench pass 75% by the start of 2026?
86% chance
BIG-bench accuracy 75% #4: Will SOTA for a single model on BIG-bench pass 75% by the start of 2027?
86% chance