MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?
12
59
แน1.2Kแน230
2027
12%
chance
1D
1W
1M
ALL
Get แน200 play money
Sort by:
I think it will resolve as NO because:
People will lose interest in MMLU after around 95%, so it's possible that GPT-x or whatever SOTA there will be will not even be evaluated on it.
There are definitely at least some ambiguous or simply incorrect questions in MMLU, I am not sure how much but I will not be surprised if it's close to 1%.
Reaching 99% is usually much harder than 98%.
Metaculus predicts 93% in mid-2025, and I personally predict 91.5%. Will still be a long way to go.
More related questions
Related questions
BIG-bench accuracy 75% #4: Will SOTA for a single model on BIG-bench pass 75% by the start of 2027?
67% chance
BIG-bench accuracy 75% #5: Will SOTA for a single model on BIG-bench pass 75% by the start of 2028?
65% chance
By 2026, will it be standard practice to sandbox SOTA LLMs?
26% chance
BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?
60% chance
BIG-bench accuracy 75% #3: Will SOTA for a single model on BIG-bench pass 75% by the start of 2026?
86% chance
Will a SOTA model be trained with Kolmogorov-Arnold Networks by 2029?
14% chance
Short-term AI 3.3: By June 2024 will SOTA on HumanEval be >= 99%?
5% chance
Short-term AI 3.4: By June 2024 will SOTA on APPS be >= 25%?
25% chance
MMLU 99% #2: Will SOTA for MMLU (average) pass 99% by the start of 2025?
15% chance
Short Term AI 3.2: By June 2024 will SOTA on MATH be >= 90%?
14% chance