MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?
Plus
12
Ṁ11652027
12%
chance
1D
1W
1M
ALL
This question is managed and resolved by Manifold.
Get
1,000and
3.00
1,0003.00Sort by:
I think it will resolve as NO because:
People will lose interest in MMLU after around 95%, so it's possible that GPT-x or whatever SOTA there will be will not even be evaluated on it.
There are definitely at least some ambiguous or simply incorrect questions in MMLU, I am not sure how much but I will not be surprised if it's close to 1%.
Reaching 99% is usually much harder than 98%.
Metaculus predicts 93% in mid-2025, and I personally predict 91.5%. Will still be a long way to go.
Related questions
Related questions
MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?
44% chance
MMLU 99% #2: Will SOTA for MMLU (average) pass 99% by the start of 2025?
12% chance
MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?
16% chance
BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?
65% chance
What will be true of the SOTA AI on the FrontierMath benchmark, before 2026?
Will there be a period of 12 contiguous months during which no new compute-SOTA LM is released, by Jan 1, 2033?
71% chance
Will a open source pure Mamaba LLM surpass 82 MMLU on MMLU (5-shot) before end of year 2024?
25% chance
Will the transformer architecture be replaced in SOTA LLMs by 2028?
66% chance
What will be true of the SOTA AI on the FrontierMath benchmark, before 2027?
By 2026, will it be standard practice to sandbox SOTA LLMs?
29% chance