BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?
8
170Ṁ98
resolved Jan 9
Resolved
N/A

  • Benchmarks

  • Only the sub benchmarks that are scored as an accuracy (i.e. from 0-100%) will be included (I think that's all of them but I'm not sure)

  • It must be a single model. If Model A achieves 75% on half and Model B achieves 75% on the other half that does not resolve the question YES

  • Ensemble models are fine but something like "run Model A on this benchmark and model B on this other benchmark" is not. If there is model selection is must be learned and it cannot include the current benchmark as an input.

  • Update 2025-05-01 (PST) (AI summary of creator comment): - If no BIG-bench results are available for any major models by the resolution date, the market will be resolved as N/A.

    • NO will not be resolved based solely on SOTA results from 2023.

    • YES will not be resolved based on personal predictions.

AI
Technical AI Timelines
Get
Ṁ1,000
to start trading!

People are also trading

BIG-bench accuracy 75% #3: Will SOTA for a single model on BIG-bench pass 75% by the start of 2026?
86% chance
BIG-bench accuracy 75% #4: Will SOTA for a single model on BIG-bench pass 75% by the start of 2027?
86% chance
BIG-bench accuracy 75% #5: Will SOTA for a single model on BIG-bench pass 75% by the start of 2028?
87% chance
Will any model get above human level on the Simple Bench benchmark before September 1st, 2025.
25% chance
MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?
6% chance
MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?
8% chance
MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?
44% chance
What will be true of the SOTA AI on the FrontierMath benchmark, before 2026?
What will be the highest score achieved on SWE-Bench Verified in 2025?
What will be true of the SOTA AI on the FrontierMath benchmark, before 2027?
© Manifold Markets, Inc.•Terms•Privacy