
Will simple-bench scores be reported in a major AI lab paper or blog post by the end of 2024?
6
1kṀ770Jan 1
29%
chance
1H
6H
1D
1W
1M
ALL
Simple-bench.com , from the YouTube channel AI Explained, asks LLMs very simple multiple choice world modeling questions. They can resemble logic puzzles but usually have an obvious gotcha. However, even when informing the LLMs about this tricky nature, most do poorly. No LLM has cracked 30%, but humans average over 90%.
The questions may undergo revision but still resolves to Yes if it is a different version of the initial benchmark by the same author.
This question is managed and resolved by Manifold.
Get
1,000 to start trading!
People are also trading
Related questions
OpenAI maintains top position on LiveBench by August 25, 2025?
58% chance
Will any model get above human level on the Simple Bench benchmark before September 1st, 2025.
15% chance
In what year will AI achieve a score of 85% or higher on the SimpleBench leaderboard?
3/4/32
In what year will AI achieve a score of 95% or higher on the SWE-bench Verified benchmark?
12/13/27
Will OpenAI models achieve ≥90% on SimpleBench by the end of 2025?
37% chance
In what year will AI achieve a score of 95% or higher on the PhysBench leaderboard?
2036
In what year will AI achieve a score of 95% or higher on the PutnamBench leaderboard?
12/31/28
Will AI Impacts publish another Expert Survey on Progress in AI by the end of 2025?
96% chance
Will we have an AI generated research paper accepted to > 1 top ML conference by 2026?
38% chance
Will it be public knowledge by EOY 2025 that a major AI lab believed to have created AGI internally before October 2023?
7% chance