Third time's the charm?
Scott Alexander has a market resolving in 2028 about whether Gary Marcus will be able to find "three extremely obvious questions, that an average human teenager could certainly answer, which a leading chatbot still fails at at least half the time when asked." (Weird adversarial hacks don't count.)
A year and a half ago, I thought maybe ChatGPT o3 was smart enough that it was there already. It was not. A few months later, I thought GPT-5-Thinking might've gotten there. Still not. So now it's been more than another year and maybe Claude Opus 5.5 is there?
To resolve YES, that it still makes egregious errors, we need, by market close, just one example I can replicate in a Temporary Chat (no access to my chat history) with effort set to max. We won't worry about "average teenagers". If the answer is clear to us and Claude gets it wrong at least half the time, this market resolves YES.
As always, ask clarifying questions before betting too hard. We'll discuss in the comments and aim for the spirit of the question, namely, thinking of this as a capabilities milestone and seeing how correct Gary Marcus was in his LLM skepticism. We'll generally but not necessarily follow precedents from the previous two incarnations of this market for o3 and GPT-5.
FAQ
1. Are images allowed this time?
Yes, with one concession for how much harder that is for an LLM. Normally it's not allowed to nudge the AI, but for visual questions we'll add "write code as needed to verify your answer".
2. What if it can't do that in a Temporary Chat?
Then we'll test another way. I've got Claude building an elaborate harness but anyone with a Claude Max plan should be able to replicate any question failure in a normal Claude session if they have no conversation history about that specific question and Claude doesn't otherwise cheat.
[Ignore any autoclarifications that appear below this line.]
1,000People are also trading
One of the images in the public set may count but haven't tested with code: https://static.remotasks.com/uploads/6a8dcf7f9a28fc4d7f8ba553/HSS_ICLR_2027%20(2).pdf
Other issue is going to whether these are all obviousness. As usual, things will be more obvious on a large screen and less on a cell phone screen
@CDBiddulph Nice, I hadn't seen clockbench before. With effort set to max and being able to write code, Opus 5.5 seems to nail things like this. I'm doing a more exhaustive replication to be sure though.
@bens The clear plastic forks photo further up is the closest so far. 8 of 9 humans in the thread counted 7 (one said 8). My five Opus max runs with "Task: count the forks in this image. Output: a single number" ended 7, 7, 12, 12, 12. dreev's first run with a hedging prompt landed on "most likely 7", so it comes down to his three-run replication with the single-number wording. Agreed the illusions are ambiguous; that is why most were dropped.
I was intrigued by @bobalobascrob's claim that examples must exist since there are unsaturated benchmarks, in particular SimpleBench. I had Claude investigate. It seems to think the evidence is weak. We don't know what effort level SimpleBench used (probably medium, and without tools like code) nor whether the misses were on questions that would be easy for humans. Apparently "trick questions" is one of SimpleBench's categories. We don't even know whether the supposed misses were really misses. Apparently there's something called the Platinum benchmarks paper that "found widespread answer-key errors in standard benchmarks".
Other excerpts from Claude's investigation:
DataLearner gives the top individual human score as 95.4%
An 11.6% average miss rate over five runs is compatible with no single question failing 3 times out of 5 (Lilemont’s “near misses” point). I’d expect misses to cluster on particular questions, so I’d grant that some probably fail most of the time, but that data isn’t published.
I think such questions probably exist, and SimpleBench raises that a little.
Sources:
SimpleBench: categories, human baseline, sampling settings (the leaderboard rows didn’t load)
DataLearner SimpleBench page: a secondary source for 88.4, “normal mode”, 213 questions and the 95.4% top human score
Anthropic docs: Effort: Opus 5.5 defaults to medium
Anthropic docs: Thinking, sampling parameters: non-default temperature/top_p rejected on Opus 5.5
Vendrow et al. 2025, “Do Large Language Model Benchmarks Test Reliability?”: abstract only
A single Claude max private run failed this “What is the shape in the center of this image, answer in only 1 word.” Most humans would say “cross” or “X” imo. Claude answered “rectangle”. I know this may not count because it needs to have the line that it can use code to verify, but idk how helpful that would be. I also only ran once, but it was fun playing around with it and optical illusions.
@BenjaminThaler The cross or X touches the edges of the image and I wouldn't call this "in the center". The shape right in the center isn't a cross, but a rectangular-ish polygon.
@BenjaminThaler I got this wrong. The shape in the center is not in the center of the cross, because the cross is off-axis. So probably the center one is the white one to the right of the middle (relative to the cross!) one, so it's rather a trapez shape. Still rectangular-ish enough to not call this wrong.
@Primer Okay yeah it seems my interpretation wasn’t universal. My purpose behind the question was to basically ask “What is the shape that emerges in the central area due to the illusion”, but I agree my phrasing to get to that point was ambiguous about what center means.
@BenjaminThaler I notice: It's neither clear how to answer such a prompt, nor how to prompt in the first place. I don't mean to single out your example here. See the discussion on the other examples, most are just not"correct". I think this should tell us something about when to label something an "egregious error".
@BenjaminThaler This is a good attempt but I, too, as a human, was staring at the center rectangle thing deciding how to answer until I saw your next sentence and then kind of smacked my forehead. So I'm guessing that a prompt that fixed that for humans would also fix it for Claude. I just tried with "What shape appears in this optical illusion?" and sure enough, Claude nails it.
@Primer i think this might be why so many people cant count the forks... im assuming some people haven't seen any clear plastic forks in the lifetime before :(
"can the grey part fit through the hole"
haven't fully tested but fable initially got this wrong.
any human can instantly recognize that after you thread the long arm through you can rotate the grey part and then get it through. so far fable and opus medium got it wrong... haven't run the max runs yet
@bobalobascrob opus vision is far above fable. Fable can't even read my circuit breaker levels correctly
if you ask it what the antonym of "unique" is, does it tell you "nonunique"?
i'm trying to think of cases like this where there is no true antonym except prefixing "non-" or something similar