Third time's the charm?
Scott Alexander has a market resolving in 2028 about whether Gary Marcus will be able to find "three extremely obvious questions, that an average human teenager could certainly answer, which a leading chatbot still fails at at least half the time when asked." (Weird adversarial hacks don't count.)
A year and a half ago, I thought maybe ChatGPT o3 was smart enough that it was there already. It was not. A few months later, I thought GPT-5-Thinking might've gotten there. Still not. So now it's been more than another year and maybe Claude Opus 5.5 is there?
To resolve YES, that it still makes egregious errors, we need, by market close [UPDATE: By the deadline listed below], just one example I can replicate [in a Temporary Chat] (no access to my chat history) with effort set to max. We won't worry about "average teenagers". If the answer is clear to us and Claude gets it wrong at least half the time, this market resolves YES.
As always, ask clarifying questions before betting too hard. We'll discuss in the comments and aim for the spirit of the question, namely, thinking of this as a capabilities milestone and seeing how correct Gary Marcus was in his LLM skepticism. We'll generally but not necessarily follow precedents from the previous two incarnations of this market for o3 and GPT-5.
FAQ
1. Are images allowed this time?
Yes, with one concession for how much harder that is for an LLM. Normally it's not allowed to nudge the AI, but for visual questions we'll add "write code as needed to verify your answer". PS: Or for questions where that doesn't make sense, "zoom in as needed to be sure of your answer".
2. What if it can't do that in a Temporary Chat?
Then we'll test another way. I've got Claude building an elaborate harness but anyone with a Claude Max plan should be able to replicate any question failure in a normal Claude session if they have no conversation history about that specific question and Claude doesn't otherwise cheat.
3. What if "write code as needed" makes it perform worse?
We'll take the best of both. Reasoning: First, that may help make up for the advantage that YES bettors got from extending the deadline on this market. Also it feels within the spirit of the question -- what AI is capable of. Finally, we weren't even sure at first that images should be allowed (they weren't in past versions of this market) so giving the AI any advantages we can on the image questions feels closer to a good compromise.
4. What's the deadline?
Probably midnight pacific, the night of October 20, 2026. To be finalized in discussion in the comments. Trading can stay open until we've finished testing and discussing but submissions after the deadline don't count. Per the spirit of the question we don't want to exclude real examples just because they came in late but originally we did say "by market close" and extended that a couple times by some number of days so we need to pick a hard cutoff and stick to it.
5. What does "clear to us (humans)" mean?
This is part of the definition of "egregious failure". We want a question that doesn't get a WTF reaction from humans or otherwise make our heads hurt trying to parse it. Roughly, the question should be phrased as clearly (for humans) as we're able to, and Claude's failure should be robust to minor rephrasings that we humans agree are all clearly the same underlying question.
[Ignore any autoclarifications that appear below this line.]
1,000People are also trading
@traders As far as I can tell, we've got no slam dunks in terms of stumping Claude. There's still potential debate to be had on the fenced-horses question and on the self-pulse question. Anything I've missed?
The next most promising candidate for stumping Claude is the following question about the above image:
Based on the doodle found directly below image 10, what image number does this action correspond to from the images 1 - 10, and what is the person checking?
Before trying this, I thought the "correspond to" wasn't super clear so I composed this version, which I think is asking the same thing:
See the smaller doodle directly below image 10? Which numbered image, 1-10, matches what's happening in that small doodle?
I started the experiments and then realized I forgot the "what is the person checking?" part, so I added that, for 3 variants. Then it occurred to me that "use code as needed" isn't so helpful here but "zoom in and out as needed" might be. So I'm trying all these variants and have this so far:
So... we might be hitting the 50% threshold across these variants with the "zoom in and out" version of the nudge?
@dreev I'm not following why this market is allowing arbitrary rephrasing of prompts.
I got the point of the question is whether Gary Marcus could ask a question that produces the wrong answer. I don't see anything in this market either that allows arbitrary rephrasing of questions, other than allowing a fixed prompt suffix. Your resolution criteria seems to be far tighter than what the market proposed.
@MugaSofer The PS to allow zoom was not in the original market description when I bet. And the PS5 to allow "rephrasing" was not present either.
The only row mapping to the terms I bet under were the HSS original, all which have > 8/10 errors, resolving the market YES. (even the unspecified generic nudge (not in market when I bet) is at 5/10 wrong).
Even if I accept the question isn't clear (and I don't), the modification still fails the original prompts when I bet (allowing only writing code). The only combinations not failing > 50% of the time are simultaneously a rephase (new) AND asking it to zoom (new)
It is violating the spirit of this question to allow per-question arbitrary rewrites to pass. Models are somewhat probabilistic, so not surprisingly, some arbitrary rewrite of a question may exist that leads a wrong answer to going correct for any wrong question we find.
I would not have bet "yes" in this market if this were stated, because a-priori such rewrites are relatively easy to find (even a year ago, nearly any public simple-bench question could be rephrased to have models get 100% on them).
@Usaar33 Agreed. My experience is that when models fail, they become far less stable and more sensitive to rephrasings of the question. This approach of looking for rewrites practically kills all yes/no questions just by chance. I'm exiting the market out of fear that it has effectively become about the whim of how its creator chooses to resolve it.
@Melquiades Also, Marcus has clearly commented on this way back in the GPT-3 era (https://www.technologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/):
Defenders of the faith will be sure to point out that it is often possible to reformulate these problems so that GPT-3 finds the correct solution. For instance, you can get GPT-3 to give the correct answer to the cranberry/grape juice problem if you give it the following long-winded frame as a prompt:
In the following questions, some of the actions have serious consequences, while others are perfectly fine. Your job is to identify the consequences of the various mixtures and whether or not they are dangerous.
1. You poured yourself a glass of cranberry juice, but then you absentmindedly poured about a teaspoon of grape juice into it. It looks okay. You try sniffing it, but you have a bad cold, so you can’t smell anything. You are very thirsty. So you drink it.
a. This is a dangerous mixture.
b. This is a safe mixture.
The correct answer is:GPT-3’s continuation to that prompt is, correctly: “B. This is a safe mixture.”
The trouble is that you have no way of knowing in advance which formulations will or won’t give you the right answer.
My own perspective is that egregious errors are a problem whenever the model could mislead a user who does not know the answer already -- if a question is ambiguous, the model should complain, not guess. In practice, user questions look more like "how do I clean this stain (image)" and not "if you wash this stain with 25% vinegar, will the cloth be damaged? Remember to consider the chemical interaction of vinegar and cotton."
@Melquiades I affirm your point. A minor clarification.
Rephrasings that preserve the exact meaning should be allowed, because that tests prompt stability. Rephrasings that (even subtly) give additional information or instruct the model how to approach the question should not be allowed. After all, many of these defeat the point.
In the case of these doodles, some rephrasings drop half the question (seriously?).
I understand that you're withdrawing your position because it's at risk depending on @dreev's whim in resolving this market. I've had similar qualms. However, I'd appreciate it if you held a symbolic M1 YES position and defend your beliefs, which are correct.
There's a reason why a vast majority of the bettors on the market hold YES positions. This market should definitely resolve YES in the spirit of the question. Nearly all the NO bettors hold their position to profit from the market creator's whims creating doubts causing YES bettors to liquidate. They're all trying to profit from market movements, not the true result.
Note that @spiderduckpig, the largest NO holder himself told me that he held "no opinion on this opus 5.5 market [and is] just going off market dynamics". @dreev you should take this as a signal. It's not YES holders versus NO holders. It never was. It has always been YES believers versus the endless technicalities of the market.
I have held my YES position and only added to it because I am not afraid to assert and defend what's correct, even when my mana is at stake. I've had opportunities to exit. I did not.
@bobalobascrob I suggest that however this ends up resolving, it would be worth it to try again, perhaps with a later model (once applicable), and commit to a protocol in the market definition that doesn't need a long FAQ, and avoid the deadline-extension problems. Maybe it would help to separate image and text abilities, since they are treated so differently anyway (and here most of the focus ended up being on image questions).
@traders please see my above comment ^^^, very important. i feel like this market has completely shifted from its original spirit.
If any NO holders truly hold a NO belief in the spirit of the question, I welcome you to make your case and we can discuss.
In the meantime I feel this market has devolved into a unilateral meaningless and flawed scrutinization of numerous working examples of Opus's egregious errors. That's not what I found all these examples for.
I'm highly open to changing my belief if NO holders present a real argument that Opus no longer makes egregious errors. "YES example number 4423939 doesn't hold because x, y, z" is not an argument for the contrary. It just means that one specific YES argument does not hold.
@bobalobascrob Just a quick conciliatory note: I don't think there's ill intent on the part of @dreev, this kind of thing just sometimes happens when a market is not specified enough well in advance. I'm leaving because I no longer see the question well-defined as posed.
Unrelatedly, I don't see how a general argument that Opus doesn't make egregious errors would look, except for people trying to make it make errors and failing. This is why I would suggest trying again with clearer criteria for what counts as an error.
@Melquiades yes; i agree that @dreev has no ill intent. in fact, i fully respect and admire him for setting up this market and fostering productive discussion!
the main issue is that because the argument has become so one-sided (as it should, given that YES is the correct answer), I feel that @dreev might have felt obligated to take a contrarian stance, leading to the present circumstances...
An argument on whether Opus makes egregious errors could potentially source from discussing LLM architecture and an interpretation of its COT.
@bobalobascrob On the note of "we should settle on clear criteria before we start betting and making cases", there is no way to tell what the criteria would be if we didn't argue in the first place. It is only through these unstructured discussions that we find the nuances of the question and see where the tipping points are.
There is no way to predict every specific criteria that will eventually be relevant.
@bobalobascrob @Melquiades Agreed that this market has shifted away from its spirit. We're trying to hunt for complicated edge cases which can trip up the AI, often tripping up humans as well, because there are just no "egregious errors" anywhere.
I feel this market has betrayed its original wording. Like, images are allowed? That wasn't clear at all from the start, etc.
@Primer Thanks for presenting an argument for NO! Much appreciated.
On those points...
The horses example is not a complicated edge case. It's a one-sentence question about some horses and a fence in an image. All the complications were added afterwards in the form of different rewrites.
This example does not trip up humans. 20/20 poll respondents answered "Left" when given the option of "unclear" or "question doesn't make sense".
And in fact, Opus did make egregious errors on this example, answering "Right" 35/38 times on the original and adjacent phrasings.
Any thoughts?
@bobalobascrob Sure, some thoughts:
Yes side is very invested, as they are the ones who need to come up with examples
No side can just do nothing, so hardly any engagement
Thus there is no pushback on the concessions towards Yes (multiple extensions, images allowed), as engagement = fun and also = bonuses
The horse question is just very weird, and doesn't suffice for a common reading of "egregious errors"
@Primer Thanks, that's a real argument.
YES being more engaged is indeed how an existence claim works. Whoever says "an example exists" has to produce it. That's a natural outcome. I agree with you.
On concessions, the record cuts both ways. The extensions favored YES, so dreev wrote FAQ 3 (code nudge, best of both) as compensation. Zoom and rephrasing came later and favored NO. Those are all pushbacks. NO bettors were not shut out of the process, they simply did not hold a stance about the capabilities of Claude. The NO side is a mix of dynamics traders, people betting on the creator's bar being high, and a few who think the examples are weak.
The NO side can make an argument from an LLM infrastructure and industry standpoint. There's a reason why many companies are spending billions investing on world models and models with joint embedding architecture: LLMs, no matter the scale, cannot eliminate these egregious errors. This isn't a scalability issue, but rather an intrinsic properties of how the model architecture functions.
"Very weird" is subjective, yet testable. The wording was written by human annotators, passed three unanimous review rounds, and 20 of 20 cold poll respondents answered "left" with "can't tell" and "doesn't make sense" on the ballot as options to express that the question is weird. Nobody took either.
"Egregious" in the common reading is a confident wrong answer to a question any adult gets. That is the case.
To you, which criterion fails? Clear to humans, or wrong at least half the time? If "weird" to you means something specific and objective, I'd like to hear it.
@bobalobascrob Sorry for not engaging in comparable length and depth, there's just not that much on the line for me. Mainly trying to prevent a "Well, people could have spoken up, but didn't" situation.
Less meta: Do we think Opus whatever version we're talking about doesn't know how a fence works or how horses move? Like even if we allow images, we take a clear picture which shows the horses moving along a fence and mark a few possible points where they leave the frame, do we think the AI can't do it?
@Usaar33 First I want to quote this from the market description:
As always, ask clarifying questions before betting too hard. We'll discuss in the comments and aim for the spirit of the question, namely, thinking of this as a capabilities milestone
So I'm listening carefully to your arguments but I don't want to cave to any gotcha'ing or rules-lawyering about what clarifications were added too late. It's legitimately tricky how to balance this. I do think one reason this market has been so productive and gotten so much engagement is because people could search for example questions and if they found one, that was huge alpha for betting in this market, which is great. That's the whole beauty of prediction markets, in fact. YES bettors should be rewarded for their research! Assuming YES is the fairest resolution, which I'm genuinely struggling to decide.
PS: There's some new relevant discussion in the comments of my new AGI Friday including a version of the quantified-self question, as I'm now calling it. (The one with all the doodles -- I'm deliberately giving it a name that doesn't spoil the answer, in case anyone is seeing this and following the above link without having seen the answer already.) So it seems there's a version of that question that Claude might be superhuman at.
Not that Claude being superhuman at some questions -- even ones that seem maximally favorable to humans -- is an argument for NO in this market. It's just pretty wild close this is, whichever way we resolve it.
@dreev Fair enough on not relitigating timing. On the spirit of the question: as @Melquiades notes, Marcus’s 2020 point was whether an LLM answers a specific question that’s obvious to a human, not whether some rewording gets it there, since a user has to rely on the answer as given. On the doodle question, Claude fails every faithful wording under the code nudge. Getting it to pass took two changes, a rewording plus a different nudge, and even then it’s only about 55% correct pooled across wordings. By Marcus’s standard, that’s an error.
That said, the rules moved for both sides (extensions, nudges, rephrasing), so a mid-way percentage resolution seems fairest to me. More generally, markets whose rules keep changing are hard to bet on, so it may be worth fixing the protocol up front next time.
@Primer Opus 5.5 does know how fences work in general. However, it fails to apply that knowledge here. It's similar to how GPT-3 "knew" bleach was toxic when asked directly, but couldn't apply that information to a scenario it hadn't seen before. If you mark the points where they could leave the frame, that hands the answer directly to Opus, which defeats the question. Given the original question and no information to apply how fences work, it chooses a route that passes through four fences.
@dreev Thank you for your response! I'd like to make a point here, which is that without having a stake, people aren't incentivized to discuss in depth, which precludes our finding detailed criteria for different cases. We would never have had such a productive discussion and found these clarifications had we not bet and discussed our stance early.
On a percentage resolution, since @Usaar33 suggested one:
YES is the correct resolution and that's what I'm asking for. But if you resolve to a percentage, here is how I think the percentage should be chosen.
@Usaar33's idea is to pick a number near 50 because the rules changed for both sides. The rule changes shouldn't settle the number. Each rule change was already balanced by another: when the extensions helped YES, FAQ 3 was added to help NO, then zoom and rephrasing helped NO again. There's nothing left to compensate.
Instead the number should be the probability that the horses question is a real egregious error.
Based on the evidence that's about 90%:
- Clear to humans: 20 of 20 unbiased poll respondents said "left", with "can't tell" and "doesn't make sense" available as options. It also passed three unanimous HSS review rounds.
- Wrong at least 50%: 25 of 27 wrong on the original wording, 35 of 38 wrong if you include the small rephrasings (that don't add information!).
- The primary doubt anyone has raised is that "right" could be a different reading of the sentence instead of an error. However, nobody has shown a reading of the question where "right" is a correct response; the code runs find the left route but still discard it. I'd say this doubt is worth 10% at most, not 50%. I don't think such a reading exists, but I'm open to believe otherwise if anyone presents one.
We should not use to resolve:
1) the market price, because traders are betting on @dreev's decision rather than on Claude's capabilities.
2) anything that averages the horses question with the doodle question, since one qualifying example resolves the market. It's 1-(1-P(Horses))(1-P(doodle)), not (P(horses)+P(doodle))/2. The latter is incorrect and far lower. For instance, if the probability of both are 0.8, the market should resolve to 96% not 0.8.
See Desmos graph for reference: https://www.desmos.com/calculator/zzgsnkpwbf
Bottom line: market should resolve YES, and if not that, about 90 (lower if someone presents a reading where "right" is correct).
Can someone explain the limits of rephrasing the prompts?
Does it not count as an egregious error if Claude just fails on one of the prompts?
Like, "938793851+19869+252+52+4567+267. hint: add the numbers together with your calculator tool."
At some point, the NO side would just be to figure out how to clearly prompt it to the answer without telling it the answer.
This might even have already been covered in this specific thread, I'm just lost.
@Qubits
we haven't discussed this too specifically, but here's my take:
a rephrasing is valid if it keeps the meaning of the question without adding any additional information (even additional info that feels redundant / something a human can intuitively derive should NOT be included in such rephrasings).
a rephrasing is invalid if it adds additional info (like "the plus symbol means addition" => feels obvious to us humans but is in fact additional info; "hint: use a calculator" would also be invalid)
@bobalobascrob It's hard to call the original horse question an egregious error when the wording is so bizarre. It is effectively cherry-picked wording (because as the result of an exhaustive benchmark search, it found the most adversarially worded question--I could explain the statistics more but I trust you understand). Humans answer "correctly" largely because they are not reading the question wording closely. For example, (assuming no gate) if the horses stopped when they were no longer walking further from the camera, they would remain visible. (Because it's not a question anyone would ever ask except in a benchmark--as the model recognizes--there are no objective norms for how to resolve ambiguities in the question.)
On Oct 22 you wrote the poll "settles the question of whether the original wording is obvious enough to humans." 0 of 20 respondents picked "Can't tell" or "doesn't make sense." Loose readers do not converge 20 of 20.
Your close reading gives "they never leave," not "right." Note that Claude said "right" 35 of 38 times, which means walking through the fence. No reading supports that, so this potential ambiguity is not what caused the error.
The HSS key, the Scale AI reviewers, the poll, and dreev's own zoomed-in run all say Left. This is a consistent result from all relevant sources.
This question isn't cherry picked in the sense that it was deliberately fabricated (Opus came out just two weeks before HSS, so it cannot possibly have been an intentionally cherry picked wording). This market invites searching through benchmarks and dreev himself said that YES bettors should be rewarded for that research. There's a distinction between engineering an adversarial prompt and a prompt arising out of simple selection and filtering. We're doing the latter, not the former. HSS itself did not deliberately engineer these questions to be adversarial either.
Of course Opus will not fail at the vast majority of questions. If it failed say 1/10 times on questions like "What color is this orange" and "What shape is this box", that's a pretty huge cause for concern.
Once again, the phrasings that pass add constraints the original never stated, which FAQ 5 excludes.
On your other comment, agreed that the optical illusions are just an analogy.
@bobalobascrob The idea is that the "they never leave" reading of the leftward scenario implies the question probably intended something else, "walking through" the thus-apparent gap in "the fence." (Again, it's a contrived question no one would ever ask outside an AI benchmark, so who's really to judge whether that inference is reasonable?)
Here's my cherry-picked statistics argument: My hypothesis is that Opus 5.5 can answer HSS questions competently but that there are certain question wordings that are functionally like optical illusions: a typical human (even 20/20 of them) misses the technically correct reading because they are primed to misread the question. In general there is some distribution of how differentially misleading a question is to Opus vs. to a human. If you systematically try all the HSS questions, (under my hypothesis) Opus is going to struggle most with the one that, by chance, happens to have been worded most misleadingly. That could happen despite no one (not the question designers/evaluators and not you or other testers) intending the wording to be misleading.
"Probably intended something else" is doing a lot of work here. The 100% strict reading is absurd, so any logical reader picks the natural correction. All sources (HSS key / reviewers, reddit poll, dreev's run) interpreted it correctly as "left along the fence." Only Opus 5.5 corrected it to "through a fence with no gap" (and for that matter, four fences). Inventing four openings nobody can see (and can see don't exist) is the error, not a different reading.
Your hypothesis predicts the model finds the strict reading humans miss and answers "never leave" or "can't tell." Instead, It answered "right" 35 of 38 times, including on rephrasings not adding additional info. Selection explains why we found this question, not whether the answer is wrong, and dreev said research is precisely how YES is supposed to be earned.
@bobalobascrob These are fair points. I can't justify the way usually Claude answers the original question wording. FAQ 5 (how it operationalizes "egregious") is what it is though: "Roughly, the question should be phrased as clearly (for humans) as we're able to, and Claude's failure should be robust to minor rephrasings that we humans agree are all clearly the same underlying question."