Based on this tweet from Spencer Schiff:
https://x.com/spencerkschiff/status/1910106368205336769?s=46&t=62uT9IruD1-YP-SHFkVEPg
and the market by @bens:
/bens/will-a-mainstream-ai-model-pass-the
(Note: the main difference with that market is that the models I accept do not need to be free to use, but can cost reasonable amounts of money, typical of current frontier models on the API.)
Resolves YES if a mainstream AI model (Currently, only Google, OpenAI, Anthropic, xAI, DeepSeek and Meta count, but if some other lab becomes similarly mainstream they will count as well) can repeatedly solve this benchmark, using this image and other similar ones I draw. I will be the judge of this.
“Solving the benchmark” means being able to match the names to the colors of the stick figures, repeatedly and with simple prompts.
I will pay for api credits if needed (limit: ~$20 per model). I will prompt each AI model a few times (6) and see if it succeeds at least 4 times out of 6, but I won't be running the same model multiple times and use some kind of cons@32 technique to aggregate multiple attempts.
The image for reference in case the tweet is deleted:
…and here is what ChatGPT gives @bens now:
…and Gemini 2.5
I will not bet in this market to remain objective. (Edit: I forgot i had written this and got a position. if the outcome is ambiguous an uninvolved third party may determine resolution)
Update 2025-04-16 (PST) (AI summary of creator comment): Prompt Guidelines Clarification:
Prompt Length: Use roughly one short sentence to describe the desired task.
Allowed Adjustments: Small amounts of prompt engineering (for example, suggesting the use of a built-in tool) are acceptable.
Disallowed Complexity: Extensive multi-step instructions (e.g. a 20-step breakdown) are not permitted.
1,000People are also trading
@yaakovgrunsfeld One-shotted it for me as well!
And then totally failed the second time with the same prompt. Womp womp I guess.
I was about to write a comment scolding someone for trying the test with the same figure over and over, only to realize that's how the market is phrased.
JFC people, 2.5y into the LLM revolution and we still don't realize that they can memorize the answer to any one question? Especially if tons of people spam the exact same question over and over, it will make it into training data?
The correct way to run this eval is to have a way to generate random stick figure / arrow diagrams. It shouldn't be too hard to vibe code an app for this.
@pietrokc This is not how the market is phrased, there is an explicit "using this image and other similar ones I draw". Presumably, makes no sense to try with other images if models don't reliably get it correct for the single example, but I would expect @Bayesian to check some other images of this type should the main example be solved reliably.
@Dulaman I'm also wondering if the new codex max model can do this. Would that meet the criteria if it ran for 4 hours on my machine writing python scripts and got the right answer?
Gemini 3 through https://gemini.google.com/app works reliably. I uploaded the image and used the prompt "Match the names to the stick figures by color."
Gemini 3 gets it in the app. Every time the same prompt: "Match each name to the color of the stick figure its arrow points to"
I ran it 3 times. Here are my multiple chats:
https://gemini.google.com/share/0e05f16bc3dd
https://gemini.google.com/share/87b6b21e8f6b
https://gemini.google.com/share/5ab928a71f4e
Then I got rid of my system prompt. Mixed. Here are those 4 chats:
Fail - https://gemini.google.com/share/970a9348ed20
Success - https://gemini.google.com/share/f40ceb342033
@JaySocrates yeah ig this is sufficient for YES resolution. PS it was public knowledge that Gemini 3.0 could solve the problem before this market was created
@JaySocrates Replication: with all personal context off, in-browser, Gemini 3 succeeded in 9 of 20 trials. Of the 11 failures, 7 switched Jack/Jimmy, 4 were wild. Certainly doesn't feel like a yes resolution to me. May prompt-engineer this over the weekend. I expect that for Gemini 3, 3-4 sentences might get a consistent correct answer but 2 would require a very special prompt.
fails for me 2/2 times for gemini 3 in AI studio. one way off; other same answers as gpt-5-pro below. (Overall abstract image understanding in my tests is coming below what benchmarks suggested)
@ian I'm pessimistic. The model doesn't know it is wrong (because it can't "see correctly"), so why would thinking help more?
@Bayesian Its NP-hard, it should take 12 minutes to solve so AI is now AGI because I said so with my perfect logic
@Bayesian woah I'm actually kind of impressed that it can almost do it?! it's almost more impressive with 2 switched than it would be if it got it perfect
@bens I mean i disagree with the latter part but it does show a level of logical consistency and in any case im very impressed too!