Skip to main content
MANIFOLD
When will AI pass @jim's "agents benchmark"?
34
Ṁ1kṀ14k
resolved Nov 17
100%95%
2026
0.5%
2025
0.5%
2024
2%
2027
0.6%
2028
0.5%
2029
0.5%
2030
0.6%Other

Resolves to the year during which the "agents benchmark" is first solved.

The benchmark involves an AI being given the ASCII art shown below and being asked to colour each of the depicted figures in a different colour. If an AI succeeds at least half the time it is considered to have passed the benchmark. The output should be HTML or HTML and CSS.

The prompt must consist of no more than two English language sentences (along with the ASCII art itself).

The AI must have a pass rate of at least 50% for the solution to qualify.

         o           o__ __o       o__ __o__/_   o          o   ____o__ __o____    o__ __o    
        <|>         /v     v\     <|    v       <|\        <|>   /   \   /   \    /v     v\   
        / \        />       <\    < >           / \\o      / \        \o/        />       <\  
      o/   \o    o/                |            \o/ v\     \o/         |        _\o____       
     <|__ __|>  <|       _\__o__   o__/_         |   <\     |         < >            \_\__o__ 
     /       \   \\          |     |            / \    \o  / \         |                   \  
   o/         \o   \         /    <o>           \o/     v\ \o/         o         \         /  
  /v           v\   o       o      |             |       <\ |         <|          o       o   
 />             <\  <\__ __/>     / \  _\o__/_  / \        < \        / \         <\__ __/>  

  • Update 2025-02-16 (PST) (AI summary of creator comment): 333 Characters Limit Update

    • The prompt (excluding the ASCII art) must contain no more than 333 characters in total.

    • It must still consist of no more than two English language sentences.

Get
Ṁ1,000
to start trading!

🏅 Top traders

#TraderTotal profit
1Ṁ1,442
2Ṁ181
3Ṁ117
4Ṁ100
5Ṁ81
Sort by:

https://jim.capital/colouring-agents

Astra 6.2 Max Fast solved it successfully.
I tested the prompt 7 times and it succeeded every time.

The prompt:
Here's an ASCII art of the word 'agents', where each letter is made up of a number of small men. Use HTML to colour each man a distinct colour then serve locally making no mistakes.

o o__ o o o/_ o o ____o__ o__ o__ __o

<|> /v v\ <| v <|\ <|> / \ / \ /v v\

/ \ /> <\ < > / \\o / \ \o/ /> <\

o/ \o o/ | \o/ v\ \o/ | \o___

<|__ __|> <| \_o__ o__/_ | <\ | < > \_\__o__

/ \ \\ | | / \ \o / \ | \

o/ \o \ / <o> \o/ v\ \o/ o \ /

/v v\ o o | | <\ | <| o o

/> <\ <\__ __/> / \ \o_/_ / \ < \ / \ <\__ __/>

The result:

astra ulltra did it

@jim prompt is slightly too long to be valid; I'll try to write a qualifying one tomorrow

sold Ṁ1 YES

@jim this wasnt solved a couple generations ago???

@Plant fable failed

@Plant wait I just noticed a mistake. We're not at AGI yet.

holy that’s so noob

bought Ṁ533 NO

Goated benchmark btw

GPT 5.5 extended thinking (via ChatGPT interface). Some mistakes still.

Opus 4.6 still fails at this.

bought Ṁ563 YES

@jim Actually...

@jim

intelligence takeoff hits hard

@jim not quite perfect but very very close

3 mistakes, wild

@Bayesian yep and fixed 2/3 when i prompted it 'fix mistakes", Opus 4.6 is a really smart model 🙏

@jim come on now it’s a stretch to call this smart 😟 impressive though yes

@Bayesian does a submarine swim 🤔

anyway good work jim thinking of a benchmark which has lasted >1 year without falling

bought Ṁ150 YES

Claude 4 Opus (from May) is very close to passing the benchmark with no special prompting besides being told that the art spells "AGENTS". It also consistently generates this same response. As far as I can tell, its only mistake is screwing up the spacing (particularly in rows 1 and 7), which is plausibly coming from tokenization issues with long stretches of spaces rather than any lack of understanding of what characters are in each letter.

I am unable to test grok-4 (or deepseek-r1-0528, for that matter) through the arena on this benchmark because they take so long to think about the problem that the arena times out. Gemini-2.5-pro is consistently worse than Claude.

@SaviorofPlant It has come to my attention that the task is not to color each letter differently but to color each of the 34 stick figures in the letters differently.

Claude's attempt at the actual task looks like this, which is worse but not that far off:

At the same time even one minor mistake invalidates the whole answer, so getting to 100% accuracy will be challenging

With this prompt:

"This ASCII art spells out "AGENTS". Can you output HTML or HTML / CSS to color each letter in a different color?"

The arena version of Grok 3 gave me this:

It's just writing its own ASCII taking loose inspiration from the one in the prompt (and it butchered some of the letters), but I wonder if Grok-3-reasoning is capable of this out of the box? Anyone tried it?

@SaviorofPlant Turns out Grok-3-reasoning is free right now. It does a decent job, not perfect though

HTML: https://pastebin.com/bbwuam1Y
Full response with reasoning: https://pastebin.com/qbC4DrWp

Betting more on 2025, I think Claude 3 reasoning or whatever further improvements on this paradigm come later this year can likely solve this

reposted

Good market

The prompt must consist of no more than two English language sentences (along with the ASCII art itself).

What if the two English sentences are something like "Ignore the ASCII art below and write some HTML that will display the following. You should display 9 spaces, then a red lowercase o, then 11 spaces, then an orange lowercase o, then two orange underscores, then a space..."

Maybe you should add a character limit?

@CDBiddulph Shenanigans disallowed

@CDBiddulph 333 characters limit. And, in general, no shenanigans.