🏅 Top traders

So Gemini beat Pokémon blue. As I specified earlier, despite some additional help from the harness I believe it is sufficiently driven by Gemini to resolve YES. I can see how the map assistance might be considered too much by some, but at the end of the day, this is a large language model making its own decisions that beat Pokémon.

It seems clear that Pokémon doesn’t make a particularly good benchmark. I think any value it had in comparing models against each other has already been lost to Goodharting. But I remember when it was deeply unclear whether a language model could complete a sudoku puzzle, and despite the subjectivity, it seems clear that these models have come a long way. Very cool!

Can we resolve please? I know Gemini got quite a lot of help, but you commented explicitly that the run counts and I bet based on that. No room for second thoughts after the fact.

@AhronMaline yup, just wanted to verify what Julian said about no additional scaffold changes.

Gemini has beaten Pokémon Blue again.

No further changes to the scaffold were made. However there was an objection made on Discord towards counting this run:

MrCheeze — 12:54 AM
(...) the automatic mapping is more like 100x as much help as Claude's navigator, its not <2x as much

Julian Bradshaw — 12:21 PM

Can you explain your position more here? (...)

MrCheeze — 12:28 PM

Its main importance is for Safari Zone where wandering at semi-random will never work

You have to be able to do relatively clean movement, which none of the models are even close to being able to figure out without having the explored map given to them

Julian Bradshaw — 12:30 PM

And Claude Navigator is making essentially clean slate attempts every time, and it can't see the whole Safari zone at once, so it won't be able to navigate all the way through?

MrCheeze — 12:31 PM

Yeah no chance

Now, as before, I think Gemini's map system counts as an allowed knowledge file system, because it's only filled out as Gemini explores, but wanted to mention the objection here. (Also afaik Claude hasn't ever reached the Safari zone, it seems possible to me that Claude could beat Safari Zone just by writing down its exploration pathways. Also o3 has already beaten Safari Zone with a different but similar mapping tool.)

@JulianBradshaw can you link the discord convo?

@Sketchy dm'd.

Just to be explicit about https://www.twitch.tv/gemini_plays_pokemon

Assuming no additional assistance, the second run of Gemini Plays Pokemon will count for the resolution of this market (if it finishes). However, it's on the edge and I will treat any additional assistance with a lot of skepticism. More discussion here.

Gemini 2.5 Pro beat Pokémon Blue earlier today. However I don't think it should count as too many substantial changes were made to the scaffolding throughout the run, including ex. allowing the main Gemini to call a separate Gemini with a prompt dedicated to solving boulder puzzles in Victory Road. (a "strategist" Gemini)

In general it was deliberately a pretty loose, experimental run. Later runs may count.

@JulianBradshaw agreed. I think it was significantly over the boundary of mid-game tweaks.

I’ll have to think about if the structure itself of map labelling is too much. Probably not.

@JulianBradshaw Would the run that's just been started where the previous scaffolding that's been developed won't be changed and the model will now run autonomously count?

@DarklyMade

From the description of the second run:

There are no changes to prompts or tooling, so this run serves as a clean test of all the improvements made during the first run. There won't be any developer interventions unless Gemini becomes hard-stuck due to a system limitation.

Yes, I think this would count. Although it's quite a bit of game-specific scaffolding, that was never disallowed and none of the structure I see is that egregiously bypassing the need to have the LLM drive itself.

@Sketchy this comment has a good summary of the scaffolding https://old.reddit.com/r/ClaudePlaysPokemon/comments/1kdjysi/gemini_beats_pokemon/mqblfeh/

In particular see

The other Gemini subagent is the Boulder Puzzle Solver, who is prompted with some pretty specific instructions on what kind of reasoning to use to solve the puzzles in Victory Road - examining the gates and switches to figure out which unsolved puzzles need to be done, and what sequence of pushes would accomplish that.

@jack yea the boulder puzzle solver is the worst part to me. But I didn’t disallow game-specific prompting or scaffolding, and it’s still LLMs making the decisions here. Since the boulder puzzle solver is baked in from the start of the run I think it’s acceptable.

It does seem like the whole benchmark is getting goodhart’ed pretty quickly but I’m trying to stick to my initial criteria.

@Sketchy The part of the criteria that seems relevant is

With minimal non-LLM programmatic assistance. I think the automatic pathfinding that Claude is using is a little bit cheating, if that helps with the spirit of this market. Something roughly twice as bad would maybe start to not count.

Which I personally thought this exceeded.

@jack The thing is, the boulder-solver is just an instance of Gemini with different prompting, not some kind of programmatic algorithm.

Ah, I guess I (mis)read that criteria as about assistance in general, not just programmatic assistance. Ok, in that case maybe the thing that stands out most for programmatic assistance is

Every tile it has seen in the map is recorded and the seen layout of the current map is given to it at all times. This includes warps and their destinations, and objects. Also, the list of which tiles can actually be reached or not in the current game state is calculated and told to Gemini.

@JulianBradshaw right this is where my head is at too. If it was a programmatic boulder solver that would be different.

@jack yea agreed. Does that feel twice as bad as the Claude path finding? I’m leaning towards no, but I do see how a map of visited places and layout much more than Claude’s pathfinding.

@Sketchy Wanted to say, as far as I know there's only been two changes to the scaffold since the start of this Gemini run (run #3):

1. when the navigator paths Gem into a moving NPC, it waits for the NPC to move instead of bonking

2. fixed a flaw in the harness where cut trees are considered non-traversible when HM01 has been deposited in the PC

According to GeminiPlaysPokemon, the first one is a QOL fix to not waste time, 2nd change is a bugfix, so they feel it's legitimate.

Citing Discord:

@Sketchy I think the main thing about Claude's pathfinding thing is it only applies to the current screen, whereas Gemini's applies to the whole map (if I understand correctly)? And Claude had tons of problems seeing the map, especially around visitability and warps, e.g. the constantly trying to get through 'gate-like structures" which Gemini's scaffold bypasses.

But I don't have a very detailed understanding of the scaffold and how much it's helping Gemini

@jack Ultimately we don't know all the details of either the Claude or Gemini scaffolds (or o3 for that matter, which also started a run recently). They're not open source. But Claude also gets some information about which tiles are reachable/walkable or not per the excalidraw diagram linked on the twitch channel.

Gemini's map recording is arguably an allowed knowledge file system, since it only gets filled out as Gemini explores and sees new map tiles.

@DarklyMade I lean towards thinking this run should count. I think the main caveat here is that Gemini's harness has gotten around a lot of the typical LLM vision problems by a.) using game RAM information to specify if tiles are warps or not; b.) presenting map info in text form.

Now, in both of these cases, Claude and o3 are also getting the same info, just with (imo) less refined harnesses, so it's really a question of whether this class of harness counts at all. And I think the original market implies yes, since it's talking about Anthropic's own measurements of Pokémon progress using this class of harness.

Re: class of harnesses, here's an exchange between ClaudePlaysPokemon and GeminiPlaysPokemon from the former's stream that's enlightening:

asdfugil: wow claude has the same issue here as gemini but it's probably just a game bug
asdfugil: an unreachable warp right here
Gemini_Plays_Pokemon: yes there's an unused warp here in the game data
Gemini_Plays_Pokemon: for whatever reason
ClaudePlaysPokemon: i love how you have also had to go through knowing WAY TOO MUCH about weird quirks of pokemon red
Gemini_Plays_Pokemon: Haha yes I learned a lot of random stuff. There's also one like this at a gatehouse and an invisible non-functional warp beside the Silph Co President too
ClaudePlaysPokemon: pokemon devs y
ClaudePlaysPokemon: the one that caused me the most pain was figuring out how there is a list of tiles you can't walk between that is handled differently than collisions
Gemini_Plays_Pokemon: are you talking about moving from elevated ground to regular ground?
ClaudePlaysPokemon: yep lol
Gemini_Plays_Pokemon: yeah that's fun, I had to add special handling for that
ClaudePlaysPokemon: same, and it took me an annoying amount of reading the disassembly to figure it out

@JulianBradshaw yep, very true, but I believe Claude's reachability info is simpler and Claude can't even blindly trust that info because things like gates/doors are marked as "not reachable".

So overall I can see arguments both ways on whether this is like "twice as much programmatic assistance"

@jack So can I. I'm going to try to be definitive and say with the current setup, the run is included. But it's on the edge and I will approach any further assistance with skepticism.