How do you'd define "over 50% of the time"? If it can play 50% of games on Steam (about half of which are probably weird visual novels with no real gameplay) does that count?

https://x.com/maxbittker/status/2019103515302346918?s=20

Feel like this qualifies for Runescape. Dev spent what seems like "a few days" making a way for ClaudeCode to interface with the game. Idk if it's "improving"? But it's definitely playing as competently as an "amateur human".

I think FPS-style games or anything requiring fast reactions will be much trickier, but "over 50%" seems very doable in the next 2 years. YES limit order at 47.

@bens it's pretty reasonable to see some ultra-fast model like

ClaudeCode with Claude-Haiku-6 meeting this criteria. Haiku 6 will probably be almost as competent as Opus 4.5/4.6, there will be substantial progress into Computer Use in the interim, and Opus 4.5/4.6 can already arguably play as well as amateur humans on random computer games if live reaction time wasn't a factor.

@bens it didn’t look like Claude was playing, just doing small in game tasks as told to by its user. If they entered something like completing a questline and it did just that at normal human level then I would agree.

Its programmers can have a short amount of time (days, not months) to connect it to the game.

Are there requirements on what the harness looks like or what its inputs and outputs are? E.g. does the AI need to take in the same image input as a player would or could the harness give it a list of on-screen enemy coordinates and such?

https://open.substack.com/pub/ramplabs/p/ai-plays-rollercoaster-tycoon

Claude can apparently manage much of Roller Coaster Tycoon pretty well, but it still has problems with spatial reasoning.

@TimothyJohnson5c16

It requires a specially made interface that aggregates data, and it can't do basic spatial reasoning in 2D.

What's the SOTA on this? Haven't heard any news on this in the last few months.

I think that real time games are going to be very hard. Approaches like AlphaStar where the AI is connected to the innards of the game engine (as opposed to simply reading the screen like a human would) shouldn't count IMO.

@VitorBosshard

NitroGEN is recent and interesting, I don't know if it's SOTA, they didn't publish benchmarks.

https://nitrogen.minedojo.org/ (seems down. Here's some alts)

https://huggingface.co/nvidia/NitroGen

https://web.archive.org/web/20251220092625/https://nitrogen.minedojo.org/

Site not found · GitHub Pages

@VitorBosshard

I would expect a future version of Sima to resolve this positive (likely this year)

@LoganZoellner this looks like it has zero capability of reacting and doing things in real time.

@robm this looks like a more serious contender. thanks.

@VitorBosshard

what exactly do you think is happening in this video?
https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/media/SIMA2_Comparison02_v03.mp4#t=0.1

@LoganZoellner Something that doesn't generalize to dodging a creeper or fighting pvp.

A major market uncertainty for me is whether the "rate at which the amateur human improves" is measured in # of games played / in game time, vs # of hours. Like, with highly parallel setups, the AI can play in an RL loop and plausibly get better quickly, but that would represent much much more in game time. I don't think it's plausible for this market to resolve YES in terms of in game time / learning efficiency being equivalent to that of a human for long term video game learning

@Bayesian my read of the market Description (which admittedly is very bad for my position as YES holder, but my honest interpretation regardless) is that it should be one agent vs one human running in similar time, no dilation or parallelisation. The developers get some time to build a harness if needed, then it just starts playing same as a human would - 1 hour of game time is 1 hour of game time, effectively measured by the game engine and not by the wall clock time.

If the game engine timers are sped up to run 60 time as fast (so you can do 1 hour of regular gameplay in 1 minute), that’s compared to what a human would do after 1 hour, not 1 minute.

If you have 60 agents running in parallel and updating each other, the same.

Just my take though.

@TomCohen yeah that would make sense. then ig the issue is that it may well be as fast for 1 hour or however long it takes before its context window fills up to practical limits, and thereafter hits a wall the human doesn't hit. hmmmm

@Bayesian unless:

1. It can create and reference memories in some form to offset context load

2. Online learning improves significantly

My best guess is that this has trended upwards due to the IMO market resolving YES, leaving AI bulls flush with cash? I'm not really aware of any developments in the last 3 years that are bullish for this market, but I'd love to be wrong!

@DanW I didn't make any new bets, but a couple of important developments are probably:

• Regular games are starting to be used as benchmarks by the big labs (the so called 'Pokemon benchmark'')

• The new ARC benchmark has an emphasis on interactivity.

• Google Deepmind are explicitly saying they're using Genie 3 to train AI models

If I were to make new bets in this direction it would probably be from a "the trend continues to hold is an update too" pov, but I'm comfortable between 50% and 70% atm.

Why is this trading up, has some progress been made?

@benjaminIkuta

insider trading... hopefully?

Gemini beat Pokemon, but that should have been priced in since it was making steady progress for a while.

The fact this was trading below 50% for this one seems surprising, considering "play video games" is a concrete external reward (the kind reasoning models excel at) and multiple major labs are clearly focused on this. Also 50% of games and amateur human are highly achievable targets.

edit:
I didn't even notice the additional "Its programmers can have a short amount of time (days, not months) to connect it to the game" in which case the scaffolding for Gemini plays Pokemon might not even be "cheating"

@LoganZoellner if you're surprised it's below 50%, what solution do you expect to exist for real time games?

@ProjectVictory

A multimodal transformer trained with reinforcement learning on a few thousand video games. It would surprise me if Google and OpenAI weren't both already working on this internally.

@LoganZoellner this solution is currently about two orders of magnitude too slow for anything realtime. To play a first person shooter somewhat competently you need latency of about 300ms at the very minimum. Transformers like Claude and Gemini take tens of seconds to make a move when playing Pokemon, keep in mind that pokemon is on the easiest end in terms of how hard it is to parse visually, so you can't just throw a super lightweight model at the problem.