https://x.com/akbirkhan/status/1899246324777972043

https://jackhopkins.github.io/factorio-learning-environment/

Looking at how much Claude is struggling with pokemon - a much simpler game with fewer decisions and less need for spatial thinking, I'm sceptical. If it happens relatively soon, I doubt it'll be achieved by general purpose AIs like LLMs.

For those betting Yes on before 2027 - do you know of any projects for game playing AIs? Genuinely curious if anything looks promising.

@ProjectVictory Claude's struggles with Pokemon feels like it could be a few "simple" advancements away from easily beating the game. Improve its vision, long-term memory, and spatial reasoning, and it should not have any issues beating Pokemon. And from there it should be able to beat any other slower paced game - just scale it up.

Of course, any one of those advancements may prove to be very difficult. My intuition tells me they will all be solved within 5 years, and possibly much sooner.

What version of factorio? They're making rocket launches cheaper/easier for Factorio: Space Age

@BoydKane Grr, good point. I'll go with "any version, whatever somebody does the experiment with."

The biggest issue seems to be the no Factorio in the training set clause. Noone training a foundation model is going to go to the trouble of excluding Factorio videos from their data set just for the sake of this market.

@ThisProfileDoesntExist Good point. To resolve ambiguity in what counts as having information "programmed into the AI": I declare it to be okay if there's Factorio stuff in the training set due to broad web scraping -- just no specialized Factorio post-training. Roughly speaking, nobody was designing the AI to be good at Factorio.

In the case of a generalised game-playing AI (which the criteria seem to describe) there needs to be a generalised reward mechanism. This is not trivial and I wonder what people think it might look like.

I think curiosity-based reward is a good candidate but I'm not sure it would result in deep mastery of arbitrary games.

@Tomoffer It's allowed to read the wiki, and this presumably knows what the objectives are.

@ThisProfileDoesntExist having read the wiki, what would make it want to follow the objectives? What would its inner motives be in general?

@Tomoffer reading the description again I guess you'd expect an auto gpt style system to be given the explicit natural language instruction to "meet all objectives" or something

I would expect Factorio to first be beaten by either:

• A giant multimodal transformer model trained on the whole internet including the parts of it that are about Factorio, which fails your criterion of not watching any videos of people playing Factorio.

• Some sort of narrow RL model trained on Factorio specifically, which is probably going to be rewarded for collecting items along the critical path.

I.e. if the team making an AI beat Factorio doesn't specifically have your rules in mind, I expect your rules to be violated and the AI to not count for this market.

@Multicore If you train a game-playing AI on every game that exists, you'll have to create a brand new game to evaluate whether that knowledge has generalized. You're probably right that the first AI to do this will be trained specifically for it, but there is a motivation to create an AI that can work out games in a blind playthrough (as it is a step in the direction of AGI). An Alpha Zero approach wouldn't count for this market, AFAICT, but it does show that work is being done towards that end.

Imo, at a glance this mostly depends on one of two things happening

• Some company decides to try training a model to play more complicated arbitrary games and Factorio is one of them

  • Plausible, but I'd maybe say 30% that it would get chosen for the list. But even if they chose Factorio for the list of games to run it on, they may not care so much about deliberately getting it to beat Factorio

  • ex: they may just do simpler shorter term goals, especially for an early attempt. Like Minecraft they may do 'build a cool mansion' and Factorio 'automate a factory producing green science'.

    • This might just be it being a shorter term reasoner, but also might be them just not bothering to apply it for that much (so it could be possible in principal but just not done)

• We get generalist enough and fast enough agents you can run on your computer. Like if it suddenly became a lot more cheaper to train a model like this, so a 'bunch' more hobbyists can enter the arena.

  • One method is GPT model hooked up to code interpreter and just let it generate code to solve for solutions ;d

I think, given infinite time and assuming deaths are okay, that factorio is ultimately an easy game. The real question is whether anyone will bother trying.

For clarity, is the AI allowed access to information about factorio after its training period (ie. can it read the wiki while it is playing)?

"The real question is whether anyone will bother trying."

Agreed! I have "2030 or later" at ~3:1; but if I learned DeepMind was working on this, I'd put most of that probability-mass much earlier.

On the other hand: "whether anyone will bother trying" is somewhat correlated with "how good AI gets at generalized game-playing": there are lots of Factorio nerds who would hear about a surprisingly-good open-source game-playing AI and point it at their hobby.



"is the AI allowed access to information about factorio after its training period (ie. can it read the wiki while it is playing)?"

Ooh. ...yes, yes it is. I'll add that to the market description's allow-list.