"Beat Factorio" means constructing a rocket.
"Figure out how" means roughly that it wasn't hand-held through the process. Roughly, the supervised part of the AI's training process must have been plausible in a world where Factorio didn't exist. (But it's allowed unlimited unsupervised play of Factorio.)
Details:
Examples of things that are okay:
a human can choose a favorable map seed
the AI can play Factorio unsupervised for any length of time
the AI can access the wiki (or other online resources) during its unsupervised play
Examples of things that aren't okay:
no knowledge of Factorio-in-particular (e.g. recipes, blueprints, "muscle memory") can have been programmed into the AI
the AI must not have had any supervised training on Factorio specifically (e.g. watching videos of people playing)
the AI's reward function must not encode any deep knowledge about Factorio specifically (rewarding novelty is fine; rewarding it each time it constructs a new kind of item is... kinda borderline, but I lean not-fine?; rewarding it each time it constructs a new kind of item along the critical path is not fine)
Every January, I'll check whether there's any credible claim that somebody's done anything-like-this; if so, I'll dig in to see if it meets my "figuring out" criteria; if so, I'll resolve this market YES. Otherwise, on 2030-01-01, I'll resolve it NO.
Related questions
The biggest issue seems to be the no Factorio in the training set clause. Noone training a foundation model is going to go to the trouble of excluding Factorio videos from their data set just for the sake of this market.
In the case of a generalised game-playing AI (which the criteria seem to describe) there needs to be a generalised reward mechanism. This is not trivial and I wonder what people think it might look like.
I think curiosity-based reward is a good candidate but I'm not sure it would result in deep mastery of arbitrary games.
@ThisProfileDoesntExist having read the wiki, what would make it want to follow the objectives? What would its inner motives be in general?
@Tomoffer reading the description again I guess you'd expect an auto gpt style system to be given the explicit natural language instruction to "meet all objectives" or something
I would expect Factorio to first be beaten by either:
A giant multimodal transformer model trained on the whole internet including the parts of it that are about Factorio, which fails your criterion of not watching any videos of people playing Factorio.
Some sort of narrow RL model trained on Factorio specifically, which is probably going to be rewarded for collecting items along the critical path.
I.e. if the team making an AI beat Factorio doesn't specifically have your rules in mind, I expect your rules to be violated and the AI to not count for this market.
@Multicore If you train a game-playing AI on every game that exists, you'll have to create a brand new game to evaluate whether that knowledge has generalized. You're probably right that the first AI to do this will be trained specifically for it, but there is a motivation to create an AI that can work out games in a blind playthrough (as it is a step in the direction of AGI). An Alpha Zero approach wouldn't count for this market, AFAICT, but it does show that work is being done towards that end.
Imo, at a glance this mostly depends on one of two things happening
Some company decides to try training a model to play more complicated arbitrary games and Factorio is one of them
Plausible, but I'd maybe say 30% that it would get chosen for the list. But even if they chose Factorio for the list of games to run it on, they may not care so much about deliberately getting it to beat Factorio
ex: they may just do simpler shorter term goals, especially for an early attempt. Like Minecraft they may do 'build a cool mansion' and Factorio 'automate a factory producing green science'.
This might just be it being a shorter term reasoner, but also might be them just not bothering to apply it for that much (so it could be possible in principal but just not done)
We get generalist enough and fast enough agents you can run on your computer. Like if it suddenly became a lot more cheaper to train a model like this, so a 'bunch' more hobbyists can enter the arena.
One method is GPT model hooked up to code interpreter and just let it generate code to solve for solutions ;d
I think, given infinite time and assuming deaths are okay, that factorio is ultimately an easy game. The real question is whether anyone will bother trying.
For clarity, is the AI allowed access to information about factorio after its training period (ie. can it read the wiki while it is playing)?
"The real question is whether anyone will bother trying."
Agreed! I have "2030 or later" at ~3:1; but if I learned DeepMind was working on this, I'd put most of that probability-mass much earlier.
On the other hand: "whether anyone will bother trying" is somewhat correlated with "how good AI gets at generalized game-playing": there are lots of Factorio nerds who would hear about a surprisingly-good open-source game-playing AI and point it at their hobby.
"is the AI allowed access to information about factorio after its training period (ie. can it read the wiki while it is playing)?"
Ooh. ...yes, yes it is. I'll add that to the market description's allow-list.