Asking GPT-3 MTG rules questions returns some rather nonsensical answers. For example:
This answer makes no sense, and those cited rules don't even exist.
This was from a prompt where I supplied it with a list of other rules questions and correct answers to them, so it does "know" that it's supposed to be answering coherently and correctly. I can also tell from other experimentation that card text and the Magic Comprehensive Rules document were a part of GPT-3's training data. GPT-3 is clearly not powerful enough to properly understand such a complicated technical system.
This market resolves to YES if, by the beginning of 2030, I have access to a system that can give me correct answers and explanations to Magic rules questions in natural English text. Specifically:
I will supply it with 20 completely random unreleased questions from RulesGuru. (Plus card text if necessary.) Over those 20 questions, it must have at least a 90% success rate on giving the right answer, and at least a 50% success rate on providing an explaination that clearly and correctly explains why it works that way. A correct explanation can leave out a small detail here or there, but it must be good enough to help a human understand the material, and avoid anything blatantly wrong like referencing parts of the rules that are irrelevant or don't exist.
Related questions
Just do the fun ones rather than the site. Humility + Opal, Season + Arbiter, Volrath's Shapeshifter being - well - the card that it is, Panglacial Wurm being the card that it is, whether the Gitrog interaction counts as slowplay (since it's technically not a loop per Toby Elliot's fantastic horsemyths post), etc etc.
I'm not that into predictions that look so far future, but I studied both AI and MTG somewhat, and IMO, this problem's too complex and unnecessary for someone to want to massage into being an accurate rules query engine. (We already have an excellent rules engine, of course, in MTG Arena, but doing both queries and responses in plain text is quite a feat.)
@TylerColeman Arena's rules engine is much simpler than the full Magic rules engine, since it's restricted to only recent cards that have been designed to work in Arena. And even then it's not perfect. For example, finding a legal declaration of blockers is NP-hard, so my understanding is that Arena uses a heuristic algorithm that may not always return perfect results. And there are other bugs here and there.
I plan to spend a few days trying to fine-tune GPT-X to answer rules questions, or train a much smaller dedicated model to do so.
I just realized a problem with this resolution process, which is that the AI system may have access to the internet and simply be able to read the answers off RulesGuru.
If there are no objections, I will change the process to use 20 unreleased questions on RulesGuru instead. (With their wording fixed up so it's clear what's being asked.)
@IsaacKing You could also change the names of the players I think, although that wouldn't slow down a reasonably general intelligence.
You could also change the names of the players I think, although that wouldn't slow down a reasonably general intelligence.
Yeah, even GPT-3 can already do that.
A bigger problem might be that RulesGuru might not exist in 2030.
It's my site, so as long as I'm still around and I haven't lost all the files and their multiple backups in some catastrophic accident, it'll be available. (May not still be online, but I'll have the files somewhere.)
Yes, I included several examples of questions answered correctly. In the past I've also tried different prompts, none of which worked significantly better.
Feel free to try it out yourself. Even if you don't know anything about Magic, you can grab questions from RulesGuru and check any rule citations that GPT-3 provides against the rules document here. Even ignoring whether the rest of the answer makes sense, if you can get GPT-3 to cite only rules that exist, that would be a marked improvement. :)