Criteria:
The game can be either a tabletop game or a video game.
It should be a PvP game - an AI beating a speedrun record doesn't count, the AI must beat the human in a direct competition.
Games where AI has already beaten top humans before the creation of this market don't count (Starcraft, DotA, Go, Chess, any FPS where simple aimbots can beat pros, etc).
There should be a video about the event with at least 10000 views on Youtube or some other platform.
The human being beaten by the AI should be someone who is generally recognized as a professional player of the game - someone on a major esports team, someone who has won a major tournament, etc.
The human must actually have been trying to win.
The evidence must convince me that the AI has higher or similar skill compared to the human, that it didn't just get lucky. (e.g. we'd need a much larger sample size for Mario Party than for chess.)
Shenanigans:
In some past "AI beats human" events, such as Alphastar's victory at Starcraft, there have been complaints that the setup was unfair in the AI's favor.
For this market, many of those shenanigans are fair game:
It's OK if the AI, like Alphastar, has a different interface to the game, as long as it does not get access to any information that could not be detected or inferred through the standard interface, and as long as it does not make any inputs that would not be possible through the standard interface.
It's OK if the AI's advantage comes partly from superhuman speed, memory, ability to perform game tree search, etc, as long as at least part of the advantage comes from the AI doing something that couldn't be done by a bot created using 2010 technology.
It's OK if there is some constraint on what types of matches the AI can play (like Alphastar only playing Protoss vs. Protoss), as long as the constrained form of the game is a situation that could reasonably occur in a normal tournament match for that game.
I recognize that some of this is problematically subjective and may scare people off from betting on this market.
Resolution:
When I hear about an event meeting these criteria, either from comments in this market or other sources, I will resolve to that game. If there's something else that technically happened first but neither I nor any traders heard about it until after resolution, too bad.
If I hear about multiple such events before resolving, I will resolve to the one that happened chronologically first, not the one I heard about first.
If an AI is announced to have beaten pros at multiple different games in the same announcement, resolves to all of them equally.
Options:
An option counts for all games in its franchise. e.g. if an AI beats a human at Call of Duty: Modern Warfare 2, that counts as "Call of Duty"
I've seeded the list with some top esports games and other plausible options. Feel free to add your own ideas.
Related questions
Poker's already been done in head to head: https://www.cardplayer.com/poker-news/21285-poker-bot-beats-humans-in-historic-match
I'd argue the constraints placed on the game of DotA do not strictly satisfy your criteria. Protoss Vs Protoss could be a constraint that could happen in a normal tournament, but the limited heros and mechanics in the Open AI Five match up are a bit of a stretch, by their own admission:
"Most of the restrictions come from remaining aspects of the game we haven’t integrated yet. Some restrictions, in particular wards and Roshan, are central components of professional-level play." - Open AI
I expect OAI felt they had sufficiently demonstrated their capabilities and could scale out to a full game with more spend, so it's not a hill I'm willing to die on but wanted to flag it.
In what sense has AI beaten top humans at starcraft? IIRC, the starcraft AI had access to the game data in a way that humans don't. I'm not aware of an AI that is only fed the screen data and only inputs mouse and keyboard actions.
While I think that SC is a bad example for AI progress, as it depends to a significant degree on speed and accuracy of execution, I still think the current status is that humans haven't been beaten.
This concern is also relevant to how progress in other games will be interpreted.
@VitorBosshard I specifically want to include things like Alphastar. I have added a section titled "Shenanigans" to the market description clarifying my intentions around things like that.
This market is already not entirely about AI progress but about what games AI creators will bother trying to break.
@VitorBosshard Likely only because DeepMind felt they had proved their point. Later versions of AlphaStar had limitations in place to bring it in line with humans (no global camera and realistic peak APM limits), and it still reached GM (top 0.2% of players) as of 2019. I'm very skeptical of anyone who would contend that, if DeepMind truly wanted nothing more than to beat humanity at Starcraft, they would not have done so already.