🏅 Top traders

Pretty surprised at how many diplomacy players there are on manifold and how many of them do stuff with AI.

Great game btw and this is one of the final frontiers for AI

Confidently betting no because being better than the best humans is reasonable, but "a 90% win rate against the best human players" is insane; my intuition is that short of hacking-human-minds stuff, even literally-optimal play results in much less than a 90% win rate against the best humans.

@ZachSteinPerlman I agree. I wasn't really thinking through the implications of multiplayer when I made the market but I didn't (and don't) want to change the resolution criteria. I'm surprised this has stayed so high for so long.

@vluzko My credence is lower than my betting suggests, I just don't expect the market to agree for a while and I care about the time value of mana. (This is a limitation of prediction markets.)

ah shit. welp

@vluzko please can you update the market description to clarify your definition of "90% win rate"? It sounds like the requirement is that in games of one AI and 6 of the top 100 humans in the world, the AI must win 90% of such games. Is that right?

I ask in part because that is not the definition that I think would be normal among Diplomacy players, who talk about joint wins and series wins and such.

And because the situation seems implausible once AI Diplomacy becomes superhuman - who would volunteer to have their mind hacked by an AI in a game that famously has no limits on manipulation?

@MartinRandall Could you describe what a joint win or a series win is to me? My guess is that joint wins might count and series wins probably won't.

@vluzko a joint wins meaning that of the seven players starting the game ended with a stalemate three players surviving and four players eliminated. Or with two players but that is really hard to arrange, apparently. It's reasonable to call that result a joint win. Add opposed to a solo win, which is rarer.

A series win meaning the players play a series of games with points awarded based on final position in each game. The player with the most points had one the series/tournament/..

My comment is based on reading tournament reports online a few years ago in a fascinated binge, so I'm likely wrong about many things.

@MartinRandall I'm inclined to count anything as a "win" if it counts as a win for the purposes of ELO calculation.

@vluzko do you have a particular ELO-like scoring system for Diplomacy rankings in mind?

@MartinRandall https://webdiplomacy.net/ghostRatings.php this is the rating system used by the platform Cicero ran on, so I'm defaulting to that.

@vluzko okay. So in that system my expected score in a 7 player game against equally skilled players is 1/7 and if I end up in a 3-way tie that gets me an actual score of 1/3, which causes my rating to go up. Does that mean it counts as a "win" for the purpose of this market?

To me, this seems possible but unlikely given my very limited sense of how much people will continue to work on this. Maybe people are going to continue to push on it more than I realize though?

@StephenMalina Even if researchers put their all into it, here's what I see as the basic argument why it won't happen. Diplomacy is a 7-player game where each player starts with 2 or 3 neighbors. Because conflict is mostly a matter of "more armies wins", any pair of players can defeat any one neighboring player early on if they so choose. Whether they so choose is substantially random and depends on whim and on tactical expediency that in turn depends on the way that moves unpredictably play out. If AI players are noticeably AI and not human, it also depends on how human players feel about allying with AI players. Throughout the game, it remains the case that success depends on your opponents not allying against you. Later on, there are stalemate lines, where if an opponent controls enough territory, there's nothing you can do to force a win. So it's hard to see how a 90% win rate is possible without highly reliable superhuman psychological manipulation. I would update a lot if someone who had played a significant amount of Diplomacy thought a 90% win rate was an attainable criterion. As it stands, I think people are just betting on the words "strongly superhuman" because they analogize it to Chess or Go in a way that I don't think is right.

@StevenK Compare to whether AI will reach a 90% win rate against top human players at three player chess. No matter how good the AI is, it seems to me that sometimes its two opponents will gang up on it at key points, and to reliably avoid that, it would need a model of how the human mind responds to board positions that's deterministic enough that it can reliably steer into board positions that cause players to behave in a given way.

Mildly superhuman version of this market: https://manifold.markets/vluzko/will-ai-for-diplomacy-be-mildly-sup

a 90% win rate against the best human players

Given that it's a seven player game, sometimes the other players happen to ally against you, and there's luck involved (in the same sense as there's luck in rock-paper-scissors, because people play mixed strategies), a 90% win rate sounds like it would almost require a mind hacking level of persuasion, but maybe I'm missing something.

@StevenK Hmm, yeah, I didn't consider the impact of multiplayer for the criteria. I think I want something more like "90% probability of not losing" against any specific player.

@StevenK My impression is also that the tactical part of the game isn't complex enough to support the possibility for much superhuman brilliance, though I haven't seen what top-level Diplomacy play looks like.

@vluzko If half the time winning the game comes down to whether your neighbors like you, and if the AI doesn't pass the Turing test, then win rate also starts depending a lot on whether the world's top Diplomacy players prefer AI wins to human wins.

@StevenK I don't know much about Diplomacy specifically but I played some similar games, and I think superhuman level is quite possible and achievable. The problem is, I imagined superhuman levels as something like "ELO higher than any human player, with some margin", which is still a much lighter threshold than "90% probability of not losing".

@l8doku Yes, ELO higher than all human players may well be achievable.

@vluzko Are you currently intending to resolve according to a literal 90% win rate in multiplayer games or some other criterion that you're still thinking about? Multiplayer seems essential to the game and I don't see any way to measure Diplomacy skill in terms of a 90% chance of not losing against any individual player. Maybe someone who has played Diplomacy a lot could weigh in?

@StevenK Note that Cicero got a 25.8% score. I think score is equivalent to win rate with a 2-way draw converted to a half win, 3-way draw converted to a one third win, and so on.

@StevenK I'm going to resolve with a literal 90% win rate for this market, and make a second market for 'weakly superhuman'. Do you have a suggestion for what 'about as superhuman at diplomacy as alphago was at go' translates to? (Never mind if it's achievable/likely)

@vluzko I don't have a suggestion, but I do have another difficulty, which is that Cicero's games weren't played all the way to a win/draw:

"For our experiments, games end at the end of 1908, and are scored according to the sum-of-squares scoring system, in which each player’s share of the score is proportional to the square of the number of SCs they control."

If future experiments also use this kind of blitz scoring, it means games will rarely play out all the way to someone winning by the standard rules.

@StevenK That is, even if the AI did very well, under this scoring, we wouldn't find out if it was actually going to win 90% of full games unless it did so in an unusually short amount of time.

@StevenK It looks like the Diplodocus experiments for no-press Diplomacy AI used a similar scoring rule, with limited but slightly longer games and sum-of-squares scoring at the end. So a question it could make sense to ask is "If turn limit + sum of squares scoring is used for future full-press Diplomacy AI on a reasonable sized sample of games against top human players, will it score at least as well as Cicero (25.8%) or Diplodocus (26-27%) did in their respective games against a wider range of players?"

@StevenK One could also ask about Elo directly. From the Diplodocus paper:

Elo ratings were computed using a standard generalization of BayesElo (Coulom, 2005) to multiple players (Hunter, 2004) (see Appendix I for details). This gives similar rankings as average score, but also attempts to correct for both the average strength of the opponents, since some games may have stronger or weaker opposition, as well as for which of the seven European powers a player was assigned in each game, since some starting positions in Diplomacy are advantaged over others. To regularize the model, a weak Bayesian prior was applied such that each player’s rating was normally distributed around 0 with a standard deviation of around 350 Elo.

The best scoring Diplodocus, which scores 27% (compared to average 1/7) has an Elo of 181 where I think the median player has 0. I haven't looked into the details, but note:

400 points in Elo systems generally corresponds to a 10-fold increase in expected winning odds or expected average score

@StevenK Maybe it's easier to score 25% against a population that scores 25% against a population that scores 25% against the general population of players than it is to score 90% against the general population of players, just because some bad luck can't be eliminated. So as another complication, maybe the assumptions behind Elo break down here.

@StevenK If I'm not mistaken, getting a 90% score would require the AI to get 54 shares to 1 share for each of 6 human players, so it ends up with 54/60=0.9, so that's a log10(54) * 400 = 693 point Elo difference. 90% score at the end of a blitz game is probably even more stringent than an eventual 90% win rate, because it means the AI has to complete its wins faster, but on the other hand, people are claiming blitz games are relatively easy for AI.

@StevenK To rephrase some of what I've said earlier in the thread: it seems much more likely to me that there will be a tower of 7 AIs on top of the best human, each of which scores points as if it had 100 more Elo when playing against the next lowest AI in the tower, than a single AI that scores points as if it had 700 more Elo when playing against the best human.