we can only hope.

What prompt will you be using? I imagine that changes their performance quite a bit

Good point! On each move, I’ll provide it the moves played so far in PGN notation, as well as the current position in FEN notation. This way both ways of representing position would be in context and in a standard format.

I think that makes the model significantly worse than it could otherwise be. I'd recommend using whatever prompt someone that claims "SOTA LLM chess" or something came up with

I’m planning to use lichess to play the game, and those are the representations it provides. In a future market this might change.

When I tested this with ChatGPT 3.0 a while back, it couldn't even remember the board position and kept making illegal moves. How will you resolve if it does this?

Let’s say three illegal moves will result in a loss. Distinctions like Rad1 vs. Rd1 won’t count towards this, but I’ll ask it for clarification.