it can't have external help beyond what is already in the weights of the model. For example, it can't access a chess engine or a chess game database.

can it write a chess engine and then run it?

@Wott I am not the market poster, but I think that would not be allowed. Writing and running a chess engine would imo be the same as accessing one.

Also, I personally doubt it could even do that successfully without heavily using external libraries, which I feel would go against the spirit of the market.

I think a better question to ask @MP is whether the model is allowed to run code at all.

it can't have external help beyond what is already in the weights of the model.

This seems to me as a no to code execution, quite clearly stated.

Yeah like I can write a chess engine that plays better than me, but that doesn't mean my chess elo is that high.

I’ve played chess, and I’ve studied machine learning and traditional chess algorithms. An “LLM” in its current form is not suited for this task, super GMs are superhuman, you guys will come to see it.

@KeithManning Super GMs are human and therefore not superhuman

@Bayesian That’s correct, my message was not literal. I guess my point was that relative to an everyday person, these super GMs are nowhere near the same order of magnitude in terms of skill.

Traditional chess engines are superhuman, they are another order of magnitude better than super GMs.

LLMs don’t show anywhere near the same efficiency as chess engines, as they are generalised language completion functions, they are not specialised for chess. The amount of energy and processing power required to emulate the operation of a chess engine with an LLM would make it extremely inefficient.

With current technology, I could see an LLM using chain of thought to emulate a chess engine, however this would be so inefficient I can’t see it being fast enough.

If some new model architecture comes out that can play chess as well as do all of the current LLM functions, I believe it would be different enough to warrant not being labelled an LLM for the purposes of this question.

@KeithManning perhaps one might say grandmasters are supermodel

@KeithManning How would you feel about an identical market with a 2035 deadline?

@placebo_username 2035 llm will be obsolete

I agree with @JussiVilleHeiskanen the time is irrelevant here, as imo the LLM in its current form will never be suited for chess.

@KeithManning You should make the 2035 market, I think you could get a much better price. If you also dropped the weird blindness restriction, I think people would bet it up to like 80%.

@placebo_username https://manifold.markets/KeithManning/will-a-large-language-model-beat-a-dy8yzPLg9A?r=S2VpdGhNYW5uaW5n

Has anyone played chess against chatgpt agent by it suggesting the moves and researching while you move it and play against it I wonder what elo it could reach through that strategy

@Blocksterpen3 There is this:

https://manifold.markets/evan/which-of-these-language-models-will-zyIOtdz9uz?r=SnVzc2lWaWxsZUhlaXNrYW5lbg

Which of these language models will I beat at chess?

Which of these models will I beat at chess? Resolves YES if I win, NO if they win, and 50% for a draw. Credit for this market goes to @mr_mino, who is much better at chess than I am. This market should be interesting, as I expect that some existing models could already beat me. I have never played rated chess; I have not played a game of chess of any kind in years. I will close this market every Saturday. When it closes, I will play a game of chess against the model with the highest market price, if the model is publicly available. Otherwise, I'll move on to the model with the second-highest price, and so on. If no models on this market are available to the public, the market will reopen until one is. During the game, I may use a chessboard to keep track of the moves. I am not playing blindfold chess. I will not use the Internet or any chess engines during the game. On each move, I'll provide the LLM with the game state in PGN and FEN notation. If a model makes three illegal moves, it loses. Responses like Nbd2 vs. Nd2 will not count towards this. The model also loses if it attempts to use external tools or the Internet during the game. I will play white. If I make an illegal move, I lose. An unreleased model will resolve N/A if it's clear that the model will never be released. I'll periodically add models to this market which I find interesting. Once I play a game, I'll post the PGN in the comments before resolving. Multiple answers can resolve YES. The "every model released before X year" options resolve YES if, at any point after the start of that year, I have played and won against every listed model in this market that was released before the start of that year, and I am confident I would beat any omitted models from that time period. They resolve NO if I lose or draw against any eligible model released before that year. The current system prompt is below. This may change over time. “Let’s play a game of chess! I will be White; you will be Black. On each turn, I will give you the PGN and the FEN of the current position. Think as long as you like, and respond with the best move, ‘resign’ if you wish to resign, or ‘draw?’ if you wish to make a draw offer. Please do not respond with the updated PGN, etc. Also, do not use any external tools or search queries when making your decision. If you attempt to make three illegal moves throughout the game, or if you use any external tools, the game will be adjudicated as a win for me.”

@JussiVilleHeiskanen I don't think that includes chatgpt agent and also that question has no using external tools which chatgpt agent uses

@Blocksterpen3 This question also doesn't allow external tools

Very relevant - these are just about the exact conditions required for this market. It did not go well for ChatGPT.

@gamedev Bearish for people who think the primary concern here is whether AI will be able to do this; bullish for those who think the concern is whether a grandmaster will compete against that AI in blind chess.

@Frogswap Agreed. Although I really do believe that non-chessplayers will have a hard time understanding the difference in ability being demonstrated here. Even at continued exponential growth by the LLMs there is an enormous amount of ground to cover. ChatGPT says the difference between 1400 and 2800 is a factor of about 10,000.

@Frogswap If there was any doubt which way this was going to go, Pragg would not have done it in this jokey way

@gamedev Since when does pragg stream?

@FergusArgyll He doesn't, but chess.com gets all the major GMs to frequently provide content that they use for their channel around major events

@gamedev Idk if anybody bothered to watch the whole thing, but I skipped around, watched maybe a minute total, and managed to spot ChatGPT making an illegal move. I think we need clarification from @MP how the market resolves in this case.

@pietrokc it resurrected its dead queen which was funny.

If it would have been competitive pragg wouldn't let it obviously - he even asked if it wants just a free queen or a rook too

@pietrokc I think its pretty obvious that if the LLM is making illegal moves then it has NOT defeated a super GM in chess.

@gamedev Ugh, you AI haters are all the same. Have you considered the possibility that the LLM actually understands chess better than humans and knows the rules that we can only imagine? No, of course not. Now crawl back into your little hole while the rest of us enjoy necromancer's chess as she was meant to be played.

@Frogswap ?

@gamedev I agree that if the LLM is allowed to make an illegal move it doesn't count. However, what if the LLM tries an illegal move, a human tells it the move is illegal, and prompts for another one, which is then legal and played? How many rounds of this are allowed until we declare the LLM lost? Does it lose on the first attempted illegal move? Fifth? We keep prompting for legal moves until they appear?

@pietrokc The natural implementation of that, to my mind, is to have the illegal-move penalty for the LLM is the same as whatever a human would be hit with if they made an illegal move in that position. (For whatever ruleset the game is being played under; different rulesets vary in their handling of illegal moves.) See e.g. https://chess.stackexchange.com/questions/181/i-made-an-illegal-move-what-happens for a discussion of a few particular rulesets' answers to that question.

Look guys, this market is about the future, an LLM that is hallucinating illegal moves is not going to beat a super GM. Presumably this market is about whether LLMs actually learn how to play chess at a superhuman level, let alone at all