Would an agent program which coordinates the responses of hundreds of LLMs, online research, calculators, coding scratchpad etc in real-time be allowed? That's the direction the offerings from the AI labs seem to be going.

I think that already most front ends which look like you're speaking to an LLM are already this, such as what you get at OpenAI, Anthropic, Gemini. Does anyone run and return just the unprocessed results of an actual LLM call? It doesn't seem like they would.

@Ernie Seems like it shouldn't be allowed. Either it's an LLM or it's not...

I think Now llms like O3 can play chess as I tested and it can beat around an 800 elo bot which is impressive considering how O1 went from barely beating 250 elo chess bots to O3 beating way higher leveled bots and rarely making illegal chess moves.I think this might become more possible

@Blocksterpen3 Carlsen better hold onto his pawns

If the LLM writes the code to train a chess engine, requisitions the compute, trains it, and then runs inference on it, does that count?

@DavidSpies "The model can write as much as it want to reason about the best move. But it can't have external help beyond what is already in the weights of the model. For example, it can't access a chess engine or a chess game database." Not op but sounds like a no to me.

rate me pls - https://manifold.markets/bananaLtsnh/will-i-make-mit-bwsi-this-year

@bananaLtsnh Hi! I see that you're fairly new here, welcome!

For your information, it's a violation of the community guidelines to advertise your markets on unrelated markets -- see https://manifoldmarkets.notion.site/Comment-guidelines-c4b4e6b0b0064b268970388f692c4745

What if an LLM uses Stockfish?

@DanielJohnston Also, what if the SGM uses Stockfish?

@CraigDemel 😂😂

@DanielJohnston The description said the model has to use only its own weights.

@MaxE What if the model is model does not use stock fish at inference time (own weights), but is trained on data generated by stockfish?

@DanielLawson I'm not sure a pure language model can grasp the logic that chess engines can employ without efficiency losses so immense that it's never worth making a llm to play high level chess before we make something better at logic

@MaxE I don’t agree, I think I scaled up version of this paper trained on stock-fish level data would work: Grandmaster-level Chess without Search: https://arxiv.org/html/2402.04494v1

You go from GM to super with just more data/larger model/similar architecture

@DanielLawson great, buy my limit order!

@CraigDemel I think it’s 100% techncialt feasible, but maybe 40-50% someone actually does it. For example, OpenAI could do it, but it’s not worth it for them to spend time/resources making a specialist trained LLM for chess. My claim is not as broad that every commercial LLM with be chess super GMs. It is that one could train an LLM trained / specialized in chess, and would reach super GM with sufficient and high-quality data.

@DanielLawson I wonder if super gms will change their strategy when playing against such a model. If you play weird positions that it wasn't trained on and play super passively to wait for a mistake

@DanielLawson that would be totally fair: grandmasters probably train by playing with stockfish too

Unless there is a loophole I'm missing here (train an ml system, fool nyt into calling it an llm), this is currently the most mispriced market on manifold.

If i can be assured this only applies to openai's o models, gemini's regular models, grok, claude etc. I would put half my manna into it. Is there a market about those specifically? (e.g. "will claude beat a super grandmaster by 2028")

I just played gemini pro-2.5-exp, it gave me it's queen after 15 moves. I think most bettors here don't play chess or don't use llms 😕

@FergusArgyll here's the game for the record. I thought < 5 seconds per move and played deliberately passively

@FergusArgyll Consider

1. Current SoTA models are not trained for chess (evidenced by GPT-3.5 Turbo being much better than GPT-4o)

2. Agents playing games is the next step for AI companies as reasoning models stabilize (OpenAI's roadmap says this explicitly; I don't know if Anthropic has publically said the same, but ClaudePlaysPokemon suggests it's top-of-mind for employees)

3. Chess is a great game for #2, requiring substantial computation while being trivial to evaluate against stockfish and requiring little context

4. "Computers beat top humans at chess... again" is great PR for a company that is promising investors AGI

5. Two years is a long time in this space

@FergusArgyll I think users assume exponential progress

@FergusArgyll I also think it is mispriced but note that it requires the human to play blind and a single win by an LLM is sufficient to resolve YES...

Blind chess should be added to the market title.
"Will a large language model beat a super grandmaster playing blind chess by 2028?"