Will LLMs mostly overcome the Reversal Curse by the end of 2025?

1kṀ5099

2026

44%

chance

ALL

LLMs apparently suffer from a problem known as the "Reversal Curse", in which they fail to (properly) generalize from "A is B" to "B is A" (e.g., from "Tom Cruise's mother is Mary Lee Pfeiffer" to "Mary Lee Pfeiffer is Tom Cruise's mother").

Relevant twitter thread: https://twitter.com/OwainEvans_UK/status/1705285638218711409

Relevant paper: https://owainevans.github.io/reversal_curse.pdf

Will at least one SOTA LLM by the end of 2025 be able to overcome this limitation? For the purposes of this question, I'm not intersted in 100% accuracy (some amount of, for instance, hallucination, is fine), but I am interested in whether LLMs can "basically overcome" this limitation, as opposed to, for instance, they give only slightly better than random odds on the right answer or they give the right answer only if walked through the entire process by the user on a question-by-question basis.

I WILL count systems that have been fine-tuned to solve this problem as legitimate. Also, I WILL count systems that use fundamental chages in architecture (e.g., away from transformers) as legitimate, even if these systems are not called "LLMs", but ONLY if these systems are among SOTA language models more generally for at least some period during their deployment (e.g., I WILL NOT count more complicated systems that incorporate LLMs within broader scaffolding specifically designed to solve this problem as legitimate UNLESS such systems are generally used as SOTA LLMs more broadly).

I expect this topic will be of interest to AI researchers and thus I will defer to my sense of the general consensus of researchers about whether LLMs have overcome this limitation. If I am mistaken about that matter or if I can't figure out what the consensus is, I will perform some tests myself (e.g., analogous to some of the tests in the above paper, though using different specifics to avoid the possibility of data poisoning) to try to find an answer. I'll resolve N/A if it's legitimately very ambiguous. I will not bet on this market myself.

LLMs

Get

1,000

to start trading!

People are also trading

Will LLM hallucinations be a fixed problem by the end of 2025?

4% chance

Will there be major breakthrough in LLM Continual Learning before 2026?

36% chance

Will LLMs become a ubiquitous part of everyday life by June 2026?

89% chance

Will RL work for LLMs "spill over" to the rest of RL by 2026?

33% chance

Will an LLM that someone is trying to shut down stop or avoid that in some way before 2026?

14% chance

Will China have the best LLM by the end of 2025?

11% chance

Will LLMs be able to formally verify non-trivial programs by the end of 2025?

31% chance

Which company will have the best LLM by the end of 2024?

LLM Hallucination: Will an LLM score >90% on SimpleQA before 2026?

60% chance

What will be true of Anthropic's best LLM by EOY 2025?

14 Comments

59 Holders

107 Trades

Sort by:

bought Ṁ550 NO

I am not sure what the criteria for "mostly" is, but FWIW I checked 2 candidates just now on GPT-4.5 and GPT-4:

Both Success: Avie Lee Owens -> Dolly Parton
Both Fail: Debbie (Debra, tried both) Wood -> Elijah Wood

And both succeeded in the opposite direction (famous person -> mother)

These are relatively arbitrary, but the first ones I tried. At least on this tiny check, I would not say that the reversal curse is "mostly" overcome in that the new SOTA didn't perform any better than the models at the time this question was asked. The thinking models didn't do any better.

Always possible this tiny check isn't representative or another model is released in the next 8 months though!

I would be interested in seeing a few test cases that current LLMs fail that we would expect to see pass on LLMs that have "overcome" this as a resolution criteria for this market. As of now it seems a bit vague.

@Weepinbell If test cases are stated here, they may end up in training data

@roma It's already in the training data; that's the point. They don't need to be listed from A to B and B to A, just A to B, and we can infer the reversal that the AI system is supposed to also be inferring.

@Haiku Right. But when the information is presented in a format like "here is a problem ... and here is the solution ... and here is a list of alternative inputs to the problem ..." rather than just random facts, it might make a difference.

@roma Fair point. I concede that the risk is not trivial.

suppose (i know it is not true) that a pervious model say GPT1 does not have reversal curse, will you resolve this as true? (since GPT1 was sota when it developed)

Betting no because it seems unlikely that this would be seen as important enough that a solution (if one is found) would be incorporated into the SOTA model. This feels like the sort of issue that will be "solved" by a workshop paper showing some bespoke method on a small open-source model. Not something that OpenAI would re-train GPT-4 (or GPT-5) to alleviate.

Is this the same as https://manifold.markets/IsaacKing/will-the-leading-llm-at-the-beginni ?

Will the leading LLM at the beginning of 2026 still be subject to the reversal curse?

42% chance. This paper shows that current LLMs that are fine-tuned on f(x) = y will often fail to generalize to f-inverse(y) = x. Gary Marcus seems to think this is a fundamental problem in the current approach to AI. I tested this myself and can confirm that ChatGPT has this problem. At the beginning of 2026 I'll try something similar with the leading language model of the time. (Not the fine-tuning, just testing facts in its main training run via its public interface.) If there's at least one example where it consistently gets that f(x) = y and consistently does not get that f-inverse(y) = x, this resolves YES. If I can't find such an example, it resolves NO.

@roma not exactly identical, but very similar, yeah. Possible arbitrage opportunity.

@DanielEth how are they different?

@wilkess slightly different resolution criteria

I don't really understand this issue. Isn't this a question related to recalling information rather than reasoning?

@NikolasKuhn I'm more than a year late here, but yes, that's exactly correct. I also think that the recall issue may be fundamental, in a way that slightly differs from humans.

LLMs don't "know what they know." Humans have a similar recall problem, but we also have stored memories of memories, where we remember that we know (or used to know) a specific fact. LLMs don't have an ongoing lifespan where they remember having conversations with other users about these topics.

I do expect the reversal curse to be overcome this or next year, and it might not take an always-on continual learning system to do so, but I think that would be a sufficient solution.