In 2024 the field of AI language models is dominated by Transformers. Many research papers are suggesting alterations and newer models, but none of them has successfully competed with Llamas and their open source friends.
The market will resolve positively as soon as the first place on the Hugging Face Open LLM Leaderboard (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) is taken by a non-Transformer model.
If Hugging Face disappears or falls into obscurity by the time this market is resolved, another similar ranking will be used.
The following modification to Transformer model are not enough to consider is a new model for the purposes of this question:
Changes to activation function
Extra dense layers
Changes to normalization/dropout/...
Changes to the number of heads/keys/queries etc.
Minor changes to how attention components are calculated (e.g. adding bias or calculating some sort of non-linearity)
Using Transformers in an ensemble
Other changes such that Wikipedia still categorizes the new model as a Transformer
An attention-based model in which attention is applied not to pairs of positions, but to some other domain, especially if it lead in significant improvement in efficiency, qualify as significant change for the purposes of this question.
I do not bet on my own questions.
Update from 2026-01-01: If a model has an unlimited context window, or is essentially recurrent, I'll consider it not a Transformer.
1,000🏅 Top traders
| # | Trader | Total profit |
|---|---|---|
| 1 | Ṁ113 | |
| 2 | Ṁ34 | |
| 3 | Ṁ33 | |
| 4 | Ṁ19 | |
| 5 | Ṁ19 |
People are also trading
@traders I'm about to resolve this market as "In 2026 or earlier" based on Kimi K3 model. While it retains the high-level block shape, there are two clear distinctions:
1. The majority of layers are using "Kimi Delta Attention", which is a version of DeltaNet and acts more like a recurrent layer, than an attention which directly correlates the current position with others.
2. It doesn't have any position encoding.
The change 1 fits the following condition from the market description:
> An attention-based model in which attention is applied not to pairs of positions, but to some other domain, especially if it lead in significant improvement in efficiency, qualify as significant change for the purposes of this question.
@OlegEterevsky I'm not going to argue with this resolution, it's probably technically correct in terms of the market description part you've highlighted. But in terms of vibes and market title it honestly feels like a superficial change.
@ProjectVictory I also chatted with with Fable about this, and its opinion is that DeepSeek V4 and GLM-5.2 are still Transformers, while Kimi K3 is borderline, tending towards "not".
With this in mind, I re-read the description of the market, and since it has a specific clause describing this exact design change, I don't see how I can judge it otherwise.
If the resolution criteria were move vibe-based, the resolution might have been different, but for what it's worth, the Wikipedia Transformer article doesn't include DeltaNet in its list of Transformer developments. So, I'm not even sure it constitutes a Transformer based on vibes.
Taking a step back, the development of LLMs is incremental, so we need to draw a line somewhere.
@OlegEterevsky what would you say about DeepSeek V4 (HCA, CSA, Engram) and Qwen3 Next (recurrent), would they count as not Transformer? I think they should.
What about GLM-5.2? It uses DSA which is not N^2 but N*2048, I would count it as not Transformer, what do you think?
@Fynn Let me have a detailed look at the architecture, but in general of its slostly recurrent, then it’s definitely not Transformer.
If it essentially uses Attention (KVQ transformations, that sort of things) and just does some tricks to optimize the performance, then maybe it’s still a Transformer.
@HastingsGreer N^2 might be a tough call if you want it to write a novel, or simulate a person living for decades.
Also below N^2 scaling is not a necessary condition for this market to resolve positively.
N^2 might be a tough call if you want it to write a novel, or simulate a person living for decades.
Gemini 1.5's 1M context window convinced me transformers can scale pretty well for that.
@singer I don't know the technical details, but I'm assuming it's not technically a "traditional" Transformer.