Will Transformer based architectures still be SOTA for language modelling by 2026? | Manifold

Will Transformer based architectures still be SOTA for language modelling by 2026?

Plus

55

Ṁ3798

2026

68%

chance

1D

1W

1M

ALL

The intent is to capture whether there is a major paradigm shift, on the same order of magnitude of RNNs to Transformers. The market resolves based on the architecture of the SOTA language model. If the architecture is still recognizable as a transformer with modifications, this market resolves as Yes. If the architecture is at least as different from Transformers as Transformers are from RNNs then it resolves No. If the evaluation numbers are ambiguous which architecture is the SOTA due to incomparable evaluations or non Pareto improvement over all evaluations, but one is obviously better overall, then I will use my judgement. If I deem it too close to call, I will resolve this market as Yes, because it indicates that no architecture has clearly surpassed Transformers.

This question is managed and resolved by Manifold.

#Technical AI Timelines

Get

1,000

and

3.00

Sort by:

I bet yes because I measure SOTA by frontier models, and I would be surprised if the labs were to invest the resources required for a large scale training run on an architecture which was new.

I like this question, added some liquidity

Is Mamba a transformer?

Would you count a model like Stanford's Monarch Mixer as a Transformer?

Is an ARDM (auto-regressive diffusion model) a transformer?

predicts YES

@ampdot Probably not; for a more definite answer you would have to elaborate more on what you have in mind.

Yes.

Plus memory and retrieval.

And hierarchical/nested/capsules (already used in video and high-end competition for long prompts)

And mixture of experts, and flash-attention and other sparsity techniques.

Will Transformer based architectures still be SOTA for language modelling by 2026?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition

Related questions

Will any 10 trillion+ parameter language model that follows instructions be released to the public before 2026?

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?

On January 1, 2027, a Transformer-like model will continue to hold the state-of-the-art position in most benchmark

Will a language model that runs locally on a consumer cellphone beat GPT4 by EOY 2026?

Will we see most new language models shifting to addition-only architectures like BitNet/BitNet 1.58b in 2024?

By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?

When will a non-Transformer model become the top open source LLM?

Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

Will transformers still be the dominant DL architecture in 2026?

Related questions

Will any 10 trillion+ parameter language model that follows instructions be released to the public before 2026?

By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?

When will a non-Transformer model become the top open source LLM?

On January 1, 2027, a Transformer-like model will continue to hold the state-of-the-art position in most benchmark

Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?

Will a language model that runs locally on a consumer cellphone beat GPT4 by EOY 2026?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

Will we see most new language models shifting to addition-only architectures like BitNet/BitNet 1.58b in 2024?

Will transformers still be the dominant DL architecture in 2026?

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules