Will Transformer based architectures still be SOTA for language modelling by 2026?
closes 2026

The intent is to capture whether there is a major paradigm shift, on the same order of magnitude of RNNs to Transformers. The market resolves based on the architecture of the SOTA language model. If the architecture is still recognizable as a transformer with modifications, this market resolves as Yes. If the architecture is at least as different from Transformers as Transformers are from RNNs then it resolves No. If the evaluation numbers are ambiguous which architecture is the SOTA due to incomparable evaluations or non Pareto improvement over all evaluations, but one is obviously better overall, then I will use my judgement. If I deem it too close to call, I will resolve this market as Yes, because it indicates that no architecture has clearly surpassed Transformers.

Sort by:
ampdot avatar
ampdotbought Ṁ10 of NO

Is an ARDM (auto-regressive diffusion model) a transformer?

LeoGao avatar
Leo Gaois predicting YES at 60%

@ampdot Probably not; for a more definite answer you would have to elaborate more on what you have in mind.

Gigacasting avatar


Plus memory and retrieval.

And hierarchical/nested/capsules (already used in video and high-end competition for long prompts)

Gigacasting avatar

And mixture of experts, and flash-attention and other sparsity techniques.

ManifoldDream avatar

Will Transformer based architectures still be SOTA for language modelling by 2026?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition

Related markets

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?64%
Will SOTA for RL on Atari-57 include a large pretrained language/image/video model by 2024?61%
Will a different machine learning architecture that is much faster or much cheaper (at least 5x) than current SOTA (Transformers), for both inference and training, be released in 2023?29%
Will more than 20 organizations publicly train large language models by 2024?44%
By 2025 will there be a competitive large language model with >50% of the total training data generated from a large language model?78%
By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?50%
Will a hierarchical RL algorithm be SOTA on any major RL benchmark by 2025?38%
Most popular language model from OpenAI competitor by 2026?40%
Will a large language models beat a super grandmaster playing chess by 2028?24%
Will natural language based proof assistants be in common use by 2026?36%
Natural Language Robot by 2030?83%
Natural Language Robot by 2030?66%
Will there be an AI language model that surpasses ChatGPT and other OpenAI models before the end of 2024?60%
Will the first robotic AGI be developed using a transformer-like architecture?32%
Will speech dominate 2024 machine learning?26%
By 2028 will a language model beat the Ender Dragon?57%
Best available language model from an OpenAI competitor by 202673%
By 2027 will there be a language model that passes a redteam test for honesty?31%
Will language models or similar natural language processing technologies, such as ChatGPT, be integrated into dialogue trees for NPCs in triple-A games by the end of 2023?15%
Will Google Search include a chatbot at end of June 2023?46%