MANIFOLD
By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?
9
Ṁ110Ṁ330
resolved Dec 31
Resolved
YES

My job is related to NLP, and I expect I will be able to judge the position on transformers in the field. No specific criteria, sorry, the question is just about my opinion.

Market context
Technology
AI
Get
Ṁ1,000
to start trading!

🏅 Top traders

#TraderTotal profit
1Ṁ16
2Ṁ14
3Ṁ12
4Ṁ11
5Ṁ11

People are also trading

Will we fully interpret a GPT-2 level language model by 2028?
14% chance
Will AI (large language models) collapse by may 2026?
10% chance
Will transformer architectures lose their dominant position in deep learning before 2028?
16% chance
By 2030, will large language models still be at the peak of AI? [DRAFT]
25% chance
By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?
10% chance
Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?
63% chance
Are Transformers “the last technology to do all of ML” (before AGI)?
Natural Language Robot by 2030?
79% chance
Natural Language Robot by 2030?
82% chance
Will Transformer-Based LLMs Make Up ≥75% of Parameters in the Top General AI by 2030?
50% chance
Sort by:

Mamba has lots of ongoing work, but didn't look like there's conclusive evidence it beats transformers yet

My guess is that we will build on transformers, perhaps significantly, but still consider the result to be under the category of 'transformer'. It will have the basic components like attention heads and mlp layers and a residual stream. It will just also have new stuff.

@NathanHelmBurger I wonder if we can pin down specific predictions about what we believe will or won't change, then

predictedYES

@wadimiusz yeah, unclear. RWKV and Hyena seem like promising advances. As the large labs start getting more cagey about their exact techniques we might not even know which techniques they are using.

https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition

People are also trading

Will we fully interpret a GPT-2 level language model by 2028?
14% chance
Will AI (large language models) collapse by may 2026?
10% chance
Will transformer architectures lose their dominant position in deep learning before 2028?
16% chance
By 2030, will large language models still be at the peak of AI? [DRAFT]
25% chance
By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?
10% chance
Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?
63% chance
Are Transformers “the last technology to do all of ML” (before AGI)?
Natural Language Robot by 2030?
79% chance
Natural Language Robot by 2030?
82% chance
Will Transformer-Based LLMs Make Up ≥75% of Parameters in the Top General AI by 2030?
50% chance
© Manifold Markets, Inc.•Terms•Privacy