Mamba has lots of ongoing work, but didn't look like there's conclusive evidence it beats transformers yet

My guess is that we will build on transformers, perhaps significantly, but still consider the result to be under the category of 'transformer'. It will have the basic components like attention heads and mlp layers and a residual stream. It will just also have new stuff.

@NathanHelmBurger I wonder if we can pin down specific predictions about what we believe will or won't change, then

@wadimiusz yeah, unclear. RWKV and Hyena seem like promising advances. As the large labs start getting more cagey about their exact techniques we might not even know which techniques they are using.

https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition