Will a Mamba-based LLM of GPT 3.5 quality or greater be open sourced in 2024?

1kṀ1874

Jan 1

73%

chance

ALL

Mamba is a next-generation architecture that seeks to improve on the shortcomings of transformers, mainly around context size and eliminating quadratic memory consumption during inference. https://arxiv.org/abs/2312.00752

YES resolution requires the Mamba LLM to match or beat GPT 3.5 on at least 5 popular benchmarks.

LLMs

Mamba

Get

1,000

to start trading!

People are also trading

Size of smallest open-source LLM marching GPT 3.5's performance in 2025? (GB)

4.40

Will there be an open source LLM as good as GPT4 by June 2024?

12% chance

Will an open-source fully functional Auto-GPT like LLM exist by the end of 2025?

89% chance

Will an uncensored open-source LLM model comparable to 2023 GPT4 be available to the public by the end of 2025?

84% chance

When will an open-source LLM be released with a better performance than GPT-4?

Will an open-source LLM beat or match GPT-4 by the end of 2024?

83% chance

Which next-gen frontier LLMs will be released before GPT-5? (2025)

Will we have an open-source model that is equivalent GPT-4 by end of 2025?

82% chance

Will any open source LLM with <20 billion parameters outperform GPT-4 on most language benchmarks by the end of 2024?

13% chance

China will make a LLM approximately as good or better than GPT4 before 2025

Sort by:

@Manifold can you resolve?
I believe the question creator is banned

@quantizor are you able to resolve?

time to resolve

does this count as Mamba-based? it's open and easily above GPT-3.5 quality

https://www.ai21.com/blog/announcing-jamba-model-family

https://huggingface.co/nvidia/mamba2-hybrid-8b-3t-4k

Here's a mamba-transformer-moe hybrid that's about as good as gpt-3.5. ai21.com/jamba.

@matt still using transformer, so I wouldn't count it : )

bought Ṁ10 NO

Looks like Gemini 1.5 used normal transformers and not Mamba, while also seeming to get around these shortcomings (1M context size). I expect this will cause interest in Mamba to wane, which decreases the chance someone will bother training and testing a Mamba LLM to GPT 3.5 level.

@adele I remain unconvinced that transformer architecture will be the long term winner due to its compute and memory-hungry nature. These are great improvements though.