The evaluation shows that BaGuaLu can train 14.5-trillion-parameter models with a performance of over 1 EFLOPS using mixed-precision and has the capability to train 174-trillion-parameter models, which rivals the number of synapses in a human brain.

https://dl.acm.org/doi/10.1145/3503221.3508417

https://keg.cs.tsinghua.edu.cn/jietang/publications/PPOPP22-Ma%20et%20al.-BaGuaLu%20Targeting%20Brain%20Scale%20Pretrained%20Models%20w.pdf

https://www.nextbigfuture.com/2023/01/ai-model-trained-with-174-trillion-parameters.html

Y Combinator using this as a joke headline in November. https://www.youtube.com/watch?v=lbJilIQhHko

If I take llama and "mixture of experts"" it a thousand times, then the resulting system has a lot of parameters, is state of the art(if not efficient in parameter use), and isn't THAT costly to run 🤔

@Mira do it

Does any open source model resolves to YES, or only entreprise models?

... can I train a 10 trillion MNIST classifier? :)

@BarrDetwix yes you can :) it'll be a tad bit costly though. And I think i should ammend the description to say that it should be competitive with SOTA of the time

Apologies for the title change, I will compensate if you want me to. I've changed title to AI model instead of LLM because we soon will have VLMs (vision language models) become popular, etc.