Finished training.
it should be competitive with SOTA of the time. Rough estimates ofc but not too far behind.
I will not trade in this market.
The evaluation shows that BaGuaLu can train 14.5-trillion-parameter models with a performance of over 1 EFLOPS using mixed-precision and has the capability to train 174-trillion-parameter models, which rivals the number of synapses in a human brain.
https://dl.acm.org/doi/10.1145/3503221.3508417
https://keg.cs.tsinghua.edu.cn/jietang/publications/PPOPP22-Ma%20et%20al.-BaGuaLu%20Targeting%20Brain%20Scale%20Pretrained%20Models%20w.pdf
https://www.nextbigfuture.com/2023/01/ai-model-trained-with-174-trillion-parameters.html
Y Combinator using this as a joke headline in November. https://www.youtube.com/watch?v=lbJilIQhHko
@BarrDetwix yes you can :) it'll be a tad bit costly though. And I think i should ammend the description to say that it should be competitive with SOTA of the time
Apologies for the title change, I will compensate if you want me to. I've changed title to AI model instead of LLM because we soon will have VLMs (vision language models) become popular, etc.