
Will loss curves on Pythia models of different sizes trained on the same data in the same order be similar?
13
1kṀ802Apr 29
77%
chance
1H
6H
1D
1W
1M
ALL
Someone in the EleutherAI discord is reporting that finetuning Pythia models of different sizes on the same data in the same order is giving spookily similar loss curves, just vertically shifted.

Will training Pythia models from scratch in the same way produce similar behaviour?
Resolves N/A if it turns out the original result was just a bug or something like that.
This question is managed and resolved by Manifold.
Get
1,000 to start trading!