Will loss curves on Pythia models of different sizes trained on the same data in the same order be similar?
13
1kṀ802
Apr 29
77%
chance

Someone in the EleutherAI discord is reporting that finetuning Pythia models of different sizes on the same data in the same order is giving spookily similar loss curves, just vertically shifted.

Will training Pythia models from scratch in the same way produce similar behaviour?
Resolves N/A if it turns out the original result was just a bug or something like that.

Get
Ṁ1,000
to start trading!
© Manifold Markets, Inc.TermsPrivacy