Note: I asked

Stella Biderman and she was going to check but I think she forgot.

This has happened twice , partly my faylt cause I also keep forgetting .

Will ask again I guess.

Yes

Some evidence https://arxiv.org/abs/2305.18411

Feature-Learning Networks Are Consistent Across Widths At Realistic Scales

We study the effect of width on the dynamics of feature-learning neural networks across a variety of architectures and datasets. Early in training, wide neural networks trained on online data have not only identical loss curves but also agree in their point-wise test predictions throughout training.…

Won't bet in case it's less obious but for the record mi prediction is like 30%.

Have a vague model about how the behaviour happens because the finetuning is affecting the models in the same way but not something else that lowers the loss significantly.

But I'm very confused and don't really get wtf is happening here tbh.