Do scaling laws happen because models experience a ton of tiny phase changes which average out to a smooth curve?

Problem 5.31 from @NeelNanda's 200 COP.

"D* 5.31 -  Hypothesis: The reason scaling laws happen is that models experience a ton of tiny phase changes, which average out to a smooth curve because of the law of large numbers. Can you find evidence for or against that? Are phase changes everywhere?"

Resolves to the best evidence available by the end of 2024.

Get Ṁ600 play money
Sort by:

This seems ~vaguely plausible but I think proving it would require a level of understanding of the individual circuits that we won't have by 2025.

@vluzko End of decade?

I'm bullish but think the odds are less than 50%

More related questions