Eric Michaud on scaling, grokking and quantum interpretability
The Inside View
00:00
The Power of Clustering in Language Models
We just looked at like the number of samples in each cluster. So we just like randomly we did some filtering but like basically we took a bunch of samples which the model got low loss on and clustered them, then looked at the size of the clusters. And like this is like really messy and there are a bunch of reasons why like even if there was a power law this could fail. But like maybe eventually it kind of looks like roughly power law like and like very roughly with like the exponent that like we would sort of expect from the like empirical scaling laws for the language models themselves.
Transcript
Play full episode
