AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
2min chapter
Eric Michaud on scaling, grokking and quantum interpretability
The Inside View
The Power of Clustering in Language Models
We just looked at like the number of samples in each cluster. So we just like randomly we did some filtering but like basically we took a bunch of samples which the model got low loss on and clustered them, then looked at the size of the clusters. And like this is like really messy and there are a bunch of reasons why like even if there was a power law this could fail. But like maybe eventually it kind of looks like roughly power law like and like very roughly with like the exponent that like we would sort of expect from the like empirical scaling laws for the language models themselves.
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode