Eric Michaud on scaling, grokking and quantum interpretability
The Inside View
00:00
Groking and Groking in Neural Networks
Grok is this phenomenon where neural networks can generalize long after they first overfit their training data. This was first discovered by some folks at OpenAI. They were training small transform models to learn basic math operations. If they like kept training the network for way longer then it took for the network to overfit eventually the network would generalize.
Transcript
Play full episode
