AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Groking and Groking in Neural Networks
Grok is this phenomenon where neural networks can generalize long after they first overfit their training data. This was first discovered by some folks at OpenAI. They were training small transform models to learn basic math operations. If they like kept training the network for way longer then it took for the network to overfit eventually the network would generalize.