Episode 20: Hattie Zhou, Mila, on supermasks, iterative learning, and fortuitous forgetting
Generally Intelligent
The Lottery Ticket Hypothesis
The lottery ticket hypothesis presented a very interesting idea to me, which is that one of the reasons or part of the reason why over-parameterization is so helpful. So if you have used lucky subnetworks, then the model could just leverage that good initial spot and go down a pretty nice path of learning. The way they find these lottery tickets in the original paper by Jonathan Franco and Michael Carbon is by doing magnitude pruning.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
