Episode 20: Hattie Zhou, Mila, on supermasks, iterative learning, and fortuitous forgetting
Generally Intelligent
00:00
The Lottery Ticket Hypothesis
The lottery ticket hypothesis presented a very interesting idea to me, which is that one of the reasons or part of the reason why over-parameterization is so helpful. So if you have used lucky subnetworks, then the model could just leverage that good initial spot and go down a pretty nice path of learning. The way they find these lottery tickets in the original paper by Jonathan Franco and Michael Carbon is by doing magnitude pruning.
Transcript
Play full episode
