Episode 20: Hattie Zhou, Mila, on supermasks, iterative learning, and fortuitous forgetting
Generally Intelligent
00:00
How to Find Sub-Networks That Are the Right Shape of the Solution?
It's not clear exactly why setting it to zero doesn't seem to harm that process. You're still able to find a similar solution. I think Jonathan Franco also showed in some of the papers that these solutions are linearly connected. So they're all arriving at the same basin in the solution space. Right, so it's almost just like the same solution you find when you train a model originally. Except now you kind of move that around a bit such that it's a sparse model.
Transcript
Play full episode
