Generally Intelligent cover image

Episode 20: Hattie Zhou, Mila, on supermasks, iterative learning, and fortuitous forgetting

Generally Intelligent

00:00

Is There a Way to Change Architectures?

The structure of the masks on a simple data set like MNIST, you could actually see the structure, especially in the first layer. That's what you knew from MNIST and I almost think that I'm pessimistic about kind of taking a sparse fee structure that you identified from lottery tickets to a general principle of how you should design architectures. Because I almost think the structure itself is just the right combinations of weight values where it just so happens that a lot of those weights are zero but they're not being used by the model. It seems like when the weights are set to zero, it's like creating some more structure in this architecture.

Transcript
Play full episode

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app