Generally Intelligent cover image

Episode 20: Hattie Zhou, Mila, on supermasks, iterative learning, and fortuitous forgetting

Generally Intelligent

CHAPTER

Is There a Way to Change Architectures?

The structure of the masks on a simple data set like MNIST, you could actually see the structure, especially in the first layer. That's what you knew from MNIST and I almost think that I'm pessimistic about kind of taking a sparse fee structure that you identified from lottery tickets to a general principle of how you should design architectures. Because I almost think the structure itself is just the right combinations of weight values where it just so happens that a lot of those weights are zero but they're not being used by the model. It seems like when the weights are set to zero, it's like creating some more structure in this architecture.

00:00
Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner