Neural Network Pruning and Training with Jonathan Frankle at MosaicML
Gradient Dissent: Conversations on AI
00:00
The Sub-Network and the Dense Network
You can't just reset the weights to what they were when you started training and then remove all other weights and get the same performance than I guess. That does work quite well, but the question is what purpose was that sub-network serving within the context of the dense network? It's entirely possible that there are two completely different dynamics going on when you have the whole network versus the sub-network. The sub-network is certainly sufficient to get good performance, but it's unclear whether it's actually necessary.
Transcript
Play full episode
