The Gradient: Perspectives on AI cover image

Hattie Zhou: Lottery Tickets and Algorithmic Reasoning in LLMs

The Gradient: Perspectives on AI

NOTE

Understanding the Lottery Ticket Hypothesis in Neural Networks

The lottery ticket hypothesis states that within a randomly initialized neural network, there exists a sub-network that can achieve the same performance as the full network with all of the weights./nThis could lead to efficiency gains by only having to train fewer weights, saving on training costs and inference costs./nOver-parameterization might be helpful in deep learning because it increases the chances of having a lucky sub-network that is a good starting point for a particular task./nIdentifying lottery tickets can be done through magnitude pruning, where only weights with large magnitudes are kept and trained on./nThis simple method of magnitude pruning raises questions about why it is able to identify these lottery tickets and what makes them good for training.

00:00
Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner