François Chollet's 'On the Measure of Intelligence' challenges the conventional understanding of artificial intelligence by focusing on skill acquisition efficiency rather than mere skill output. The book argues that true intelligence lies in the ability to learn and adapt to new tasks efficiently, using limited data. Chollet introduces the concept of 'generalization' as a key aspect of intelligence, contrasting it with the rote memorization often seen in current AI systems. He proposes the Abstraction and Reasoning Corpus (ARC) challenge as a benchmark for measuring this type of intelligence. The book's central theme is the importance of understanding the underlying processes of learning and adaptation in AI, rather than solely focusing on performance metrics.
This research area focuses on understanding how neural networks optimize to the same linearly connected minimum under different conditions, such as SGD noise, and how sparse subnetworks (winning tickets) can achieve full model performance. It involves concepts like iterative magnitude pruning and error landscape geometry.
This research compares fine-tuning with weight rewinding and learning rate rewinding in neural network pruning. Both rewinding techniques outperform fine-tuning, offering a network-agnostic approach that matches state-of-the-art pruning methods in terms of accuracy and compression ratios.
The lottery ticket hypothesis proposes that dense neural networks contain sparse subnetworks, or 'winning tickets,' that can be trained to achieve comparable accuracy to the full network. This concept is explored through experiments demonstrating the effectiveness of these subnetworks when identified and trained independently. The hypothesis has implications for neural network pruning and training efficiency.
In this episode of Machine Learning Street Talk, we chat with Jonathan Frankle, author of The Lottery Ticket Hypothesis. Frankle has continued researching Sparse Neural Networks, Pruning, and Lottery Tickets leading to some really exciting follow-on papers! This chat discusses some of these papers such as Linear Mode Connectivity, Comparing and Rewinding and Fine-tuning in Neural Network Pruning, and more (full list of papers linked below). We also chat about how Jonathan got into Deep Learning research, his Information Diet, and work on developing Technology Policy for Artificial Intelligence!
This was a really fun chat, I hope you enjoy listening to it and learn something from it!
Thanks for watching and please subscribe!
Huge thanks to everyone on r/MachineLearning who asked questions!
Paper Links discussed in the chat:
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks: https://arxiv.org/abs/1803.03635
Linear Mode Connectivity and the Lottery Ticket Hypothesis: https://arxiv.org/abs/1912.05671
Dissecting Pruned Neural Networks: https://arxiv.org/abs/1907.00262
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs: https://arxiv.org/abs/2003.00152
What is the State of Neural Network Pruning? https://arxiv.org/abs/2003.03033
The Early Phase of Neural Network Training: https://arxiv.org/abs/2002.10365
Comparing Rewinding and Fine-tuning in Neural Network Pruning: https://arxiv.org/abs/2003.02389
(Also Mentioned)
Block-Sparse GPU Kernels: https://openai.com/blog/block-sparse-gpu-kernels/
Balanced Sparsity for Efficient DNN Inference on GPU: https://arxiv.org/pdf/1811.00206.pdf
Playing the Lottery with Rewards and Multiple Languages: Lottery Tickets in RL and NLP: https://arxiv.org/pdf/1906.02768.pdf
r/MachineLearning question list: https://www.reddit.com/r/MachineLearning/comments/g9jqe0/d_lottery_ticket_hypothesis_ask_the_author_a/ (edited)
#machinelearning #deeplearning
Machine Learning Street Talk (MLST) 
