The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) A Universal Law of Robustness via Isoperimetry with Sebastien Bubeck - #551
Jan 10, 2022
Sebastian Bubeck, a Senior Principal Research Manager at Microsoft, discusses his award-winning paper on the universal law of robustness via isoperimetry. He explains the significance of convex optimization in machine learning and its applications to multi-armed bandit problems. The conversation delves into the necessity of overparameterization in neural networks for data interpolation and its implications for adversarial robustness. Bubeck also explores isoperimetry’s connection to neural networks and the challenges of scaling training methods.
AI Snips
Chapters
Transcript
Episode notes
Overparameterization Trend
- In 2011, Hinton's neural network used 60 million parameters for a 10-million data point problem.
- This overparameterization, exceeding classical theory's recommendations, foreshadowed the trend of increasingly larger models like GPT-3.
Overparameterization and Smoothness
- Overparameterization is necessary for smooth fitting, where predictions don't change drastically with slight input variations.
- Classical theory suggests fewer parameters for generalization, creating tension between these concepts.
Adversarial Robustness and Model Size
- Adversarial examples arise from non-smoothness, where small input changes drastically alter outputs.
- The law of robustness suggests smaller models are more susceptible to adversarial attacks, potentially explaining current vulnerabilities.