Navigating Overparameterization in Neural Networks

Today we’re joined by Sebastian Bubeck a sr principal research manager at Microsoft, and author of the paper A Universal Law of Robustness via Isoperimetry, a NeurIPS 2021 Outstanding Paper Award recipient. We begin our conversation with Sebastian with a bit of a primer on convex optimization, a topic that hasn’t come up much in previous interviews. We explore the problem that convex optimization is trying to solve, the application of convex optimization to multi-armed bandit problems, metrical task systems and solving the K-server problem. We then dig into Sebastian’s paper, which looks to prove that for a broad class of data distributions and model classes, overparameterization is necessary if one wants to interpolate the data. Finally, we discussed the relationship between the paper and the work being done in the adversarial robustness community.

The complete show notes for this episode can be found at twimlai.com/go/551