Gradient Descent Theories in Neural Networks

Simon Shaolei Du is an Assistant Professor at the University of Washington. His research focuses on theoretical foundations of deep learning, representation learning, and reinforcement learning. Simon's PhD thesis is titled "Gradient Descent for Non-convex Problems in Modern Machine Learning", which he completed in 2019 at Carnegie Mellon University. We discuss his work related to the theory of gradient descent for challenging non-convex problems that we encounter in deep learning. We cover various topics including connections with the Neural Tangent Kernel, theory vs. practice, and future research directions. Episode notes: https://cs.nyu.edu/~welleck/episode23.html Follow the Thesis Review (@thesisreview) and Sean Welleck (@wellecks) on Twitter, and find out more info about the show at https://cs.nyu.edu/~welleck/podcast.html Support The Thesis Review at www.patreon.com/thesisreview or www.buymeacoffee.com/thesisreview