Machine Learning Street Talk (MLST)

061: Interpolation, Extrapolation and Linearisation (Prof. Yann LeCun, Dr. Randall Balestriero)

Jan 4, 2022

Yann LeCun, Meta's Chief AI Scientist and Turing Award winner, joins Randall Balestriero, a researcher at Meta AI, to dive into the complexities of interpolation and extrapolation in neural networks. They discuss how heavily dimensional data challenges traditional views, presenting their groundbreaking paper on high-dimensional extrapolation. Yann critiques the notion of interpolation in deep learning, while Randall emphasizes the geometric principles that can redefine our understanding of neural network behavior. Expect eye-opening insights into AI's evolving landscape!

03:19:43

Creator website

Episode guests

RANDALL BALESTRIERO

Yan LeCun

Highlights

AI Chapters

Episode notes

The Importance of Regularization in Neural Network Training

01:48

Hybrid learning systems with continuous and discrete elements

04:08

The Importance of Problem-Specific Concepts in AI

02:02

Do We Really Want Distinctive Reasoning?

01:28

Intro

2min

Revisiting High-Dimensional Neural Networks: Interpolation vs. Extrapolation

4min

Understanding the Evolving Landscape of Deep Learning

2min

Optimizing Machine Learning with Weights and Biases

3min

Understanding Neural Networks through Geometry

16min

Navigating Interpolation and Extrapolation in Machine Learning

28min

Celebrating a Pioneer in Deep Learning

2min

The Geometry of Learning: Interpolation vs. Extrapolation

6min

Unpacking Deep Learning: Interpolation and Extrapolation

23min

10.

Exploring Data Augmentation and Its Impact on Model Robustness

3min

11.

Navigating Neural Network Complexities

20min

12.

Redefining Interpolation in Deep Learning

13min

13.

Navigating the Complexities of Model Training

15min

14.

Navigating Latent Spaces and Neural Approaches

18min

15.

Dimensionality in Machine Learning: MNIST vs. ImageNet

9min

16.

Rethinking Interpolation in High-Dimensional Spaces

3min

17.

Navigating Generalization in Machine Learning

20min

18.

Understanding Latent Spaces in Neural Networks

9min

19.

Initialization and Evolutionary Algorithms in Neural Networks

3min

We are now sponsored by Weights and Biases! Please visit our sponsor link: http://wandb.me/MLST

Patreon: https://www.patreon.com/mlst

Yann LeCun thinks that it's specious to say neural network models are interpolating because in high dimensions, everything is extrapolation. Recently Dr. Randall Balestriero, Dr. Jerome Pesente and prof. Yann LeCun released their paper learning in high dimensions always amounts to extrapolation. This discussion has completely changed how we think about neural networks and their behaviour.

[00:00:00] Pre-intro

[00:11:58] Intro Part 1: On linearisation in NNs

[00:28:17] Intro Part 2: On interpolation in NNs

[00:47:45] Intro Part 3: On the curse

[00:48:19] LeCun

[01:40:51] Randall B

YouTube version: https://youtu.be/86ib0sfdFtw

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.