Machine Learning Street Talk (MLST)

Deep Learning is Not So Mysterious or Different - Prof. Andrew Gordon Wilson (NYU)

28 snips
Sep 19, 2025
Guest
Andrew Gordon Wilson
Professor Andrew Gordon Wilson from NYU highlights the misconceptions in AI, particularly around model complexity and the bias-variance trade-off. He challenges the traditional view that complexity leads to overfitting, arguing that larger models can actually prefer simpler functions. Wilson discusses the importance of inductive biases and how they can improve generalization. He shares insights on practical model construction, advocating for a blend of expressiveness and simplicity to enhance performance across different data scales.
Ask episode
AI Snips
Chapters
Books
Transcript
Episode notes
INSIGHT

Deep Learning Explainsable By Classic Biases

  • Deep learning's odd behaviors can be explained by soft inductive biases and classical generalization frameworks.
  • Many phenomena attributed uniquely to deep nets appear in other model classes once you examine induced function distributions.
INSIGHT

Scale Produces A Simplicity Bias

  • Larger models often develop a stronger bias toward simpler solutions despite being more expressive.
  • This simplicity bias explains phenomena like double descent and benign overfitting.
ADVICE

Favor Flexible Models With Soft Biases

  • Honestly represent your beliefs and prefer flexible models with soft biases rather than hard constraints.
  • Use expressiveness plus Occam-like compression so models adapt automatically to small or large datasets.
Get the Snipd Podcast app to discover more snips from this episode
Get the app