Latent Space: The AI Engineer Podcast cover image

Latent Space: The AI Engineer Podcast

The End of Finetuning — with Jeremy Howard of Fast.ai

Oct 19, 2023
Jeremy Howard, co-creator of Fast.ai and a leading voice in machine learning, shares his journey from skepticism to success in AI. He discusses the groundbreaking ULMFiT approach to fine-tuning language models and how it faced initial resistance despite its effectiveness. Howard emphasizes the importance of democratizing AI, creating accessible tools, and fostering community engagement. He also explores the evolution of training dynamics in language models and the power of technology to empower diverse communities, advocating for open-source initiatives.
01:09:15

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

  • Language models tend to memorize the training set quickly, leading to catastrophic forgetting and the need for improved fine-tuning techniques.
  • Fast AI aims to democratize AI by making it accessible to a wider audience through courses, software libraries, and open-source projects.

Deep dives

Research in Language Models

The podcast episode delves into the research behind language models, specifically focusing on fine-tuning and pre-training. One of the key insights discussed is the phenomenon of memorization occurring in language models, wherein the models tend to memorize the training set after just one pass, leading to issues like catastrophic forgetting. The podcast also explores the challenges in fine-tuning language models and suggests a shift towards continued pre-training rather than fine-tuning. The discussion highlights the need for more accessible and innovative approaches to language models that do more with less data, less training time, and minimal overfitting.

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner