Latent Space: The AI Engineer Podcast cover image

Latent Space: The AI Engineer Podcast

The End of Finetuning — with Jeremy Howard of Fast.ai

Oct 19, 2023
Jeremy Howard, co-creator of Fast.ai and a leading voice in machine learning, shares his journey from skepticism to success in AI. He discusses the groundbreaking ULMFiT approach to fine-tuning language models and how it faced initial resistance despite its effectiveness. Howard emphasizes the importance of democratizing AI, creating accessible tools, and fostering community engagement. He also explores the evolution of training dynamics in language models and the power of technology to empower diverse communities, advocating for open-source initiatives.
01:09:15

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

  • Language models tend to memorize the training set quickly, leading to catastrophic forgetting and the need for improved fine-tuning techniques.
  • Fast AI aims to democratize AI by making it accessible to a wider audience through courses, software libraries, and open-source projects.

Deep dives

Research in Language Models

The podcast episode delves into the research behind language models, specifically focusing on fine-tuning and pre-training. One of the key insights discussed is the phenomenon of memorization occurring in language models, wherein the models tend to memorize the training set after just one pass, leading to issues like catastrophic forgetting. The podcast also explores the challenges in fine-tuning language models and suggests a shift towards continued pre-training rather than fine-tuning. The discussion highlights the need for more accessible and innovative approaches to language models that do more with less data, less training time, and minimal overfitting.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode