AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
AI Stories
Build LLMs From Scratch with Sebastian Raschka #52
Nov 21, 2024
Sebastian Raschka, a Senior Staff Research Engineer at Lightning AI and bestselling author, dives into the art of building large language models. He shares insights on two significant open-source libraries, PyTorch Lightning and LitGPT, that enhance LLM training and deployment. The discussion shifts to his new book, where he outlines essential steps in LLM training and contrasts models like GPT-2 with the latest Llama 3. Sebastian also explores the universe of multimodal LLMs and their potential, highlighting exciting developments on the horizon.
01:06:03
Episode guests
AI Summary
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- Sebastian Raschka emphasizes the importance of quality data and advanced training techniques in the evolution of contemporary large language models (LLMs).
- The transition from absolute to rotational positional embeddings marks a significant architectural advancement that enhances LLMs' contextual encoding capabilities.
Deep dives
The Evolution of LLMs
The transition from early large language models (LLMs) to contemporary versions has been marked by significant advancements in architecture and training techniques. Modern LLMs have shifted from using absolute positional embeddings to rotational positional embeddings, enhancing their ability to encode contextual information. Furthermore, the introduction of multi-query attention has simplified key and value sharing, which helps in optimizing computational efficiency without compromising performance. These architectural changes, combined with increased model sizes and refined datasets, have contributed to the remarkable improvements in language modeling capabilities.
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode