Spotify: A Product Story Explained: The conspiracy to make AI seem harder than it is! by Gustav Söderström
Jul 31, 2023
Gustav Söderström, Co-president at Spotify, breaks down the complexities of generative AI in this insightful talk. He explains how Large Language Models like ChatGPT function through next-word prediction and the transformative power of transformer architectures. Discover how creativity in AI is influenced by temperature sampling, and learn about the fascinating role of embeddings in recommendations. Gustav also articulates the mechanics behind neural networks and diffusion models, showcasing how they generate stunning images and music from simple prompts.
AI Snips
Chapters
Transcript
Episode notes
LLMs Are Just Next-Token Number Models
- Large Language Models (LLMs) predict the next token in a sequence by learning statistics from massive text corpora.
- They are fundamentally number-sequence models, not mysterious language-only systems.
Use Transformers For Large Context
- Use the transformer architecture to handle long context efficiently by weighting attention across tokens.
- Prefer transformers for sequence tasks that need thousands of tokens of context.
Temperature Controls Creativity
- Deterministic generation picks the single most likely next token; randomness comes from sampling around top choices.
- Raising temperature increases novelty but also the chance of incoherence.