Vanishing Gradients Episode 52: Why Most LLM Products Break at Retrieval (And How to Fix Them)
7 snips
Jul 2, 2025 Eric Ma, who leads data science research at Moderna, dives into the challenges of aligning retrieval with user intent in LLM-powered systems. He argues that most features fail not at the model level but with context. Eric reveals how a simple YAML-based approach can outperform complex pipelines and discusses the pitfalls of vague user queries. He also emphasizes the importance of evolving retrieval workflows to meet user needs and when it's sufficient to rely on intuition versus formal evaluation in refining these systems.
AI Snips
Chapters
Transcript
Episode notes
Manual Context Retrieval Works
- Eric Ma built an automatic git commit message writer that reads git diff output instead of relying on cosine similarity or vector databases.
- For documentation, he uses simple YAML headers to manually specify intents and linked files for generating content automatically.
Cosine Similarity Falls Short
- Dumping scientific papers into a vector database and querying with cosine similarity often causes hallucinations.
- Vague queries like 'what's this all about' have no relevance to specific text chunks and yield poor retrieval results.
Knowledge Graphs Enhance Retrieval
- Auto-constructed knowledge graphs enhance query understanding and help answer broad queries better.
- Using knowledge graphs with LLMs can improve retrieval for complex queries across multiple papers.