Contrarian Guide to AI: Jason Liu on Betting Against Agents while Doubling Down on RAG & Fine-Tuning
High Agency: The Podcast for AI Builders
Importance of Embeddings in Information Retrieval
In information retrieval, the gap between the author and the question asker determines the necessity of using embeddings. For data written and queried by the same person, techniques like BM25 work well due to text similarity. However, when there is a gap between author and searcher, embeddings become crucial for finding matches, like 'jungle' and 'forest'. Testing different methods using baseline metrics and having a language model generate questions is recommended for optimal retrieval results.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
