The Analytics Engineering Podcast Bring Your Own Data to LLMs (W/ Jerry Liu of LlamaIndex)
Aug 25, 2023
Jerry Liu, CEO and co-founder of LlamaIndex, discusses how companies are bringing their data to tailor large language models (LLMs) for their needs. Topics include working on LLMs versus autonomous systems, skill set and data preparation for LOMs, using databases for storing embeddings, capabilities of LMs in analyzing user questions, and exploring agents and specialized microservices in analytics engineering.
AI Snips
Chapters
Transcript
Episode notes
Context Window as Neural CPU Cache
- The context window of an LLM acts like a limited cache, restricting how much text it can process at once.
- To handle large data, the system retrieves relevant info into this cache for the LLM to reason over effectively.
Sales Bot Use Case Challenge
- Jerry built a sales bot that used GPT-3 to synthesize customer conversations and summarize action items.
- He faced challenges feeding all data into the 4,000 token context window, prompting strategies to retrieve relevant information.
Two-Step LLM Application Strategy
- Build LLM applications via a two-step approach: first retrieve relevant context from vector databases, then synthesize it in prompts.
- This simple retrieval augmented generation model is effective and widely used today.