The Data Exchange with Ben Lorica Beyond the Chatbot: What Actually Works in Enterprise AI
Sep 11, 2025
Jay Alammar, Director and Engineering Fellow at Cohere and co-author of "Hands-on Large Language Models," delves into enterprise AI. He discusses the challenges in understanding large language models and the adoption of GraphRag, emphasizing the gap between vendor enthusiasm and real-world application. Alammar highlights the balance between self-directed and collaborative learning in AI, and the critical role of evaluation processes for effective AI development. He also explores the potential of smaller AI models, showcasing their efficiency in addressing specific tasks.
AI Snips
Chapters
Books
Transcript
Episode notes
Start Small Before Chat Deployments
- Do not start enterprise LLM projects with chat as the first deployment; begin with simpler, predictable tasks like summarization or extraction.
- Use those building blocks to compose more complex applications instead of launching a chat UI first.
Models Are Probabilistic — Ground Them
- Models are probabilistic and will confidently produce incorrect outputs, so you cannot rely on a model's knowledge alone.
- Grounding via retrieval augmented generation (RAG) reduces hallucination by providing context from authoritative sources.
Make RAG Knowledge-Seeking And Multi-Query
- Improve RAG by adding query rewriting, multi-query search, and multi-step retrieval rather than a single raw document pull.
- Let models decide when they have gathered sufficient information and use tools (search, SQL) to collect multiple targeted pieces of evidence.
