Navigating Production Challenges in LLMs

Kyle Nesbit, a longtime Googler and AI expert, joins us on Deployed to share lessons from his 17+ years at the forefront of distributed systems, machine learning, and AI-driven product innovation. 

Kyle has helped build foundational technologies like BigQuery and worked on early large language model (LLM) development at Google, giving him a unique perspective on how teams can successfully transition from traditional engineering to modern AI-focused workflows.

In this episode, we explore:

• The challenges and opportunities of transitioning traditional engineering teams to LLM development
• Why starting with evaluation metrics is the foundation for success
• Practical strategies for iterative improvement and guardrail design
• How to balance product priorities and quality trade-offs when scaling AI systems
• The real story behind AI demos and how to communicate progress effectively
• Foundational issues in data discovery and access—and why solving them matters more than chasing trends