Phillip Carter, a Principal PM at Honeycomb with a robust background in .NET at Microsoft, shares his insights on building AI infrastructure. He discusses the evolution of cloud development and the critical transitions from corporate roles to dynamic startups. The conversation covers advancements in natural language querying and the importance of OpenTelemetry for unified observability. Carter also highlights how AI is transforming data instrumentation and enhancing developer workflows, making it an exciting time for innovation in tech.

Phillip Carter is a Principal Product Manager at Honeycomb, which develops a software debugging product for distributed systems. Phillip recently published one of the most interesting blog posts I’ve read titled “All the Hard Stuff Nobody Talks About when Building Products with LLMs”. The post is excellent and everyone should give it a read. In this episode, we dive deep into what the hard stuff actually is, the pros and cons of Large Language Models (LLMs) and what teams need to think about when using LLMs in their products. We also talk about the real world results that come from shipping an LLM-enabled product, including the conversion increases that Honeycomb has seen as a result of Query Assistant and how the decision to use LLMs came about. For all the teams wondering how to use LLMs now, this is the episode for you.Where to Find Phillip:* LinkedIn: https://www.linkedin.com/in/phillip-carter-4714a135/* Twitter: https://twitter.com/_cartermp * Personal Website: https://phillipcarter.dev/* Blog Post Referred to in Podcast: https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llmWhere to Find Shomik:* Twitter: https://twitter.com/shomikghosh21* LinkedIn: https://www.linkedin.com/in/shomik-ghosh-a5a71319/* Software Snack Bites Newsletter: https://www.shomik.substack.com* Software Snack Bites Podcast: Apple Podcasts, Spotify, Google.In this episode, we cover:(00:49) - Phillip’s Background and What Honeycomb Does(06:37) - How the Idea to Use LLMs for the Query Assistant Product Occurred(08:45) - Managing Context Windows and Token Counting(14:35) - Few Shot Prompting & Schema Context(16:32) - The Conversion Results from Using LLMs to Power a New UI(21:36) - Are LLMs Mainly Useful for Search & UI/UX(28:09) - Should Companies Have an “AI Strategy”(31:55) - Best Efforts Results for LLM Output(36:10) - Managing Direct Database Access for LLMs(40:51) - How Embeddings Can Help with Context Windows(44:31) - Guiding UI/UX Principles for LLM Features(47:52) - The Magic That LLM Features from a UI Perspective Can ProvideHow to Subscribe:Available on Apple Podcasts, Spotify, or Google. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit shomik.substack.com

Phillip Carter, Principal Product Manager at Honeycomb, discusses observability for large language models. They delve into how observability helps in testing, refining functionality, debugging, and enabling incremental development for LLMs. Carter offers tips on implementing observability and highlights current technology limitations.