#12 - Phillip Carter (Principal PM @ Honeycomb) - All The Hard Stuff when Building Products with LLMs, Actual Results from Leveraging AI

Phillip Carter is a Principal Product Manager at Honeycomb, which develops a software debugging product for distributed systems. Phillip recently published one of the most interesting blog posts I’ve read titled “All the Hard Stuff Nobody Talks About when Building Products with LLMs”. The post is excellent and everyone should give it a read. In this episode, we dive deep into what the hard stuff actually is, the pros and cons of Large Language Models (LLMs) and what teams need to think about when using LLMs in their products. We also talk about the real world results that come from shipping an LLM-enabled product, including the conversion increases that Honeycomb has seen as a result of Query Assistant and how the decision to use LLMs came about. For all the teams wondering how to use LLMs now, this is the episode for you.

Where to Find Phillip:

* LinkedIn: https://www.linkedin.com/in/phillip-carter-4714a135/

* Twitter: https://twitter.com/_cartermp

* Personal Website: https://phillipcarter.dev/

* Blog Post Referred to in Podcast: https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm

Where to Find Shomik:

* Twitter: https://twitter.com/shomikghosh21

* LinkedIn: https://www.linkedin.com/in/shomik-ghosh-a5a71319/

* Software Snack Bites Newsletter: https://www.shomik.substack.com

* Software Snack Bites Podcast: Apple Podcasts, Spotify, Google.

In this episode, we cover:

(00:49) - Phillip’s Background and What Honeycomb Does

(06:37) - How the Idea to Use LLMs for the Query Assistant Product Occurred

(08:45) - Managing Context Windows and Token Counting

(14:35) - Few Shot Prompting & Schema Context

(16:32) - The Conversion Results from Using LLMs to Power a New UI

(21:36) - Are LLMs Mainly Useful for Search & UI/UX

(28:09) - Should Companies Have an “AI Strategy”

(31:55) - Best Efforts Results for LLM Output

(36:10) - Managing Direct Database Access for LLMs

(40:51) - How Embeddings Can Help with Context Windows

(44:31) - Guiding UI/UX Principles for LLM Features

(47:52) - The Magic That LLM Features from a UI Perspective Can Provide

How to Subscribe:

Available on Apple Podcasts, Spotify, or Google.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit shomik.substack.com