Using an LLM as Judge for Evaluation

How can you move from manually writing prompts for an LLM application toward defining them programmatically? This week on the show, Brett Kennedy returns to discuss his new book “Building LLM Applications with DSPy.”

With large language models becoming embedded in production software, manual prompt engineering becomes brittle, time-consuming, and difficult to maintain. Brett shares how DSPy replaces manually writing and adjusting prompts with structured prompt programming.

We dig into how DSPy uses declarative signatures to define inputs and outputs. We also discuss how developers programmatically compile, tune, and optimize prompts for specific LLM-based tasks within an application.

Topics:

• 00:00:00 – Introduction
• 00:01:50 – Working on a new book
• 00:02:53 – Writing with a partner
• 00:04:16 – Building LLM Applications with DSPy
• 00:05:22 – What is DSPy?
• 00:06:58 – Working with signatures and typing
• 00:09:53 – Prompt engineering failing to work for a new model
• 00:13:43 – What is an optimizer?
• 00:19:04 – Moving toward a new level of abstraction in prompting
• 00:23:19 – Prompts for the LLM within a consumer facing application
• 00:28:26 – Video Course Spotlight
• 00:30:21 – Example of using with RAG
• 00:38:45 – Example of a business plan tool
• 00:43:06 – The Bitter Lesson
• 00:46:03 – Structure of the book
• 00:49:55 – Advice on starting with DSPy
• 00:54:15 – Example of a transformed prompt
• 00:59:37 – What are you excited about in the world of Python?
• 01:01:47 – What do you want to learn next?
• 01:04:11 – What’s the best way to follow your work online?
• 01:04:40 – Thanks and goodbye

Show Links:

• Building LLM Applications with DSPy - Serj Smorodinsky and Brett Kennedy
• DSPy
• DSPy: The framework for programming—not prompting—language models - GitHub
• Tutorials Overview - DSPy
• GEPA optimization - DSPy
• Let the LLM Write the Prompts: An Intro to DSPy in Compound AI Pipelines - YouTube
• Episode #208: Detecting Outliers in Your Data With Python
• Tabular Models Benchmark: Performance Across 19 Datasets 2026
• Brett Kennedy on Manning
• Serj Smorodinsky on Manning
• W Brett Kennedy – Medium
• Brett Kennedy - LinkedIn

Level up your Python skills with our expert-led courses:

• Building Type-Safe LLM Agents With Pydantic AI
• Accessing Multiple AI Models With the OpenRouter API
• Using LlamaIndex for RAG in Python

Support the podcast & join our community of Pythonistas