Evaluating AI Integration in Business

Supported by Our Partners

• Swarmia — The engineering intelligence platform for modern software organizations.

• Graphite — The AI developer productivity platform. 

• Vanta — Automate compliance and simplify security with Vanta.

—

On today’s episode of The Pragmatic Engineer, I’m joined by Chip Huyen, a computer scientist, author of the freshly published O’Reilly book AI Engineering, and an expert in applied machine learning. Chip has worked as a researcher at Netflix, was a core developer at NVIDIA (building NeMo, NVIDIA’s GenAI framework), and co-founded Claypot AI. She also taught Machine Learning at Stanford University.

In this conversation, we dive into the evolving field of AI Engineering and explore key insights from Chip’s book, including:

• How AI Engineering differs from Machine Learning Engineering 

• Why fine-tuning is usually not a tactic you’ll want (or need) to use

• The spectrum of solutions to customer support problems – some not even involving AI!

• The challenges of LLM evals (evaluations)

• Why project-based learning is valuable—but even better when paired with structured learning

• Exciting potential use cases for AI in education and entertainment

• And more!

—

Timestamps

(00:00) Intro 

(01:31) A quick overview of AI Engineering

(05:00) How Chip ensured her book stays current amidst the rapid advancements in AI

(09:50) A definition of AI Engineering and how it differs from Machine Learning Engineering 

(16:30) Simple first steps in building AI applications

(22:53) An explanation of BM25 (retrieval system) 

(23:43) The problems associated with fine-tuning 

(27:55) Simple customer support solutions for rolling out AI thoughtfully 

(33:44) Chip’s thoughts on staying focused on the problem 

(35:19) The challenge in evaluating AI systems

(38:18) Use cases in evaluating AI 

(41:24) The importance of prioritizing users’ needs and experience 

(46:24) Common mistakes made with Gen AI

(52:12) A case for systematic problem solving 

(53:13) Project-based learning vs. structured learning

(58:32) Why AI is not the end of engineering

(1:03:11) How AI is helping education and the future use cases we might see

(1:07:13) Rapid fire round

—

The Pragmatic Engineer deepdives relevant for this episode:

• Applied AI Software Engineering: RAG https://newsletter.pragmaticengineer.com/p/rag 

• How do AI software engineering agents work? https://newsletter.pragmaticengineer.com/p/ai-coding-agents 

• AI Tooling for Software Engineers in 2024: Reality Check https://newsletter.pragmaticengineer.com/p/ai-tooling-2024 

• IDEs with GenAI features that Software Engineers love https://newsletter.pragmaticengineer.com/p/ide-that-software-engineers-love

—

See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠

—

Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.