Dev Interrupted Why enterprise AI lives or dies on applied research | Contextual AI’s Elizabeth Lingg
Sep 16, 2025
In this discussion, Elizabeth Lingg, Director of Applied Research at Contextual AI, shares insights from her esteemed career at Microsoft and Apple. She explores the challenges of turning AI research into reliable products and emphasizes the importance of correlating accuracy with customer satisfaction. Elizabeth highlights the necessity of specialized AI tailored to unique business needs and advocates for a collaborative approach between research and engineering teams. Her expert advice on measuring AI impact through diverse metrics provides a roadmap for effective enterprise AI integration.
AI Snips
Chapters
Books
Transcript
Episode notes
Metrics Can Drive The Wrong Behavior
- Two engineering managers showed up at a conference with every AI tool and a dashboard but felt stuck on measuring real impact.
- Their execs obsessed over PR throughput, creating pressure to game metrics rather than improve developer quality.
Why Models Hallucinate
- OpenAI's paper argues hallucinations stem from model incentives and post-training evaluation pressure.
- Models prefer making confident guesses over saying "I don't know" because evals penalize blank answers.
Link Internal Metrics To Customer Outcomes
- Correlate inner-loop metrics (accuracy, recall) with outer-loop metrics (usage, satisfaction) to measure real impact.
- Use regression and multiple metrics to determine which model features drive customer outcomes.
