Measuring AI's Impact on Software Engineering

Supported by Our Partners

•⁠ Statsig ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more.

• Graphite — The AI developer productivity platform.

—

There’s no shortage of bold claims about AI and developer productivity, but how do you separate signal from noise?

In this episode of The Pragmatic Engineer, I’m joined by Laura Tacho, CTO at DX, to cut through the hype and share how well (or not) AI tools are actually working inside engineering orgs. Laura shares insights from DX’s research across 180+ companies, including surprising findings about where developers save the most time, why devs don’t use AI at all, and what kinds of rollouts lead to meaningful impact.

We also discuss: 

• The problem with oversimplified AI headlines and how to think more critically about them

• An overview of the DX AI Measurement framework

• Learnings from Booking.com’s AI tool rollout

• Common reasons developers aren’t using AI tools

• Why using AI tools sometimes decreases developer satisfaction

• Surprising results from DX’s 180+ company study

• How AI-generated documentation differs from human-written docs

• Why measuring developer experience before rolling out AI is essential

• Why Laura thinks roadmaps are on their way out

• And much more!

—

Timestamps

(00:00) Intro

(01:23) Laura’s take on AI overhyped headlines 

(10:46) Common questions Laura gets about AI implementation 

(11:49) How to measure AI’s impact 

(15:12) Why acceptance rate and lines of code are not sufficient measures of productivity

(18:03) The Booking.com case study

(20:37) Why some employees are not using AI 

(24:20) What developers are actually saving time on 

(29:14) What happens with the time savings

(31:10) The surprising results from the DORA report on AI in engineering 

(33:44) A hypothesis around AI and flow state and the importance of talking to developers

(35:59) What’s working in AI architecture 

(42:22) Learnings from WorkHuman’s adoption of Copilot 

(47:00) Consumption-based pricing, and the difficulty of allocating resources to AI 

(52:01) What DX Core 4 measures 

(55:32) The best outcomes of implementing AI 

(58:56) Why highly regulated industries are having the best results with AI rollout

(1:00:30) Indeed’s structured AI rollout 

(1:04:22) Why migrations might be a good use case for AI (and a tip for doing it!) 

(1:07:30) Advice for engineering leads looking to get better at AI tooling and implementation 

(1:08:49) Rapid fire round

—

The Pragmatic Engineer deepdives relevant for this episode:

• AI Engineering in the real world

• Measuring software engineering productivity

• The AI Engineering stack

• A new way to measure developer productivity – from the creators of DORA and SPACE

—

See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠

—

Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.