Measuring AI code assistants and agents with the AI Measurement Framework

61 snips

Aug 15, 2025

This discussion dives into the challenge of measuring developer productivity with the rise of AI. Experts reveal how AI’s anticipated benefits often don't match reality, emphasizing the need for reliable data. They introduce an AI Measurement Framework to assess productivity while addressing pitfalls of traditional metrics. Insights into code maintainability highlight the importance of quality over mere acceptance rates. The conversation also stresses viewing AI as a partner to developers, ensuring it enhances rather than complicates workflows.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Measure AI Against Existing Baselines

AI changes some measurements but fundamentals of software performance remain constant.
Use baseline pre-AI metrics to compare post-AI impact over time.

ADVICE

Use Three Dimensions: Utilization, Impact, Cost

Track utilization, impact, and cost together to form a complete picture of AI adoption.
Avoid over-indexing on a single metric and interpret metrics as a basket of signals.

ADVICE

Watch Downstream Quality Signals

Measure downstream quality signals like maintainability and change failure rate to detect long-term harm.
Combine these with developer satisfaction to spot trade-offs between speed and sustainability.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

In this episode of Engineering Enablement, DX CTO Laura Tacho and CEO Abi Noda break down how to measure developer productivity in the age of AI using DX’s AI Measurement Framework. Drawing on research with industry leaders, vendors, and hundreds of organizations, they explain how to move beyond vendor hype and headlines to make data-driven decisions about AI adoption.

They cover why some fundamentals of productivity measurement remain constant, the pitfalls of over-relying on flawed metrics like acceptance rate, and how to track AI’s real impact across utilization, quality, and cost. The conversation also explores measuring agentic workflows, expanding the definition of “developer” to include new AI-enabled contributors, and avoiding second-order effects like technical debt and slowed PR throughput.

Whether you’re rolling out AI coding tools, experimenting with autonomous agents, or just trying to separate signal from noise, this episode offers a practical roadmap for understanding AI’s role in your organization—and ensuring it delivers sustainable, long-term gains.

Where to find Laura Tacho:

• X: https://x.com/rhein_wein

• LinkedIn: https://www.linkedin.com/in/lauratacho/

• Website: https://lauratacho.com/

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

• Substack: https://substack.com/@abinoda

In this episode, we cover:

(00:00) Intro

(01:26) The challenge of measuring developer productivity in the AI age

(04:17) Measuring productivity in the AI era — what stays the same and what changes

(07:25) How to use DX’s AI Measurement Framework

(13:10) Measuring AI’s true impact from adoption rates to long-term quality and maintainability

(16:31) Why acceptance rate is flawed — and DX’s approach to tracking AI-authored code

(18:25) Three ways to gather measurement data

(21:55) How Google measures time savings and why self-reported data is misleading

(24:25) How to measure agentic workflows and a case for expanding the definition of developer

(28:50) A case for not overemphasizing AI’s role

(30:31) Measuring second-order effects

(32:26) Audience Q&A: applying metrics in practice

(36:45) Wrap up: best practices for rollout and communication

Referenced: