The Problem With Reductionist Metrics for Productivity

3min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

Using reductionist metrics like number of commits or pull requests to measure productivity can lead to several issues. Firstly, such metrics overlook the human element, incentivizing individuals to game the system rather than focusing on meaningful work. This can distort the process and lead to undesired outcomes. Secondly, simplifying productivity measurement to a few convenient metrics fails to capture essential trade-offs such as velocity and quality. By focusing solely on one aspect, the broader picture and important considerations like bug count, reliability, and robustness are neglected. Additionally, a myopic measurement strategy can result in high attrition rates as employees may feel pressured to deliver quantity over quality, impacting long-term retention. In essence, using reductionist metrics for productivity can lead to skewed results, miss critical trade-offs, and disregard the impact on individuals in the organization.

This week we’re joined by Ciera Jaspan and Collin Green, who lead the Engineering Productivity Research team at Google. Ciera and Collin have written several papers from studies they’ve conducted, and this discussion covers the insights from their research as well as their work more broadly at Google.

Discussion points:

(1:19) About the Engineering Productivity Research team
(3:57) How the team interacts with the rest of the organization
(5:58) The different backgrounds included on the team
(13:11) How Google measures developer productivity
(18:54) Evaluating discrepancies between qualitative and quantitative data
(28:40) Google’s quarterly developer survey
(32:02) Distributing survey results back to the organization
(40:25) Misunderstandings about surveys
(43:51) Ciera and Collin’s paper on why measuring productivity is difficult
(50:35) Reductionist metrics for measuring productivity
(55:26) Examples of other fields that have struggled with measurement
(59:00) Google’s study on measuring technical debt
(1:08:05) Human judgment in measurement

Mentions and links:
Follow Ciera and Collin on LinkedIn
A Human-Centered Approach to Measuring Developer Productivity - Paper, Abi’s summary
Enabling the Study of Software Development with Cross-Tool Logs - Paper
Defining, Measuring, and Managing Tech Debt - Paper, Abi’s summary
Google’s Goals, Signals, Metrics framework - Paper, Abi’s summary