Google emphasizes a human-centered approach to measuring developer productivity.
Survey data should be corroborated with objective metrics for credibility.
Avoid reductionist metrics like lines of code to assess productivity effectively.
Consider human factors and structural solutions for comprehensive productivity analysis.
Deep dives
Understanding Developer Productivity at Google
The podcast episode features Colin Green and Sierra Jasmine from Google, who lead the engineering productivity research team. The team focuses on improving developers' productivity by understanding their experiences and needs. They employ a mixed methods research approach, combining software engineers and UX researchers from diverse backgrounds to gain a comprehensive understanding. Through various research methods, such as surveys, interviews, and logs analysis, they aim to triangulate on developer productivity, emphasizing the importance of considering human factors alongside technical metrics.
Measuring Developer Productivity
The podcast delves into Google's approach to measuring developer productivity, emphasizing the interplay between speed, ease, and quality. By employing multiple metrics and validating them against each other, Google ensures a holistic view of productivity. They highlight the significance of correlating survey-based measures with objective data sources to bolster credibility and understanding. The team encourages stakeholders to focus on specific goals, signals, and metrics tailored to their objectives, fostering a nuanced and context-specific measurement strategy.
Challenges and Insights on Survey-Based Research
Discussions revolve around the challenges and insights gained from survey-based research and the human-centered approach to understanding productivity. The team stresses the complexity of measuring developer productivity and cautions against oversimplification. They highlight the value of transparent data sharing, stakeholder engagement, and corroboration between survey data and objective metrics to enhance credibility and utility. By integrating human factors with structural solutions, they address human-centric issues within the productivity framework.
Importance of Addressing Human Factors
The episode underlines the importance of considering human factors in productivity analysis, showcasing how structural solutions can mitigate human-related challenges. By recognizing the intertwined nature of technical and human issues within the development environment, Google advocates for a comprehensive understanding. They emphasize empathy towards developers, privacy considerations, and the need to avoid reductionist approaches that overlook the nuanced interplay between structural elements and human experiences.
Fostering Stakeholder Buy-in and Validating Survey Data
Google's approach to developer productivity research involves fostering stakeholder buy-in, educating leaders on the validity of survey data, and championing a human-centered approach to measurement. By emphasizing the need for systematic and scientific productivity assessment while avoiding reductionist practices, the team navigates the challenges of integrating human-centric research methods with technical metrics. They advocate for a holistic understanding of productivity that encompasses human nuances and structural dynamics for effective organizational improvements.
Importance of Considering Human Factors in Research
It is crucial for researchers to remember that the data they study represents real people and organizations, not just numbers. Using convenient metrics like lines of code or pull requests to measure productivity can lead to unintended consequences. Human behavior can be influenced by how they are incentivized, such as encouraging more lines of code for productivity but possibly sacrificing quality and other important aspects of work.
Challenges with Reductionist Metrics in Engineering Productivity
Relying solely on reductionist metrics like lines of code to measure engineering productivity can overlook critical trade-offs and nuances in the development process. This approach may incentivize behaviors that prioritize quantity over quality, potentially leading to long-term issues such as high attrition rates. Understanding the complexity of human judgment and considering multiple dimensions beyond simplistic metrics is essential for effective management of technical debt and productivity in engineering teams.
This week we’re joined by Ciera Jaspan and Collin Green, who lead the Engineering Productivity Research team at Google. Ciera and Collin have written several papers from studies they’ve conducted, and this discussion covers the insights from their research as well as their work more broadly at Google.
Discussion points:
(1:19) About the Engineering Productivity Research team
(3:57) How the team interacts with the rest of the organization
(5:58) The different backgrounds included on the team
(13:11) How Google measures developer productivity
(18:54) Evaluating discrepancies between qualitative and quantitative data
(28:40) Google’s quarterly developer survey
(32:02) Distributing survey results back to the organization
(40:25) Misunderstandings about surveys
(43:51) Ciera and Collin’s paper on why measuring productivity is difficult
(50:35) Reductionist metrics for measuring productivity
(55:26) Examples of other fields that have struggled with measurement
(59:00) Google’s study on measuring technical debt
(1:08:05) Human judgment in measurement
Mentions and links: Follow Ciera and Collin on LinkedIn A Human-Centered Approach to Measuring Developer Productivity - Paper, Abi’s summary Enabling the Study of Software Development with Cross-Tool Logs - Paper Defining, Measuring, and Managing Tech Debt - Paper, Abi’s summary Google’s Goals, Signals, Metrics framework - Paper, Abi’s summary
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
