Engineering Enablement by DX

This week we’re joined by Ciera Jaspan and Collin Green, who lead the Engineering Productivity Research team at Google. Ciera and Collin have written several papers from studies they’ve conducted, and this discussion covers the insights from their research as well as their work more broadly at Google. Discussion points:(1:19) About the Engineering Productivity Research team(3:57) How the team interacts with the rest of the organization(5:58) The different backgrounds included on the team(13:11) How Google measures developer productivity(18:54) Evaluating discrepancies between qualitative and quantitative data (28:40) Google’s quarterly developer survey(32:02) Distributing survey results back to the organization (40:25) Misunderstandings about surveys(43:51) Ciera and Collin’s paper on why measuring productivity is difficult(50:35) Reductionist metrics for measuring productivity(55:26) Examples of other fields that have struggled with measurement(59:00) Google’s study on measuring technical debt(1:08:05) Human judgment in measurementMentions and links: Follow Ciera and Collin on LinkedInA Human-Centered Approach to Measuring Developer Productivity - Paper, Abi’s summaryEnabling the Study of Software Development with Cross-Tool Logs - PaperDefining, Measuring, and Managing Tech Debt - Paper, Abi’s summaryGoogle’s Goals, Signals, Metrics framework - Paper, Abi’s summary

This week we’re joined by Jason Kennedy, Senior Engineering Manager of Developer Experience at One Medical. Jason’s team takes a uniquely customer-driven approach to improving the developer experience, and in this episode he describes their philosophy and how it works in practice. Jason explains how they shadow developers, how they run surveys, and more. Discussion points:(1:02) Renaming from Engineering Efficiency to Engineering Experience(4:17) How Platform and DevEx teams differ (5:38) How One Medical’s approach to customer experience inspires this team’s work(7:01) Mapping out the developer journey(11:14) Jason’s career transition from VPE to a line manager role(14:14) Challenges some companies face with getting buy-in for a DevEx team(16:22) Taking a customer service approach to DevEx(19:12) Jason’s experience with DORA metrics(22:19) Lessons learned about ownership(24:18) The “Gemba” practice used at One Medical (28:02) How information from the Gemba practice is stored(30:59) Using weekly polls to surface pain points(34:03) Tracking trends in the poll(35:00) Using a quarterly NPS survey for overall sentiment(37:08) How sentiment is measured and evaluated(41:44) The biggest challenges with surveys Mentions and links:Follow Jason on LinkedInListen to the podcast episode with Jasmine JamesBook about Disney: Be Our Guest

Matthew and Luke lead Extend’s Developer Experience team, a team that has approached their work in a way that is more forward-thinking than most. In this episode, they cover how they deliver impact at multiple levels of the organization, their journey with productivity metrics, and how they’ve made DevEx a C-level concern. Discussion points:(1:40) How the DevEx team started and where it fits at Extend(5:08) Tradeoffs of DevEx reporting into Platform (6:40) The mandate and tasks they focus on(12:07) The impact of learning and development efforts(16:33) How to drive team-level improvements (18:44) Why developer experience is becoming more prevalent(26:17) How they made DevEx a C-level concern(30:27) Their journey with productivity metrics (33:10) Advice for presenting DevEx data to executives (34:52) The team’s experience using git metrics tools(48:30) Being rigorous in leveraging metrics Mentions and links: Connect with Matthew and Luke on LinkedInOther podcasts mentioned: Manuel Pais; Peloton’s DevEx survey

Manuel Pais delves into one of the concepts covered in his book “Team Topologies”: platform and enabling work. Manuel shares how he views the strategy behind when and how to invest in platform or enabling work. This conversation also goes into each type of work in more detail, covering topics such as measuring cognitive load and where platform engineering may be heading in the future. (2:13) How enabling teams and platform teams are different (10:28) What it looks like for a team to own both platform and enabling work (17:04) How to deliver enabling work in an organization(22:28) Whether enabling teams should be temporary(30:10) Platform team anti-patterns(47:10) Measuring cognitive load

Thansha Sadacharam, who leads Tech Learning and Insights at Peloton walks us through the journey of building the company’s developer experience survey. She shares what went into the survey’s design, rollout, and maintenance, as well as the different teams involved.Discussion points: (1:19) Where the idea for running a developer survey originated(6:36) Advice for other leaders getting buy-in for these initiatives(11:27) The first steps in designing the survey(18:21) How the survey incorporated benchmarking(20:30) Measuring developer satisfaction(22:37) Refining the question items (25:50) How long the survey was(26:50) What was involved in trimming the questions (29:28) Writing survey questions (33:12) How much time was spent developing the survey(35:19) The communication plan for launching the survey(42:05) Driving participation rates  (45:21) Sampling and how often surveys are being sent (49:21) How the information was presented (54:10) Feeling nervous about sending out surveys Mentions and linksFollow Thansha on LinkedIn

In this episode, Abi is interviewed by Laura Tacho about the new paper he co-authored with Dr. Nicole Forsgren, Dr. Margaret-Anne Storey, and Dr. Michaela Greiler. Abi and Laura discuss the pitfalls of some of the common metrics organizations use, and how the new paper builds on prior frameworks such as DORA and SPACE to offer a new approach to measuring and improving developer productivity. Discussion topics:(2:20) Laura’s background(3:59) Laura’s view on git metrics(11:05) What developer experience (DevEx) is (14:37) How the authors came together for this paper (18:55) How DORA and SPACE are different(22:38) Limitations of DORA metrics (24:43) Employing the DORA metrics at GitHub(27:47) What the SPACE framework is(30:44) Whether to use DORA or SPACE or both(33:54) Limitations of the SPACE framework(37:29) The need for a new approach (38:46) What the new DevEx paper solves (40:13) The three dimensions of developer experience (40:54) Flow state (43:10) Feedback loops(43:52) Cognitive load (44:51) Why developer sentiment matters(47:58) Using both perceptual and workflow measures(50:59) Examples of perceptual and workflow measures (54:05) How to collect metrics (59:47) How other companies are measuring and improving developer experience(01:02:56) Advice for earlier-stage or growing organizationsResources for learning more about the DevEx framework:Read the new paper on ACM QueueRead Abi’s announcement about the new paper Read how top companies measure developer productivity Connect with Abi and Laura Sign up for Laura’s course, Measuring Development Team PerformanceConnect with Laura on LinkedIn or TwitterConnect with Abi on LinkedIn or Twitter

Tara Hernandez, the VP of Developer Productivity at MongoDB, joins the podcast to give an inside look at what the developer experience looks like at an organization that develops a database. Here, Tara shares what it looks like to develop, test, and release changes at MongoDB, while also providing insight into how the company invests in developer productivity more broadly. Discussion points: (0:57) What was going on at the time Tara joined (4:37) Tara’s perspective on the buzz of platform engineering(7:38) What’s involved in building and testing a database(10:11) The development environment at MongoDB(13:14) How testing works(16:50) What the release process looks like(19:27) What goes into performance testing a release(21:31) MongoDB’s investment in engineering enablement (22:39) Takeaways from working on databases(24:24) Affecting cultural change(26:40) Opportunities Tara’s team identified to change culture(29:12) Managing technical debt(33:06) MongoDB’s culture around developer experience (34:59) Why Evergreen CI is open sourceMentions and links: Follow Tara on LinkedIn or TwitterRead more about MongoDB’s “Evergreen” Continuous Integration Visit MongoDB’s engineering blog 

Max Kanat-Alexander, the Tech Lead for the Developer Productivity and Insights Team at LinkedIn, shares an inside look at LinkedIn’s metrics platform and how teams across the organization use it. Discussion points: (1:31) Why Max shares how his team is measuring productivity(3:20) Why some teams use metrics and some don’t (6:03) The types of metrics Max’s team focuses on(12:59) The role of TPMs(17:05) How Max would measure productivity if he weren’t at LinkedIn(25:04) Surprises in how teams are using metrics at LinkedIn(31:27) The tooling required to enable metrics for teams to use(36:41) Qualitative versus quantitative metrics(40:39) Measuring code quality at Google (46:16) Whether a centralized team should own measurementMentions and links:Connect with Max on LinkedIn or TwitterRead the article, Measuring Developer Productivity and Happiness at LinkedInListen to the first interview with Max and his colleague Or Michael Berlowitz: Episode 23Abi’s blog post on the Three-Bucket Framework for Engineering Metrics

Mike Fisher, the former CTO at Etsy, spearheaded a multi-year developer experience initiative aimed at improving developer happiness and efficiency during his time at Etsy. Here, he shares the story of that initiative, including the pillars of the program and the investment that went into it. Towards the end of the conversation, Mike also shares his perspective on measuring developer productivity. Discussion points:(1:31) What was happening at Etsy when Mike joined (4:08) The scaling challenges Etsy faced(6:08) Deciding on the term “developer experience” (9:35) Whether developer experience is a new approach(11:24) The pillars of Etsy’s DevEx initiative (15:49) Converting the length of time required for this initiative(18:11) The investment allocated to the initiative (20:04) Talking about the ROI of devex initiatives (22:50) Who was actually leading this work(24:37) Etsy’s experience with platform teams (30:42) Advice for leaders championing DevEx initiatives(34:45) Framing the conversation about getting budget for a DevEx initiative(37:45) How leaders can address the efficiency conversation(42:00) Measuring productivity (45:49) The “experiment velocity” metric ‍Mentions and links:Follow Mike on LinkedIn or TwitterSubscribe to Mike’s newsletter, Fish Food for Thought

Karl’s team at American Airlines were early adopters of Backstage, and in this episode he shares their journey of implementing and rolling out a developer portal. He also describes two of the extensions his team has built for their portal. Discussion points:(1:24) Where the idea of building a developer portal came from(7:24) What the developer experience looked like before the portal (10:41) Initiating the project(14:16) The decision to choose Backstage (16:28) The V1 scope for the portal (19:14) Getting adoption for the portal(23:35) Defining success for the portal’s adoption (28:04) The ideal state for how developers will use the portal(30:56) Who should or shouldn’t invest in building a developer portal (33:14) Custom extensions Karl’s team has developed for their portal(37:46) What’s difficult about developing a new plugin for the backstage platform‍Mentions and links:Follow Karl on LinkedInThe Runway platform at American Airlines Read more on the engineering blog from American Airlines