Measuring developer productivity at Airbnb | Christopher Sanson (Airbnb)
Jan 3, 2024
auto_awesome
Christopher Sanson, a product manager at Airbnb dedicated to enhancing developer productivity and tooling, discusses Airbnb's measurement journey, including their implementation of DORA metrics. He covers topics such as their approach before DORA metrics, planning the delivery of new metrics to the organization, and how Airbnb uses DORA metrics to review data and measure specific initiatives.
Metrics, including Dora metrics, play a key role in measuring developer productivity at multiple levels and complement other data sources like surveys, customer feedback, and interviews.
Implementing Dora metrics at Airbnb came with challenges, such as defining how to measure metrics and the complexities of tracking lead time.
Deep dives
Overview of Developer Productivity at Airbnb
The developer productivity team at Airbnb is part of the larger developer platform team, which aims to help developers be productive, ship value, and enjoy their work. Developer productivity fits within the broader organization by understanding what tools to build and the impact on developers. Metrics, including Dora metrics, play a key role in measuring developer productivity at multiple levels and complement other data sources like surveys, customer feedback, and interviews to provide a holistic view of the developer experience at Airbnb.
Using Dora Metrics
Dora metrics are used at Airbnb to measure lead time, deployment frequency, change fail rate, and mean time to restore services. The focus is on quantitative data, but it is complemented by qualitative data like surveys and customer feedback. The metrics provide a snapshot of developer productivity and guide the team's roadmap. The metrics are set as goals within the Commitment to Craft initiative, and teams are held accountable for improving these metrics.
Challenges and Adjustments in Implementing Dora Metrics
Implementing Dora metrics at Airbnb came with challenges, such as defining how to measure metrics and the complexities of tracking lead time. The team adjusted the approach by decoupling PR and release lead time. The metrics are calculated using the existing tech stack, including streaming data from GitHub and Spinnaker to a database, and using an internal metrics dashboard for reporting.
Leveraging Multiple Signals and Granular Metrics
Dora metrics are viewed as part of a larger puzzle, and are used alongside other signals, such as survey data, team-specific metrics, and data on work environment factors. By combining quantitative and qualitative data, teams can have a more comprehensive understanding of developer productivity and focus on actionable improvements. Granular metrics, like build times and code review wait times, are used for specific project measurements and key results (KRs), while Dora metrics provide a high-level perspective on trends and bottlenecks.
Christopher Sanson is a product manager at Airbnb who is dedicated to enhancing developer productivity and tooling. Today, we learn more about Airbnb's developer productivity team and how various teams use metrics, both within and outside the organization. From there, we dive even deeper into their measurement journey, highlighting their implementation of DORA metrics and the challenges they overcame throughout the process.
Discussion points:
(2:43) Who is the developer productivity customer
(4:49) The evolution of developer productivity at Airbnb
(9:26) Approach before DORA metrics
(14:29) Getting buy-in for DORA metrics
(17:49) Planning how to deliver new metrics to the organization
(21:12) How Airbnb calculates deployment frequency
(23:29) Implementing a proof of concept
(27:20) Statistical measurement strategies and tactics