Measuring specific initiatives in projects can be challenging due to the complexity of various changing factors. To address this, it is essential to use leading indicators such as metrics on flakiness, build times, and code review wait times that teams can directly influence. Lagging indicators like survey data and dorm metrics should not be used for key results (KRs) as they are less concrete and take longer to show results. These leading indicators are crucial as they provide measurable insights that can potentially influence outcomes over time. They serve as valuable complements, offering a high-level overview of the project's direction, identifying bottlenecks, and areas for prioritization. It's important to understand that survey metrics and quant metrics are influenced by numerous factors, and improvements may have a delayed impact on internal customers' perceptions and experiences.
Christopher Sanson is a product manager at Airbnb who is dedicated to enhancing developer productivity and tooling. Today, we learn more about Airbnb's developer productivity team and how various teams use metrics, both within and outside the organization. From there, we dive even deeper into their measurement journey, highlighting their implementation of DORA metrics and the challenges they overcame throughout the process.
Discussion points:
- (2:43) Who is the developer productivity customer
- (4:49) The evolution of developer productivity at Airbnb
- (9:26) Approach before DORA metrics
- (14:29) Getting buy-in for DORA metrics
- (17:49) Planning how to deliver new metrics to the organization
- (21:12) How Airbnb calculates deployment frequency
- (23:29) Implementing a proof of concept
- (27:20) Statistical measurement strategies and tactics
- (31:11) Operationalizing developer productivity metrics
- (34:26) How Airbnb reviews data
- (35:41) How Airbnb uses DORA metrics
Mentions and links: