Code with Jason 228 - Flaky Tests with Alan Ridlehoover and Fito von Zastrow
Jul 30, 2024
Join engineering leaders Alan Ridlehoover and Fito von Zastrow from Cisco Meraki as they tackle the challenges of flaky tests in extensive Rails applications. Alan discusses the staggering issue of 800 flaky tests disrupting builds, while Fito shares insights on their origins and management. Dive into the complexities of testing and learn why duplication in tests can often lead to clearer outcomes. They reveal effective strategies for maintaining test reliability and eliminating hidden dependencies, ultimately promoting better development practices.
AI Snips
Chapters
Transcript
Episode notes
Scale Of Flakiness At Meraki
- Cisco Meraki ran ~120,000 RSpec tests and found ~800 flaky tests causing constant re-verification.
- Alan described the pain and scale of flakiness in a large Rails monolith with hundreds of developers.
Give Ownership To Fix Flakes
- Assign clear ownership for flaky tests and notify owners automatically so they fix them.
- Alan described adding Slack notifications and team-based ownership to drive fixes and reduce flakiness.
Three Root Categories Of Flakiness
- Flaky tests group into categories: leaked state, environment differences (parallelism), and non-determinism like randomness or network.
- Fito emphasized that many flakes arise from shared state, CI-versus-local differences, and external dependencies.