The Analytics Power Hour #290: Always Be Learning
12 snips
Feb 3, 2026 Mårten Schultzberg, product manager and staff data scientist at Spotify known for experimentation and co-author of Spotify's EWL framework, explores how organizations learn from experiments. He discusses valuing learning beyond wins, defining wins/regressions/neutrals, powering and monitoring tests, preventing metric fishing, and when neutral results still justify shipping. Practical lessons on scaling experimentation culture and tooling are highlighted.
AI Snips
Chapters
Transcript
Episode notes
Experiments Prevent Damage As Much As Drive Gains
- Experiments are as valuable for detecting regressions and avoiding bad launches as they are for finding wins.
- Mårten says spotting negatives early is one of the biggest recurring wins of experimentation.
Neutral Results Are Real Learning
- Spotify counts three learning outcomes: clear winners, detected regressions, and well-powered neutral results.
- A neutral result from a properly powered test is considered meaningful learning about user impact.
Monitor Outcome Distribution To Guide Strategy
- Track the distribution of experiment outcomes and compare win, neutral, and regression rates regularly.
- Use that distribution to diagnose strategy: too many neutrals may signal diminishing returns or a need to rethink approach.