The Power of Experimentation

In one of our most anticipated conversations of the year, we got the chance to sit down with three of Spotify’s eng leaders: Krisitan Lindwall, Director of Engineering, Data, Insights, Experimentation, & ML Infrastructure; Mark Grey, Senior Staff Engineer; and Pooja Dave, Director of Engineering, Music Promotion. They share insights from their experience building a culture of experimentation & innovation at a massive scale and what elements are necessary for experimentation at scale. They share the origin story of Spotify’s experimentation platform, how to develop eng leaders to think strategically & execute effectively, mistakes to avoid while scaling your experimentation capabilities, and navigating the balance between structured processes vs. unstructured time to ideate.

ABOUT KRISTIAN LINDWALL

Kristian (@klindwall) is an engineering leader at Spotify. He has over 10 years of experience managing and coaching a broad variety of engineering and product teams. He is currently based in New York where he manages the engineering teams working on the company’s data, insights, and machine learning platforms. Prior to that, he led parts of the agile coach practice at Spotify for a few years and has been very active in supporting the growth of a strong agile and lean approach in the company. Before Spotify, Kristian spent 8 years in fintech in Stockholm where he built and grew the engineering team at the largest online broker company in Sweden.

"How we behave in the organization is really what reinforces and drives the culture and I think there's a few things driving that culture of innovation. Connecting hands to heads, meaning give people an opportunity to engage in ideation and make sure people are involved in strategy work and in the full process of figuring out where we're heading.”

- Kristian Lindwall   

ABOUT POOJA DAVE

With over a decade of experience building and leading several R&D organizations in ad tech, marketing tech, and platforms servicing those, Pooja currently runs the organization at Spotify that helps artists engage and grow their fanbase. Prior to Spotify, she worked at Microsoft on several products including Devices, Browser Rendering Engine, and Advertising/MarTech SDKs. This diversity in experiences has given her a well-rounded exposure to engineer solutions and lead teams with strong backend architecture, client, machine learning, and data practices.

“For Spotify, failure is the paradox to success.”

- Pooja Dave   

ABOUT MARK GREY

Mark is a Senior Staff Engineer at Spotify, where for a decade he has worked on a broad range of distributed systems related to experimentation, data processing, and analytics. Having operated and scaled solutions at all stages of growth, his primary focus is on technical strategy and platformization.

Prior to Spotify, Mark worked at the New York Times on personalization infrastructure such as near-realtime article recommendations.

"We want to ideally maximize the throughput on those things that we see pan out or don't pan out. So kind of a fail faster, double down model and there's all kinds of practices and tools that we put in place that Confidence is just one among many that allow us to increase that throughput. So try and derive insights from a small experiment, validate your hypothesis quickly, and then proceed and scale up from there.”

- Mark Grey   

This episode is brought to you by incident.io

incident.io is trusted by hundreds of tech-led companies across the globe, including Etsy, monday.com, Skyscanner and more to seamlessly orchestrate incident response from start to finish. Intuitively designed, and with powerful and flexible built-in workflow automation, companies use incident.io to supercharge incident response and up-level the entire organization.

Learn more about how you can better identify, learn from, and respond to incidents at incident.io

Interested in joining an ELC Peer Group?

ELCs Peer Groups provide a virtual, curated, and ongoing peer learning opportunity to help you navigate the unknown, uncover solutions and accelerate your learning with a small group of trusted peers.

Apply to join a peer group HERE: sfelc.com/peerGroups

SHOW NOTES:

• About Spotify’s experimentation platform, Confidence (3:46)
• Why Spotify decided to offer Confidence externally (5:43)
• What experimentation without a platform looked like in the early days @ Spotify (6:24)
• Understanding the scale of the Confidence platform (8:58)
• Challenges eng leaders face when scaling testing / experimentation processes (10:51)
• Strategies for determining which experiments & features are most impactful (13:23)
• How to build a stronger culture of innovation / experimentation at scale (15:47)
• Frameworks to help develop eng leaders to be both thinkers & doers (19:11)
• Facilitating conversations around data ideation (23:13)
• An example of how Spotify ideates around data (26:10)
• Mistakes to avoid when scaling up & defining the experiment (28:36)
• How to prioritize experiments when there are conflicts (32:22)
• Recommendations for capturing ideas & turning them into features (35:32)
• Create breathing space within eng teams to help bolster innovation (40:10)
• Why it’s also key to implement structured processes for experimentation (42:57)
• What good coaching looks like when orgs are scaling their experiments (45:24)
• Knowing when you need to platformize something (48:55)
• How generalizing platform capabilities can enable greater speed (51:27)
• Learn to think outside the box & don’t get in the way of experimentation (55:11)
• Rapid fire questions (57:10)

LINKS AND RESOURCES

• Becoming - In her memoir, a work of deep reflection and mesmerizing storytelling, Michelle Obama invites readers into her world, chronicling the experiences that have shaped her—from her childhood on the South Side of Chicago to her years as an executive balancing the demands of motherhood and work, to her time spent at the world’s most famous address.
• Unsolicited Feedback - A new podcast by Reforge that invites you into closed-door conversations between growth and product leaders. Brian Balfour (Reforge, HubSpot), Fareed Mosavat (Reforge, Slack), and friends give targeted feedback around recent features and releases across the product and growth multiverse.
• Lenny’s Podcast - Lenny Rachitsky (author of #1 business newsletter on Substack with 500k+ subscribers) interviews world-class product leaders and growth experts to uncover concrete, actionable, and tactical advice to help you build, launch, and grow your own product.

This episode wouldn’t have been possible without the help of our incredible production team:

Patrick Gallagher - Producer & Co-Host

Jerry Li - Co-Host

Noah Olberding - Associate Producer, Audio & Video Editor https://www.linkedin.com/in/noah-olberding/

Dan Overheim - Audio Engineer, Dan’s also an avid 3D printer - https://www.bnd3d.com/

Ellie Coggins Angus - Copywriter, Check out her other work at https://elliecoggins.com/about/