DataFramed
#289 How I Nearly Got Fired For Running An A/B Test with Vanessa Larco, Former Partner at New Enterprise Associates
Mar 6, 2025
In this engaging discussion, Vanessa Larco, a former partner at NEA with an impressive background in product leadership from giants like Microsoft and Disney, shares her insights on A/B testing. She explores the challenges of balancing short-term gains with long-term user experience, particularly in gaming. Vanessa also highlights the pitfalls of misaligned metrics and the need for clear communication and cross-team collaboration. Her anecdotes underline the importance of understanding user behavior and leveraging data to drive meaningful changes in product development.
55:02
Episode guests
AI Summary
AI Chapters
Episode notes
Podcast summary created with Snipd AI
Quick takeaways
- A-B testing can yield valuable insights but should be applied cautiously, recognizing when it might lead to negative outcomes.
- In gaming, A-B testing optimizes user acquisition by testing various marketing themes, enhancing engagement and lowering acquisition costs.
Deep dives
The Complexity of A-B Testing
A-B testing is a powerful technique that can yield valuable insights, but its effectiveness varies depending on the context. It is crucial to recognize when it is appropriate to use A-B testing versus when it can lead to negative outcomes. For instance, while small changes like button colors may benefit from A-B testing, major shifts in product design or user experience, such as a complete redesign, may not be conducive to this method. Comprehensive knowledge about users and the existing product is essential, as A-B testing conducted without understanding the user base often leads to misguided decisions.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
