#289 How I Nearly Got Fired For Running An A/B Test with Vanessa Larco, Former Partner at New Enterprise Associates
Mar 6, 2025
auto_awesome
In this engaging discussion, Vanessa Larco, a former partner at NEA with an impressive background in product leadership from giants like Microsoft and Disney, shares her insights on A/B testing. She explores the challenges of balancing short-term gains with long-term user experience, particularly in gaming. Vanessa also highlights the pitfalls of misaligned metrics and the need for clear communication and cross-team collaboration. Her anecdotes underline the importance of understanding user behavior and leveraging data to drive meaningful changes in product development.
A-B testing can yield valuable insights but should be applied cautiously, recognizing when it might lead to negative outcomes.
In gaming, A-B testing optimizes user acquisition by testing various marketing themes, enhancing engagement and lowering acquisition costs.
Successful A-B testing relies on stakeholder buy-in and collaborative input from cross-functional teams to ensure meaningful insights.
Deep dives
The Complexity of A-B Testing
A-B testing is a powerful technique that can yield valuable insights, but its effectiveness varies depending on the context. It is crucial to recognize when it is appropriate to use A-B testing versus when it can lead to negative outcomes. For instance, while small changes like button colors may benefit from A-B testing, major shifts in product design or user experience, such as a complete redesign, may not be conducive to this method. Comprehensive knowledge about users and the existing product is essential, as A-B testing conducted without understanding the user base often leads to misguided decisions.
Insights from Gaming: User Acquisition Strategies
In gaming, A-B testing can be instrumental in optimizing user acquisition and maximizing revenue. By testing different marketing themes and advertising strategies, teams can identify which concepts resonate most with target demographics, potentially leading to lower customer acquisition costs. For example, one game was launched using various themes like 'Gnome Town' and 'Candyland,' where unexpected popularity for the gnome-themed concept proved highly cost-effective. This trial-and-error approach allows product teams to align their marketing efforts with actual user preferences, resulting in a stronger return on investment.
Empathy and Understanding Users Through A-B Testing
A-B testing not only facilitates product optimization but also fosters a deeper understanding of user preferences and behaviors. By incorporating activities such as betting on outcomes among team members, product managers encourage engagement with user sentiment and expectations. This playful method reinforces the team’s focus on what resonates with users, such as distinct character themes or the effectiveness of specific marketing strategies. Ultimately, testing reveals what users truly appreciate, providing critical insights that guide product development and future iterations.
Navigating Long-Term vs. Short-Term Focus
Although A-B testing excels at examining short-term user interactions, it poses challenges when applied to long-term strategy and product development. Decisions based solely on immediate data may not account for broader user experience or sustainability over time. Product managers are encouraged to think beyond immediate results by considering the implications of changes for long-term user engagement and satisfaction. Balancing short-term testing outcomes with long-term strategic goals often requires additional research and thoughtful consideration of user feedback.
Implementing Effective A-B Testing Protocols
Ensuring the success of A-B testing programs relies heavily on obtaining buy-in from stakeholders and establishing clear protocols. Engaging cross-functional teams—from customer support to engineering—can help identify potential concerns and optimize the testing process. Additionally, understanding the context within which tests are conducted, including not inadvertently affecting critical user accounts, is vital for ensuring a positive experience for end users. A collaborative approach, where all relevant team members contribute to defining essential metrics and objectives, enhances the potential for impactful insights from A-B testing.
The rise of A-B testing has transformed decision-making in tech, yet its application isn't without challenges. As professionals, how do you navigate the balance between short-term gains and long-term sustainability? What strategies can you employ to ensure your testing methods enhance rather than hinder user experience? And how do you effectively communicate the insights gained from testing to drive meaningful change within your organization?
Vanessa Larco is a former partner at NEA where she led Series A and Series B investment rounds and worked with major consumer companies like DTC jewelry giant Mejuri, menopause symptom relief treatment Evernow, and home-swapping platform Kindred as well as major enterprise SaaS companies like Assembled, Orby AI, Granica AI, EvidentID, Rocket.Chat, Forethought AI. She is also a board observer at Forethought, SafeBase, Orby AI, Granica, Modyfi, and HEAVY.AI. She was a board observer at Robinhood until its IPO in 2021. Before she became an investor, she built consumer and enterprise tech herself at Microsoft, Disney, Twilio, and Box as a product leader.
In the episode, Richie and Vanessa explore the evolution of A-B testing in gaming, the balance between data-driven decisions and user experience, the challenges of scaling experimentation, the pitfalls of misaligned metrics, the importance of understanding user behavior, and much more.