Experiment Nation: The Podcast

Kelly Wortham dropped some serious knowledge bombs in our recent Experiment Nation chat. Here’s what stood out: - CRO isn’t just for marketers: Lawyers, project managers, even developers can thrive in this field. It’s all about a love of learning and adaptability. - Don’t underestimate the power of mentorship: Kelly has THREE mentors, each focusing on different aspects of her career. Find your own advisors to help you grow! - Certifications are nice, but experience trumps all: While CXL is great, hands-on testing platform experience is the real deal. - UX psychology is CRUCIAL: Understand your users’ motivations and behaviors to create truly effective tests.

Simar, CEO and Founder of OptiPhoenix, shared invaluable insights on A/B testing and CRO at Experiment Nation. Here are 5 key takeaways: Don’t rely solely on macro goals: Micro goals reveal user behavior, driving better testing strategies and informing future hypotheses. Iterative testing trumps scrapping: Learn from failed tests by adjusting hypotheses and running new experiments, boosting your win rate. QA is critical, but often neglected: Invest time in thorough QA, ensuring your test is accurate and integrated correctly. Outsourcing AB test development can be beneficial: Consider outsourcing for faster scale-up, access to expert resources, and reduced development burden. Prioritize experimentation within your workflow: Make A/B testing an integral part of your product development process, leading to better decisions and faster results.

Avishek Basu Mallick, a Senior PM at CheapOair, talks about: - Experimentation at scale: Even with high traffic, rigorous experimentation is crucial. - Idea generation: User research, NPS surveys, and internal stakeholders are valuable sources of experiment ideas. - Communicating progress: Regular updates with leadership build trust and ensure alignment. - Personalization is nuanced: Distinguish between customization, segmentation, and true personalization. - Generative AI’s potential: Explore its use for simplifying the travel planning experience.

Michael St Laurent, a seasoned CRO expert, shared his insights on large-scale testing Here are 5 key takeaways for marketers: Prioritization is key: Large-scale testing requires prioritizing tests based on potential impact and aligning with overall business goals. AI is the future, but humans guide it: Mike emphasizes the power of AI in automating processes and analyzing data, but highlights the importance of human expertise and guidance. Data quality is crucial: AI models rely heavily on the quality of data. Invest in building a robust data infrastructure to ensure accurate results. People are the biggest challenge: Navigating internal politics and managing stakeholder expectations is often the biggest hurdle for large-scale testing. Building a testing culture is a win: The ultimate goal of large-scale testing is to create a data-driven culture that fosters continuous improvement and better user experiences.

Ellie Hughes, a top CRO expert and Experimentor of the Year, shares vital insights for enhancing your experimentation strategies. She emphasizes starting small while remaining curious, focusing on impactful activities rather than mundane tasks. Clear client communication and setting expectations are essential for success. Ellie advocates for embracing iteration and encourages thinking big to maximize impact. Learn how personal frustrations can spark growth and discover the importance of data in understanding customer behavior for effective decision-making.

In this discussion, Chris Mercer, a conversion rate optimization expert for low-traffic websites, shares invaluable insights into the art of measurement. He emphasizes the importance of understanding user behavior before any optimization efforts. Sequential testing is highlighted as a more effective approach than traditional A/B testing for sites with limited traffic. Chris also stresses the necessity of focusing on metrics that drive lead generation and revenue, while also advocating for feedback-driven strategies and the automation of measurement using Google Analytics 4.

Join Nick Murphy, Senior Research and Experimentation Lead at Further, as he shares insights into building effective learning libraries that enhance experimentation. He emphasizes capturing the ‘why’ behind tests to foster a culture of growth. Discover how organizing your library can facilitate cross-team collaboration and inform decision-making. Nick also discusses the importance of early documentation and how macro learnings can support consultancy practices, paving the way for a deeper understanding of consumer behavior.

In this episode, senior customer experience leader, Courtney LeBlanc, reveals the ideal partnership between CX and experimenters and shares invaluable insights on: Forging powerful partnerships between CX and experimenters Navigating the daily demands of a CX leader Simplifying CX workflows with customer insights ... and so much more! Don't miss out on this enlightening conversation from an expert in the CX game! Find Courtney on LinkedIn ⤵ LinkedIn: https://www.linkedin.com/in/courtney-leblanc-426920b3/ 00:00 Introduction and Mutual Appreciation 00:27 Meet Courtney: Customer Support and CX Expert 03:38 Building an Alliance with Customer Support 06:11 Challenges and Strategies in CX Collaboration 14:25 Data Analysis and Insights Sharing 23:01 The Ideal Partnership Between CX and Experimentation 24:47 Balancing Customer Insights and Business Priorities 26:46 The Emotional Impact of Customer Support 28:29 Communicating Customer Stories Effectively 30:04 Leveraging NPS Data for Product Improvement 32:08 Empowering Teams with Data Access 33:06 The Importance of Experiment Communication 37:53 Fostering Collaboration Between CX and Product Teams 48:58 Reflecting on Growth and Collaboration 50:22 Conclusion and Final Thoughts

How do you get Product Managers to adopt Experimentation? Shagun Aulakh, the Director of Product Management - Experimentation at American Express shares with Rommil here tips. Here are just a few of the takeaways: 1- Leadership advocacy is critical for embedding experimentation practices in a highly matrixed organization. 2 - Prioritize education and internal PR to influence and persuade teams to embrace experimentation. 3- Define success metrics for your experimentation program and map out a long-term plan for evolving maturity. 5 - Identify champions within the organization to inspire and motivate other teams to adopt experimentation practices. 5 - Democratize insights and share success stories to showcase the value and impact of experimentation. Watch Shagun here: Experimentation Elite in Birmingham UK: https://experimentationelite.com/#spe... Growth Marketing Summit in Frankfurt, Germany: https://www.growthmarketingsummit.com/en

In this episode, we have Joey Herzberg who shares his thoughts on how AI fits in the experimentation life cycle, the utility of keeping easily referenced records of test results, and the importance of experimentation strategy with regard to sample ratios.