Profit-Maximizing Marketing Experiments with Elea Feit
Jul 20, 2023
auto_awesome
This week’s guest is Elea Feit, an Associate Professor of Marketing at Drexel University’s LeBow College of Business. With a rich background in market research from General Motors and Amazon, she dives into the evolving world of marketing experiments. Elea explains the power of randomized controlled trials and the innovative 'test and roll' framework, emphasizing how even limited data can yield significant insights. She also discusses the importance of moving beyond mere statistical significance to focus on impactful marketing changes, and the latest advancements in marketing mix modeling.
Advanced testing methods like randomized controlled trials and holdout testing enable marketers to assess the effectiveness of their strategies beyond traditional A/B tests.
The Test and Roll framework highlights the value of analyzing smaller datasets effectively, allowing marketers to make informed decisions without relying solely on large sample sizes.
Deep dives
The Importance of Experimentation in Marketing
Marketers frequently rely on A-B testing for optimizing websites and conversions, but there's a growing interest in evaluating the effectiveness of marketing strategies themselves through more advanced methods. Techniques such as randomized controlled trials, holdout testing, and geo-lift experiments offer powerful ways to analyze marketing impact, yet many marketers find these methods inaccessible due to a lack of supporting tools. Understanding which marketing efforts yield positive returns can significantly influence budget allocation and strategy decisions. Gaining insights from these experiments is crucial for marketers aiming to refine their approach and ensure that their advertising spends deliver maximum value.
Test and Roll Framework for Marketing Experiments
The discussion introduces the Test and Roll framework as a practical strategy for marketers to plan and execute experiments effectively. This framework emphasizes that even a smaller dataset can be useful for gaining insights if appropriately analyzed, moving past the traditional focus on achieving statistical significance. By prioritizing the results derived from available data rather than waiting for a large sample size, marketers can make informed decisions that maximize profitability. The framework also considers input from past tests to help determine the optimal test size for new experiments, enhancing the decision-making process.
Choosing the Right Size for Marketing Tests
Deciding how large a test needs to be is a critical assessment for any marketing team. A back-of-the-envelope calculation can help teams gauge the potential financial implications of decisions before executing a full-scale marketing campaign. For impactful tests, especially concerning costly advertising channels, marketers should consider fiscal returns when determining sample size and budget allocation. If a project has significant potential upside, allocating a more substantial budget during the testing phase may be justified to reveal conclusive results about channel effectiveness.
Evaluating In-Platform Testing vs. Traditional Testing
In-platform testing offers a streamlined approach for advertisers to assess their ad campaigns within platforms like Google and Meta, providing a level of convenience and user-level randomization that can be difficult to achieve independently. However, marketers must be cautious, as these platforms may have biases in how they execute tests and analyze results. While platform-controlled experiments can simplify the logistics of running tests, the potential lack of transparency could affect trust in the findings. Marketers should thus weigh the benefits of efficiency against the need for robust and unbiased data when considering their testing methods.
A lot of marketers are familiar with A/B testing, usually with regards to website testing and conversion rate optimization.
But there’s another kind of testing that marketers are becoming more interested in lately, and that’s testing of the marketing itself. It goes by many names - randomized controlled trials, geo-lift experiments, conversion lift studies, holdout testing - maybe you’ve heard of these? But maybe you weren’t sure where to start with them?
Elea Feit is here to help! She has a bachelor’s in mathematics, a master of science in industrial engineering, and a PhD in Marketing. She’s applied those learnings at companies like General Motors and Amazon, and continues to train the next generation of marketers and analysts as Associate Professor of Marketing at Drexel University’s LeBow College of Business.