Nerd Alert: Where A/B Testing Goes Wrong

Welcome to Nerd Alert, a series of special episodes bridging the gap between marketing academia and practitioners. We’re breaking down highly involved, complex research into plain language and takeaways any marketer can use.

In this episode, Elena and Rob explore how sophisticated advertising algorithms can skew A/B test results, potentially leading marketers to draw misleading conclusions about ad performance.

Topics covered:   

• [00:45] "Where A-B Testing Goes Wrong”
• [02:00] Divergent delivery in online advertising platforms
• [03:00] How user types and targeting policies impact tests
• [04:10] Pizza party analogy explaining divergent delivery
• [05:40] Marketing Architects' approach to testing
• [06:50] Using synthetic audiences for more controlled testing 

To learn more, visit marketingarchitects.com/podcast 
 

Resources: 
Braun, M., & Schwartz, E. M. (2024). Where A-B Testing Goes Wrong: How Divergent Delivery Affects What Online Experiments Cannot (and Can) Tell You About How Customers Respond to Advertising. Journal of Marketing. 
 

Get more research-backed marketing strategies by subscribing to The Marketing Architects on Apple Podcasts, Spotify, or wherever you listen to podcasts.