Ecommerce Playbook: Numbers, Struggles & Growth We Ran Incrementality Tests... Meta’s Tool Didn’t Win
Jul 31, 2025
Tony Chop, VP of Paid Media at Common Thread and an expert in incrementality testing, shares insights from testing Meta’s Incremental Attribution Tool. He reveals that the three conducted tests raised questions about Meta’s reported impacts, especially for new customer acquisition. Tony discusses high-fidelity geo holdout tests using real revenue data, emphasizing a cautious approach toward adopting new tools. Additionally, he highlights the effectiveness of campaigns targeting existing customers and the importance of robust measurement strategies in advertising.
AI Snips
Chapters
Transcript
Episode notes
Meta vs Real Revenue Testing
- Meta's conversion lift studies use platform-reported revenue, which is a critique since they grade their own homework.
- CTC's geo holdout test using actual Shopify revenue offers a higher fidelity measurement of incrementality.
Incremental Attribution Skews Warm
- Meta's incremental attribution product seems to skew towards showing ads to warmer audiences, impacting mainly returning customers.
- This explains why it behaves more like a retention campaign rather than an acquisition campaign.
Be Cautiously Curious with Meta
- Do not shift all your Meta ad spend immediately to the incremental attribution product; maintain a conservative curiosity stance.
- Continue to run 5-10% of your spend on incrementality tests to better understand and adapt.