Evaluating Long-term Impact of Experiments on Cohort Value

Archie Abrams is the VP of Product and Head of Growth at Shopify, where he leads a 600+ person growth org across product, design, engineering, data, ops, and growth marketing. Shopify powers over 10% of e-commerce in the United States, with $235 billion in GMV in 2023 (roughly the size of Finland’s economy). He previously led Consumer product and growth at Lyft and was at Udemy for 8 years as SVP of Product having joined the company when it was 10 people. In our conversation, we discuss:

• Why Shopify optimizes for churn

• Why the core product team doesn’t use metrics-based goals

• Why they keep multi-year experiment holdouts

• How they structure their growth team

• The benefits of not having a CMO

• Lessons learned about integrating sales into a product-led growth model

• The power of discounting as a growth lever

• Much more

—

Brought to you by:

• Explo—Embed customer-facing analytics in your product

• Dovetail—The customer insights hub for product teams

—

Find the transcript at: https://www.lennysnewsletter.com/p/shopifys-growth-archie-abrams

—

Where to find Archie Abrams:

• X: https://x.com/archieabrams

• LinkedIn: https://www.linkedin.com/in/archie-abrams-b6aa8b6/

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• X: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Archie’s background 

(02:30) Shopify’s impressive growth

(06:17) Shopify’s unique approach to churn and retention

(08:43) Monetization model and success metrics

(11:08) Long-term experimentation and metrics

(23:00) Examples of big wins that Archie’s team has shipped

(26:42) Monetary friction

(27:14) Metrics

(29:47) Shopify’s growth team structure

(33:03) Goal setting and forecasting

(37:10) Examples of long-term results within Shopify

(41:36) Shipping neutral experiments

(42:05) Building a hundred-year company

(48:04) Why Shopify doesn’t use KPIs

(51:30) Shopify’s “Get s**t done” framework

(54:30) Cross-team collaboration 

(58:48) The importance of an opinionated founder 

(01:01:12) Growth and sales integration

(01:06:42) Shopify’s marketing structure

(01:08:49) Insights on discounting from Udemy

(01:11:09) Lightning round

—

Referenced:

• Shopify: https://www.shopify.com/

• Tobias Lütke on LinkedIn: https://www.linkedin.com/in/tobiaslutke

• Gross Merchandise Value: Calculation and Best Practices: https://www.shopify.com/retail/gross-merchandise-value

• Brian Chesky’s new playbook: https://www.lennysnewsletter.com/p/brian-cheskys-contrarian-approach

• Glen Coates on LinkedIn: https://www.linkedin.com/in/glcoates

• Harley Finkelstein on LinkedIn: https://www.linkedin.com/in/harleyf

• Udemy: https://www.udemy.com/

• Scientific Advertising: https://www.amazon.com/Scientific-Advertising-Original-Claude-Hopkins/dp/1640954252

• Four-Minute Mile: https://www.amazon.com/Four-Minute-Mile-Roger-Bannister/dp/1493038753/

• The Sopranos on HBO: https://www.hbo.com/the-sopranos

• Suno: https://suno.com/

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

—

Lenny may be an investor in the companies discussed.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.lennysnewsletter.com/subscribe