Running 121 AB Tests: Challenges and Setup

On the podcast: Scaling to $5M in ARR on paid ads, positive and negative results from 121 A/B tests, and why they still haven’t built an Android app. 

💡 The Power of a Single Metric: Concentrating on just one key metric can be remarkably effective. In the early stages, it's common to take on too much. By zeroing in on a solitary metric, it becomes simpler to iterate and conduct large-scale testing.

👀 Subscriptions as a Market Fit Gauge: The subscription model acts as a litmus test for your app's value. Gating access early and observing if users are willing to pay offers clear evidence of your app's worth.

💬 Flexibility in Attribution Methods: There's no one-size-fits-all approach to attribution. Various apps adopt diverse strategies. Often, straightforward methods like asking users about their discovery path during onboarding can be the most reliable.

🤳🏼 Prioritizing Creativity in Paid Marketing: If your strategy leans heavily on paid marketing, expect to dedicate at least 80% of your efforts to crafting creative content. For smaller teams, collaborating with content creators and influencers can be an effective strategy for scaling. 

📱 Choosing a Single Platform for Initial Launch: Easing early-stage challenges is feasible by focusing on a single platform. Many startups overextend by launching on multiple platforms simultaneously. Opting for a single platform, such as iOS, allows you to concentrate your limited resources on achieving initial milestones before considering a broader launch.

About Guest:

👨‍💻 Founder and CEO of Opal, an app that helps users limit their screen time and find focus.

📱 Even though iOS includes screen time management tools, Kenneth and his team believed they could improve the experience – and that users would pay for it.

💡 “Essentially, if you get people to pay for your products… that's a pretty strong signal that you have something pretty valuable.”

👋  LinkedIn

Links & Resources:

• Connect with Kenneth via LinkedIn
• Check out Opal: https://www.opal.so/ 
• X: https://twitter.com/kschlenker 

Episode Highlights:

[1:09] A three-phase approach: How the Opal team tackled building a subscription business for a screen time management app.

[4:54] When ad spend is worth it: Opal implemented paid marketing from the beginning – and it paid off.

[10:44] Attribution made easy: Sometimes the simplest method of finding out where users came from – just asking them! – is the most useful.

[13:08] Contracting creatives: Consider hiring independent contractors who care about your mission to build your ad content.

[17:22] From premium to freemium: Many apps (like Duolingo) start out free, then add paid subscriptions later. Opal is doing the opposite.

[25:53] Work smarter, not harder: Releasing your app on a single platform (instead of iOS, Mac, web, and Android all at once) can save your team a lot of time and money.

[30:07] Lessons from 121 A/B tests: Prioritize the bigger swings that will significantly increase your uplift early on.