AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
4min snip
Lessons from scaling Uber and Opendoor | Brian Tolkin (Head of Product at Opendoor, ex-Uber)
Lenny's Podcast: Product | Growth | Career
Challenges of A-B Testing with Low Transaction Volumes
A-B testing is challenging for organizations with low transaction volumes, as they struggle to generate significant results within short time frames. It is critical to acknowledge this limitation and conduct power analysis beforehand to assess the potential for obtaining meaningful insights. In some cases, it may be acceptable to run experiments over longer periods, like six months, aligning with future planning needs. Understanding that not all tests can yield quick answers helps mitigate the frustration of inconclusive results. Creating conviction in hypotheses is essential; thus, alternative methods should be employed when traditional A-B tests are impractical. Engaging with customers can provide valuable insights, while observational data and statistical techniques can enhance understanding despite lower rigor. Adjusting confidence levels and running experiments with 80% confidence instead of the traditional 95% may be a feasible trade-off when necessary. Leveraging long-term holdouts to validate intuition, or simply trusting one's instinct to proceed with a solution, may sometimes be the best course of action when significant results remain elusive.
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode