Experiment Nation: The Podcast

S3E15: A/B Testing Statistics issues CROs should ignore with Timothy Chan

May 5, 2023

40:36

Creator website

AI Chapters

Episode notes

Introduction

3min

The Role of a Data Scientist in Facebook's Growth

3min

Facebook's Democratization of Experimentation

2min

How Facebook Approached QA and Experimentation

3min

How a Data Scientist Influenced Experimentation

2min

Facebook's Culture of Overlapping Tests

3min

The Purist Perspective on Interaction Effects

2min

How to Succeed at Facebook's Scale Experiment

3min

How to Measure CRO Performance With Minimum Sample Size

2min

10.

The Power of Experimentation at Facebook

2min

11.

How to Join a Startup

2min

12.

The Data Scientist's Perspective on Statistics

2min

13.

How Facebook's Stats Engine Works

5min

14.

How to Set a Custom Confidence Interval

3min

15.

The Effects of Testing on False Positives

2min

16.

The Cost of Making a Change on a Website

3min

Experimenters obsess about many A/B Testing Statistics concepts - but not all of them are worth worrying about. Statsig's Timothy Chan shares his thoughts about test isolation, interaction effects and confidence levels - some of his thoughts may surprise you. #experimentation #product #statistics