Experiment Nation: The Podcast cover image

S3E15: A/B Testing Statistics issues CROs should ignore with Timothy Chan

Experiment Nation: The Podcast

00:00

How to Set a Custom Confidence Interval

I'm a big fan of 95% confidence intervals I know folks will criticize it as it's arbitrary and it absolutely is but it's a great solid default. While I am totally I'm not against folks who want to set a custom confidence interval of like 90% 80% or 99%. You should do that before you set up the experiment but I think you should also have strong justifications in picking those confidence intervals because to be clear those are arbitrary as well. There's certainly circumstances that warrant for example if you wanted to really squash your false positive rate because you're exceptionally risk-inverse then yeah 99% great. On the flip side if all you're more worried about missing out on

Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner