
S3E15: A/B Testing Statistics issues CROs should ignore with Timothy Chan
Experiment Nation: The Podcast
00:00
How to Set a Custom Confidence Interval
I'm a big fan of 95% confidence intervals I know folks will criticize it as it's arbitrary and it absolutely is but it's a great solid default. While I am totally I'm not against folks who want to set a custom confidence interval of like 90% 80% or 99%. You should do that before you set up the experiment but I think you should also have strong justifications in picking those confidence intervals because to be clear those are arbitrary as well. There's certainly circumstances that warrant for example if you wanted to really squash your false positive rate because you're exceptionally risk-inverse then yeah 99% great. On the flip side if all you're more worried about missing out on
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.