Quick takeaways

• A/B testing should be the preferred method for comparing different versions of a product or service, while differences and differences should be used when A/B testing is not feasible.

• Toxic behaviors in experimentation, such as cutting corners and engaging in statistical theater, compromise the integrity of the experiment process and require the use of the right data, good hypotheses, and proper statistical application to overcome.

• Small companies with limited data should avoid running experiments due to low statistical power, as it increases the likelihood of false positives, and selecting metrics that are volatile but not impactful can lead to misleading results.

Deep dives