The Importance of Minimum Effective Interest in Tests

One of the major findings of the previous meta-analysis was that a lot of tests were actually not very well-powered. Two thirds of tests had, I mean, the minimum effect below 10% lift. So only 25% of tests get the ability to detect a lift of 10% and 10% is huge if you look at the distribution of the actual effect sizes. But with under power tests, you would miss those true effects of 2% and 3% way too often,. 80% of the time, which is unacceptable.