The Importance of Hypothesis Shopping

If you test 20 hypotheses that are completely false, one of them will show up with a p-value below 5% by random chance. So if I do my red car colon cancer experiment, it's probably going to be higher than the p-value and there'll be a lot of statistical noise. A lot of the bad studies you read are exactly done that way. What you do is give people a questionnaire with 500 items and then you check every pair of items.