The Effects of Testing on False Positives

I think there's an argument to be made that the control group is the default for a reason shipping new experiences can sometimes introduce like new technical overhead. If you're not worried about that and you see an experiment where the test group is beating the control but in a not statistically significant manner I'm okay with like picking a winner as long as you know that's not a statistically sound result. After four weeks if you haven't seen the effect you needed to it's also possible that that effect is so small that is not worth pursuing.