Continuous Integration

It's an awful system for several reasons, especially if you're running in continuous integration. Different tests have different noise levels and some are going to be noisier than others. It also wouldn't tell us when performance changed, just that performance did change. We put a lot of bandades and adjustments on it, but it never got easy to work with. The speed at which you can detect and fix something is much better as find out when things change.