The High Failure Rate of Experiments at Microsoft, Bing, and Airbnb

Many people at Microsoft were initially in denial about the low success rate of their product planning and releasing. However, when an experimentation platform was built, it revealed that about 60-70% of ideas failed to make any significant impact on key metrics. In the more optimized domain of Bing, this failure rate rose to 85%. Similarly, at Airbnb, 92% of machine learning models designed to optimize search failed to improve metrics.