Tips For Your Personalization and Experimentation Program [Ronny Kohavi, Airbnb]
Apr 30, 2020
auto_awesome
Ronny Kohavi, an expert in experimentation and personalization, discusses the role of online experiments during the current crisis and the need for a trustworthy environment for testing. He shares great examples from Amazon and Microsoft, and explores the connection between experimentation, personalization, and machine learning.
Experimentation is crucial for evaluating and refining personalization efforts and machine learning algorithms.
Metrics play a vital role in online control experiments, and defining and measuring the right metrics is essential for optimization and testing.
Deep dives
The Role of Experimentation in Personalization
Experimentation and personalization are closely connected in improving user experiences. Personalization can be beneficial but should not be expected to dramatically increase revenues. Controlled experiments are essential for evaluating and refining personalization efforts, as well as other machine learning algorithms. The use of experimentation ensures that changes are tested, validated, and properly implemented.
The Importance of Metrics in Online Control Experiments
Metrics play a crucial role in online control experiments. It is important to define and measure the right metrics for optimization and testing. The overall evaluation criterion (OEC) serves as an indicator of success, but it's crucial to balance it with countervailing metrics to avoid unintended consequences. The book, 'Trustworthy Online Control Experiments: A Practical Guide to A/B Testing,' offers insights on metrics, institutional memory, ethics, and instrumentation, allowing readers to understand and apply these concepts in practice.
The Relationship Between Personalization, Experimentation, and Innovation
Personalization and experimentation work hand in hand, especially in industries like travel and e-commerce. Personalization efforts can benefit from experimentation, as it helps validate and refine algorithms and models. A portfolio of bold and innovative experiments, along with incremental improvements, is key to driving innovation. Experimentation allows for the rapid testing and validation of ideas, helping organizations make informed decisions and avoid pitfalls.
The Impact of Speed and Performance in User Experience
Speed and performance are crucial factors in user experience and can significantly impact metrics and revenue. Even small improvements or degradations in page loading times can influence user interactions and abandonment rates. The book highlights the importance of speed and provides examples from Google, Amazon, and Bing that demonstrate the financial implications of optimizing page performance.
Personalization, artificial intelligence and machine learning are hot topics. But how can you measure if your personalization initiatives and your algorithms really work?
Before he joined Airbnb, Ronny grew the experimentation platform (ExP) team at Microsoft to over 110 data scientists, developers, and program managers to accelerate innovation at Microsoft through trustworthy analysis and experimentation.
But Ronny is not just an experimentation expert! In the past, he was Director of Data Mining and Personalization at Amazon, so he also has tons of experience in the area of personalization.
Listen to the full Diggintravel Podcast interview with Ronny to learn more about:
What is the role of online experiments in the current crisis?
Why do you need a trustworthy environment for testing?
How can you define the right metrics (OEC – overall evaluation criteria)?
OEC and other great examples from Amazon and Microsoft
How are experimentation, personalization and machine learning connected?
You can find the link to the full interview article with Ronny here: