Coffee Sessions #32 with D. Sculley of Google, The Godfather Of MLOps.

//Bio
D is currently a director in Google Brain, leading research teams working on robust, responsible, reliable and efficient ML and AI. In his time at Google, D worked on nearly every aspect of machine learning, and have led both product and research teams including those on some of the most challenging business problems.

// Links to D. Sculley's Papers
ML Test Score: https://research.google/pubs/pub46555/
Machine Learning: The high-interest credit card of technical debt
https://research.google/pubs/pub43146/
Google Scholar:
https://scholar.google.com/citations?user=l_O64B8AAAAJ&hl=en


--------------- ✌️Connect With Us ✌️ -------------
Join our slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/
Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/
Connect with D. Sculley on LinkedIn: https://www.linkedin.com/in/d-sculley-90467310/

Timestamps:
[00:00] Introduction to D. Sculley
[00:40] Biggest Papers were written by D for Machine Learning
[02:08] What's changed since you wrote those papers?
[02:56] "No 1, there is an MLOps community."
[04:38] Old best practices
[05:12] "The fact that there are jobs titled MLOps, this is different than it was 5 or 6 years ago."
[06:30] Machine Learning Systems then and now
[07:08] "There wasn't the level of general infrastructure that was looking to offer the large scale integrated solutions."   
[07:57] ML Test Score
[11:09] "The Test Score was really written for situations where you don't care about one prediction. You care about millions or billions of predictions per day."
[12:27] "In the end, it's not about the score. It's about the process of asking the questions making sure that each of the important questions that you're asking yourself, you have a good answer to."  
[13:04] What else is needed in the Test Score?
[14:36] Stratified testing  
[17:05] Counterfactual testing
[18:34] Boundaries
[19:15] Dark ages
[20:27] How do you try in Triage?
[21:10] "Reliability is important. There are no small mistakes. If there are errors, they're going to get spotted and publicised. They're going to impact user's lives. The bar is really high and it's worth the effort to ensure strong reliability."
[23:11] How do you build that interest stress test?
[24:39] "I believe that stress test is going to look like a useful way to encode expert knowledge about domain areas."
[25:37] How do I bring robustness?
[26:47] "Because we don't know how to specify the behaviour in advance, testing the behaviour that we wanted to have is a fundamentally hard problem."
[27:22] Underspecification Paper
[30:58] "It's important to be evaluating models on this auto domain stress test and make sure that we understand the implications of what we're thinking about while we are in deployment land."
[32:27] Principal challenges in productionizing Machine Learning
[34:57] "As we expose our models to more specifics, this means there are more potential places our models might be exhibiting unexpected or undesirable behaviour."
[42:37] Splintering of ML Engineering
[46:00] Communities shaping the MLOps sphere
[46:42] "It's much better to have one large community than three smaller communities because of those edufacts."
[47:47] Concept of technical debt in machine learning.
[49:28] "The good idea is to tend to make their way into the community if they are in a form that people can digest and share."