Coffee Sessions #37 with Ariel Biller of ClearML, MLOps Memes. //Abstract The Meme king of MLOps joins us to talk about why we need more MLOps memes and how he got so damn good at being able to zoom out and see things from a metta level them make a meme about it! //Bio A researcher first, developer second, in the last 5 years Ariel worked on various projects from the realms of quantum chemistry, massively parallel supercomputing, and deep-learning computer vision. With AllegroAi, he helped build an open-source R&D platform (Allegro Trains), and later went on to lead a data-first transition for a revolutionary nanochemistry startup (StoreDot). Answering his calling to spread the word on state-of-the-art research best practices, He recently took up the mantle of Evangelist at ClearML. Ariel received his Ph.D. in Chemistry in 2014 from the Weizmann Institute of Science. With a broad experience in computational research, he made the transition to the bustling startup scene of Tel-Aviv, and to cutting-edge Deep Learning research. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Ariel on LinkedIn: https://www.linkedin.com/in/LSTMeow/  //Other Links https://youtu.be/1C_l5ICJlEo https://youtu.be/yTtTrwXEhN4 https://youtu.be/F4Ghp-phFuI Timestamps: [00:00] Introduction to Ariel Biller [01:20] Ariel's background [03:40] Story behind Memeing [06:36] "Memes can be as extreme as you want because people don't know if they're going to take you seriously or you're joking." [07:21] MLOps memes and more [10:15] MLOps fear [13:00] MLOps being more complicated than DevOps. [13:10] "A meme material is a social commentary about what there is and what there is now." [16:00] Standardization [18:18] "Would we have MLOps' code in a sweeping way or not?" [18:26] "I'm not sure as a community of builders, we have the right perspective that will walk for all the cases." [20:26] Journey into evangelism [26:45] "Feature stores are a big meme." [27:08] "Memeing is like a muscle. If you flex it daily it creates tensions." [31:26] We need to de-jargonize MLOps and ML engineering [35:55] Current Israeli tech scene [39:16] "The deficit is that there's a limited amount of people doing MLOps right now." [43:14] Tooling space [46:57] "Concentrate on the basic stuff that will survive forever and if you need to reach out for a tool, don't reach out for a tool, reach out for obstruction." [51:47] Standardization of ID Tree [52:43] "Everybody is doing whatever they want because it works for them. Someday, someone would come out with some good obstruction and good toolchain that works across the board that will click for everyone and will use it from that time on." [55:20] Ecosystem support

Coffee Sessions #36 with Luigi Patruno of 2U, Luigi in Production Part 2.    //Abstract Learning Voraciously: We talk a lot in the community about how to learn and upskill in an efficient way. Luigi provided great insight into how he applies certain principles to his learning practices. One tip he shared is to rigorously read and digest books. Luigi himself has used books to address his knowledge gaps in areas like product, finance, etc. I appreciated the emphasis on books. A lot of the reason we feel inundated by new learning resources is that they are online. Emphasizing books, which are often far higher-quality than blog posts, can slow things down and focus our learning.   Leadership Patience: Lately, Luigi has been spending more time managing projects and the data science team at 2U. He shared a lot of his insights into how to manage data science and machine learning properly. One of the most important things he emphasized to us was his patient attitude towards solving problems important to leadership. Turning around organizations is hard work. It's slow, it takes energy, and it is a nonlinear process. As he has course-corrected at various times as a data science leader, Luigi has brought admirable patience to the task, which has helped him be more successful on the things that matter to the entire company.   Communication Flows: It's easy to imagine Luigi as a great communicator, given his experience running MLInProduction.com. In our conversation, he showed us how he puts it to use in his management style. Luigi shared the importance of understanding how communication flows across an organization. Being aware of this is crucial to working on the right, most impactful things. Having a pulse on what different groups and leaders are thinking about can help you evaluate your impact as a team. //Bio Luigi Patruno is a Data Scientist focused on helping companies utilize machine learning to create competitive advantages for their business. As the Director of Data Science at 2U, Luigi leads the development of machine learning models and MLOps infrastructure for predicting student success outcomes across 2U’s portfolio of university partners. As the Founder of MLinProduction.com, Luigi creates and curates content to educate machine learning practitioners about best practices for running resilient machine learning systems in production. Luigi has consulted on data science and machine learning at Fortune 500 companies and start-ups and has taught graduate-level courses in Statistics and Big Data Engineering. He has an M.S. in Computer Science and a B.S. in Mathematics. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Luigi on LinkedIn: https://www.linkedin.com/in/luigipatruno91/ Timestamps: [00:00] Introduction to Luigi Patruno [01:12] Update about Luigi [04:08] Luigi's transition [07:18] Problem-focused [11:00] New problem [12:51] Rational platform strategy [18:18] Bringing the learnings to the team [20:57] Formulating and communicating vision [25:40] Problem-driven mindset [35:53] Organizational blind spots [41:12] Continous learning   [42:46] "Default to reading." [44:44] The Lindy effect [46:20] "You'll fail less often on the easy problems."   [46:25] Act upon reading [51:48] Ethical implications of ML [53:24] "Machine Learning is predicated on leveraging data to uncover insights that went to otherwise be able to be uncovered."

Coffee Sessions #35 with Nick Masca of Marks and Spencer, War Stories Productionising ML. //Abstract A conversation with MLOps war stories. Better said, a war story conversation. The kind that informs modern MLOps best practices.   Nick shared how to make MLOps organizational changes at large companies. I loved one tidbit he mentioned--"it's an evolution, not a revolution". That's a frank observation about the speed of practical change. As we all know it doesn't happen overnight.   Another great learning Nick shared focused on the value of delivering incremental results regularly. Oftentimes, ML projects suffer because of a focus on delivering too much too soon. This can then lead to a trough of disappointment with the way things actually pan out. Nick shared his experience on how to avoid such pitfalls with us so you don't have to learn the hard way. //Bio Nick currently serves as a Head of Data Science at Marks and Spencer, a large retailer based in the UK.  With a background originally in statistics, he transitioned into data science in 2014 and has picked up many battle scars and learnings since. //Link to the MLOps War Stories https://www.linkedin.com/posts/dpbrinkm_what-is-your-mlops-war-story-activity-6772604800971370496-LxtX --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Nick on LinkedIn: www.linkedin.com/in/nick-masca-09454956/ Timestamps: [00:00] Introduction to Nick Masca [01:36] Nick's background in tech [05:01] Nick's current job [06:19] Building the basics [08:18] "If you can gain trust and demonstrate value early, you could also freeze you up to the tidy marks later." [09:19] Strategy on long-running vision [10:25] "Historically, the legacy waterfall processes in the business where teams have specialist responsibilities." [11:14] KPI's [12:36] KPI translations into action plans [15:43] Data scientists call [17:13] Nick's nightmarish story   [22:52] Making the case on such a nightmarish story [25:06] Tools used by Marks and Spencer in 2015   [27:15] More complicated process [28:08] Takeaways from experience [30:57] Obstacles in deploying [34:53] Simplifying models [37:31] Combining environments into one [38:45] "Having written standards can be quite helpful to take ownership and responsibility around that."   [40:23] M&S team interaction [41:31] "It's an evolution, it's not a revolution I'd say at the moment but there's definitely real emphasis where we are to improve things and work towards goals to enable our team to work quicker, empower them." [42:10] Team moralizing [43:11] Takeaways from war stories [43:30] "The biggest takeaway for me is to start small, keep things simple, try things and it can be surprising sometimes what you'll find. Something simple can give you surprising results." [44:35] Opinions on Data Science and Machine Learning businesses democratize and commoditize

MLOps community meetup #60! Last Wednesday we talked to Vishnu Prathish, Director Of Engineering, AI Products, Innovyze. //Abstract The way Data Science is done is changing. Notebook sharing and collaboration were messy and there was minimal visibility or QA into the model deployment process. Vishnu will talk about building an ops platform that deploys hundreds of models at-scale every month. A platform that supports typical features of MLOps (CI/CD, Separated QA, Dev and PROD environment, experiments tracking, Isolated retraining, model monitoring in real-time, Automatic Retraining with live data) and ensures quality and observability without compromising the collaborative nature of data science. //Bio With 10 years in building production-grade data-first software at BBM & HP Labs, I started building Emagin's AI platform about three years ago with the goal of optimizing operations for the water industry. At Innovyze post-acquisition, we are part of the org building world-leading water infrastructure data analytics product. //Takeaways Why is MLOps necessary for model building at scale?   What are various cloud-based models for MLOps?   Where can ops help in various points in the ML pipeline Data Prep, Feature Engineering, Model building, Training, Retraining, Evaluation and inference ----------- Connect With Us ✌️-------------   Join our Slack community:   https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vishnuprathish/ Timestamps: [00:00] Introduction to Vishnu Prathish [00:16] Vishnu's background [04:18] Use cases on wooden pipes for freshwater [04:55] Virtual representation of actual, physical, tangible assets [06:56] Platform built by Vishnu [08:30] Build a reliable representation of network [11:52] Pipeline architecture [16:17] "MLOps is still an evolving discipline. You need to try and fail many times before you figure out what's right for you." [17:11] Open-sourcing [18:17] Platform for virtual twin [20:02] Entirely Amazon Stagemaker [20:43] Data quality issues [23:21] Reproducibility [23:40] "Reproducibility is important for everybody. Most of the frameworks do that for you." [25:00] Reproducibility as Innovyze's core business. [26:38] Each model is individual to each customer [27:50] Solving reproducibility problems [28:24] "Reproducibility applies to the process of training pipelines. It starts with collected from historical raw data from customers. In real-time, there's also this data being collected directly from sensors coming from a certain pipeline." [31:55] "Reusable training is step one to attaining automated retraining." [32:17] Collaboration of Vishnu's team [36:23] War stories [41:36] Data prediction [44:24] "A data scientist is the most expensive hire you can make." [47:55] 3 Tiers [48:53] MLOps problems [52:25] Automatically retraining [52:34] "Because of the numbers of models that go through this pipeline, it's impossible for somebody to manually monitor and retrain as necessary. It's not easy, it takes a lot of time." [54:22] Metrics on retraining [56:42] "Retraining is a little less prevalent for our industry compared to a turned prediction model that changes a lot. There are external factors that depend on it but a pump is a pump."

Coffee Sessions #34 with Geoff Sims of Atlassian, Machine Learning at Atlassian. //Abstract As one of the world's most visible software companies, Atlassian's vast data and deep product suite pose an interesting MLOps challenge, and we're grateful to Geoff for taking us behind the curtain. //Bio Geoff is a Principal Data Scientist at Atlassian, the software company behind Jira, Confluence & Trello. He works with the product teams and focuses on delivering smarter in-product experiences and recommendations to our millions of active users by using machine learning at scale. Prior to this, he was in the Customer Support & Success division, leveraging a range of NLP techniques to automate and scale the support function. Prior to Atlassian, Geoff has applied data science methodologies across the retail, banking, media,  and renewable energy industries. He began his foray into data science as a research astrophysicist, where he studied astronomy from the coldest & driest location on Earth: Antarctica. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Geoff on https://www.linkedin.com/in/geoff-sims-0a37999b/ Timestamps: [00:00] Introduction to Geoff Sims [01:20] Geoff's background [04:00] Evolution of ML Ecosystem in Atlassian [06:50] Figure out by necessity [08:47] Machine Learning not priority number one and disconnected to MLOps [11:53] Atlassian being behind or advanced? [16:38] Serious switch of Atlassian around machine learning [17:47] What data org did it come from? [20:00] Consolidation of the stack [21:21] Tooling - blessing and curse [24:37] Tackling play out [29:38] Staying on the same page [30:48] Priority of needs [31:55] How did it evolve? [35:12] Where is Atlassian now? [40:21] "Architecturally, Tecton is very very similar (to ours), it was just way more mature." [41:17] What unleashed you to do now? [41:36] "The biggest thing is independence from a data science perspective. Less reliance and less dependence on an army of engineers to help deploy features and models." [44:25] Have you bought other tools? [45:43] "At any given time, there's something that's a bottleneck. Look where the bottleneck is, then fix it and move on to the next thing."   [48:20] Atlassian bringing a model into production [50:01] "When we undertake whatever the project is, its days or weeks to go to a prototype rather than months or quarters." [53:10] "Conceptually, you're struggling walking towards that place because that's the place you want to be. If that's your problem, that's good. That's the promised land." [54:45] "Using our own tools is paramount because we are customers as well. So we see and feel the pain which helps us identify the problems and understand them."

MLOps community meetup #59! Last Wednesday was the celebration of the MLOps Community 1 Year Anniversary! This has been a conversion of Demetrios Brinkmann, David Aponte and Vishnu Rachkonda! //Abstract Over the past year Demetrios, David and Vishnu have interviewed many of the top names in MLOps. During this time they have been able to apply these learnings at their jobs and see what works for them. In this one year anniversary meetup the three of them will discuss some of the most impacting advice they have received in the last year and how they have put it into practice. //Bio Demetrios Brinkmann At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter. David Aponte David is one of the organizers of the MLOps Community. He is an engineer, teacher, and lifelong student. He loves to build solutions to tough problems and share his learnings with others. He works out of NYC and loves to hike and box for fun. He enjoys meeting new people so feel free to reach out to him! Vishnu Rachakonda Vishnu is the operations lead for the MLOps Community and co-hosts the MLOps Coffee Sessions podcast. He is a machine learning engineer at Tesseract Health, a 4Catalyzer company focused on retinal imaging. In this role, he builds machine learning models for clinical workflow augmentation and diagnostics in on-device and cloud use cases. Since studying bioengineering at Penn, Vishnu has been actively working in the fields of computational biomedicine and MLOps. In his spare time, Vishnu enjoys suspending all logic to watch Indian action movies, playing chess, and writing. ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup:  https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Timestamps: [01:07] Big shoutout to everybody that's in these meetups! [02:03] Big shoutout to Ivan Nardini for leading the Engineering Labs and to everyone who took part in the Engineering Labs! [02:26] Big shoutout to Charlie You leading the Reading Group and to everyone who takes part in it! [02:39] Big shoutout to everyone who takes part in the Office Hours! [02:49] Big shoutout to the people who are helping with shaping the website! [03:34] Thanks to all the people in Slack! Laszlo, Ariel, and people answering Slack questions. [04:10] Big thanks to all our Sponsors FiddlerAI, Algorithmia, and Tecton!   [06:13] David's Background [08:08] Vishnu's Background [09:55] High-Level Points [15:57] Starting small [24:05] Over-optimization - the root of all evil [26:42] Keeping text deck open [36:45] Missing from current MLOps tooling [48:00] How to communicate in these data products?

Coffee Sessions #33 with Sarah Catanzaro of Amplify Partners, MLOps Investments. //Bio Sarah Catanzaro is a Partner at Amplify Partners, where she focuses on investing in and advising high potential startups in machine intelligence, data management, and distributed systems. Her investments at Amplify include startups like RunwayML, Maze Design, OctoML, and Metaphor Data among others. Sarah also has several years of experience defining data strategy and leading data science teams at startups and in the defense/intelligence sector including through roles at Mattermark, Palantir, Cyveillance, and the Center for Advanced Defense Studies. //We had a wide-ranging discussion with Sarah, three takeaways stood out: The relationship between unstructured data and structured data is due for change. In most settings, you have some form of structured data (i.e. a metadata table) and unstructured data (i.e. images, text, etc.) Managing the relationship between these forms of data can constitute the bulk of MLOps. Because of this difficulty, Sarah forecasted new tooling arising to make data management easier. Academic benchmarks suffer from a lack of transparency on production/industry use cases. In conversation with Andrew Ng, Sarah shared her lesson that despite all the blame industry professionals place on academics for narrowly optimizing to benchmarks with little practical meaning, they also share the blame for making it difficult to create meaningful benchmarks. Companies are loath to share realistic data and the true context in which ML has to operate. MLOps is due for consolidation, especially as companies adopt platform-driven strategies. As many of you all know, there are tons and tons of MLOps tools out there. As more companies address these challenges, Sarah predicted that many of the point solutions would start to be consolidated into larger platforms. // Other Links https://amplifypartners.com/team/sarah/ https://projectstoknow.amplifypartners.com/ml-and-data https://twitter.com/sarahcat21/status/1360105479620284419 --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Sarah on LinkedIn: https://www.linkedin.com/in/sarah-catanzaro-9770b98/ Timestamps: [00:00] Introduction to Sarah Catanzaro [02:07] Sarah's background in tech [06:00] Staying engineer oriented despite being an investment firm [08:50] Tools you wished you had earlier in your career [12:36] 2 Motives of ML Engineers and ML Platform Team [16:36] Open-sourcing [21:29] Startup focus on resources [23:57] Playout of open-source project [27:32] Consolidation [33:18] Finding solutions [36:18] Evolution of MLOps industry in the coming years [42:36] Frameworks   [43:14] Structure data sets available to researchers. Meaningful advances of deep learning applied to structure data as well.

MLOps community meetup #58! Last Wednesday we talked to Ben Wilson, Practice Lead Resident Solutions Architect, Databricks. Model Monitoring Deep Dive with the author of Machine Learning Engineering in Action. It was a pleasure getting to talk to Ben about difficulties in monitoring in machine learning. His expertise obviously comes from experience and as he said a few times in the meetup, I learned the hard way over 10 years as a data scientist so you don't have to! Ben was also kind enough to give us a 35% off promo code for his book! Use the link: http://mng.bz/n2P5 //Abstract A great deal of time is spent building out the most effectively tuned model, production-hardened code, and elegant implementation for a business problem. Shipping our precious and clever gems to production is not the end of the solution lifecycle, though, and many-an-abandoned projects can attest to this. In this talk, we will discuss how to think about model attribution, monitoring of results, and how (and when) to report those results to the business to ensure a long-lived and healthy solution that actually solves the problem you set out to solve. //Bio Ben Wilson has worked as a professional data scientist for more than ten years. He currently works as a resident solutions architect at Databricks, where he focuses on machine learning production architecture with companies ranging from 5-person startups to global Fortune 100. Ben is the creator and lead developer of the Databricks Labs AutoML project, a Scala-and Python-based toolkit that simplifies machine learning feature engineering, model tuning, and pipeline-enabled modelling. He's the author of Machine Learning Engineering in Action, a primer on building, maintaining, and extending production ML projects. //Takeaways Understanding why attribution and performance monitoring is critical for long-term project success Borrowing hypothesis testing, stratification for latent confounding variable minimization, and statistical significance estimation from other fields can help to explain the value of your project to a business Unlike in street racing, drifting is not cool in ML, but it will happen. Being prepared to know when to intervene will help to keep your project running. ----------- Connect With Us ✌️------------- Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup:  https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Ben on LinkedIn: www.linkedin.com/in/benjamin-wilson-arch/ Timestamps: [00:00] Introduction to Ben Wilson [00:11] Ben's background in tech [03:40] Human aspect of Machine Learning in MLOps [05:51] MLOps is an organizational problem [09:27] Fragile Models [12:36] Fraud Cases [15:21] Data Monitoring [18:37] Importance of knowing what to monitor for [22:00] Monitoring for outliers [24:16] Staying out of Alert Hell [29:40] Ground Truth [31:25] Model vs Data Drift on Ground Truth Unavailability [34:25] Benefit to monitor system or business level metrics [38:20] Experiment in the beginning, not at the end [40:30] Adaptive windowing [42:22] Bridge the gap [46:42] What scarred you really bad?

MLOps community meetup #57! Last Wednesday we talked to Josh Tobin, Founder, Stealth-Stage Startup. // Abstract: Machine learning is quickly becoming a product engineering discipline. Although several new categories of infrastructure and tools have emerged to help teams turn their models into production systems, doing so is still extremely challenging for most companies. In this talk, we survey the tooling landscape and point out several parts of the machine learning lifecycle that are still underserved. We propose a new category of tool that could help alleviate these challenges and connect the fragmented production ML tooling ecosystem. We conclude by discussing similarities and differences between our proposed system and those of a few top companies. // Bio: Josh Tobin is the founder and CEO of a stealth machine learning startup. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley advised by Pieter Abbeel. // Other Links https://josh-tobin.com course.fullstackdeeplearning.com ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Josh on LinkedIn: https://www.linkedin.com/in/josh-tobin-4b3b10a9/ Timestamps: [00:00] Introduction to Josh Tobin [01:18] Background of Josh into tech [08:27] We're you guys behind the Rubik's Cube? [09:26] Rubik's Cube Project [09:51] "Research is meant to show you what's possible to solve." [11:07] "That's one of the things that's started to change and I think the MLOps world is maybe a part of that. What I'm excited about this is that people are focusing on the impact of their models." [13:18] Insights on Testing [17:11] Evaluation Store [18:33] "Production Machine Learning is data-driven products that have predictions in the loop." [23:40] Analyzing and moving forward [24:02] "My medium term mindset how machine learning is created is that is there's still gonna be humans involved but humans will be more efficient by tools." [25:50] Is there a market for this? [27:40] "The long tale of machine learning use cases is becoming part of every products and service more or less the companies create but it's the same way the software part of the products and services the companies create these days. It's going to create an enormous amount of value." [30:09] Talents [32:52] Organizational by-ends and knowledge [35:16] Tools used for Evaluation Store 39:59] Difference from Monitoring Tool [42:10] Who is the right person to interact in Evaluation Store? [50:05] Technical challenges of Apple and Tesla [53:30] "As Machine Learning use cases are getting more and more complicated, higher and higher dimensional data, bigger and bigger models, larger training sets many companies would need in order to continually improve their systems over time."

Coffee Sessions #32 with D. Sculley of Google, The Godfather Of MLOps. //Bio D is currently a director in Google Brain, leading research teams working on robust, responsible, reliable and efficient ML and AI. In his time at Google, D worked on nearly every aspect of machine learning, and have led both product and research teams including those on some of the most challenging business problems. // Links to D. Sculley's Papers ML Test Score: https://research.google/pubs/pub46555/ Machine Learning: The high-interest credit card of technical debt https://research.google/pubs/pub43146/ Google Scholar: https://scholar.google.com/citations?user=l_O64B8AAAAJ&hl=en --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with D. Sculley on LinkedIn: https://www.linkedin.com/in/d-sculley-90467310/ Timestamps: [00:00] Introduction to D. Sculley [00:40] Biggest Papers were written by D for Machine Learning [02:08] What's changed since you wrote those papers? [02:56] "No 1, there is an MLOps community." [04:38] Old best practices [05:12] "The fact that there are jobs titled MLOps, this is different than it was 5 or 6 years ago." [06:30] Machine Learning Systems then and now [07:08] "There wasn't the level of general infrastructure that was looking to offer the large scale integrated solutions."    [07:57] ML Test Score [11:09] "The Test Score was really written for situations where you don't care about one prediction. You care about millions or billions of predictions per day." [12:27] "In the end, it's not about the score. It's about the process of asking the questions making sure that each of the important questions that you're asking yourself, you have a good answer to."   [13:04] What else is needed in the Test Score? [14:36] Stratified testing   [17:05] Counterfactual testing [18:34] Boundaries [19:15] Dark ages [20:27] How do you try in Triage? [21:10] "Reliability is important. There are no small mistakes. If there are errors, they're going to get spotted and publicised. They're going to impact user's lives. The bar is really high and it's worth the effort to ensure strong reliability." [23:11] How do you build that interest stress test? [24:39] "I believe that stress test is going to look like a useful way to encode expert knowledge about domain areas." [25:37] How do I bring robustness? [26:47] "Because we don't know how to specify the behaviour in advance, testing the behaviour that we wanted to have is a fundamentally hard problem." [27:22] Underspecification Paper [30:58] "It's important to be evaluating models on this auto domain stress test and make sure that we understand the implications of what we're thinking about while we are in deployment land." [32:27] Principal challenges in productionizing Machine Learning [34:57] "As we expose our models to more specifics, this means there are more potential places our models might be exhibiting unexpected or undesirable behaviour." [42:37] Splintering of ML Engineering [46:00] Communities shaping the MLOps sphere [46:42] "It's much better to have one large community than three smaller communities because of those edufacts." [47:47] Concept of technical debt in machine learning. [49:28] "The good idea is to tend to make their way into the community if they are in a form that people can digest and share."