MLOps community meetup #65! Last Wednesday we talked to Kseniia Melnikova, Product Owner (Data/AI), SoftwareOne. //Abstract In this MLOps Meetup, we talked about the Machine Learning model lifecycle and development stages and then analyze the main mistakes that everybody does at each stage. Kseniia also provided the audience with solutions to the mistakes and we discussed existing tools for experiment management. //Bio Kseniia is a product owner for Data/AI-based products. Right now, she is working mostly with numeric data analysis, customer insights, and product recommendations. Previously Kseniia worked at Samsung Research with the biometrics team. She was studying computer science in Russia (Moscow) and a little bit of management in South Korea (Seoul). One of the most interesting directions of research - Model Lifecycle Management Systems and Reproducibility. ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup:  https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Kseniia on LinkedIn: https://www.linkedin.com/in/kseniia-melnikova/ Timestamps: [00:00] Introduction to Kseniia Melnikova [02:00] MLOps World Conference Announcement [03:40] AI Development process: Common Mistakes [07:45] Step 1: Planning [07:48] Mistake #1: Personal Decisions - Teamwork [08:31] Mistake #1: Cases [09:00] Mistake #1: Solution [11:52] Scrum [12:50] "In Scrum, it's hard to plan because especially in research, you don't know which result affects new tasks that's why it might be a little slow for Machine Learning." [14:28] Step 2: Data Processing [14:34] Mistake #2: Chaos with Datasets [15:26] Mistake #2: Cases [16:48] Mistake #2: Solution [20:12] Step 3: Experiments [20:21] Mistake #3: Lack of Experiments Tracking [22:13] Mistake #3: Case - Manual Experiments Tracking [24:10] Mistake #3: Solutions [25:57] Experiments Tracking Tools Example: MLFlow UI [26:46] Awareness of Existing Tools [28:21] Tools' Features [29:21] Possible Combination [29:48] Another Possible Combination [30:24] Best Practice [31:42] Mistake #0: Lack of Information Sharing [32:26] Mistake #0: Solution - Organize more meetings/standups!   [34:18] Find Your Mistakes [34:41] Mistake #0: Solution - Organize more meetings/standups!   [35:35] Audio Data [39:32] Experiment tracking of only 1 ML engineer [41:38] "I prefer reproducibility tools because it's automatic and it also takes a lot of time to manually upload the results into conference." [43:03] AI Development Check-list [43:40] Check-list Results [44:52] "I think it's always interesting to rate yourself to share the results with other people to compete out of it." [45:10] Why to Implement [45:17] "If we have more automation on experimentations for data sets versioning, it will lead to less manual work." [45:28] "AI Development process implementation will have the possibility to reproduce and compare experiments." [45:37] "AI Development process implementation will make you comfortable on solving the issues you'll face every day." [45:52] "AI Development process implementation will lead to a faster commercialization cycle because you will take less time on the process and more time for the results." [46:03] "If we will take all the principles of AI Development process implementation, it will lead to easy communication between team members. You'll gain trust, have great teamwork, and everyone will have respect for each other."   [46:50] War stories prior to having AI Development process [49:50] Calculating the lost money

Coffee Sessions #42 with Amit Paka of Fiddler AI, Model Performance Monitoring. //Abstract Machine Learning accelerates business growth but is prone to performance degradation due to its high reliance on data. Moreover, MLOps is often fragmented in many organizations, causing frictions to debug models in production. With new rules from the EU that focus on trust and transparency, it’s becoming more important to keep track of model performance. But how? We propose a new framework, a centralized ML Model Performance Management powered by Explainable AI. Learn more about how you can stay compliant while maximizing your model performance at all times with explainability and continuous monitoring. //Bio Amit is the co-founder and CPO of Fiddler, a Machine Learning Monitoring company that empowers companies to efficiently monitor and troubleshoot ML models with Explainable AI. Prior to founding Fiddler, Paka led the shopping apps product team at Samsung. Paka founded Parable, the Creative Photo Network, now part of the Samsung family. He also led PayPal's consumer in-store mobile payments launching innovations like hardware beacon payments and has developed successful startup products particularly in online advertising - paid search, a contextual, ad exchange, and display advertising. Paka has passions for actualizing new concepts, building great teams, and pushing the envelope, and aims to leverage these skills to help define how AI can be fair, ethical, and responsible. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Amit on LinkedIn: https://www.linkedin.com/in/amitpaka/ Timestamps: [00:00] Thank you to Fiddler AI! [00:46] Introduction to Amit Paka [05:04] Amit's background in tech [09:55] EU Regulation [12:39] "The goal that the EU seems to be going for is they want to go for helping build human-centric and responsible AI."   [13:28] 4 AI Categories:               1. Unacceptable risk applications 2. High-risk applications 3. Limited risk applications 4. Minimal risk applications   [14:58] Deep dive into High-risk applications [17:28] Digital Services Act (DSA) and Digital Marketing Act (DMA) [19:02] Military   [19:33] "They don't know what they don't know and they probably wanted the door open."   [21:13] US on JIC Team - transparency and increasing trustworthiness on AI [23:06] Diversity of industries and Explainability   [24:22] "The urgent need for Explainability comes from verticals that are facing the problems today on the ground and cannot run their business." [30:09] Model Performance Management (MPM) [34:05] "When your model is facing issues, you now have to root-cause it within life." [35:40] Control Theory [36:10] "Control Theory means that you do not just measure it but you can influence it so you can actually keep it." [38:14] Abstraction into being useful [43:23] "You can train a model that accurately represents the reality." [44:00] Data scientist doing ML Flow [49:55] Amit's favorite surprise! [53:04] Banking and Insurance adoption of ML [55:48] Advise ML Scientists and Data Scientists in terms of Explainable AI [58:25] "Models are incredibly hard to debug. You're just training a model for high accuracy but you don't know how that accuracy is distributed." [59:49] Linking of EU Regulation and MPM

Coffee Sessions #41 with Monmayuri Ray of Gitlab, CI/CD in MLOPS. //Abstract We all are familiar with the concept of MVP. In the world of DevOps, one is also familiar with Minimal Viable Feature and further Minimal Viable change. CI/CD is the orchestrator and the underlying base to enable automated experimentation, to start small, and build an idea for production. Now if we use the same fundamentals in MLOps, what does that mean? The podcast will take the audience on a journey in understanding the fundamentals of orchestrating machine predictions using responsible CI/CD in MLOps in this ever-changing, agile world of software development. One shall hope to learn how to excel at the craft of CI for Machine Learning (ML), lowering the cost of deployment through a robust CI/CD/CT/CF framework. //Bio Monmayuri is an advisor,  data scientist, and researcher specializing in MLops/DevOps at GitLab in Sydney. She builds creative, products to solve challenges for companies in industries as diverse as financial services, healthcare, and human capital. Along the way, Mon has built expertise in Natural Language Processing, scalable feature engineering, MLOps transformation and digitization, and the humanization of technology. With a background in applied mathematics in biomedical engineering, she likes to describe the essence of AI as “low-cost prediction” and MLOps as “low-cost transaction” and believes the world needs the collaboration of poets, historians, artists, psychoanalysts and scientists, engineers to unlock the potential of these emerging technologies where one works in making a machine think like humans and be efficient automated fortune tellers. //Takeaways Key Takeaways include how to incorporate the best CI/CD practice in your MLOPS lifecycle. Things to do and things not to do. How best to get the DevOps engineer, ML engineer, and data scientists to speak the same language and automate CI for pipeline and models. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Mon on LinkedIn: https://www.linkedin.com/in/monmayuri-ray-713164a0/ Timestamps: [00:00] Introduction to Monmayuri Ray [00:57] Mon's background in tech [02:50] MLOps being approached at Gitlab [07:00] CI/CD for MLOPS Definition [07:57] "AI is the dropping cost of machine prediction." [10:25] MLOps and other tools fitting into Gitlab [12:18] "If you want to have an MLOps first strategy, anything you are putting first needs to be substituted with what you had before first. It's really important then to know your priorities." [15:24] Process of how to build [18:16] "Before getting into even understanding the maturity, understand the outcome." [18:45] Challenges in CI/CD for MLOps [19:50]" Automation also empowers collaboration." [24:15] Keeping up [28:33] "I think, the best tools and frameworks are to give people the freedom to be the best version of who they are. As a system, being governed, having that controlled freedom, you can be more Human." [31:20] Resources to suggest in terms of MLOps Education [32:12] Understand the business outcomes of MLOps - Understanding the economics of AI and Machine Learning - Cultural shift   [35:57] Effectiveness of understanding the business outcomes of MLOps to Gitlab customers. [39:42] "It's judgment, action, outcome, and how does this fully impact the overall workflow." [40:00] Enabling vs Keeping the guardrails on [43:26] Best practices

MLOps community meetup #64! Last Wednesday we talked to Christopher  Bergh, CEO, DataKitchen. //Abstract Working on a shared technically difficult problem there will be some things that are important no matter what industry you are in. Whether it's building cars in a factory, using agile or scrum methodology, or productionizing ML models you need a few basics. Chris gives us some of his best practices in the conversation. //Bio Chris Bergh is the CEO and Head Chef at DataKitchen. Chris has more than 25 years of research, software engineering, data analytics, and executive management experience. At various points in his career, he has been a COO, CTO, VP, and Director of Engineering. Chris is a recognized expert on DataOps. He is the co-author of the "DataOps Cookbook” and the “DataOps Manifesto,” and a speaker on DataOps at many industry conferences. //Takeaways Your model is not an island. For success, Data science requires a high level of technical collaboration with other parts of the data organization. //Other Links On-Demand Webinar - Your Model is Not an Island:  Operationalizing Machine Learning at Scale with ModelOps   https://info.datakitchen.io/watch-on-demand-webinar-operationalize-machine-learning-at-scale-with-modelops ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup:  https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Chris on LinkedIn: https://www.linkedin.com/in/chrisbergh/ Timestamps: [00:00] Introduction to Christopher Bergh [02:57] MLOps community in partnership with MLOps World Conference [04:34] Chris' Background [07:59] "When we started with the company, I realized that the problem I have is generalizable to everyone. I'm getting enough there in years and I wanted to remove the amount of pain that other people have." [09:53] DataOps vs MLOps [10:15] "I don't really honestly care what Ops you use, right? Hahaha! Call it your favorite Ops 'cause first of all as an engineer, I want precise definitions. I look at it from a completely odd-ball way so you could call it whatever Ops term you want." [12:45] Best practices of companies [14:16] "When that code runs in production, monitor and check to see if it's right. Absorb it, monitor it because the model could go out of tune. The data going into it could be wrong. The data transformation could break. Shit happens and don't trust your data providers." [19:00] The whole is still greater than its part [20:26] "It is harder to focus on the results than just under a piece of the task. Don't spend too much time on doing the wrong thing." [23:50] DevOps Principles and Agile [27:17] DataOps Manifesto - DataOps is Data Management reborn [27:45] "The 'Ops' term is ending up encompassing the work that you do in addition to the system you build to do the work." [30:45] Standardization   [32:22] "I think that there's a lack of perception of the need to spend time on doing the operations part of the equation." [34:15] Tools as lego blocks [34:49] "Good interphases make good neighbors." [36:23] "Standards can help but they're not the panacea." [36:30] Cultural side - You build it, you own it, you ship it [39:28] Value chain [44:19] Ripple effect of testing [48:03] Google on "One tool to rule them all" [49:50] "Legacy happens if you're gonna live in the real world and not start greenfield projects." [53:47] Starting MLOps in the legacy system

Coffee Sessions #40 with Srivatsan Srinivasan of AIEngineering, Scaling AI in Production.   //Abstract //Bio 20+ years of intense passion for building data-driven applications and products for top financial customers. Srivatsan has been a trusted advisor to a senior-level executive from business and technology, helping them with complex transformation in the data and analytics space. Srivatsan also run a YouTube Channel (AIEngineering) where he talks about data, AI and MLOps. //Takeaways Understand the role and need of MLOps Prioritize MLOps capability Model deployment Importance of K8s //Other Links AI and MLOps free courses - https://github.com/srivatsan88 Youtube channel: bit.ly/AIEngineering --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Srivatsan on LinkedIn: https://www.linkedin.com/in/srivatsan-srinivasan-b8131b/ Timestamps: [00:00] Introduction to Srivatsan Srinivasan [01:41] Background on Youtube AIEngineering [03:17] Tips on learning MLOps and start with the field [06:00] "Focus on your key challenges and that will drive your capability that you need to implement." [06:50] Tips on starting CI/CD [08:46] "Start with DevOps and see what additional capabilities you will require for the Machine Learning aspect of it." [09:24] Staying general in different environments [10:43] "Focus on the core concepts of it. The concepts are similar."    [12:10] Testing systems robustly [20:00] Trends within MLOps space [20:31] "Everybody can fail fast but you need to fail smart because Machine Learning is a huge investment." [23:21] GCP Auto ML [26:54] Deployment [27:06] "It's not only the tools, but it's also the patterns." [29:34] Kubernetes perspective [31:21] Favorite model release strategy [36:22] Annotation, labeling, and concept of ground truth [38:10] Best practices in Architecture and systems design in the context of ML [41:29] "You learn a lot, at the same time the complexity also increases, so work with multiple teams in this process to learn it."   [42:35] "Your speed increases based on the way you envision your architecture." [42:55] Software engineering lifecycle vs machine learning development life cycle [44:55] Youtube experience [45:50] "My focus has always been from intermediate to experts." [46:24] Content creation [47:17] "You cannot do everything in MLOps at one stretch. You have to see what is critical for you." [47:23] "For me, continuous training is not that critical because I don't want to take the freedom out of the data scientists." [48:31] New contents planned [48:40] IoT and Edge Analytics - Predictive maintenance   [50:21] "It's a two-way process. I learn then I teach."

Coffee Sessions #39 with Stephen Galsworthy of Quby, MLOps: A leader's perspective. //Abstract   //Bio Dr. Stephen Galsworthy is a data leader skilled at building high-performing teams and passionate about developing data-powered products with lasting impact on users, businesses, and society.   Most recently he was the Chief Data and Product Officer at Quby, an Amsterdam-based tech company offering data-driven energy services. He oversaw its transformation from a hardware-based business to a digital organization with data and AI at its core. He put in place a central cloud-based data infrastructure and unified analytics platform to collect and take advantage of petabytes of IoT data. His team deployed real-time monitoring and energy insight services for 500k homes across Europe.    Stephen has a Master’s degree and Ph.D. in Mathematics from Oxford University and has been leading data science teams since 2011. //Takeaways MLOps as a process, people, and technological problem.   Experiences from a team working at the forefront of data and AI. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Stephen on LinkedIn: https://www.linkedin.com/in/galsworthy/

MLOps community meetup #63! Last Wednesday we talked to Felipe Campos Penha, Senior Data Scientist, Cargill. //Abstract Can one learn anything useful by creating content online? The usual answer is a sounding YES. But what about live coding an MLOps project on Twitch? Can anything good come out of it? //Bio Felipe Penha creates content about Data Science regularly on the Data Science Bits channel on YouTube and Twitch. He has 8+ years of experience with hands-on data-related work, starting with his doctorate in Astroparticle Physics. His career in the private sector has been devoted to bringing value to various segments of the Food and Beverages Industry through the use of Analytics and Machine Learning. ----------- Connect With Us ✌️-------------    Join our Slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Felipe on LinkedIn: https://www.linkedin.com/in/fcpenha/ Timestamps: [00:00] Introduction to Felipe Campos Penha [01:30] Felipe's background [05:36] Developing models in physics vs developing models for companies [08:07] Felipe's transition from Jupyter Notebook to Operational ML [09:34] "The thought of business basically for customers, they always wanted to see the value and try to roll out more manual work like spreadsheets so they could try out that model on the field." [12:07] Felipe's software engineering development learning [14:10] Catalyst on Youtube and Twitch [18:06] Elements of Twitch [20:02] Non-polished versions of Twitch [21:16] "Twitch was not made for coding, it was for gamers." [26:17] Felipe's audience impact on Twitch [28:02] Logistical pieces [30:43] Words of wisdom on live streaming [30:56] "Don't be afraid to start. There are many streamers that are actually learning from scratch and they are showing the process of learning online. They are learning faster because the help is faster." [33:16] Blog post as other means to Twitch [33:50] "I'm a perfectionist when I'm writing. The shortest it is, the hardest it could get. You want to polish it a lot to make nice figures. I learned a lot but for me, I feel that process is too slow because you're thinking about one subject for a long time trying to polish it while in live streaming, it's very dynamic and fast." [34:25] Twitch affecting Felipe's career [36:36] "Exposing yourself, showing your mistakes, vocalizing your thoughts, I think all of this makes you a better programmer."   [37:12] Getting through a problem [39:41] Recommended streamers that caught Felipe's interest [41:00] Community aspect and importance of Twitch [42:42] Role of community on Twitch [45:16] "Twitch is becoming such a trend that even companies are following."

MLOps community meetup #62! Last Wednesday we talked to Oguzhan Gencoglu, Co-founder & Head of AI, Top Data Science. //Abstract Starting the AI adoption with AI Proof-of-Concepts (PoCs) is the most common choice for most companies. Yet, a significant percentage of AI PoCs do not make it into production whether they were successful or not. Furthermore, running yet another AI PoC follows the law of diminishing returns in various aspects. This talk will revolve around this theme. //Bio Oguzhan "Ouz" Gencoglu is the Co-founder and Head of AI at Top Data Science, a Helsinki-based AI consultancy. With his team, he delivered more than 70 machine learning solutions in numerous industries for the past 5 years. Before that, he used to conduct machine learning research in several countries including the USA, Czech Republic, Turkey, Denmark, and Finland. Oguzhan has given more than 40 talks on machine learning to audiences of various backgrounds. ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup:  https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Oguzhan on LinkedIn: https://www.linkedin.com/in/ogencoglu/ Timestamps: [00:00] Introduction to Oguzhan Gencoglu [00:47] Ouz's background [01:47] Recurring/repetitive problem patterns [03:16] "When you solve a repetitive task in an automatic way, that's Scalability." [04:32] Evolution expected of Machine Learning [05:10] "People are quite confused about the titles and what's worst, those titles don't have a common definition in different companies. If you feel a little bit overwhelmed, that's normal." [08:04] Proof-of-Concepts [10:35] Successful PoCs but not Productionized [16:03] Productionize as soon as possible [16:47] "In your Proof-of-Concepts, it's not only technical, but it's also a mindset." [20:00] Framework of a successful PoCs [24:28] Taking too much on PoCs [28:05] Proof-of-Concepts after Proof-of-Concepts and Proof-of-Concepts hell [31:30] Wholistic view [34:00] Operationalizing PoCs [37:17] "The teams also need to adjust themselves to these new tools, new paradigms, and the different needs of the whole industry."   [37:26] Horror stories [39:54] Open communication tips 43:31] "Open communication should not only be from the technical perspective but also down to the business and strategy perspective." [44:20] Translation tips [44:39] "I believe the most crucial part of today's ML scientists' role is not building a machine learning model but translating a real-life problem into a machine learning problem. It's crucial because it's a scarce talent and skill." [49:30] Realistic budget for small PoCs [50:18] "You need at least 1 month of work of proof of value but that doesn't mean things will go to production." [51:40] Understanding the questions fully [52:55] "That translation skill is the greatest skill to have in this industry because you can't auto ML that or whatever. It stands the test of time because that will be needed all the time."

Coffee Sessions #38 with Adam Sroka of Origami Energy, Organisational Challenges of MLOps. //Abstract Deploying data science solutions into production is challenging for both small and large organizations. From platform and tooling wars to architecture and design pattern trade-offs it can get overwhelming for inexperienced teams. Furthermore, many organizations will only go through the painful discovery process once. Adam will share some of his experiences from consulting and leading data teams to successfully deploying machine learning solutions, highlighting some of the more difficult challenges to overcome. You might not be surprised to hear it’s not all down to the tech. //Bio Dr. Adam Sroka, Head of Machine Learning Engineering at Origami Energy, is an experienced data and AI leader helping organizations unlock value from data by delivering enterprise-scale solutions and building high-performing data and analytics teams from the ground up. Adam shares his thoughts and ideas through public speaking, tech community events, on his blog, and in his podcast. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Adam on LinkedIn: https://www.linkedin.com/in/aesroka/ Timestamps: [00:00] Introduction to Adam Sroka [01:53] Adam's background in tech [08:06] 2 blog posts of Adam: Why So Many Data Scientists Quit Good Jobs at Great Companies and Why You Shouldn’t Hire More Data Scientists [08:31] High turn rate Adam has in the data science role [13:50] Avoiding hiring talents with deficits and coaching people [16:05] "I can't teach you to care about the standard of your core of what you're doing. It's quite hard to teach people charisma. Everything else, you pick up." [16:45] Resume-driven development, the idea of not playing the game, and politics in the workplace. [17:57] "You have to realize, other people, don't have the same experience in the context that you do."   [19:59] Exit, Voice, Loyalty and Neglect Model [22:35] You probably don't need a data scientist [23:40] "Data scientists can do everything slower and more expensively than everyone else but they can do everything and that's the important bit." [27:54] "My success is just driven by who I am as much as what I can do." Vishnu [28:24] Being Candor [30:37] Disconnect between the senior stakeholders and data scientists [32:30] "Before you come out to bring someone in some expensive talent search, engage with the consultancy. Do a four-week PRC, get them to tell you like." [34:18] Educational experiences as a consultant [37:35] Adam's journey into MLOps, productionize ML models when you are a data scientist and tips [43:16] "Beginners can help beginners. Your perspective is really important. The value is not in the content. The value is in your perspective of the content." [45:21] Educating clients on uncertainty [48:34] Decision making process [52:32] Organizational problems [53:43] "All models are wrong, but some are useful." George Box

MLOps community meetup #61! Last Wednesday we talked to Lex Beattie, Michael Munn, and Mike Moran. //Abstract We started out talking about some of the main bottlenecks they have encountered over the years of trying to push data products into production environments. Then things started to heat up as we dove into the topic of monitoring ML and inevitably the word explainability started being thrown around. Turns out Lex is currently doing a Ph.D. on the subject so there was much to talk about. We had to ask if explainability is now table-stakes when it comes to monitoring solutions on the market? The short answer from the team. Yes!   Please excuse the bit of sound trouble we had with google mike at the beginning. //Bio Lex Beattie - ML Engagement Lead, Spotify In the last year, Lex has helped over 40 different teams across Spotify understand ML best practices, productionize ML workflows and implement impactful ML in their products. Lex is also a Ph.D. candidate at the University of Oklahoma, focusing on feature importance and interpretability in deep neural networks. Beyond her passion for all things ML, she enjoys exploring the great outdoors in Montana with her German Wirehaired Pointer, Bridger. Michael Munn - ML Solutions Engineer, Google Michael is an ML Solutions Engineer with Google Cloud and Google's Advanced Solutions Lab. In his role, he works with customers to build and deploy end-to-end ML solutions with Google Cloud. Within the Advanced Solutions Lab, he teaches these skills to customers. Mike Moran - Principal Engineer, Skyscanner Mike has worked across many dimensions; in large/tiny companies, back-end/front-end, with many languages, and as a sys-admin /engineer/manager. Mike has a healthy skepticism for most things and likes solving problems through applying System thinking. Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Lex on LinkedIn: https://www.linkedin.com/in/lexbeattie/ Connect with Michael on LinkedIn: https://www.linkedin.com/in/munnm/ Connect with Mike on LinkedIn: https://www.linkedin.com/in/mrmikemoran/ Timestamps: [00:00] Introduction to Lex, Michael, & Mike [02:46] Common roadblocks [05:25] Consolidating knowledge [07:02] Bottlenecks on failures [09:58] Don't go on a detour [12:22] Bringin on complexity signs [19:33] Explainable AI [21:34] "There are different ways to approach Explainable AI. It starts to get more complicated when you start working with more complicated models." Lex [24:43] "If there are a lot of disparate sources out there about Explainability, I'd found myself hunting down various resources to simplify it for customers I'd worked with." Michael [26:46] "Being clear about who you're explaining it to because in our context, sometimes the organization needs to explain it to a regulator." Mike [28:04] Monitoring solution [31:00] ML Canvas [33:24] Explainable AI Resources [34:48] Explainable Predictions by Michael [36:48] Purpose of Explainable Model [39:40] Work in the same language [42:46] Use of War Stories [49:11] Hot seat! [49:15] Mike - Skyscanner pricing [50:30] Lex - Spotify recommendation sudden stop [51:35] Michael - NLP models on emails