MLOps Coffee Sessions #76 with Mohamed Elgendy, Build a Culture of ML Testing and Model Quality. // Abstract Machine learning engineers and data scientists spend most of their time testing and validating their models’ performance. But as machine learning products become more integral to our daily lives, the importance of rigorously testing model behavior will only increase. Current ML evaluation techniques are falling short in their attempts to describe the full picture of model performance. Evaluating ML models by only using global metrics (like accuracy or F1 score) produces a low-resolution picture of a model’s performance and fails to describe the model performance across types of cases, attributes, scenarios. It is rapidly becoming vital for ML teams to have a full understanding of when and how their models fail and to track these cases across different model versions to be able to identify regression. We’ve seen great results from teams implementing unit and functional testing techniques in their model testing. In this talk, we’ll cover why systematic unit testing is important and how to effectively test ML system behavior. // Bio Mohamed is the Co-founder & CEO of Kolena and the author of the book “Deep Learning for Vision Systems”. Previously, he built and managed AI/ML organizations at Amazon, Twilio, Rakuten, and Synapse. Mohamed regularly speaks at AI conferences like Amazon's DevCon, O'Reilly's AI conference, and Google's I/O. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletter and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Adam on LinkedIn: https://www.linkedin.com/in/aesroka/ Connect with Mohamed on LinkedIn: https://www.linkedin.com/in/moelgendy/

MLOps Coffee Sessions #75 with Shreya Shankar, Towards Observability for ML Pipelines. // Abstract Achieving observability in ML pipelines is a mess right now. We are tracking thousands of means, percentiles, and KL divergences of features and outputs in a haphazard attempt to figure out when and how to retrain models. In this session, we break down current unsuccessful approaches and discuss the path towards effectively maintaining ML models in production. Along the way, we introduce mltrace -- a preliminary open source project striving towards "bolt-on" observability in ML pipelines. // Bio Shreya Shankar is a computer scientist living in the Bay Area. She's interested in building systems to operationalize machine learning workflows. Shreya's research focus is on end-to-end observability for ML systems, particularly in the context of heterogeneous stacks of tools. Currently, Shreya is doing her Ph.D. in the RISE lab at UC Berkeley. Previously, she was the first ML engineer at Viaduct, did research at Google Brain, and completed her BS and MS in computer science at Stanford University. // Related Links Shreya Shankar's blogposts: https://www.shreya-shankar.com/ Shreya Shankar's Podcasts: https://www.listennotes.com/top-episodes/shreya-shankar/ The deployment phase of machine learning by Benedict Evans: https://www.ben-evans.com/benedictevans/2019/10/4/machine-learning-deployment Shreya Shrankar's mltrace blogpost: https://www.shreya-shankar.com/introducing-mltrace/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Shreya on LinkedIn: https://www.linkedin.com/in/shrshnk Timestamps: [00:00] Introduction to Shreya Shankar [01:12] Shreya's background   [03:22] Contrast in scale influence [05:28] Embedding ML and building machine learning infused products [07:26] Management structure and professional incentive [08:25] Organizational side of MLOps retros [10:15] Tooling implementations [12:00] Structured rational investment hardships [13:17] Working at a start-up [14:02] Academic work and entrepreneurial ambitions   [16:00] ML Monitoring Observability interest [17:14] Where to get started [20:47] Realization while at Viaduct [23:30] Preventing alert fatigue   [27:04] Tooling bridging the gap [30:40] Juncture at overall MLOps ecosystem [33:58] The deployment phase of machine learning - it's the new SQL by Benedict Evans [35:30] Model monitoring [36:16] mltrace [38:28] Introducing mltrace blog post series [41:25] Tips to our content creators/writers [43:47] Monitoring through the lens of the database [47:37] Advice about picking up ML engineering and ML systems development in 2022 [49:36] Database low down the stack [50:51] Most excited about 2022 [52:13] What MLOps space/ecosystem should change? [53:21] Funding has changed the incentives around innovation   [54:52] Competition in million-dollar rounds [55:25] Starting a company [56:30] Wrap up

MLOps Coffee Sessions #74 with Jesse Johnson, Scaling Biotech. // Abstract Scaling a biotech research platform requires managing organization complexity - teams, functions, projects - rather than just the traditional volume, velocity, and variety. By examining the processes and experiments that drive the platform, you can focus your work where it matters the most by finding the ideal balance for each type of experiment along with a number of common trade-offs. // Bio Jesse Johnson is head of Data Science and Data Engineering at Dewpoint Therapeutics, an R&D-stage biotech startup. His interest in exploring complex systems, understanding what makes them tick, then using this understanding to improve and scale them led him from academic mathematics, into software engineering (Google, Verily Life Sciences), and then to Biotech (Sanofi, Cellarity, Dewpoint). His goal is to identify ways to scale biotech research through better software and organizational design. // Related Links Jessie's blogposts: scalingbiotech.com --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Jesse on LinkedIn: https://www.linkedin.com/in/jesse-johnson-51619a7/ Timestamps: [00:00] Introduction to Jesse Johnson [05:10] Jesse's background [05:52] Biotech environments [06:31] Jesse's background in Biotech companies [09:21] Jesse's journey from academic to software engineering [12:20] Transition from primary output insights/research into writing code [14:54] Actual hands-on use case in practice [19:19] Jesse's career trajectory [23:57] Where we're at state-of-the-art data engineering and its outstanding challenges [26:50] Dewpoint's data and machine learning challenges and tooling [29:04] Dewpoint's team structure [30:20] Jesse being the VP of Data Science and Data Engineering [33:24] New biotech data makes it hard to design a data platform [35:35] Changes in how biotech data is viewed [35:54] Experiment data output [40:19] Solving challenges in structuring real-world context into interpretable data fields [44:16] Maturity between the current data engineering and MLOps tooling space   [47:31] Achieving a blogpost mission in 2022 [49:50] Wrap up

MLOps Coffee Sessions #73 with Breno Costa and Matheus Frata, On Structuring an ML Platform 1 Pizza Team. // Abstract Breno and Matheus were part of an organizational change at Neoway in recent years. With the creation of cross-functional and platform teams in order to improve the value stream generated by these. They share their experience in creating a machine learning platform team. The challenges they faced along the way, how they approached using product thinking and the results achieved so far. // Bio Matheus Frata Matheus is an Electronics Engineer that got into Data Science by accident! During his graduation, Matheus joined Neoway as a Data Scientist, but during that time he saw a lot of problems that were related to engineers! This was Matheus' beginning with MLOPS.  Today, Matheus works as a Machine Learning Engineer helping their Data Scientists to FLY!!! Breno Costa Breno uses his mixed background in Computer Science and Mathematical Modeling to design and develop ML-based software products. A brief period as an entrepreneur gives a different look at how to approach problems and generate more value. He has worked at Neoway for three years and currently works as a machine learning engineer on the Platform team. // Related links https://mlops.community/building-neoways-ml-platform-with-a-team-first-approach-and-product-thinking/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletter and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Breno on LinkedIn: https://www.linkedin.com/in/breno-c-costa/ Connect with Matheus on LinkedIn: https://www.linkedin.com/in/matheus-frata/ Timestamps: [00:00] Introduction to Breno Costa & Matheus Frata [02:08] Breno's background in Neoway [03:23] What does Neoway do and Matheus' background in Neoway [05:43] Organizational structure of Neoway [07:31] Concept of redesign [10:47] Getting the structure right as a priority [15:26] Designing the teams [20:28] Three different ways of setting up the cells interaction [23:58] Platform differences [25:33] Technical components before redesigning and organizational overhauling [31:50] Supporting platform teams [33:23] Settling tech stack managing technical needs [42:10] Building internal tools [50:10] Wrap up

MLOps Coffee Sessions #72 with Vishnu Rachakonda and Demetrios Brinkmann, 2021 MLOps Year in Review. // Abstract Vishnu and Demetrios sit down to reflect on some of the biggest news and learnings from 2021 from the biggest funding rounds to best insights. The two finish out the chat by talking about what to expect in 2022. // Bio Demetrios Brinkmann At the moment Demetrios is immersing himself in Machine Learning by interviewing experts from around the world in the weekly MLOps.community meetups. Demetrios is constantly learning and engaging in new activities to get uncomfortable and learn from his mistakes. He tries to bring creativity into every aspect of his life, whether that be analyzing the best paths forward, overcoming obstacles, or building lego houses with his daughter. Vishnu Rachakonda Vishnu is the operations lead for the MLOps Community and co-hosts the MLOps Coffee Sessions podcast. He is a machine learning engineer at Tesseract Health, a 4Catalyzer company focused on retinal imaging. In this role, he builds machine learning models for clinical workflow augmentation and diagnostics in on-device and cloud use cases. Since studying bioengineering at Penn, Vishnu has been actively working in the fields of computational biomedicine and MLOps. In his spare time, Vishnu enjoys suspending all logic to watch Indian action movies, playing chess, and writing. //Related links Dr. Angela Duckworth's book on Grit featuring Cody Coleman: https://www.scribd.com/book/311311935/Grit?utm_medium=cpc&utm_source=google_search&utm_campaign=3Q_Google_DSA_NB_RoW&utm_device=c&gclid=CjwKCAjw0a-SBhBkEiwApljU0klle1jhwhK1hrCtdOzR2NIqNu1Y1D9kkGhFg5k2jvo5cCft7UOCqBoCsigQAvD_BwE You don't need Kafka Vicki Boykis blog:   https://vicki.substack.com/p/you-dont-need-kafka?s=r The Informed Company book:   https://www.amazon.com/Informed-Company-Cloud-Based-Explore-Understand/dp/1119748003 --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Timestamps: [01:03] Campfire [01:31] What are you most interested in learning about? [02:00] Learning about serving models [03:42] 2021 MLOps Community growth [04:22] Engaging people coming back to the community [05:41] Consistently high-quality interactions [07:07] Vishnu's 2021 favorite moment in the Coffee Sessions [10:05] Dr. Angela Duckworth's book on Grit featuring Cody Coleman [11:43] Biggest surprise over the year for Demetrios [13:48] You don't need Kafka Vicki Boykis blog [16:26] What excites Vishnu in 2022 [18:04] The Informed Company book [20:48] What excites Demetrios in 2022 [26:28] News and blurbs   [33:25] Spinouts [34:30] Last year's cool events [36:02] Community progress [38:47] Community highlights [41:28] New projects [44:26] A controversial blogpost [46:03] Milestones [46:57] Lessons [50:00] Shout out and thanks to our sponsors!

Loblaws is one of Canada’s largest grocery store chains, Mefta's team at Loblaw Digital runs several ML systems such as search, recommendations, inventory, and labor prediction on production. In this conversation, he shares his experience setting up their ML platform on GCP using Vertex AI and open-source tools.  The goal of this platform is to help all the data science teams within their organization to take ML projects from EDA to production rapidly while ensuring end-to-end tracking of these ML pipelines. We also talk about our overall platform architecture and how the MLOps tools fit into the end-to-end ML pipeline. //Bio Mefta Sadat is a Senior ML Engineer at Loblaw Digital. He has been here for over three years building the Data Engineering and Machine Learning platform. He focuses on productionizing ML services, tools, and data pipelines. Previously Mefta worked at a Toronto-based Video Streaming Company and designed and built the recommendation system for the Zoneify App from scratch. He received his MSc in Computer Science from Ryerson University focusing on research to mitigate risk in Software Engineering using ML. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletter and more: https://mlops.community/ Timestamps: [00:00] Introduction to Mefta Sadat [01:04] Mefta's background   [02:45] Mefta's journey in ML Engineering [04:19] Use cases of Machine Learning at Loblaws [06:00] Loblaws' team operation [07:37] Number of people in the team and number of users in the platform [08:40] Software engineering process [10:47] Data platform vs ML platform [13:10] Timeline leveraging machine learning in Loblaws products and business [15:01] Transition from legacy systems to the cloud [16:47] Recommendation System use case - Legacy Style Stack and its impact on the business [21:01] Biggest challenges and pain points [24:31] Choices of tools to use   [27:31] Dealing with data access [30:39] The good, the bad, and the ugly   [32:48] Setting up alerts on image classification models [33:53] Productionizing ML passion [36:00] Post-deployment monitoring of recommendation systems [37:47] Wrap up

MLOps Coffee Sessions #70 with Reah Miyara, 2022 Predictions for MLOps and the Industry. // Abstract MLOps has moved fast in the last year. What will 2022 be like in the MLOps ecosystem? Raeh from Arize AI comes on to talk to us about what he expects for the new year.   Arize is kindly offering 20 free subscriptions to their tool. No marketing BS these are design partners. First come first serve https://arize.com/mlops-signup/! // Bio Reah Miyara is a Senior Product Manager at Arize AI, a leading ML monitoring and observability platform counted on by top enterprises to track billions of predictions daily. Reah joins Arize from Google AI, where he led product strategy for the Algorithms and Optimization organization. His experience as a team and product leader is extensive, touching a broad cross-section of the AI technology landscape. Reah played pivotal roles in ML and AI initiatives at Google, IBM Watson, Intuit, and NASA Jet Propulsion Laboratory and his work have directly contributed to many important innovations and successes that have moved the broader industry forward. Reah also co-led the Google Research Responsible AI initiative, confronting the risks of AI being misused and taking steps to minimize AI’s negative influence on the world. // Relevant Links Subscription - https://arize.com/mlops-signup/ https://arize.com/blog/welcome-to-arize-reah/ https://arize.com/blog/best-practices-in-ml-observability-for-monitoring-mitigating-and-preventing-fraud/ https://www.reah.me/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletter and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Reah on LinkedIn: https://www.linkedin.com/in/reah/ Timestamps: [00:00] Introduction to Reah Miyara [01:57] Wrong predictions [03:41] Real predictions for 2022 [04:00] One: AI fairness and bias issues will get worse before they get better. [07:27] Two: Enterprises will stop shipping AI blinds [10:51] Three: The Citizen Data Scientist will rise [17:07] Four: The ML infrastructure ecosystem will get more complex [22:28] Five: Unleash the power of unstructured data [26:34] Six: Robustness of ML Models against changes [33:18] We want to have the best ML monitoring and observability tool out there. [34:07] Demetrios' prediction: More talks about laws and regulations will happen but nothing will actually get done. [35:27] Wrap up

MLOps Coffee Sessions #69 with James Lamb, Building for Small Data Science Teams co-hosted by Adam Sroka. // Abstract In this conversation, James shares some hard-won lessons on how to effectively use technology to create applications powered by machine learning models. James also talks about how making the "right" architecture decisions is as much about org structure and hiring plans as it is about technological features. // Bio James Lamb is a machine learning engineer at SpotHero, a Chicago-based parking marketplace company. He is a maintainer of LightGBM, a popular machine learning framework from Microsoft Research, and has made many contributions to other open-source data science projects, including XGBoost and prefect. Prior to joining SpotHero, he worked on a managed Dask + Jupyter + Prefect service at Saturn Cloud and as an Industrial IoT Data Scientist at AWS and Uptake. Outside of work, he enjoys going to hip hop shows, watching the Celtics / Red Sox, and watching reality TV (he wouldn’t object to being called “Bravo Trash”). // Relevant Links James keeps track of conference and meetup talks he has given at https://github.com/jameslamb/talks#gallery. The audience for this podcast might be most interested in "Scaling LightGBM with Python and Dask" and "How Distributed LightGBM on Dask Works". --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Adam on LinkedIn: https://www.linkedin.com/in/aesroka/ Connect with James on LinkedIn: https://www.linkedin.com/in/jameslamb1/ Timestamps: [00:00] Introduction to James Lamb [01:11] James' background in the machine learning space [03:24] LightGBM [09:56] Community behind LightGBM [13:36] Background of James in SpotHero [20:06] Experience in Maturity Models [22:40] Bottlenecks of tradeoffs between speed and confidence [28:28] Tools to be excited about [31:46] To code your own that's already out there [36:33] Building design decisions   [39:36] Risk of the unicorn [42:44] Cross team empathy [47:18] Proudest technical accomplishment and/or biggest frustration less proud of lessons learned [50:53] SpotHero is hiring! [51:20] Wrap up [51:53] Please like, subscribe, and you can leave a review!

MLOps Coffee Sessions #68 with Chris Albon, Wikimedia MLOps co-hosted by Neal Lathia. // Abstract // Bio Chris spent over a decade applying statistical learning, artificial intelligence, and software engineering to political, social, and humanitarian efforts. He is the Director of Machine Learning at the Wikimedia Foundation. Previously, Chris was the Director of Data Science at Devoted Health, Director of Data Science at the Kenyan startup BRCK, cofounded the AI startup Yonder, created the data science podcast Partially Derivative, was the Director of Data Science at the humanitarian non-profit Ushahidi, and was the director of the low-resource technology governance project at FrontlineSMS. Chris also wrote Machine Learning For Python Cookbook (O’Reilly 2018) and created Machine Learning Flashcards. Chris earned a Ph.D. in Political Science from the University of California, Davis researching the quantitative impact of civil wars on health care systems. He earned a B.A. from the University of Miami, where he triple majored in political science, international studies, and religious studies. // Relevant Links --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, Feature Store, Machine Learning Monitoring and Blogs: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Neal on LinkedIn: https://www.linkedin.com/in/nlathia/ Connect with Chris on LinkedIn: https://www.linkedin.com/in/chrisralbon/ Timestamps: [00:00] Introduction to Chris Albon [00:28] Do you sleep? :-)   [02:43] ML at Wikimedia [09:27] Wikimedia workflow [15:00] Creating a repeatable process [19:11] Wikimedia element team size [20:47] Wikimedia workflow and hardware [23:56] Evaluating open source [29:20] Lacking in ML source tooling [33:11] Wikimedia's separate data platform [38:14] Abstractions [41:50] Experimentation aspect of getting models into production [44:05] Stack of Abstraction in ML [47:16] Chris' proudest model [49:10] How Wikimedia work with communities [55:24] Large language models [1:02:16] Beautiful vision [1:03:23] Wrap up

MLOps Coffee Sessions #67 with John Crousse, ML Stepping Stones: Challenges & Opportunities for Companies co-hosted by Adam Sroka. // Abstract In this coffee session, John shares his observations after working with multiple companies which were in the process of scaling up their ML capabilities. John's observations are mostly around changes in practices, successes, failures, and bottlenecks identified when building ML products and teams from scratch. John shares a few thoughts on building long-term products vs short-term projects, on the important non-ML components, and the most common missing pieces he sees in today's ecosystem. John also elaborates on how those challenges and solutions can differ for different company sizes. // Bio John always liked CS/ML/AI but wasn't such a hot topic back then. He found opportunities to work on models in the Financial industry as a consultant from 2007 to 2017 then he went freelance to move outside of the financial industry, and focus on AI/ML.   John likes to do things efficiently, and MLOps is the bottleneck, so he ended up spending more time on MLOPs than models lately.   John finished his CS degree in 2007.   // Relevant Links   --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, Feature Store, Machine Learning Monitoring and Blogs: https://mlops.community/   Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Adam on LinkedIn: https://www.linkedin.com/in/aesroka Connect with John on LinkedIn: https://www.linkedin.com/in/john-crousse-31219b9 Timestamps: [00:00] Introduction to John Crousse [01:11] Main trends in Machine Learning [03:07] Symptoms of Machine Learning product [05:05] Proper product with limited resources [08:52] Going into production mindsets [11:22] Bottlenecks and challenges [14:55] Business case for Machine Learning or MLOps in small organizations [17:04] Gathering feedbacks best suited to product owners [19:14] More substantial role   [20:11] Data factory [24:03] Delivery patterns or tech stacks [26:06] Bottleneck metrics [27:28] Concept of evaluation store [32:18] The biggest gap to bridge [34:42] Hindrance to people's development [35:23] "The last mile of the machine learning projects" [36:40] MLOps assessment survey [40:10] Who owns the product and path to recommend [41:34] Datamesh community [44:41] Tips on balancing between pure autonomy [45:58] Wrap up