MLOps.community

MLOps community meetup #48! Last Wednesday, we talked to Manoj  Agarwal, Software Architect at Salesforce. // Abstract: Serving machine learning models is a scalability challenge at many companies. Most applications require a small number of machine learning models (often < 100) to serve predictions. On the other hand, cloud platforms that support model serving, though they support hundreds of thousands of models, provision separate hardware for different customers. Salesforce has a unique challenge that only very few companies deal with; Salesforce needs to run hundreds of thousands of models sharing the underlying infrastructure for multiple tenants for cost-effectiveness. // Takeaways: This talk explains Salesforce hosts hundreds of thousands of models on a multi-tenant infrastructure to support low-latency predictions. // Bio: Manoj Agarwal is a Software Architect in the Einstein Platform team at Salesforce. Salesforce Einstein was released back in 2016, integrated with all the major Salesforce clouds. Fast forward to today and Einstein is delivering 80+ billion predictions across Sales, Service, Marketing & Commerce Clouds per day. //Relevant Links https://engineering.salesforce.com/flow-scheduling-for-the-einstein-ml-platform-b11ec4f74f97 https://engineering.salesforce.com/ml-lake-building-salesforces-data-platform-for-machine-learning-228c30e21f16 ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup:  https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Manoj on LinkedIn: https://www.linkedin.com/in/agarwalmk/ Timestamps: [00:00] Happy birthday Manoj! [00:41] Salesforce blog post about Einstein and ML Infrastructure [02:55] Intro to Serving Large Number of Models with Low Latency [03:34] Manoj' background [04:22] Machine Learning Engineering: 99% engineering + 1% machine learning - Alexey Gregorev on Twitter [04:37] Salesforce Einstein [06:42] Machine Learning: Big Picture [07:05] Feature Engineering [07:30] Model Training [08:53] Model Serving Requirements [13:01] Do you standardize on how models are packaged in order to be served and if so, what standards Salesforce require and enforce from model packaging? [14:29] Support Multiple Frameworks   [16:16] Is it easy to just throw a software library in there? [27:06] Along with that metadata, can you breakdown how that goes?   [28:27] Low Latency [32:30] Model Sharding with Replication [33:58] What would you do to speed up transformation code run before scoring? [35:55] Model Serving Scaling [37:06] Noisy Neighbor: Shuffle Sharding [39:29] If all the Salesforce Models can be categorized into different model type, based on what they provide, what would be some of the big categories be and what's the biggest? [46:27] Retraining of the Model: Does that deal with your team or is that distributed out and your team deals mainly this kind of engineering and then another team deal with more machine learning concepts of it? [50:13] How do you ensure different models created by different teams for data scientists expose the same data in order to be analyzed? [52:08] Are you using Kubernetes or is it another registration engine? [53:03] How is it ensured that different models expose the same information?

**Private data, Data Science friendly** Data Scientists are always eager to get their hands on more data, in particular, if that data has any value that can be extracted. Nevertheless, in real-world situations, data does not exist in the abundance that we thought existed, in other situations, the data might exist, but not possible to share it with different entities due to privacy concerns, which makes the work of data scientists not only hard, but sometimes even impossible. // Abstract: In the last episode of this series, we've decided to bring not one, but two guests to tells us how Synthetic data can unlock the use of data for Data Science teams whenever privacy concerns are a reality.  Jean-François Rajotte, Researcher and Resident data Scientist at the University of Columbia and Sumit Mukherjee, Senior Applied Scientist at Microsoft's AI for Good, bring us into more detail their expertise not only, in Synthetic data generation, but in it's mind blowing combination with Federated Learning to take the healthcare sector into the next level of AI adoption. //Other links to check on Jean-François Rajotte: https://venturebeat.com/2021/01/20/microsofts-felicia-taps-ai-to-enable-health-providers-to-share-data-anonymously/ https://dsi.ubc.ca/ https://leap-project.github.io/ //Other links to check on Sumit Mukherjee: www.sumitmukherjee.com (Sumit research) https://arxiv.org/abs/2101.07235 https://arxiv.org/pdf/2009.05683.pdf https://github.com/microsoft/privGAN (PrivGan) //Final thoughts Feel free to drop some questions into our slack channel (https://go.mlops.community/slack)  Watch some of the other podcast episodes and old meetups on the channel: https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ ----------- Connect With Us ✌️------------- Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Fabiana on LinkedIn: https://www.linkedin.com/in/fabiana-clemente/ Connect with Jean-François on LinkedIn: https://www.linkedin.com/in/jfraj/ Connect with Sumit on LinkedIn: https://www.linkedin.com/in/sumitmukherjee2/

Coffee Sessions #26 with Vishnu Rachakonda of Tesseract Health, Daniel Galinkin of iFood, Matias Dominguez of Rappi & Simarpal Khaira of Intuit, Feature Store Master Class. //Bio Vishnu Rachakonda Machine Learning Engineer at Tesseract Health. Coffee sessions co-host but this time his role is one of the all-stars guest speakers. Daniel Galinkin One of the co-founders of Hekima, one of the first companies in Brazil to work with big data and data science, with over 10 years of experience in the field. At Hekima, Daniel was amongst the people responsible for dealing with infrastructure and scalability challenges. After iFood acquired Hekima, he became the ML Platform Tech Lead for iFood. Matias Dominguez   A 29-year-old living in Buenos Aires, past 4.5 years working on fraud prevention.  Previously at MercadoLibre and other random smaller consulting shops. Simarpal Khaira Simarpal is the product manager driving product strategy for Feature Management and Machine Learning tools at Intuit. Prior to Intuit, he was at Ayasdi, a machine learning startup, leading product efforts for machine learning solutions in the financial services space. Before that, he worked at Adobe as a product manager for Audience Manager, a data management platform for digital marketing. --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Daniel on LinkedIn: https://www.linkedin.com/in/danielgalinkin/ Connect with Matias on LinkedIn: https://www.linkedin.com/in/mndominguez/ Connect with Simarpal on LinkedIn: https://www.linkedin.com/in/simarpal-khaira-6318959/  Timestamps: [00:00] Introduction to guest speakers. [00:33] Vishnu Rachakonda Background [01:00] Guest speakers' Background [03:13] Are Feature Stores for everyone? [04:02] Guest speakers' Feature Store background [17:09] How do you go about gathering requirements for a Feature Store and customize it? [17:34] Guest speakers' process for Feature Store [31:14] What solution are we actually trying to build? [36:42] How do you ensure consistency in your transformation logic and in your process generating features? [43:39] In terms of versioning that transformation logic and knowledge that goes into creating Feature Stores and allowing them to be reusable and consistent, how are you going to grapple with that? [48:06] How do you bake in best practices into the services that you offer? [49:34] "It's too possible for you to do something wrong. You have to specify that wrong thing. That makes it harder to do that wrong thing." Daniel [51:54] "It starts with changing the mindset. Making people getting the habit of what is the value here. Then you are producing features for consumers because tomorrow you could become a consumer. Write it in a way as you want to consume somebody's feature." Simar [56:51] "As part of that process, it should come with everyone's best practices to actually improve all features" Matias

MLOps community meetup #47! Last Wednesday, we talked to Adrià Romero, Founder and Instructor at ProductizeML. // Abstract: In this talk, we tackled:   - Motivations and mission behind ProductizeML. - Common friction points and miscommunication between technical and management/product teams, and how to bridge these gaps.   - How to define ML product roadmaps, (and more importantly, how to get it signed off by all your team). - Best practices when managing the end-to-end ML lifecycle. / Takeaways: - Self-study guide that reviews the end-to-end ML lifecycle starting with some ML theory, data access and management, MLOps, and how to wrap up all these pieces in a viable but still lovable product.   - Free and collaborative self-study guide built by professionals with experience on different stages from the ML lifecycle. // Bio: Adrià is an AI, ML, and product enthusiast with more than 4 years of professional experience on his mission to empower society with data and AI-driven solutions. Born and raised in the beautiful and sunny Barcelona, he began his journey in the AI field as an applied researcher at the Florida Atlantic University, where he published some of the first deep learning works in the healthcare sector. Attracted by the idea of deploying these ideas to the real world, he then joined Triage, a healthcare startup building healthcare solutions powered by AI, such a smartphone app able to detect over 500 skin diseases from a picture. During this time, he has given multiple talks at conferences, hospitals, and institutions such as Novartis and Google. Previously, he interned at Huawei, Schneider Electric, and Insight Center for Data Analytics. Early this year, he started crafting ProductizeML, An Instruction and Interactive Guide for Teams Building Machine Learning Products where he and a team of AI & product specialists carefully prepare content to assist on the end-to-end ML lifecycle. ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup:  https://go.mlops.community/register   Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Adria on LinkedIn: https://www.linkedin.com/in/adriaromero/ References mentioned on this episode: https://en.wikipedia.org/wiki/ImageNet https://twitter.com/productizeML https://course.productize.ml/ https://github.com/ProductizeML/gitbook https://adria756514.typeform.com/to/V4BDqjYA - Newsletter Signup https://www.buymeacoffee.com/ Timestamps: [00:00] Introduction to Adrià Romero   [00:32] How did you get into tech? [02:16] ImagiNet Project (Visual Recognition Challenge) [06:49] Visual Recognition with Skin Lesions [07:05] Fundamental vs Applied Research (Academia experience) [08:44] Motivation for technology [14:55] Transition to ProductizeML [19:09] ProductizeML Context [23:50] What was its that made you think that Education is probably more powerful? [24:21] ProductizeML Objective [26:55] Ethics: Do you want to put that in there later? [30:12] ProductizeML Content Format and Tools [34:07] ProductizeML Catalogue [39:28] ProductizeML Audience Target [42:54] "Buy me a coffee" platform [48:29] Do you ever foresee with the educational being more vertical-specific?

The revolution of Federated Learning - And we're back with another episode of the podcast When Machine Learning meets Privacy! For the episode #8 we've invited Ramen Dutta, a member of our community and founder of TensoAI. // Abstract: In this episode,  Ramen explain us the concept behind Federated Learning, all the amazing benefits and it's applications in different industries, particularly in agriculture. It's all about not centralizing the data, sound awkward? Just listen to the episode. //Other links to check on Ramen: https://www.linkedin.com/in/tensoai/ https://www.tensoai.com/ https://twitter.com/tensoAI //Final thoughts Feel free to drop some questions into our slack channel (https://go.mlops.community/slack)  Watch some of the other podcast episodes and old meetups on the channel: https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ ----------- Connect With Us ✌️------------- Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Fabiana on LinkedIn: https://www.linkedin.com/in/fabiana-clemente/ Connect with Ramen on LinkedIn: https://www.linkedin.com/in/tensoai/

Coffee Sessions #25 with Marian Ignev of CloudStrap.io & SashiDo.io, Most Underrated MLOps Topics. //Bio Marian a passionate entrepreneur, backend dude & visionary. These are the three main things described to Marian very well: Marian's everyday routines include making things happen and motivating people to work hard and learn all the time because I think success is a marathon, not just a sprint! Marian loves to communicate with bright creative minds who want to change things.   His favorite professional topics are backend stuff, ops, infra, IoT, AI, Startups, Entrepreneurship.   In his free time, he loves to cook for my family. --------------- ✌️Connect With Us ✌️ -------------  Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Marian on LinkedIn: https://www.linkedin.com/in/mignev/ Timestamps: [00:00] Introduction to Marian Ignev [01:57] Marian's background [10:06] Who do you need and what should they be doing? [18:05] How are you solving problems at your company? [27:22] What are your thoughts around ML tooling? Why that hasn't happened yet and why it will change? [33:16] You can't actually figure out what's the main focus of ML tooling services.   [34:14] "Start small and start simple to focus only on a small problem that you can bring more than the others." [37:14] How are you making the case for how standardization to occur in your initial MLOps? [38:08] "If you're doing a mistake somewhere, do it everywhere because it will be very easy to find and replace it after that."    [41:50] How do you model monitoring? [47:19] How would you recommend people to get started? [49:00] Ecosystem of Machine Learning in Eastern Europe Other links you can check on Marian: https://www.sashido.io/ https://www.cloudstrap.io/ https://twitter.com/mignevm.ignev.net/blog/  (Blog) m.ignev.net/  (Personal Website) fridaycode.net/  (FridayCode)

MLOps community meetup #46! Last Wednesday, we talked to Hendrik Brackmann, Director of Data Science and Analytics at Tide. // Abstract: Tide is a U.K.-based FinTech startup with offices in London, Sofia, and Hyderabad. It is one of the first, and the largest business banking platform in the UK, with over 150,000 SME members. As of 2019, one of Tide’s main focuses is to be data-driven. This resulted in the forming of a Data Science and Analytics Team with Hendrik Brackmann at its head. Let's witness Hendrik's personal anecdotes in this episode! // Bio: After studying probability theory at the University of Oxford, Hendrik joined MarketFinance, an SME lender, in order to develop their risk models. Following multiple years of learning, he joined Finiata, a Polish and German lender in order to build out their data science function. Not only did he succeed in improving the risk metrics of the company he also learnt to manage a different department as interim Head of Marketing. Hendrik's job as Director of Data Science and Analytics at business bank Tide is to oversee data engineering, data science, insights and analytics and data governance functions of Tide. // Final thoughts Please feel free to drop some questions you may have beforehand into our slack channel (https://go.mlops.community/slack) Watch some old meetups on our youtube channel: https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ ----------- Connect With Us ✌️-------------    Join our Slack community:  https://go.mlops.community/slack Follow us on Twitter:  @mlopscommunity Sign up for the next meetup:  https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Hendrik on LinkedIn: https://www.linkedin.com/in/hendrik-brackmann-b2b5477a/ Timestamps: [00:00] Introduction to Hendrik Brackmann [01:54] Hendrik's background into tech [03:22] First Phase of the three epic journeys of Hendrik [08:05] Were there some hiccups you were running into as you're trying to make things better?   [10:50] Any other learnings that you got from that job that you want to pass along to us? [11:50] You were doing all batch at that point, right? [12:35] Phase 2: of Hendrik's epic journey [15:11] Did you eventually cut down at the time that it took? [15:50] Breakdown of Transformation terminologies and its importance [19:03] What are some things that you would never do again? [20:32] How did you see things more clearly? [22:30] Phase 3: Moving on to Tide [24:46] Have you only worked with teams with one programming language? [30:47] Did you try to open-source solutions or did you just go right out to buy it? [33:12] What is real-time for you? How much latency is there? How much time do you need? [37:18] What stage did you realize to get the feature store?   [40:09] What would you recommend from a maturity standpoint to get a feature store? [41:20] Can you summarize some of the greatest problems that the feature stores solve for you?    [42:22] What problems does a feature store introduces if any? [44:39] Where do the model and the feature start from the perspective of a system in the engineering? [49:15] You need a good data management in feature stores [50:21] Have you ever used or built any feature stores that explicitly handle units and does dimensional analysis on derived features? [54:46] What kind of models do you have up at the moment and how do you test and monitor and deploy the models?

Coffee Sessions #24 with Sara Robinson of Google, Machine Learning Design Patterns co-hosted by Vishnu Rachakonda. //Bio Sara is a Developer Advocate for Google Cloud, focusing on machine learning. She inspires developers and data scientists to integrate ML into their applications through demos, online content, and events. Before Google, she was a Developer Advocate on the Firebase team. Sara has a Bachelor’s degree from Brandeis University. When she’s not writing code, she can be found on a spin bike or eating frosting. --------------- ✌️Connect With Us ✌️ -------------  Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Sara on LinkedIn: https://www.linkedin.com/in/sara-robinson-40377924/ Timestamps: [00:00] Introduction to Sara Robinson   [01:38] Sara's Background into tech [04:54] What were some things that jumped out at you right away with Machine Learning that is different? [07:44] Sara's Transition to the Machine Learning realm. [08:36] What is the role of a Developer Advocate? [11:41] Compared to traditional software developer advocacy, what stands out to you as being different, unique, perhaps more fun about working in the Machine Learning realm as a Developer Advocate? [13:40] "No one person has it right." [15:27] Given how new this space is, how did you go about writing a book? What leads you to write this book (Machine Learning Design Patterns)?  [19:00] Process of deciding to write the book [21:46] What is it that made the focus of these design patterns? [25:07] Who's the reader that you think who's gonna have this book on their shelf as a reference? [26:42] How would you advise readers to go about reconciling this domain-based needs and the design patterns that you may suggest or identify? [31:20] Can you tell us about a time that some of the design patterns as you're learning with your co-authors has been useful to you? [36:50] Workflow Pipeline breakdown in the book [42:23] How do you think about that level of maturity in terms of thinking about the design patterns? [46:06] How do I communicate in design pattern? What if there is resistance to formalization or implementational structure because it might prevent creativity or reiteration? [49:32] Pre-bill and custom components of Pipeline Frameworks [51:28] How do we know to do the next step or stay in Feature Store patterns? [56:07] Are we going to see the convergence of tools and frameworks soon? Resources referenced in this episode: https://www.oreilly.com/library/view/machine-learning-design/9781098115777/ https://www.amazon.com/Machine-Learning-Design-Patterns-Preparation/dp/1098115783 https://books.google.com.ph/books/about/Machine_Learning_Design_Patterns.html?id=djwDEAAAQBAJ&redir_esc=y https://amzn.to/38tM22C https://sararobinson.dev/2020/11/17/writing-a-technical-book.html

Coffee Sessions #23 with Todd Underwood of Google, Followups from OPML Talks on ML Pipeline Reliability co-hosted by Vishnu Rachakonda. //Bio Todd is a Director at Google and leads Machine Learning for Site Reliability Engineering Director. He is also Site Lead for Google’s Pittsburgh office. ML SRE teams build and scale internal and external ML services and are critical to almost every Product Area at Google. Before working at Google, Todd held a variety of roles at Renesys.  He was in charge of operations, security, and peering for Renesys’s Internet intelligence services that are now part of Oracle's Cloud service. He also did product work for some early social products that Renesys worked on. Before that Todd was Chief Technology Officer of Oso Grande, an independent Internet service provider (AS2901) in New Mexico. //Other links referenced by Todd: --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/ Connect with Todd on LinkedIn: https://www.linkedin.com/in/toddunder/ Timestamps: [00:00] Intro to Todd Underwood [02:04] Todd's background [08:54] What's kind of vision do you "paint"? [14:54] Playing a little bit "devil's advocate." Do you think that's even possible? [19:36] "Start serving to make sure of having the possibility to get it out." How do you feel about that? [23:56] What advise could you give to other people who wanted to bring in ML professionals into their companies to make ML useful for them? [29:53] Is it useful to use these new models?   [32:25] Do you feel like there would be a point where there would be a standard procedure? [35:50] How machine learning breaks [40:44] As an engineering leader, what's your advice to other engineering leaders in terms of how to make that reflection on your team needs and failures...?   [48:42] It's the design that you're looking at as the problem, not the person. [56:27] Do we think that people sold a bunch of stuff and now we were left with the results?      [1:00:46] Recommendations on readings, things to do to better hone our craft. [1:03:35] The more you explore, the more you realize, what's going on? Where can I learn from? [1:05:00] Since you are in the mode of predicting things and philosophical background, where are you seeing the industry going in the next 5 years as we create it? Resources referenced in this episode: https://www.youtube.com/watch?v=Nl6AmAL3i08&feature=emb_title&ab_channel=USENIX https://www.youtube.com/watch?v=hBMHohkRgAA&ab_channel=USENIX https://youtu.be/0sAyemr6lzQ https://youtu.be/EyLGKmPAZLY https://www.usenix.org/conference/opml20/presentation/papasian https://www.usenix.org/system/files/login/articles/02_underwood.pdf https://storage.googleapis.com/pub-tools-public-publication-data/pdf/da63c5f4432525bcaedcebeb50a98a9b7791bbd2.pdf

MLOps community meetup #45! Last Wednesday, we talked to Joe Reis, CEO/Co-Founder of Ternary Data. // Abstract: The fact is that most companies are barely doing BI, let alone AI. Joe discussed ways for companies to build a solid data foundation so they can succeed with machine learning. This meetup covers the continuum from cloud data warehousing to MLOps. // Bio: Joe is a Data Engineer and Architect, Recovering Data Scientist, 20 years in the data game.  Joe enjoys helping companies make sense of their culture, processes, and architecture so they can go from dreaming to doing. He’s certified in both AWS and Google Cloud. When not working, you can find Joe at one of the two groups he co-founded—The Utah Data Engineering Meetup and SLC Python. Joe also sits on the board of Utah Python, a non-profit dedicated to advocating Python in Utah. // Other links to check on Joe: https://www.youtube.com/channel/UC3H60XHMp6BrUzR5eUZDyZg https://josephreis.com/ https://www.ternarydata.com/ https://www.linkedin.com/pulse/what-recovering-data-scientist-joe-reis/ https://www.linkedin.com/pulse/should-you-get-tech-certification-depends-joe-reis/ ----------- Connect With Us ✌️-------------    Join our Slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register   Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Joe on LinkedIn: https://www.linkedin.com/in/josephreis/   Timestamps: [00:23] How did you get into tech? What brought you on to the journey into data? [04:50] You got into the auto ML and you decided to branch out and do your own thing? How did that happen? [08:18] What is it with BI and then making that jump to ML? [11:00] How have you seen Machine Learning fall flat with trying to shoehorn Machine Learning on top of the already weak foundation of BI? [13:45] Let's imagine we're doing BI fairly well and now we want to jump to Machine Learning. Do we have to go out and reinvent the whole stack or can we shoehorn it on? [15:36] How do you move from BI to ML? [18:24] What do you mean by realtime?   [20:35] Managed Services in DevOps [23:30] The maturity isn't there yet [26:03] Where would you draw the line between BI and AI? [30:45] What are the things is Machine Learning an overkill for? [33:43] Are you thinking about what data sets to collect and how different do those vary? [35:18] "Software Engineering and Data Engineering are basically going to merge into one." [38:27] What do you usually recommend moving from BI to AI? [40:45] What is "strong data foundation" in your eyes? [42:47] "MLFlow to gateway drug." What's your take on it?   [46:25] In this pandemic, how easy is it for you to pivot to a new provider? [49:10] Vision of companies starts coming together on different parts of the stack in the Machine Learning tools.