Differentiating User Preferences and Scalability Issues in ML Pipelines

Jake Watson is the writer of thedataplatform.substack.com⁠ and Principal Data Engineer at The Oakland Group.

MLOps podcast #207 with Jake Watson, Principal Data Engineer at The Oakland Group, How Data Platforms Affect ML & AI. // Abstract I’ve always told my clients and colleagues that traditional rule-based software is difficult, but software containing Artificial Intelligence (AI) and/or Machine Learning (ML)* is even more difficult, sometimes impossible. Why is this the case? Well, software is difficult because it’s like flying a plane while building it at the same time, but because AI and ML make rules on the fly based on various factors like training data, it’s like trying to build a plane in flight, but some parts of the plane will be designed by a machine, and you have little idea what that is going to look like till the machine finishes. This double goes for more cutting-edge AI models like GPT, where only the creators of the software have a vague idea of what it will output. This makes software with AI / ML more of a scientific experiment than engineering, which is going to make your project manager lose their mind when you have little idea how long a task is going to take. But what will make everyone’s lives easier is having solid data foundations to work from. Learn to walk before running. // Bio Jake has been working in data as an Analyst, Engineer, and/or Architect for over 10 years. Started as an analyst in the UK National Health Service converting spreadsheets to databases tracking surgical instruments. Then continued as an analyst at a consultancy (Capita) reporting on employee engagement in the NHS and dozens of UK Universities. There Jake moved reporting from Excel and Access to SQL Server, Python with frontend websites in d3.js. At Oakland Group, a data consultancy, Jake worked as a Cloud Engineer, Data Engineer, Tech Lead, and Architect depending on the project for dozens of clients both big and small (mostly big). Jake has also developed and productionised ML solutions as well in the NLP and classification space. Jake has experience in building Data Platforms in Azure, AWS, and GCP (though mostly in Azure and AWS) using Infrastructure as Code and DevOps/DataOps/MLOps. In the last year, Jake has been writing articles and newsletters for my blog, including a guide on how to build a data platform: https://thedataplatform.substack.com/p/how-to-build-a-data-platform // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://thedataplatform.substack.com/ How Data Platform Foundations Impact AI and ML Applications blog: https://thedataplatform.substack.com/p/issue-29-how-data-platform-foundations AI in Production Conference: https://home.mlops.community/public/events/ai-in-production-2024-02-15 How to Build a Data Platform blog: https://thedataplatform.substack.com/p/how-to-build-a-data-platform --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Jake on LinkedIn: https://www.linkedin.com/in/jake-watson-data/ Timestamps: [00:00] Jake's preferred coffee [00:26] AI in Production Conference teaser [02:38] Takeaways [04:00] Please like, share, and subscribe to our MLOps channels! [04:17] Data Engineer's Crucial Role [05:44] Jake's background [06:44] Data Platform Foundations blog [10:34] Data mesh organizational side of things [17:58] Importance of data modeling [20:13] Dealing with the sprawl [22:03] Data quality [23:59] Data hierarchy on building a platform [29:34] ML Platform Team Structure [31:47] Don't reinvent the wheel [34:04] Data pipelines synergy [37:31] Wrap up