Josh Tobin discusses the shift in focus from model development to machine learning systems, the evolution of modeling in the machine learning ecosystem, the capabilities of Gantry in enhancing model performance and maintenance, core capabilities and flexible support for machine learning, innovative approaches and challenges in building and deploying machine learning models, and when to choose Gantry for model development and maintenance.
Read more
AI Summary
AI Chapters
Episode notes
auto_awesome
Podcast summary created with Snipd AI
Quick takeaways
Machine learning practices are shifting from traditional models to machine learning systems focused on real products that interact with end users.
The role of modeling in machine learning is becoming smaller, with a growing focus on applying machine learning at the process and data analytics level to solve specific business problems.
Deep dives
The Evolution of Machine Learning Practices
Machine learning practices have evolved from traditional models to the current ML ecosystem. In the early days, machine learning involved optimizing a fixed data set and metric. However, this approach is rapidly changing as companies are tasked with building real products that interact with end users. This shift requires considering the dynamic nature of user behavior and the changing world. There is a growing focus on building machine learning-powered products based on foundation models like GPT-3. These models allow for quick prototyping and adaptation to specific tasks without extensive training. Companies are also benefiting from the reduced entry barriers to adopt machine learning and the expanding capabilities of state-of-the-art models.
Challenges in Model Building and Validation
While the availability of foundation models and advances in machine learning technology make it easier to get started with model building, challenges still exist in model validation, operation, and maintenance. The transition from an MVP model to a reliable and high-performing model can be difficult. Failure modes and limitations of foundation models need to be understood and addressed. Data availability, reduced barriers to entry, and rapidly expanding capabilities are driving the adoption of machine learning in various industries. However, companies need to be aware of potential platform and technology risks, including constraints introduced by model providers and the failure modes of large language models.
The Changing Role of Modeling and ML Expertise
As machine learning becomes more accessible, the role of modeling in companies is gradually changing. While training machine learning models remains a crucial skill, it is becoming a smaller part of the machine learning workflow. Companies are increasingly adopting higher level abstractions that utilize machine learning under the hood. Instead, the focus is shifting towards applying machine learning at the process and data analytics level to solve specific business problems. Although the need for ML expertise may diminish, understanding the fundamental aspects of model training can still be valuable. Gantry, as a tool, supports teams in maintaining and improving deployed models, allowing for better performance and decreased maintenance costs.
Gantry's Role in Model Maintenance and Improvement
Gantry is a machine learning tool designed to support the maintenance, improvement, and performance of deployed models. It addresses the challenge of taking a generic model and adapting it to the specific business context and problem at hand. Gantry provides an analytics suite for understanding model performance, diagnosing failures, and identifying opportunities for improvement. It also enables the folding of insights gained from production data into model retraining, allowing for continual learning and evolution. While currently used by machine learning experts, Gantry aims to empower non-experts in the future as models become more accessible and specialized roles emerge in ML-based product development.
Summary The focus of machine learning projects has long been the model that is built in the process. As AI powered applications grow in popularity and power, the model is just the beginning. In this episode Josh Tobin shares his experience from his time as a machine learning researcher up to his current work as a founder at Gantry, and the shift in focus from model development to machine learning systems. Announcements
Hello and welcome to the Machine Learning Podcast, the podcast about machine learning and how to bring it from idea to delivery.
Your host is Tobias Macey and today I'm interviewing Josh Tobin about the state of industry best practices for designing and building ML models
Interview
Introduction
How did you get involved in machine learning?
Can you start by describing what a "traditional" process for building a model looks like?
What are the forces that shaped those "best practices"?
What are some of the practices that are still necessary/useful and what is becoming outdated?
What are the changes in the ecosystem (tooling, research, communal knowledge, etc.) that are forcing teams to reconsider how they think about modeling?
What are the most critical practices/capabilities for teams who are building services powered by ML/AI?
What systems do they need to support them in those efforts?
Can you describe what you are building at Gantry and how it aids in the process of developing/deploying/maintaining models with "modern" workflows?
What are the most challenging aspects of building a platform that supports ML teams in their workflows?
What are the most interesting, innovative, or unexpected ways that you have seen teams approach model development/validation?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Gantry?
When is Gantry the wrong choice?
What are some of the resources that you find most helpful to stay apprised of how modeling and ML practices are evolving?
From your perspective, what is the biggest barrier to adoption of machine learning today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@themachinelearningpodcast.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers