Why You Need More Than Airflow // Ketan Umare // Coffee Sessions #109
Jul 23, 2022
auto_awesome
Ketan Umare, Co-founder and CEO of Union.ai, shares insights from his extensive experience at Lyft, Oracle, and Amazon. He discusses the limitations of Airflow in machine learning, emphasizing the need for ML-specific orchestration tools. The conversation covers the complexities of data pipelines, the importance of effective feature management, and the challenges of model drift. Ketan also highlights cloud-native solutions, security in modern engineering, and innovative programming collaborations, all while offering book recommendations that tie historical lessons to today's tech landscape.
The complexity of machine learning workflows can lead to interconnected pipelines, necessitating robust orchestration tools to manage dependencies effectively.
Reinforcement learning poses unique challenges that require specific tools to address real-time decision-making and validation processes distinct from supervised learning.
Union.ai offers an evolution in ML orchestration by simplifying user experiences and addressing gaps left by traditional tools, promoting efficient project management.
Deep dives
The Complexity of Machine Learning Workflows
Machine learning workflows can become exceedingly complex very quickly, as highlighted by the discussion around dependencies and the challenges faced by teams managing large numbers of pipelines. It was mentioned that once a small-scale model begins to grow, it can lead to hundreds of interconnected pipelines, creating a tangled web that becomes difficult to debug. As new models are introduced, existing pipelines might break due to unexpected changes or updates, reflecting the need for strong orchestration tools to manage these dependencies effectively. The conversation emphasizes the importance of preparing for this complexity early in the development process to avoid future headaches.
The Unique Challenges of Reinforcement Learning
Reinforcement learning, as discussed, presents unique challenges that differentiate it from traditional machine learning practices. Unlike conventional models, reinforcement learning requires real-time decision-making based on a model's changing behavior, complicating the validation process. The talk emphasized that to effectively deploy reinforcement learning in production, a new set of technical problems must be addressed, distinct from those encountered in supervised learning. This differentiation underlines the need for tools that cater specifically to the unique requirements of reinforcement learning.
The Evolution of Pipelines and Workflow Orchestration
The conversation delved into the evolution of machine learning workflow orchestration tools, particularly comparing Airflow with the newer Flight platform. While Airflow offers a solid foundation for data engineering tasks, it lacks the flexibility needed for rapidly changing machine learning environments. The host noted that integrating tools like Flight enables organizations to handle more complex, compute-intensive workflows without the bottlenecks present in traditional systems. The transition from existing tools like Airflow to Flight is framed as an opportunity to rethink how workflows can be optimized for machine learning.
The Role of Union in Workflow Management
Union was discussed as a solution that enhances the capabilities offered by Flight, aiming to simplify the user experience and best practices in machine learning orchestration. It seeks to fill the gaps left by traditional tools, allowing for smoother deployments and a comprehensive approach to managing machine learning projects and workflows. The goal of Union is not just to provide an orchestrator but to serve as a complete environment that supports engineers and data scientists in their tasks. This reflects a growing recognition within the industry of the need for dedicated solutions tailored to the dynamic nature of machine learning.
The Future of Machine Learning Infrastructure
The discussion explored the rapidly changing landscape of machine learning infrastructure, emphasizing the need for adaptability and innovation. The host predicts that as organizations increasingly adopt cloud-native solutions like Kubernetes, there will be greater demand for orchestrators designed for the unique challenges of machine learning. It was suggested that the ability to plug into existing systems seamlessly while offering advanced features will set successful tools apart in the coming years. This forecast aligns with the broader trends in the industry that dictate a shift toward more flexible, efficient machine learning operations.
MLOps Coffee Sessions #109 with Ketan Umare, Co-founder and CEO of Union.ai, Why You Need More Than Airflow co-hosted by George Pearse.
// Abstract
Airflow is a beloved tool by data engineers and Machine Learning Engineers alike. But when doing ML what are the shortcomings and why is an orchestration tool like that not always the best developer experience? In this episode, we break down what some key drivers are for using an ML-specific orchestration tool.
// Bio
Ketan Umare is the CEO and co-founder at Union.ai. Previously he had multiple Senior roles at Lyft, Oracle, and Amazon ranging from Cloud, Distributed storage, Mapping (map-making), and machine-learning systems. He is passionate about building software that makes engineers' lives easier and provides simplified access to large-scale systems. Besides software, he is a proud father, and husband, and enjoys traveling and outdoor activities.
