MLOps.community Why You Need More Than Airflow // Ketan Umare // Coffee Sessions #109
Jul 23, 2022
Ketan Umare, Co-founder and CEO of Union.ai, shares insights from his extensive experience at Lyft, Oracle, and Amazon. He discusses the limitations of Airflow in machine learning, emphasizing the need for ML-specific orchestration tools. The conversation covers the complexities of data pipelines, the importance of effective feature management, and the challenges of model drift. Ketan also highlights cloud-native solutions, security in modern engineering, and innovative programming collaborations, all while offering book recommendations that tie historical lessons to today's tech landscape.
AI Snips
Chapters
Books
Transcript
Episode notes
ML Products Need Unique Infrastructure
- Machine learning products face constant changes unlike traditional software, requiring a different infrastructure approach.
- Existing software and data tools don't fit ML's flux, necessitating new solutions like Flight.
Choose Flight For ML Pipelines
- Use specialized tools like Flight for compute-intensive ML workloads, as Airflow lacks quick iteration and versioning needed.
- Reuse components, delay feature materialization, and manage pipeline complexity proactively at scale.
Complexity of ML Pipelines at Lyft
- At Lyft, one ML team managed 600 pipelines for just five models, creating immense complexity.
- Pipelines coexist, getting deprecated or revived, making debugging and dependency tracking very difficult.
