

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte
May 23, 2022
Joining the discussion are Ketan Umare, CEO and co-founder at Union, who initiated Flyte at Lyft, and Haytham Abuelfutuh, Union's CTO, who also built Flyte there. They dive into the complexities of data orchestration in machine learning, comparing traditional tools to Flyte's innovative engine on Kubernetes. The conversation highlights the architectural design for user-friendly workflows and applications of Flyte in diverse sectors, including biotech and gaming. They also discuss the balance between open-source principles and sustainable business models.
AI Snips
Chapters
Transcript
Episode notes
Flight's Origin at Lyft
- Ketan Umare shared how Flight started at Lyft to solve the real-time ETA and ETD model deployment challenges.
- They rebuilt a workflow engine supporting rapid model retraining and deployment under high operational demand.
ML Workflows Need New Tools
- Machine learning pipelines need fundamentally different tooling than traditional data workflows.
- Flight is designed as a serverless, language-agnostic platform focused on machine learning's rapid iteration and robustness needs.
Choose Flight for ML on Kubernetes
- Use Flight if you want a Kubernetes native, serverless platform with robust execution guarantees.
- For pure query transformations use tools like dbt; Flight excels at machine learning workflows requiring reproducibility and scalability.