The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) Scalable and Maintainable Workflows at Lyft with Flyte w/ Haytham AbuelFutuh and Ketan Umare - #343
7 snips
Jan 30, 2020 
In this discussion, Ketan Umare and Haytham AbuelFutuh, both software engineers at Lyft, dive into the innovative Flyte project they contribute to. Ketan shares the motivation behind developing Flyte, while Haytham highlights its Kubernetes-native design. They explore strong typing's role in improving user experience, the challenges of managing machine learning workflows, and Flyte's open-source journey to foster community engagement. The conversation also touches on data provenance and optimizing computational efficiency in large-scale data processing.
AI Snips
Chapters
Transcript
Episode notes
Laptop Modeling
- Ketan Umare's team member at Lyft used to run machine learning models for ETA prediction on his laptop.
- This highlighted the need for a more robust and scalable system.
Lost Model
- After a research scientist left Lyft, his model was lost, forcing the team to recreate it, which took three months.
- Recreating the model revealed the hidden efforts of the original scientist and emphasized the importance of model reproducibility.
ML and Data Dependency
- There's an artificial divide between machine learning and data, but they are deeply interconnected.
- ML models both use and produce data, creating a cyclical dependency that needs to be managed.