Why was Flyte built at Lyft?
What sorts of requirements does a ML infrastructure team have at lyft?
What problems does it solve / use cases?
Where does it fit in in the ML and Data ecosystem?
What is the vision?
Who should consider using it?
Learnings as the engineering team tried to bootstrap an open-source community.
Ketan Umare is a senior staff software engineer at Lyft responsible for technical direction of the Machine Learning Platform and is a founder of the Flyte project. Before Flyte he worked on ETA, routing and mapping infrastructure at Lyft. He is also the founder of Flink Kubernetes operator and contributor to Spark on kubernetes. Prior to Lyft he was a founding member of Oracle Baremetal Cloud and lead teams building Elastic Block Storage. Prior to that, he started and lead multiple teams in Mapping and Transportation optimization infrastructure at Amazon. He received his Masters in Computer Science from Georgia Tech specializing in High-performance computing and his Bachelors in Engineering in Computer Science from VJTI Mumbai.
Besides work, he enjoys spending time with his daughter and wife. He loves the Pacific Northwest outdoors and will try anything new.
Lyft
Pricing, Locations, Estimated Time of Arrivals (ETA), Mapping, Self-Driving (L5), etc.
What sort of scale, storage, network bandwidth are we looking at?
Tens of thousands of workflows, hundreds of thousands of executions, millions of tasks, and tens of millions of containers!
Flyte: more than 900k workflow executed a month and more than 30+ million container executions per month
Typical flow of information?
What are the user stories you’re typically dealing with at lyft?
How do you set it up?
On-prem, cloud, etc.
Helm installable?
Why Golang?
What problems does it solve?
Complex data dependencies? Why
Orchestrated compute on demand
Reuse and sharing
Key features
Multi-tenant, hosted, serverless
Parametrized, data lineage, caching
Additionally, if the run invokes a task that has already been computed before, regardless of who executed it, Flyte will smartly use the cached output, saving you both time and money.
Versioning, sharing
Modular, loosely coupled
Seems like you guys recognize that the best task for the job might be hosted elsewhere, so it was important to integrate other solutions into flyte.
Flyte extensions
Backend plugins - is it true you can create and manage k8s resources like CRDs for things like spark, sagemaker, bigquery?
Drop a Star
https://flyte.org
Flyte community
----------- Connect With Us ✌️-------------
Join our slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Connect with Ketan on LinkedIn: https://www.linkedin.com/in/ketanumare/
Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/
