MLOps Coffee Sessions #109 with Ketan Umare, Co-founder and CEO of Union.ai, Why You Need More Than Airflow, co-hosted by George Pearse.

// Abstract
Airflow is a beloved tool by data engineers and Machine Learning Engineers alike. But when doing ML, what are the shortcomings, and why is an orchestration tool like that not always the best developer experience? In this episode, we break down what some key drivers are for using an ML-specific orchestration tool.

// Bio
Ketan Umare is the CEO and co-founder at Union.ai. Previously, he had multiple Senior roles at Lyft, Oracle, and Amazon, ranging from Cloud, distributed storage, Mapping (map-making), and machine-learning systems. He is passionate about building software that makes engineers' lives easier and provides simplified access to large-scale systems. Besides software, he is a proud father and husband, and enjoys traveling and outdoor activities.

// MLOps Jobs board
jobs.mlops.community

MLOps Swag/Merch
https://mlops-community.myshopify.com/

// Related Links
Zero to One: Notes on Startups, or How to Build the Future, Hardcover by Peter Thiel and Blake Masters:
https://www.amazon.com/Zero-One-Notes-Startups-Future/dp/0804139296

--------------- ✌️Connect With Us ✌️ -------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with George on LinkedIn: https://www.linkedin.com/in/george-pearse-b7a76a157/?originalSubdomain=uk
Connect with Ketan on LinkedIn: https://www.linkedin.com/in/ketanumare/

Timestamps:

[00:00] Introduction to Ketan Umare

[01:08] Takeaways

[04:03] Changes with Ketan after 2 years

[10:06] Diametrically opposed software and data infrastructure

[13:06] Airflow shortcomings

[21:14] Core abstractions

[24:56] Machine Learning specific packages

[28:25] Core target users

[30:50] Airflow foundational level

[33:46] Better visualization of data science pipelines

[36:39] Flyte for RL Deployment

[38:36] Role of signals in the system

[40:59] Everything in Flyte is run through an API

[42:20] Manual intervention in the system

[45:11] Union vs Flyte

[50:16] Security precautions

[51:56] Role of Union cloud

[56:20] Distinctiveness of Flyte pipelines

[01:01:11] Ketan in his current company

[01:02:25] Learning from Airflow

[01:04:58] Lightning round with Ketan

[01:09:59] Wrap up

Ho, Ho, Ho-Cast