The Data Scientist Show - Daliana Liu cover image

How he carved his own path at Airbnb, from data engineer to CEO of Mage - Tommy Dang - the data scientist show #056

The Data Scientist Show - Daliana Liu

00:00

Airflow - Data Warehouse Scaling Made Simple

Airflow is a tool that lets you orchestrate and be able to see all your pipelines. We also integrate natively with Spark so that you can handle large, large data sets. And then finally, we treat data as a first class citizen. There's tools like Airflow who are great at orchestrating generic workloads. You can use it for anything. But we just wanted to focus only on data, data workloads, data pipelines,. integrating data, transforming data. Every step in your pipeline actually produces data product. So although this tool has a lot of functionality, what if I only want to use a few or I want to customize a feature? Yes.

Transcript
Play full episode

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app