The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

How Airflow orchestration decisions impact Spark performance

6 snips

Aug 13, 2026

Meni Shmueli, Co-founder and CEO of DataFlint and former data engineer focused on Spark performance. He explores how Airflow orchestration choices can make Spark jobs faster or far more costly. Short stories show parallelism pitfalls and serial-run waste. He explains DataFlint’s Airflow/Astro integrations and how AI enables holistic scheduling, cost-aware optimization, and huge real-world savings.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Airflow Orchestrates While Spark Does The Heavy Lifting

Airflow schedules and Spark execute complementary roles: Airflow decides when and how pipelines run while Spark performs heavy distributed compute.
Meni Shmueli says most customers use Airflow to order pipelines and Spark as the underlying engine moving data at scale.

ANECDOTE

Parallelizing Tasks Made Spark Slower And Costlier

Parallelizing independent Spark tasks in Airflow led one customer to slower, unstable, and far more expensive runs because tasks contended for the same cluster resources.
Meni explains contention caused spills to disk and crashes; sequential runs used memory and ran faster.

ADVICE

Benchmark Pipelines By Duration And Underlying Spark Cost

Do benchmark pipelines with both Airflow and Spark contexts; measure duration alongside actual Spark resource usage and cost.
Meni warns a faster DAG runtime can hide a jump from 100 to 1,000 cluster machines and huge cost increases.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive. In this episode, Kenten is joined by [Meni Shmueli](linkedin.com/in/meni-shmueli-dataflint), Co-Founder and CEO at [DataFlint](dataflint.io), to dig into how Airflow and Spark fit together, where teams go wrong, and how AI is changing the way they reason about cost and performance.

Key Takeaways:

00:00 Introduction.
01:40 Meni's background as a data engineer and what led him to start DataFlint, a production observability agent for Apache Spark.
02:53 Why the Airflow plus Spark combination is so common, and how each tool plays to its strengths.
04:23 How orchestration decisions in Airflow directly impact Spark performance and cost.
04:44 A customer story where parallelizing Airflow tasks made Spark jobs slower, less stable, and more expensive, and the opposite case where sequential runs left compute on the table.
07:02 The number one mistake teams make benchmarking pipelines: only looking at the Airflow side and ignoring underlying Spark cost and resource usage.
08:13 What DataFlint's Airflow and Astro integration gives teams, and how it brings production context into AI agents.
10:46 How AI is changing pipeline optimization, including holistic scheduling across hundreds of pipelines and connecting context from Spark, FinOps, and cloud.
12:21 A customer migration from Databricks to EMR that cut workflow costs by 80%, with examples of up to 100x optimizations.
14:23 Where the Airflow and Spark story could be better, including Spark Declarative Pipelines and tighter feedback between the two projects.

Resources Mentioned:

[Orchestrate Everything](https://astronomer.link/data-flowcast-oe)
[DataFlint](dataflint.io)
[Apache Airflow](airflow.apache.org)
[Apache Spark](spark.apache.org)
[Astro](astronomer.io/astro)
[Airflow Summit](airflowsummit.org)

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #ApacheAirflow