AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Notebook Orchestration vs. Pipeline Orchestration
This chapter explores the cost and efficiency disparities between notebook orchestration and pipeline orchestration, emphasizing the importance of selecting the most efficient compute engine for tasks. It highlights the need for engineers to assess different design architectures for optimal cost savings and efficiency, discussing the substantial differences in assumptions when transitioning work to spark notebooks.