Alter Everything cover image

Alter Everything

148: Present & Future of Data Engineering

Dec 20, 2023
Megan Dibble, a data operations expert, and Nick Schrock, founder of Dagster Labs, delve into the dynamic world of data engineering. They clarify the distinctions between data engineering and data analytics, and introduce the hybrid role of analytics engineer. The discussion uncovers the evolution of roles in data engineering, spotlighting the shift towards a software engineering mindset. They also tackle challenges like vendor fatigue, the need for quality data, and strategies for effective data orchestration, emphasizing its vital role in decision-making.
23:55

Podcast summary created with Snipd AI

Quick takeaways

  • Data engineering has evolved into a crucial discipline over the last decade, becoming essential for managing the complexity of modern data systems.
  • The emergence of orchestration tools is vital for integrating diverse data technologies, addressing the challenge of vendor fatigue in the industry.

Deep dives

Understanding Data Engineering vs. Data Analytics

Data engineering and data analytics are closely related yet distinct fields, each with different focuses. Data analysts typically work on business problems outlined by stakeholders, conducting ad hoc analyses to extract insights from existing datasets and translating those insights into dashboards. Conversely, data engineers concentrate on the foundational layer of managing the flow of data, built on software engineering principles, to store and retrieve data efficiently. Emerging roles like analytics engineers blur these lines by bridging the gap, but fundamentally, data engineering prioritizes maintaining and developing robust data pipelines over business-oriented tasks.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode