A Reflection On Learning A Lot More Than 97 Things Every Data Engineer Should Know
Data Engineering Podcast
How to Define a Data Lake
When working with large data systems and complex data systems, you need to have a much more detailed understanding of what is the data that i'm collecting. What transformations are processing doing need to do on that data? What is the format and the storage location? You need these well defined ons of how the data is going to be stored and used so that you don't end up having to re engineer things.
Summary
The Data Engineering Podcast has been going for five years now and has included conversations and interviews with a huge number of guests, covering a broad range of topics. In addition to that, the host curated the essays contained in the book "97 Things Every Data Engineer Should Know", using the knowledge and context gained from running the show to inform the selection process. In this episode he shares some reflections on producing the podcast, compiling the book, and relevant trends in the ecosystem of data engineering. He also provides some advice for those who are early in their career of data engineering and looking to advance in their roles.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
- StreamSets DataOps Platform is the world’s first single platform for building smart data pipelines across hybrid and multi-cloud architectures. Build, run, monitor and manage data pipelines confidently with an end-to-end data integration platform that’s built for constant change. Amp up your productivity with an easy-to-navigate interface and 100s of pre-built connectors. And, get pipelines and new hires up and running quickly with powerful, reusable components that work across batch and streaming. Once you’re up and running, your smart data pipelines are resilient to data drift. Those ongoing and unexpected changes in schema, semantics, and infrastructure. Finally, one single pane of glass for operating and monitoring all your data pipelines. The full transparency and control you desire for your data operations. Get started building pipelines in minutes for free at dataengineeringpodcast.com/streamsets. The first 10 listeners of the podcast that subscribe to StreamSets’ Professional Tier, receive 2 months free after their first month.
- Your host is Tobias Macey and today I’m doing something a bit different. I’m going to talk about some of the lessons that I have learned while running the podcast, compiling the book "97 Things Every Data Engineer Should Know", and some of the themes that I’ve observed throughout.
Interview
- Introduction
- How did you get involved in the area of data management?
- Overview of the 97 things book
- How the project came about
- Goals of the book
- What are the paths into data engineering?
- What are some of the macroscopic themes in the industry?
- What are some of the microscopic details that are useful/necessary to succeed as a data engineer?
- What are some of the career/team/organizational details that are helpful for data engineers?
- What are the most interesting, innovative, or unexpected outcomes/feedback that I have seen from running the podcast and working on the book?
- What are the most interesting, unexpected, or challenging lessons that I have learned while working on the Data Engineering Podcast and 97 things book?
- What do I have planned for the future of the podcast?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
- 97 Things Every Data Engineer Should Know
- Buy on Amazon (affiliate link)
- Read on O’Reilly Learning
- O’Reilly Learning 30 Day Free Trial
- Podcast.__init__
- Pipeline Academy data engineering bootcamp
- Hadoop
- Object Relational Mapper (ORM)
- Singer
- Airbyte
- Data Mesh
- Data Contracts Episode
- Designing Data Intensive Applications
- Data Council
- Data Engineering Weekly Newsletter
- Data Mesh Learning
- MLOps Community
- Analytics Engineering Newsletter
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
The AI-powered Podcast Player
