An Exploration Of What Data Automation Can Provide To Data Engineers And Ascend's Journey To Make It A Reality
Data Engineering Podcast
00:00
What's the Ripple Effect?
The existing model of e t l pipe lines is backwards, where we're very focused on these push based floes. As a data consumer, i just care if my dashboard's up to date. I don't understand all the ramifications of kicking off whatever jobs are going to fill that in. We actually run check sums on all code and all data as it moves through pipe lines. It generally does require more complex traversal back through to all the upstream systems. This is one of those things that i think does become really exciting when we think about the mount o meta data requiredOus a busiss level, pe then at the code level and at a operations level,. To determine known how
Transcript
Play full episode
Transcript
Episode notes
Summary
The dream of every engineer is to automate all of their tasks. For data engineers, this is a monumental undertaking. Orchestration engines are one step in that direction, but they are not a complete solution. In this episode Sean Knapp shares his views on what constitutes proper automation and the work that he and his team at Ascend are doing to help make it a reality.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
- Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
- The only thing worse than having bad data is not knowing that you have it. With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. Go to dataengineeringpodcast.com/bigeye today to sign up and start trusting your analyses.
- Your host is Tobias Macey and today I’m interviewing Sean Knapp about the role of data automation in building maintainable systems
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what you mean by the term "data automation" and the assumptions that it includes?
- One of the perennial challenges of automation is that there are always steps that are resistant to being performed without human involvement. What are some of the tasks that you have found to be common problems in that sense?
- What are the different concerns that need to be included in a stack that supports fully automated data workflows?
- There was recently an interesting article suggesting that the "left-to-right" approach to data workflows is backwards. In your experience, what would be required to allow for triggering data processes based on the needs of the data consumers? (e.g. "make sure that this BI dashboard is up to date every 6 hours")
- What are the tasks that are most complex to build automation for?
- What are some companies or tools/platforms that you consider to be exemplars of "data automation done right"?
- What are the common themes/patterns that they build from?
- How have you approached the need for data automation in the implementation of the Ascend product?
- How have the requirements for data automation changed as data plays a more prominent role in a growing number of businesses?
- What are the foundational elements that are unchanging?
- What are the most interesting, innovative, or unexpected ways that you have seen data automation implemented?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on data automation at Ascend?
- What are some of the ways that data automation can go wrong?
- What are you keeping an eye on across the data ecosystem?
Contact Info
- @seanknapp on Twitter
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Ascend
- Google Sawzall
- CI/CD
- Airflow
- Kubernetes
- Ascend FlexCode
- MongoDB
- SHA == Secure Hash Algorithm
- dbt
- Materialized View
- Great Expectations
- Monte Carlo
- OpenLineage
- Open Metadata
- Egeria
- OOM == Out Of Memory Manager
- Five Whys
- Data Mesh
- Data Fabric
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Bigeye:  Bigeye is an industry-leading data observability platform that gives data engineering and science teams the tools they need to ensure their data is always fresh, accurate and reliable. Companies like Instacart, Clubhouse, and Udacity use Bigeye’s automated data quality monitoring, ML-powered anomaly detection, and granular root cause analysis to proactively detect and resolve issues before they impact the business. Go to [dataengineeringpodcast.com/bigeye](https://www.dataengineeringpodcast.com/bigeye) today and start trusting your data.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
