Maintain Your Data Engineers' Sanity By Embracing Automation
Data Engineering Podcast
00:00
Data Quality Checks in Cloud Data Warehouses
There are different levels of sophistication or commitment in the data ecosystem, particularly with some of the tools that you mentioned. We kind of took a very bifercated approach, and we checked stuff up front, right at the beginning, and then on the other end of the spectrum, check stuff right at the end in the data warehouse. I think another thing to consider, and this was a huge issue for us at wee days, cost. Just because it passed once doesn't mean it's going to pass again to morrow. And one of the interesting challenges of the data aspect is that there isn't usually one sort of like, make this check now, and it's good for all
Play episode from 28:31
Transcript
Transcript
Episode notes
Summary
Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows. In order to make this a tractable problem it is essential that engineers embrace automation at every opportunity. In this episode Chris Riccomini shares his experiences building and scaling data operations at WePay and LinkedIn, as well as the lessons he has learned working with other teams as they automated their own systems.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
- Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
- Your host is Tobias Macey and today I’m interviewing Chris Riccomini about building awareness of data usage into CI/CD pipelines for application development
Interview
- Introduction
- How did you get involved in the area of data management?
- What are the pieces of data platforms and processing that have been most difficult to scale in an organizational sense?
- What are the opportunities for automation to alleviate some of the toil that data and analytics engineers get caught up in?
- The application delivery ecosystem has been going through ongoing transformation in the form of CI/CD, infrastructure as code, etc. What are the parallels in the data ecosystem that are still nascent?
- What are the principles that still need to be translated for data practitioners? Which are subject to impedance mismatch and may never make sense to translate?
- As someone with a software engineering background and extensive experience working in data, what are the missing links to make those teams/objectives work together more seamlessly?
- How can tooling and automation help in that endeavor?
- A key factor in the adoption of automation for application delivery is automated tests. What are some of the strategies you find useful for identifying scope and targets for testing/monitoring of data products?
- As data usage and capabilities grow and evolve in an organization, what are the junction points that are in greatest need of well-defined data contracts?
- How can automation aid in enforcing and alerting on those contracts in a continuous fashion?
- What are the most interesting, innovative, or unexpected ways that you have seen automation of data operations used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on automation for data systems?
- When is automation the wrong choice?
- What does the future of data engineering look like?
Contact Info
- Website
- @criccomini on Twitter
- criccomini on GitHub
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- WePay
- Enterprise Service Bus
- The Missing README
- Hadoop
- Confluent Schema Registry
- Avro
- CDC == Change Data Capture
- Debezium
- Data Mesh
- What the heck is a data mesh? blog post
- SRE == Site Reliability Engineer
- Terraform
- Chef configuration management tool
- Puppet configuration management tool
- Ansible configuration management tool
- BigQuery
- Airflow
- Pulumi
- Monte Carlo
- Bigeye
- Anomalo
- Great Expectations
- Schemata
- Data Engineering Weekly newsletter
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
