Reduce The Overhead In Your Pipelines With Agile Data Engine's DataOps Service
Data Engineering Podcast
Airflow and Steroids: A Platform for Custom Solutions
Almost all components have been coded by ourselves so our development team has done wonderful work there. We used to have several, at least a couple open source software as a base for our tool. So we let it go and call it that from scratch as well and it actually works really, really well. One thing we still have is Airflow. It's the only open source software that is part of our tool and we call it Airflow and steroids because we have all our own libraries there.
Summary
A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. DataOps has arisen as a parallel set of practices to that of DevOps teams as a means of reducing wasted effort. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow. In this episode Tevje Olin explains how the platform is implemented, the features that it provides to reduce the amount of effort required to keep your pipelines running, and how you can start using it in your own team.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack
- Your host is Tobias Macey and today I'm interviewing Tevje Olin about Agile Data Engine, a platform that combines data modeling, transformations, continuous delivery and workload orchestration to help you manage your data products and the whole lifecycle of your warehouse
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Agile Data Engine is and the story behind it?
- What are some of the tools and architectures that an organization might be able to replace with Agile Data Engine?
- How does the unified experience of Agile Data Engine change the way that teams think about the lifecycle of their data?
- What are some of the types of experiments that are enabled by reduced operational overhead?
- What does CI/CD look like for a data warehouse?
- How is it different from CI/CD for software applications?
- Can you describe how Agile Data Engine is architected?
- How have the design and goals of the system changed since you first started working on it?
- What are the components that you needed to develop in-house to enable your platform goals?
- What are the changes in the broader data ecosystem that have had the most influence on your product goals and customer adoption?
- Can you describe the workflow for a team that is using Agile Data Engine to power their business analytics?
- What are some of the insights that you generate to help your customers understand how to improve their processes or identify new opportunities?
- In your "about" page it mentions the unique approaches that you take for warehouse automation. How do your practices differ from the rest of the industry?
- How have changes in the adoption/implementation of ML and AI impacted the ways that your customers exercise your platform?
- What are the most interesting, innovative, or unexpected ways that you have seen the Agile Data Engine platform used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Agile Data Engine?
- When is Agile Data Engine the wrong choice?
- What do you have planned for the future of Agile Data Engine?
Guest Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
About Agile Data Engine
Agile Data Engine unlocks the potential of your data to drive business value - in a rapidly changing world.
Agile Data Engine is a DataOps Management platform for designing, deploying, operating and managing data products, and managing the whole lifecycle of a data warehouse. It combines data modeling, transformations, continuous delivery and workload orchestration into the same platform.
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Rudderstack:  RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team. RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again. Visit [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack) to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.
The AI-powered Podcast Player
