Data Engineering Podcast
Keep Your Data Lake Fresh With Real Time Streams Using Estuary
Podcast summary created with Snipd AI
Quick takeaways
- Gazette and Flow systems by Estuary simplify real-time data lake building
- Estuary supports high-scale Change Data Capture for efficient database handling
- Estuary's diverse integration options and transformation approaches enhance data movement capabilities
Deep dives
Streamlining Data Management with RutterStack's CDC Solution
RutterStack offers a cost-effective and secure Customer Data Platform (CDP) solution built on top of your data warehouse. They provide technical controls to fully harness the power of customer data, ensuring real-time data lake efficiency. Their platform enables capturing data from various APIs via HTTP connectors, supporting diverse data sources for unified data transformation.
Evolution of Data Transformation Process at Estuary
Estuary's evolution showcases the co-founders' transition from marketing and advertising technology to real-time data solutions. Their focus on enabling high-scale Change Data Capture (CDC) ensures handling large databases with backfilling efficiency. The platform supports hybrid workflows, empowering users to transform data in real time, while ensuring schema drift management for continuous data flow.
Empowering Streaming Use Cases with API Connectivity
Estuary's HTTP file connector facilitates data integration from diverse API sources, addressing unique data stream challenges. Noteworthy integrations include Salesforce for real-time data updates and Google Sheets for live data materializations. The platform's streamlined HTTP-injust connector offers push-based capture for microservices, enriching data movement capabilities.
Esteury's Data Transformation Flexibility and Road Ahead
Esteury's focus on diverse transformation approaches, such as SQL, TypeScript, and Python integration, paves the way for AI pipeline advancements and efficient data management. The platform aims to democratize data access within organizations, fostering collaborative data practices across technical and non-technical teams. Future developments include platform unification for holistic data engineering work.
Addressing Data Management Challenges and Future Innovations
Addressing gaps around data democratization and platform collaboration remains crucial in data management solutions. Esteury emphasizes collaborative data practices within organizations, aiming for seamless data workflow orchestration. Future data innovations may revolve around more integrated data management platforms, fostering inclusive data practices and robust analytics.
Summary
Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. In order to remove that barrier, the team at Estuary have built the Gazette and Flow systems from the ground up to resolve the pain points of other streaming engines, while providing an intuitive interface for data and application engineers to build their streaming workflows. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack
- Your host is Tobias Macey and today I'm interviewing David Yaffe and Johnny Graettinger about using streaming data to build a real-time data lake and how Estuary gives you a single path to integrating and transforming your various sources
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Estuary is and the story behind it?
- Stream processing technologies have been around for around a decade. How would you characterize the current state of the ecosystem?
- What was missing in the ecosystem of streaming engines that motivated you to create a new one from scratch?
- With the growth in tools that are focused on batch-oriented data integration and transformation, what are the reasons that an organization should still invest in streaming?
- What is the comparative level of difficulty and support for these disparate paradigms?
- What is the impact of continuous data flows on dags/orchestration of transforms?
- What role do modern table formats have on the viability of real-time data lakes?
- Can you describe the architecture of your Flow platform?
- What are the core capabilities that you are optimizing for in its design?
- What is involved in getting Flow/Estuary deployed and integrated with an organization's data systems?
- What does the workflow look like for a team using Estuary?
- How does it impact the overall system architecture for a data platform as compared to other prevalent paradigms?
- How do you manage the translation of poll vs. push availability and best practices for API and other non-CDC sources?
- What are the most interesting, innovative, or unexpected ways that you have seen Estuary used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Estuary?
- When is Estuary the wrong choice?
- What do you have planned for the future of Estuary?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Estuary
- Try Flow Free
- Gazette
- Samza
- Flink
- Storm
- Kafka Topic Partitioning
- Trino
- Avro
- Parquet
- Fivetran
- Airbyte
- Snowflake
- BigQuery
- Vector Database
- CDC == Change Data Capture
- Debezium
- MapReduce
- Netflix DBLog
- JSON-Schema
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Rudderstack:  RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team. RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again. Visit [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack) to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.
Remember Everything You Learn from Podcasts
