Keep Your Data Lake Fresh With Real Time Streams Using Estuary
Data Engineering Podcast
Challenges and Solutions in Streaming Data Processing
Streaming applications face challenges in integrating with tools that cannot handle event-by-event processing efficiently. For example, updating a Google Sheet at a rate of once per second with a torrent of data is not feasible one event at a time. Additionally, partitioning in systems like Kafka is critical as it impacts scalability, and once set, can be challenging to modify. The modern data stack has effectively addressed these challenges.
Summary
Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. In order to remove that barrier, the team at Estuary have built the Gazette and Flow systems from the ground up to resolve the pain points of other streaming engines, while providing an intuitive interface for data and application engineers to build their streaming workflows. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack
- Your host is Tobias Macey and today I'm interviewing David Yaffe and Johnny Graettinger about using streaming data to build a real-time data lake and how Estuary gives you a single path to integrating and transforming your various sources
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Estuary is and the story behind it?
- Stream processing technologies have been around for around a decade. How would you characterize the current state of the ecosystem?
- What was missing in the ecosystem of streaming engines that motivated you to create a new one from scratch?
- With the growth in tools that are focused on batch-oriented data integration and transformation, what are the reasons that an organization should still invest in streaming?
- What is the comparative level of difficulty and support for these disparate paradigms?
- What is the impact of continuous data flows on dags/orchestration of transforms?
- What role do modern table formats have on the viability of real-time data lakes?
- Can you describe the architecture of your Flow platform?
- What are the core capabilities that you are optimizing for in its design?
- What is involved in getting Flow/Estuary deployed and integrated with an organization's data systems?
- What does the workflow look like for a team using Estuary?
- How does it impact the overall system architecture for a data platform as compared to other prevalent paradigms?
- How do you manage the translation of poll vs. push availability and best practices for API and other non-CDC sources?
- What are the most interesting, innovative, or unexpected ways that you have seen Estuary used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Estuary?
- When is Estuary the wrong choice?
- What do you have planned for the future of Estuary?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Estuary
- Try Flow Free
- Gazette
- Samza
- Flink
- Storm
- Kafka Topic Partitioning
- Trino
- Avro
- Parquet
- Fivetran
- Airbyte
- Snowflake
- BigQuery
- Vector Database
- CDC == Change Data Capture
- Debezium
- MapReduce
- Netflix DBLog
- JSON-Schema
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Rudderstack:  RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. RudderStack’s warehouse-first approach means it does not store sensitive information, and it allows you to leverage your existing data warehouse/data lake infrastructure to build a single source of truth for every team. RudderStack also supports real-time use cases. You can Implement RudderStack SDKs once, then automatically send events to your warehouse and 150+ business tools, and you’ll never have to worry about API changes again. Visit [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack) to sign up for free today, and snag a free T-Shirt just for being a Data Engineering Podcast listener.
Remember Everything You Learn from Podcasts
