X-Ray Vision For Your Flink Stream Processing With Datorios
Data Engineering Podcast
Real-Time Data Processing and Challenges in Apache Flink
This chapter delves into the importance of real-time data processing in the context of security agencies and the journey into working with data, leading to the creation of a company focused on real-time data processing. The challenges of data observability in Apache Flink are discussed, emphasizing the need for understanding system complexities for troubleshooting and ensuring data quality for AI solutions. It also explores the unique challenges of handling streaming data, discussing issues like managing statefulness, identifying data quality problems, and the debate between batch and stream processing in data engineering.
00:00
Transcript
Play full episode
Transcript
Episode notes
Summary
Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. In this episode Ronen Korman and Stav Elkayam discuss how the increased understanding provided by purpose built observability improves the usefulness of Flink.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- This episode is supported by Code Comments, an original podcast from Red Hat. As someone who listens to the Data Engineering Podcast, you know that the road from tool selection to production readiness is anything but smooth or straight. In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. There are 3 seasons of great episodes and new ones landing everywhere you listen to podcasts. Search for "Code Commentst" in your podcast player or go to dataengineeringpodcast.com/codecomments today to subscribe. My thanks to the team at Code Comments for their support.
- Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Trusted by teams of all sizes, including Comcast and Doordash. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
- Your host is Tobias Macey and today I'm interviewing Ronen Korman and Stav Elkayam about pulling back the curtain on your real-time data streams by bringing intuitive observability to Flink streams
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Datorios is and the story behind it?
- Data observability has been gaining adoption for a number of years now, with a large focus on data warehouses. What are some of the unique challenges posed by Flink?
- How much of the complexity is due to the nature of streaming data vs. the architectural realities of Flink?
- How has the lack of visibility into the flow of data in Flink impacted the ways that teams think about where/when/how to apply it?
- How have the requirements of generative AI shifted the demand for streaming data systems?
- What role does Flink play in the architecture of generative AI systems?
- Can you describe how Datorios is implemented?
- How has the design and goals of Datorios changed since you first started working on it?
- How much of the Datorios architecture and functionality is specific to Flink and how are you thinking about its potential application to other streaming platforms?
- Can you describe how Datorios is used in a day-to-day workflow for someone building streaming applications on Flink?
- What are the most interesting, innovative, or unexpected ways that you have seen Datorios used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Datorios?
- When is Datorios the wrong choice?
- What do you have planned for the future of Datorios?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Starburst:  This episode is brought to you by Starburst - an end-to-end data lakehouse platform for data engineers who are battling to build and scale high quality data pipelines on the data lake. Powered by Trino, the query engine Apache Iceberg was designed for, Starburst is an open platform with support for all table formats including Apache Iceberg, Hive, and Delta Lake. Trusted by the teams at Comcast and Doordash, Starburst delivers the adaptability and flexibility a lakehouse ecosystem promises, while providing a single point of access for your data and all your data governance allowing you to discover, transform, govern, and secure all in one place. Want to see Starburst in action? Try Starburst Galaxy today, the easiest and fastest way to get started using Trino, and get $500 of credits free. Go to [dataengineeringpodcast.com/starburst](https://www.dataengineeringpodcast.com/starburst)
- Red Hat Code Comments Podcast:  Putting new technology to use is an exciting prospect. But going from purchase to production isn’t always smooth—even when it’s something everyone is looking forward to. Code Comments covers the bumps, the hiccups, and the setbacks teams face when adjusting to new technology—and the triumphs they pull off once they really get going. Follow Code Comments [anywhere you listen to podcasts](https://link.chtbl.com/codecomments?sid=podcast.dataengineering).
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
