Data Engineering Podcast
X-Ray Vision For Your Flink Stream Processing With Datorios
Episode guests
Podcast summary created with Snipd AI
Quick takeaways
- Observability is essential for enhancing real-time data processing capabilities in Flink.
- Understanding real-time data context and detecting anomalies are crucial for ensuring data quality in streaming data.
Deep dives
Challenges Faced in Managing Real-Time Data Streams
Real-time data streams present challenges due to their complexity and the need for observability. Understanding operational missions with real-time data is crucial, as it allows for reacting to information quickly. Adapting data use to changing missions is necessary to develop data products that meet specific demands.
Adoption of Apache Flink for Real-Time Data Processing
Datorios initially developed its real-time processing engine but later transitioned to using Apache Flink due to its emerging prominence in processing real-time data. Flink's strength lies in processing data for real-time AI applications, providing the ability to process data into models in real-time. The decision to switch to Flink was deemed significant in enhancing real-time data processing capabilities.
Challenges of Data Observability in Streaming Data and Flink
Observing streaming data, especially in Flink, poses multiple challenges due to Flink's complex dynamics. Flink's stateful processing, hidden states, and complex distributed solutions make it hard to pinpoint issues when data processing faces challenges. Ensuring data quality in streaming data requires understanding the context and detecting anomalies in real-time scenarios.
Prominence and Growth of Real-Time Data and Flink Ecosystem
Real-time data and Flink are gaining traction due to increasing demands for real-time applications. Flink offers capabilities to process data with high quality and relevance, making it instrumental for real-time AI systems. The market is evolving to adopt real-time data, and solutions like Datorios provide observability to enhance data processing and usage across industries.
Summary
Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. In this episode Ronen Korman and Stav Elkayam discuss how the increased understanding provided by purpose built observability improves the usefulness of Flink.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- This episode is supported by Code Comments, an original podcast from Red Hat. As someone who listens to the Data Engineering Podcast, you know that the road from tool selection to production readiness is anything but smooth or straight. In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. There are 3 seasons of great episodes and new ones landing everywhere you listen to podcasts. Search for "Code Commentst" in your podcast player or go to dataengineeringpodcast.com/codecomments today to subscribe. My thanks to the team at Code Comments for their support.
- Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Trusted by teams of all sizes, including Comcast and Doordash. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
- Your host is Tobias Macey and today I'm interviewing Ronen Korman and Stav Elkayam about pulling back the curtain on your real-time data streams by bringing intuitive observability to Flink streams
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Datorios is and the story behind it?
- Data observability has been gaining adoption for a number of years now, with a large focus on data warehouses. What are some of the unique challenges posed by Flink?
- How much of the complexity is due to the nature of streaming data vs. the architectural realities of Flink?
- How has the lack of visibility into the flow of data in Flink impacted the ways that teams think about where/when/how to apply it?
- How have the requirements of generative AI shifted the demand for streaming data systems?
- What role does Flink play in the architecture of generative AI systems?
- Can you describe how Datorios is implemented?
- How has the design and goals of Datorios changed since you first started working on it?
- How much of the Datorios architecture and functionality is specific to Flink and how are you thinking about its potential application to other streaming platforms?
- Can you describe how Datorios is used in a day-to-day workflow for someone building streaming applications on Flink?
- What are the most interesting, innovative, or unexpected ways that you have seen Datorios used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Datorios?
- When is Datorios the wrong choice?
- What do you have planned for the future of Datorios?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Starburst:  This episode is brought to you by Starburst - an end-to-end data lakehouse platform for data engineers who are battling to build and scale high quality data pipelines on the data lake. Powered by Trino, the query engine Apache Iceberg was designed for, Starburst is an open platform with support for all table formats including Apache Iceberg, Hive, and Delta Lake. Trusted by the teams at Comcast and Doordash, Starburst delivers the adaptability and flexibility a lakehouse ecosystem promises, while providing a single point of access for your data and all your data governance allowing you to discover, transform, govern, and secure all in one place. Want to see Starburst in action? Try Starburst Galaxy today, the easiest and fastest way to get started using Trino, and get $500 of credits free. Go to [dataengineeringpodcast.com/starburst](https://www.dataengineeringpodcast.com/starburst)
- Red Hat Code Comments Podcast:  Putting new technology to use is an exciting prospect. But going from purchase to production isn’t always smooth—even when it’s something everyone is looking forward to. Code Comments covers the bumps, the hiccups, and the setbacks teams face when adjusting to new technology—and the triumphs they pull off once they really get going. Follow Code Comments [anywhere you listen to podcasts](https://link.chtbl.com/codecomments?sid=podcast.dataengineering).
Remember Everything You Learn from Podcasts
