Ep 2 - Processing Without Pause: Continuous Stream Processing and Apache Flink®

7 snips

Feb 18, 2025

Anna McDonald and Abhishek Walia from Confluent share their expertise on stream processing, highlighting its advantages over batch processing. They dive into tools like Apache Flink and Kafka Streams, explaining their unique approaches and applications. Real-world examples showcase the transformative power of real-time data in industries such as banking and healthcare. The discussion also covers the importance of proofs of concept and observability in optimizing data workflows, while emphasizing adaptability in an ever-evolving tech landscape.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Stream Processing vs. Batch Processing

Stream processing, unlike batch processing, runs continuously without pauses.
This requires a different mindset for architecture and planning.

ANECDOTE

Real-World Stream Processing

Anna McDonald shared examples of real-time stream processing.
These include sleep apnea masks adjusting based on real-time data and sensors detecting couches on expressways.

ADVICE

Choosing Stream Processing Frameworks

KSQL and Flink run as clusters, while KStreams is a Java library.
Choose based on your development interface and architectural needs.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

We’re diving even deeper into the fundamentals of data streaming to explore stream processing—what it is, the best tools and frameworks, and its real-world applications.

Our guests, Anna McDonald, Distinguished Technical Voice of the Customer at Confluent, and Abhishek Walia, Staff Customer Success Technical Architect at Confluent, break down what stream processing is, how it differs from batch processing, and why tools like Flink are game changers.

You’ll learn:

The key differences between stream and batch processing
How different frameworks like Flink, Kafka Streams, and ksqlDB approach stream processing
The role of POCs and observability in real-time data workflows
And more…

About the Guests:

Anna McDonald is the Distinguished Technical Voice of the Customer at Confluent. She loves designing creative solutions to challenging problems. Her focus is on event-driven architectures, reactive systems, and Apache Kafka®.

Abhishek Walia is a Staff Customer Success Technical Architect at Confluent. He has years of experience implementing innovative, performance-driven, and highly scalable enterprise-level solutions for large organizations. Abhishek specializes in architecting, designing, developing, and delivering integration solutions across multiple platforms.

Guest Highlights:

“Flink is more approachable because it blends approaches together and says, ‘If you need this, you still can use this.’ It's the most powerful at this point.” - Abhishek Walia

“If you're somebody who's ever gone from normal to eventing, at some point you probably would have gone, ‘When does [the data] stop?’ It doesn't stop.” - Anna McDonald

“ Start with a fully managed service. That's probably going to save a lot of cycles for you.” - Abhishek Walia

Episode Timestamps:

*(01:35) - Anna & Abhishek’s Journey in Data Streaming

*(12:30) - Data Streaming 101: Stream Processing

*(26:30) - The Playbook: Tools & Tactics for Stream Processing

*(50:20) - Voices from the World of Data Streaming

*(56:13) - Quick Bytes

*(58:57) - Top 3 Takeaways

Links & Resources:

Connect with Joseph: @thedatagiant
Joseph’s LinkedIn: linkedin.com/in/thedatagiant
Anna’s LinkedIn: linkedin.com/in/jbfletch
Abhishek’s LinkedIn: linkedin.com/in/abhishek-walia
Introducing Derivative Event Sourcing
Designing Event-Driven Systems
Learn more at Confluent.io

Our Sponsor:

Your data shouldn’t be a problem to manage. It should be your superpower. The Confluent Data Streaming Platform transforms organizations with trustworthy, real-time data that seamlessly spans your entire environment and powers innovation across every use case. Create smarter, deploy faster, and maximize efficiency with a true Data Streaming Platform from the pioneers in data streaming. Learn more at confluent.io.