Data Engineering Podcast

Feldera: Bridging Batch and Streaming with Incremental Computation

Nov 4, 2024

Leonid Ryzhyk, CTO of Feldera, along with CEO Lalith Suresh and Chief Science Officer Mihai Budiu, dive into the world of incremental computation. They discuss how Feldera bridges batch and streaming data seamlessly, revolutionizing real-time machine learning applications like fraud detection. The trio highlights the architecture's evolution, emphasizing historical data analysis and feature engineering. They also tackle the skepticism surrounding traditional streaming technologies and explore Feldera's potential in both the open-source community and enterprise solutions.

47:36

Creator website

Episode guests

Leonid Ryzhyk

Lalith Suresh

Mihai Budiu

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Feldera's incremental compute engine revolutionizes data processing by efficiently merging batch and streaming capabilities, enhancing real-time ML and AI workloads.

The founders emphasize the importance of educating users about incremental computation to overcome misconceptions and promote the technology's operational efficiency.

Deep dives

Introduction to Feldera and Incremental Computation

Feldera is an incremental compute engine designed for continuous computation in data, machine learning (ML), and artificial intelligence (AI) workflows. It enhances efficiency by enabling fast query execution while avoiding redundant calculations typical of batch processing. The founders, with extensive experience in computer science, aimed to leverage their background to create a technology that optimizes data processing through incremental updates. This eliminates the need for substantial computational resources for frequent, minor data changes, offering both batch and real-time analytical capabilities.

Intro

2min

Introduction to Feldera and Incremental Computation

17min

Exploring Incremental Computation with Feldera

15min

Enhancing Real-Time Machine Learning with Streaming Data

2min

Innovative Use Cases of Feldera and Open Source Development

2min

Navigating Streaming Technology Challenges

8min

Bridging Gaps in Data Management and Incremental Computation

2min

Summary
In this episode of the Data Engineering Podcast, the creators of Feldera talk about their incremental compute engine designed for continuous computation of data, machine learning, and AI workloads. The discussion covers the concept of incremental computation, the origins of Feldera, and its unique ability to handle both streaming and batch data seamlessly. The guests explore Feldera's architecture, applications in real-time machine learning and AI, and challenges in educating users about incremental computation. They also discuss the balance between open-source and enterprise offerings, and the broader implications of incremental computation for the future of data management, predicting a shift towards unified systems that handle both batch and streaming data efficiently.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Imagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at dataengineeringpodcast.com/datafold today!
As a listener of the Data Engineering Podcast you clearly care about data and how it affects your organization and the world. For even more perspective on the ways that data impacts everything around us you should listen to Data Citizens® Dialogues, the forward-thinking podcast from the folks at Collibra. You'll get further insights from industry leaders, innovators, and executives in the world's largest companies on the topics that are top of mind for everyone. They address questions around AI governance, data sharing, and working at global scale. In particular I appreciate the ability to hear about the challenges that enterprise scale businesses are tackling in this fast-moving field. While data is shaping our world, Data Citizens Dialogues is shaping the conversation. Subscribe to Data Citizens Dialogues on Apple, Spotify, Youtube, or wherever you get your podcasts.
Your host is Tobias Macey and today I'm interviewing Leonid Ryzhyk, Lalith Suresh, and Mihai Budiu about Feldera, an incremental compute engine for continous computation of data, ML, and AI workloads

Interview

Introduction
Can you describe what Feldera is and the story behind it?
DBSP (the theory behind Feldera) has won multiple awards from the database research community. Can you explain what it is and how it solves the incremental computation problem?
Depending on which angle you look at it, Feldera has attributes of data warehouses, federated query engines, and stream processors. What are the unique use cases that Feldera is designed to address?
- In what situations would you replace another technology with Feldera?
- When is it an additive technology?
Can you describe the architecture of Feldera?
- How have the design and scope evolved since you first started working on it?
What are the state storage interfaces available in Feldera?
- What are the opportunities for integrating with or building on top of open table formats like Iceberg, Lance, Hudi, etc.?
Can you describe a typical workflow for an engineer building with Feldera?
You advertise Feldera's utility in ML and AI use cases in addition to data management. What are the features that make it conducive to those applications?
What is your philosophy toward the community growth and engagement with the open source aspects of Feldera and how you're balancing that with sustainability of the project and business?
What are the most interesting, innovative, or unexpected ways that you have seen Feldera used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Feldera?
When is Feldera the wrong choice?
What do you have planned for the future of Feldera?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

Data Engineering Podcast

Feldera: Bridging Batch and Streaming with Incremental Computation

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Introduction to Feldera and Incremental Computation

Comparison with Existing Technologies

Feldera's Architectural Capabilities

Challenges and Future Directions

Remember Everything You Learn from Podcasts