Reinventing Stream Processing: From LinkedIn to Responsive with Apurva Mehta
Mar 6, 2025
auto_awesome
In this installment, Apurva Mehta, co-founder and CEO of Responsive, shares insights from his journey in stream processing at LinkedIn and Confluent. He breaks down the evolution of stream processing from simple tasks to powering complex applications. Apurva clarifies the concept of 'real time,' emphasizing low latency over instant responses. He discusses the pitfalls of traditional databases in handling high-update rates and explains how Responsive innovates by decoupling state from compute to enhance efficiency and operational simplicity.
Apurva Mehta shares his journey from LinkedIn to Responsive, highlighting stream processing's evolution and growing importance across industries.
The podcast clarifies the concept of 'real-time' processing, emphasizing low latency as a more realistic expectation for many applications.
Innovative state management strategies using remote storage solutions like S3 are discussed, aimed at improving efficiency in complex stream processing systems.
Deep dives
Background and Career Journey in Stream Processing
The speaker describes their career trajectory, emphasizing their extensive experience in stream processing, which began around 2013 at LinkedIn. They contributed to the development of Kafka, particularly in implementing transaction capabilities, and later focused on stream processing jobs for indexing within LinkedIn's graph database and search index. At Confluent, they further honed their expertise, leading teams that developed stream processing solutions and observing the growing importance of these technologies in various industries. Their background sets the foundation for understanding the evolution of stream processing applications and the challenges that come with them.
Complexity of Real-Time Stream Processing
The nuances of what constitutes 'real-time' processing are explored, highlighting the distinction between low-latency requirements and traditional real-time definitions. It is suggested that low latency is often more realistic for many use cases involving stream processing, where milliseconds or seconds are tolerable. Additionally, the speaker discusses practical implementations, like managing real-time updates to search indexes or social networks, illustrating how complex computations are necessary to refresh data accurately and efficiently. The conversation emphasizes the sophisticated nature of these systems and the need for robust architectures to support them.
Challenges of Stateful Stream Processing
The podcast delves into the necessity of stateful stream processing, which is essential for applications requiring the background of past events to influence future decisions. Use cases such as inventory management illustrate the importance of preserving state to handle high updates and complex operations effectively. The speaker highlights that while stateless solutions may serve simpler scenarios, many business applications require maintaining memory of events to function effectively. This necessity underscores the complexity and the operational challenges in managing state within distributed systems.
Innovations in State Management and Architecture
A discussion on evolving state management strategies reveals insights into using RocksDB and the idea of separating storage from compute to enhance efficiency. The speaker proposes that moving state storage to systems like S3 helps alleviate operational burdens and simplifies the complexity of maintaining a distributed database. This shift allows for leveraging Kafka's transactional features to create more reliable systems while enabling greater flexibility in managing state. By pursuing these innovations, the industry aims to enhance the user experience for developers and address some of the significant pain points in stream processing.
Future Perspectives on Stream Processing Applications
Looking ahead, the speaker anticipates a surge in recognition and adoption of stream processing applications as enhancements in technology become more mainstream. The notion of stream processing being intertwined with app development is emphasized, suggesting that improvements in tooling and infrastructure will enable businesses to realize more complex capabilities. Currently viewed as a 'wild west,' the speaker sees a future where developers have robust resources to build sophisticated applications efficiently. Overall, these predictions suggest a transformative shift in how businesses can leverage streaming technologies to enhance their operational frameworks.
In this episode, Apurva Mehta, co-founder and CEO of Responsive, recounts his extensive journey in stream processing—from his early work at LinkedIn and Confluent to his current venture at Responsive.
He explains how stream processing evolved from simple event ingestion and graph indexing to powering complex, stateful applications such as search indexing, inventory management, and trade settlement.
Apurva clarifies the often-misunderstood concept of “real time,” arguing that low latency (often in the one- to two-second range) is more accurate for many applications than the instantaneous response many assume. He delves into the challenges of state management, discussing the limitations of embedded state stores like RocksDB and traditional databases (e.g., Postgres) when faced with high update rates and complex transactional requirements.
The conversation also covers the trade-offs between SQL-based streaming interfaces and more flexible APIs, and how Responsive is innovating by decoupling state from compute—leveraging remote state solutions built on object stores (like S3) with specialized systems such as SlateDB—to improve elasticity, cost efficiency, and operational simplicity in mission-critical applications.
Chapters
00:00 Introduction to Apurva Mehta and Streaming Background 08:50 Defining Real-Time in Streaming Contexts 14:18 Challenges of Stateful Stream Processing 19:50 Comparing Streaming Processing with Traditional Databases 26:38 Product Perspectives on Streaming vs Analytical Systems 31:10 Operational Rigor and Business Opportunities 38:31 Developers' Needs: Beyond SQL 45:53 Simplifying Infrastructure: The Cost of Complexity 51:03 The Future of Streaming Applications
