Data Engineering Podcast Simplifying Continuous Data Processing Using Stream Native Storage In Pravega with Tom Kaitchuck - Episode 63
Summary
As more companies and organizations are working to gain a real-time view of their business, they are increasingly turning to stream processing technologies to fullfill that need. However, the storage requirements for continuous, unbounded streams of data are markedly different than that of batch oriented workloads. To address this shortcoming the team at Dell EMC has created the open source Pravega project. In this episode Tom Kaitchuk explains how Pravega simplifies storage and processing of data streams, how it integrates with processing engines such as Flink, and the unique capabilities that it provides in the area of exactly once processing and transactions. And if you listen at approximately the half-way mark, you can hear as the hosts mind is blown by the possibilities of treating everything, including schema information, as a stream.
Preamble
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
- Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
- Your host is Tobias Macey and today I’m interviewing Tom Kaitchuck about Pravega, an open source data storage platform optimized for persistent streams
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by explaining what Pravega is and the story behind it?
- What are the use cases for Pravega and how does it fit into the data ecosystem?
- How does it compare with systems such as Kafka and Pulsar for ingesting and persisting unbounded data?
- How do you represent a stream on-disk?
- What are the benefits of using this format for persisted streams?
- One of the compelling aspects of Pravega is the automatic sharding and resource allocation for variations in data patterns. Can you describe how that operates and the benefits that it provides?
- I am also intrigued by the automatic tiering of the persisted storage. How does that work and what options exist for managing the lifecycle of the data in the cluster?
- For someone who wants to build an application on top of Pravega, what interfaces does it provide and what architectural patterns does it lend itself toward?
- What are some of the unique system design patterns that are made possible by Pravega?
- How is Pravega architected internally?
- What is involved in integrating engines such as Spark, Flink, or Storm with Pravega?
- A common challenge for streaming systems is exactly once semantics. How does Pravega approach that problem?
- Does it have any special capabilities for simplifying processing of out-of-order events?
- For someone planning a deployment of Pravega, what is involved in building and scaling a cluster?
- What are some of the operational edge cases that users should be aware of?
- What are some of the most interesting, useful, or challenging experiences that you have had while building Pravega?
- What are some cases where you would recommend against using Pravega?
- What is in store for the future of Pravega?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
- Pravega
- Amazon SQS (Simple Queue Service)
- Amazon Simple Workflow Service (SWF)
- Azure
- EMC
- Zookeeper
- Bookkeeper
- Kafka
- Pulsar
- RocksDB
- Flink
- Spark
- Heron
- Lambda Architecture
- Kappa Architecture
- Erasure Code
- Flink Forward Conference
- CAP Theorem
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA