In this video I talk to Philip Fried from Estuary about Batch vs Realtime Stream Processing. Philip brings a ton of experience in the world of data processing and has shared some of the best practices in implementing these systems. We dive deep into the world of data processing, covering batch and streaming systems, their challenges, tradeoffs and use cases. Chapters: 00:00 Batch vs Realtime Stream Processing 03:25 What is Batch and Reatlime processing? 18:29 How does Batch and Realtime compare in terms of Latency and Throughput? 27:24 Where is the cost saving coming from? Compute?Storage? or Network? 31:38 Moving from Batch to Stream processing 37:50 How is Idempotency implemented in Streaming systems? 48:50 How do we approach Schema evolution in Batch and Streaming systems? 57:16 Summary - key points to keep in mind Do checkout Estuary if you deal with a ton of data, and don't want to deal with the painful operations, infrastructure management, schema migrations etc and only want to focus on building highly scalable and resilient applications. References: Estuary: https://estuary.dev/ Flow documentation: https://docs.estuary.dev If you like this video please hit the like button, share it with your network (whoever works with a ton of data) and subscribe to the channel. Feel free to watch related episodes in the playlist: • Distributed Syste... Modern Databases: • Modern Databases Software Engineering: • Software Engineering Distributed Systems: • Distributed Systems Cheers, The GeekNarrator

Make Your Minutes Count | Rafa Conde