Monday Morning Data Chat #108 - Switching From a Batch to Streaming Mindset w/ Chip Huyen
4 snips
Jan 9, 2023 AI Snips
Chapters
Transcript
Episode notes
Streaming vs. Batch
- Companies are hesitant to adopt streaming due to misconceptions about abandoning batch processing.
- Streaming and batch processing can coexist, with each serving different use cases based on data freshness needs.
Team's Streaming Struggle
- Chip Huyen shares an anecdote about a team struggling with the concept of long-running streaming jobs.
- They found the continuous nature of streaming and the need to maintain a service daunting.
Data Freshness
- No company desires stale data; data freshness is always preferred.
- Maintaining separate batch and streaming pipelines for the same data is inefficient and undesirable.