Building a Reliable Kafka Data Processing Pipeline With Lily Mara
The InfoQ Podcast
00:00
The Nature of Message Lag
When things start to get to be in a bad state, we can get tens or I think in rare cases we may have even hundreds of millions of messages sitting in Kafka waiting to be processed. This is not a good thing and this is an indication that your system is not healthy because you have a bunch of messages that you expected to be able to process that you somehow are lacking the capacity to process. We've recently added a metric that reports the timestamp of the most recently processed message on the consumer side. And if you have an elevated production rate, you have more HTTP requests than can be handled by your Kafka consumer coming in for a sustained period of time, you're maybe going to see
Transcript
Play full episode
