Building a Reliable Kafka Data Processing Pipeline With Lily Mara
The InfoQ Podcast
00:00
Postgres: A Virtual Queue for Concurrency
We use a property on each event, which is the subscription ID, more or less the row ID in Postgres. We hash that and we make a queue in memory that's for each one of the worker threads. So this means that one customer's app, they are always going to have all of their Kafka events go to the same Kafka partition. And data sets which are going to the same Postgres server, they are going to be either on the same or slightly different partitions.
Transcript
Play full episode
