Real-Time Data Migration and Processing Challenges

This week we’re looking at Debezium - an open source project that taps into a huge number of databases and lets you stream data to other systems in real time. It’s a huge project that covers a wide range of uses: Some people use it to replicate from Oracle to MySQL, others to do smart cache invalidation, and others to build a bridge from an existing relational database to the event-sourcing world. If you’re working on a system that has more than one kind of database, it may be an essential tool. But what exactly does it do, and how does it do it?

Joining us for a deep dive is Debezium expert and former project lead, Gunnar Morling. He takes us through all things Debezium, from who uses Debezium and why; which systems you can connect to and what data you get out from it; and how a project of this scope is developed.

Gunnar on Twitter: https://twitter.com/gunnarmorling
Gunnar on LinkedIn: https://www.linkedin.com/in/gunnar-morling-2b44b7229/
Gunnar’s blog: https://www.morling.dev/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Gunnar’s talk on the new incremental snapshot mechanism: https://events.bizzabo.com/468544/agenda/session/1136877
Getting into Real Time data: https://youtu.be/ftAVFxa5AwI
Stripe’s talk at Flink Forward: https://www.slideshare.net/FlinkForward/squirreling-away-640-billion-how-stripe-leverages-flink-for-change-data-capture
Francesco’s “When JDBC Goes Wrong” talk: https://www.confluent.io/events/kafka-summit-london-2022/jdbc-source-connector-what-could-go-wrong/

#debezium #microservicesarchitecture #cdc #database #postgresql #oracle #mysql #eventsourcing #kafka #jdbc #realtime