Data Council Week (Ep 6) - All About Debezium and Change Data Capture With Gunnar Morling of Decodable
Apr 27, 2023
auto_awesome
Gunnar Morling discusses Debezium's replication of data, working with Kafka, importance of documentation in open-source projects, and the vision moving forward. They cover the challenges of CDC open-source solutions and the importance of building a diverse system with common interfaces.
Debezium aims to expand beyond Kafka with new connectors for diverse data streaming platforms.
Open-source CDC solutions for diverse databases are rare but may evolve with platforms like Decodable and potential offerings from companies like Netflix.
Deep dives
Gunner Morling's Background and Decodable Journey
Gunner Morling, a senior engineer at Decodable, joined the company after spending a decade at Red Hat, specializing in projects like Bezm and Hibernate. His move was driven by a desire for new challenges and a broader data journey perspective involving data movement beyond Kafka. Decodable aims to facilitate the entire data platform journey, including data processing and Flink capabilities.
Challenges in Leave and Red Hat and Start-up Experience
Gunner Morling left Red Hat after a decade to seek new challenges and a more agile startup experience. The decision stemmed from personal reasons and a desire to explore different technical interests. Despite his fondness for Red Hat, he felt the need for a change and a fresh perspective in a smaller, faster-paced environment.
Evolution and Challenges of Debezium Project
The Debezium project, evolving into an end-to-end data platform, faces the challenge of expanding beyond a tight integration with Kafka. Efforts include developing a JDBC sync connector and a Debezium server to enhance connectivity with various data streaming platforms beyond Kafka, promoting easier setups for end-to-end data flows.
Open Source CDC Solutions and Future Prospects
While there are few open-source CDC solutions like Maxwell Demon for MySQL, fully comprehensive options for diverse databases are scarce. Possible future trends include Netflix's potential open-sourcing of their internal CDC solution and the evolution of platforms like Decodable in managing and expanding CDC offerings across different databases and platforms.
Setting the vision in early days of Red Hat and spearheading Debezium (6:20)
Replication of data in Debezium (9:47)
The patterns and processes of Debezium (16:21)
Debezium working with Kafka (19:03)
Building a diverse system while incorporating common interfaces (24:09)
The importance of documentation in open-sourced projects (27:59)
Debezium’s vision moving forward (31:32)
Why aren’t there more CDC open-sourced solutions? (34:35)
Connecting with Gunnar (37:27)
The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.
RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode