Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

From change data capture (CDC) to business development, connecting Apache Kafka® environments, and customer success stories, Graham Hainbach discusses the possibilities of data integration with Kafka and Attunity using Replicate, Compose, and Enterprise Manager. He also shares real-life examples of how Attunity best leverages Kafka in their systems.EPISODE LINKSApache Kafka Transaction Data Streaming for DummiesJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Colin McCabe shares about what it’s like being a distributed systems engineer on the Core Kafka team at Confluent, where he has worked previously, and how that led to his interest in Apache Kafka®. As an active member of the Apache open source community, he describes that the community is a place that both welcomes newcomers and fosters different ideas that help make the product the best that it can be for everyone.Being a distributed systems engineer versus a full stack engineer comes with its own unique challenges. Colin offers some advice for those interested in working with Kafka and what the interview process is like at Confluent. It’s not all about what you know, but rather how you collaborate and contribute to the team, and how you get to the answer. Part of finding the answer is getting involved with Apache projects themselves by engaging with others and helping with bug fixes as much as possible, because it’ll help you gain a better grasp on a technology that is ever-changing.EPISODE LINKSKIP-500: Apache Kafka Without ZooKeeper ft. Colin McCabe and Jason GustafsonJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Lena Hall joins Tim Berglund in the studio to talk about Apache Kafka®, the various ways to run Kafka on Microsoft Azure, Kafka on Kubernetes (K8s), and some exciting events that are happening in the Kafka world. Lena shares about serving double duty as both a senior software engineer and senior cloud developer advocate for Azure Engineering, including her unique roles and responsibilities, and how she balances engineering with advocacy. From writing tech articles to her experience with fuzzing and presence on YouTube, Lena is a strong community supporter and believes in the importance of staying rooted in the world of code as an advocate, because it helps you better understand common challenges and gives you insight as an engineer trying to fix them. It’s important to ask what's good about it and how it can be improved.They also discuss Kubernetes, the benefits of running Kafka on Kubernetes, why it’s popular, and using systems that can integrate with it. With Confluent Operator, it’s faster to spin up new environments, as well as easier to support a larger number of clusters in addition to scaling and configuration changes. As the Kafka ecosystem continues to grow and progress, one of the most notable updates of all is KIP-500 involving ZooKeeper. EPISODE LINKSApache Kafka on Kubernetes – Could You? Should You?Ask Confluent #1: Kubernetes, Confluent Operator, Kafka and KSQLFuzzing, Confluent Operator, StatefulSets, and Azure Kubernetes Service (AKS)Distributed Datastores on Kubernetes (on StatefulSets) at GOTO ChicagoRunning a Distributed Database on Kubernetes on AzureKIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum | PodcastProject Springfield, the Microsoft Project that Lena worked on in regard to KIP-500What is Apache Kafka in Azure HDInsightUse Azure Event Hubs from Apache Kafka applicationsSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

The focus of KIP-402 is to improve fairness in how Apache Kafka® processes connections and how network threads pick up requests and new data. Gwen Shapira (Engineering Manager for Cloud-Native Kafka, Confluent) outlines the details of this KIP and her team’s efforts to make user-facing Kafka improvements. Halfway through the episode, Gwen shares how to send metadata and produce client messages. EPISODE LINKSKIP-402: Improve fairness in SocksetServer processorsJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Helping users be successful when it comes to using Apache Kafka® is a large part of Dani Traphagen’s role as a senior systems engineer at Confluent. Whether she’s advising companies on implementing parts of Kafka or rebuilding their systems entirely from the ground up, Dani is passionate about event-driven architecture and the way streaming data provides real-time insights on business activity. She explains the concept of a stream, topic, key, and stream-table duality, and how each of these pieces relate to one another. When it comes to data modeling, Dani covers importance business requirements, including the need for a domain model, practicing domain-driven design principles, and bounded context. She also discusses the attributes of data modeling: time, source, key, header, metadata, and payload, in addition to exploring the significance of data governance and lineage and performing joins.EPISODE LINKSConvert from table to stream and stream to table Distributed, Real-Time Joins and Aggregations on User Activity Events Using Kafka StreamsKSQL in Action: Real-Time Streaming ETL from Oracle Transactional DataKSQL in Action: Enriching CSV Events with Data from RDBMS into AWSJourney to Event Driven – Part 4: Four Pillars of Event Streaming MicroservicesJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Joy Gao chats with Tim Berglund about all things related to streaming ETL—how it works, its benefits, and the implementation and operational challenges involved. She describes the streaming ETL architecture at WePay from MySQL/Cassandra to BigQuery using Apache Kafka®, Kafka Connect, and Debezium.  EPISODE LINKSCassandra Source Connector DocumentationStreaming Databases in Real Time with MySQL, Debezium, and KafkaStreaming Cassandra at WePayChange Data Capture with Debezium ft. Gunnar MorlingJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Todd Palino, a senior SRE at LinkedIn, talks about the start of Apache Kafka® at LinkedIn, what learning to use Kafka was like, how Kafka has changed, and what he and others in the community hope for in the future of Kafka. If you’re curious about life as an SRE, Todd shares the details on that too, and goes into how Kafka is used at LinkedIn, as well as several wins and challenges over the years with the product. EPISODE LINKSKafka: The Definitive Guide by Neha Narkhede, Gwen Shapira & Todd PalinoURP? Excuse You! The Three Metrics You Have to Know Join the Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Neil Buesing (Director of Real-Time Data, Object Partners) discusses what a day in his life looks like and how Kafka Streams helps analyze flight data.EPISODE LINKSUsing Location Data to Showcase Keys, Windows, and Joins in Kafka Streams DSL and KSQLKafka: The Definitive Guide by Neha Narkhede, Gwen Shapira & Todd PalinoRead the Confluent blogJoin the Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Tim Berglund sits down with Colin McCabe and Jason Gustafson to talk about KIP-500. The pair, who work on the Kafka Core Engineering Team, discuss the history of Kafka, the creation of KIP-500, and what it will do for the community as a whole. They break down ZooKeeper's role in Kafka, the implications of removing ZooKeeper dependency, replacing it with a self-managed metadata quorum, and how they've been combatting security, stability, and compatibility issues. With pending improvements towards scalability and inter-broker communication, and now that KIP-500 has been adopted within the community—there's a lot covered in this episode that you won't want to miss!EPISODE LINKSKIP-500: Replace ZooKeeper with a Self-Managed Metadata QuorumKIP-497: Add inter-broker API to alter ISRZooKeeper Atomic BroadcastThe Atomic Broadcast ProblemRAFT Animated GuideRAFT (Computer Science)Join the Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

When it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. Kubernetes provides all the building blocks that are needed, and a lot of thought is required to truly create an enterprise-grade Apache Kafka® platform that can be used in production. But before running Kafka on Kubernetes, there are some factors to consider. What are the maturing stages of Kubernetes adoption? How did Datadog experience these stages? Balthazar Rouberol shares what to think about before hopping on Kubernetes hype train.EPISODE LINKSKafka-Kit: Tools for Scaling KafkaRunning Production Kafka Clusters in KubernetesJoin the Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating. 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.