Developer Voices

What problems does Apache Kafka Solve? (with Neil Buesing)

Sep 6, 2023

Neil Buesing, a seasoned Apache Kafka® user and respected voice from the Kafka community, discusses the problems Kafka can solve, including queuing, real-time transitioning, and software integration headaches. He also explores the challenges of adopting new technology, the relationship between Kafka and microservices, and achieving client satisfaction. The significance of structured business data and the future prospects of adopting new technologies are also examined.

01:02:16

Creator website

Episode guests

Neil Buesing

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Kafka can function as a database, message queue, batch job optimizer, and data integration suite.

Kafka separates work done by different programs or applications, enabling independent data processing without complex infrastructure.

Organizations adopt Kafka to transition from batch updates to real-time streaming, enabling data enrichment and event-driven systems.

Deep dives

Kafka is a versatile tool with multiple use cases

Kafka is a tool that can function as a database, a message queue, a solution for batch job optimization, and a data integration suite. It is essential for organizations dealing with large amounts of data or multiple departments trying to collaborate effectively.

Introduction

2min

Introduction to Kafka and its Key Components

19min

Challenges of Adopting New Technology and Paradigm Shift in Problem-Solving

2min

The Relationship Between Apache Kafka and Microservices

3min

Challenges and Solutions in Writing Data to Apache Kafka

23min

Achieving Client Satisfaction and Success Stories

4min

The Importance and Benefits of Structured Business Data with Apache Kafka

7min

The Future Prospects of Adopting New Ways of Thinking and Technologies

3min

Neil Buesing is a seasoned Apache Kafka® user, and a respected voice from the Kafka community who specialises in helping companies make the best use of Kafka. And that makes him the ideal person to ask the $64,000 question: What problems can Kafka actually solve for me? Because Kafka's definitely interesting, and it can be fun, but to earn a place in the toolbox it has to make life easier.

In answering that question, Neil covers a tonne of ground, from queuing and quasi-databases, transitioning from batch to real-time, and solving general software integration headaches.

If you have data problems, big or small, join us to figure out if Kafka is the answer.

Kinetic Edge: https://www.kineticedge.io
Neil on Twitter: https://twitter.com/nbuesing
Neil on LinkedIn: https://www.linkedin.com/in/nbuesing/
Kris on Twitter: https://twitter.com/krisajenkins
Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Building an Enterprise Eventing Framework (Conference Talk): https://www.confluent.io/en-gb/kafka-summit-san-francisco-2019/building-an-enterprise-eventing-framework/
Synchronous Kafka (Conference Talk): https://www.confluent.io/resources/kafka-summit-2020/synchronous-commands-over-apache-kafka/
Dev Local: https://github.com/kineticedge/dev-local
KIP-714 - Client metrics and observability: https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

Developer Voices

What problems does Apache Kafka Solve? (with Neil Buesing)

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Kafka is a versatile tool with multiple use cases

Kafka's core functionality: recording events and enabling independent data processing

Kafka is not a traditional database but offers database-like features

Common use cases: real-time streaming and data enrichment

Barriers and challenges in adopting Kafka

Remember Everything You Learn from Podcasts