Exploring NATS: A Multi-Paradigm Connectivity Layer for Distributed Applications
Data Engineering Podcast
Scalability Challenges in Jetstream Design
This chapter explores the scalability issues encountered during the design of Jetstream, emphasizing the need for adaptive solutions to handle high consumer demands. It also contrasts the applicability of NATS and Kafka in diverse workloads and discusses future advancements in NATS for edge computing.
00:00
Transcript
Play full episode
Transcript
Episode notes
Summary
In this episode of the Data Engineering Podcast Derek Collison, creator of NATS and CEO of Synadia, talks about the evolution and capabilities of NATS as a multi-paradigm connectivity layer for distributed applications. Derek discusses the challenges and solutions in building distributed systems, and highlights the unique features of NATS that differentiate it from other messaging systems. He delves into the architectural decisions behind NATS, including its ability to handle high-speed global microservices, support for edge computing, and integration with Jetstream for data persistence, and explores the role of NATS in modern data management and its use cases in industries like manufacturing and connected vehicles.
Announcements
Parting Question
In this episode of the Data Engineering Podcast Derek Collison, creator of NATS and CEO of Synadia, talks about the evolution and capabilities of NATS as a multi-paradigm connectivity layer for distributed applications. Derek discusses the challenges and solutions in building distributed systems, and highlights the unique features of NATS that differentiate it from other messaging systems. He delves into the architectural decisions behind NATS, including its ability to handle high-speed global microservices, support for edge computing, and integration with Jetstream for data persistence, and explores the role of NATS in modern data management and its use cases in industries like manufacturing and connected vehicles.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- Your host is Tobias Macey and today I'm interviewing Derek Collison about NATS, a multi-paradigm connectivity layer for distributed applications.
- Introduction
- How did you get involved in the area of data management?
- Can you describe what NATS is and the story behind it?
- How have your experiences in past roles (cloud foundry, TIBCO messaging systems) informed the core principles of NATS?
- What other sources of inspiration have you drawn on in the design and evolution of NATS? (e.g. Kafka, RabbitMQ, etc.)
- There are several patterns and abstractions that NATS can support, many of which overlap with other well-regarded technologies. When designing a system or service, what are the heuristics that should be used to determine whether NATS should act as a replacement or addition to those capabilities? (e.g. considerations of scale, speed, ecosystem compatibility, etc.)
- There is often a divide in the technologies and architecture used between operational/user-facing applications and data systems. How does the unification of multiple messaging patterns in NATS shift the ways that teams think about the relationship between these use cases?
- How does the shared communication layer of NATS with multiple protocol and pattern adaptaters reduce the need to replicate data and logic across application and data layers?
- Can you describe how the core NATS system is architected?
- How have the design and goals of NATS evolved since you first started working on it?
- In the time since you first began writing NATS (~2012) there have been several evolutionary stages in both application and data implementation patterns. How have those shifts influenced the direction of the NATS project and its ecosystem?
- For teams who have an existing architecture, what are some of the patterns for adoption of NATS that allow them to augment or migrate their capabilities?
- What are some of the ecosystem investments that you and your team have made to ease the adoption and integration of NATS?
- What are the most interesting, innovative, or unexpected ways that you have seen NATS used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on NATS?
- When is NATS the wrong choice?
- What do you have planned for the future of NATS?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
- NATS
- NATS JetStream
- Synadia
- Cloud Foundry
- TIBCO
- Applied Physics Lab - Johns Hopkins University
- Cray Supercomputer
- RVCM Certified Messaging
- TIBCO ZMS
- IBM MQ
- JMS == Java Message Service
- RabbitMQ
- MongoDB
- NodeJS
- Redis
- AMQP == Advanced Message Queueing Protocol
- Pub/Sub Pattern
- Circuit Breaker Pattern
- Zero MQ
- Akamai
- Fastly
- CDN == Content Delivery Network
- At Most Once
- At Least Once
- Exactly Once
- AWS Kinesis
- Memcached
- SQS
- Segment
- Rudderstack
- DLQ == Dead Letter Queue
- MQTT == Message Queueing Telemetry Transport
- NATS Kafka Bridge
- 10BaseT Network
- Web Assembly
- RedPanda
- Pulsar Functions
- mTLS
- AuthZ (Authorization)
- AuthN (Authentication)
- NATS Auth Callouts
- OPA == Open Policy Agent
- RAG == Retrieval Augmented Generation
- Home Assistant
- Tailscale
- Ollama
- CDC == Change Data Capture
- gRPC
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
