Exploring NATS: A Multi-Paradigm Connectivity Layer for Distributed Applications
Apr 28, 2025
auto_awesome
Derek Collison, the creator of NATS and CEO of Synadia, shares insights from his impressive background at Google and VMware. He discusses how NATS revolutionizes messaging systems with innovative features like the circuit breaker pattern and Jetstream. Derek highlights NATS’s advantages in edge computing, emphasizing its resilience and data persistence capabilities. He also addresses the challenges of open-source technology and shares thoughts on the future of connectivity in modern distributed systems, proving NATS's versatility across various industries.
Derek Collison, creator of NATS, emphasizes its evolution as a connectivity layer designed for reliability and efficiency in distributed systems.
NATS distinguishes itself from traditional messaging systems by offering robust delivery mechanisms while minimizing the chance of overload in data processing.
The integration of NATS with technologies like Jetstream and its adaptability for use cases in industries such as manufacturing and IoT highlights its evolving relevance.
Deep dives
The Problem of Data Migrations
Data migrations are often lengthy and resource-intensive, causing significant strain on teams during the process. Companies frequently find themselves overwhelmed when migrating data across platforms, which can take months or even years. The frustration experienced by teams is compounded by the potential for morale to plummet when they encounter obstacles or delays. However, solutions like Datafold’s AI-powered migration agent are addressing these challenges by facilitating faster migrations through code translation and automated validation.
Derek Collison and the Genesis of NATS
Derek Collison, the founder of Zinedia and creator of NATS, has a background in distributed systems, which informs his perspectives on messaging technologies. NATS originated from Collison's work on Cloud Foundry, where there was a demand for a messaging system that could provide reliability and efficiency. Unlike RabbitMQ, which he felt was not adequately designed to protect against failures, NATS aimed to function as a reliable, resilient layer for communication among distributed applications. The result is a system that strives to provide a consistent 'dial tone' amidst various operational demands.
Features and Advantages of NATS
NATS offers several key features designed to improve message delivery and system robustness. One distinctive feature is the ability to deliver messages to a group of consumers in a distributed manner, minimizing the chance of overload and ensuring that messages are efficiently handled. NATS also emphasizes a protective approach where the system safeguards itself from overwhelming loads, unlike traditional messaging systems. By leveraging such characteristics, NATS provides an agile environment for developing scalable applications while addressing the challenges of modern data flow.
The Future Vision of NATS
The vision for NATS is to serve as a foundational technology that modernizes how distributed systems are architected, accommodating the rapidly evolving demands of edge computing and IoT. As NATS expands its capabilities to include things like Jetstream, which enables the storage and retrieval of messages, it also seeks to simplify integration with existing systems and promote flexibility in deployment. By creating a wide range of connectors for various services, the aim is to make NATS a central hub for data flows across diverse environments. This forward-thinking approach ensures NATS remains relevant and adaptable to emerging technological landscapes.
Real-World Applications and Lessons Learned
NATS is being utilized in a variety of innovative ways that illustrate its flexibility and resilience across different domains. From manufacturing and connected cars to AI inference at the edge, the adaptability of NATS has surprised many in the industry as they push its limits in deployment scenarios. A key insight is that while NATS aims to be user-friendly, it is essential for teams to understand the complexities involved in deploying distributed systems effectively. Organizations that have integrated NATS have experienced transformative changes, although they also face the challenge of ensuring their architecture scales with their increasingly complex demands.
Summary In this episode of the Data Engineering Podcast Derek Collison, creator of NATS and CEO of Synadia, talks about the evolution and capabilities of NATS as a multi-paradigm connectivity layer for distributed applications. Derek discusses the challenges and solutions in building distributed systems, and highlights the unique features of NATS that differentiate it from other messaging systems. He delves into the architectural decisions behind NATS, including its ability to handle high-speed global microservices, support for edge computing, and integration with Jetstream for data persistence, and explores the role of NATS in modern data management and its use cases in industries like manufacturing and connected vehicles.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Derek Collison about NATS, a multi-paradigm connectivity layer for distributed applications.
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what NATS is and the story behind it?
How have your experiences in past roles (cloud foundry, TIBCO messaging systems) informed the core principles of NATS?
What other sources of inspiration have you drawn on in the design and evolution of NATS? (e.g. Kafka, RabbitMQ, etc.)
There are several patterns and abstractions that NATS can support, many of which overlap with other well-regarded technologies. When designing a system or service, what are the heuristics that should be used to determine whether NATS should act as a replacement or addition to those capabilities? (e.g. considerations of scale, speed, ecosystem compatibility, etc.)
There is often a divide in the technologies and architecture used between operational/user-facing applications and data systems. How does the unification of multiple messaging patterns in NATS shift the ways that teams think about the relationship between these use cases?
How does the shared communication layer of NATS with multiple protocol and pattern adaptaters reduce the need to replicate data and logic across application and data layers?
Can you describe how the core NATS system is architected?
How have the design and goals of NATS evolved since you first started working on it?
In the time since you first began writing NATS (~2012) there have been several evolutionary stages in both application and data implementation patterns. How have those shifts influenced the direction of the NATS project and its ecosystem?
For teams who have an existing architecture, what are some of the patterns for adoption of NATS that allow them to augment or migrate their capabilities?
What are some of the ecosystem investments that you and your team have made to ease the adoption and integration of NATS?
What are the most interesting, innovative, or unexpected ways that you have seen NATS used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on NATS?
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
