Data Engineering Podcast

Overcoming Redis Limitations: The Dragonfly DB Approach

Mar 30, 2025

Roman Gershman, CTO and founder of Dragonfly DB, shares his journey from Google to creating a high-speed alternative to Redis. He dives into the challenges of developing in-memory databases, focusing on performance, scalability, and cost efficiency. Roman discusses operational complexities users face, while highlighting Dragonfly's compatibility with Redis and innovations like SSD tiering. He also explores programming trade-offs between C++ and Rust, emphasizing adaptability in database development and the importance of community feedback in shaping future advancements.

43:58

Creator website

Episode guests

Roman Gershman

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Dragonfly DB was developed to overcome the limitations of Redis, focusing on performance, scalability, and operational simplicity for better user experience.

The podcast emphasizes the importance of cost efficiency and memory management innovations like SSD tiering to support various applications and future growth.

Deep dives

Challenges in Data Migration

Data migrations can be extensive and resource-intensive, often leading to prolonged project durations that demoralize teams. Traditional migration processes often take months or even years, consuming significant resources without assurances of timely completion. In contrast, new AI-powered solutions, such as Datafold's migration agent, promise to expedite this process, allowing companies to complete migrations up to ten times faster than conventional methods. These innovations not only enhance efficiency but also provide written guarantees for migration timelines, fundamentally shifting how companies approach data migrations.

Intro

3min

The Genesis of Dragonfly DB

27min

Language Choices in Memory-Intensive Systems

2min

Navigating Challenges in Database Development

3min

Enhancing Dragonfly: Interface Innovations and Challenges

8min

Final Insights and Community Engagement

2min

Summary
In this episode of the Data Engineering Podcast Roman Gershman, CTO and founder of Dragonfly DB, explores the development and impact of high-speed in-memory databases. Roman shares his experience creating a more efficient alternative to Redis, focusing on performance gains, scalability, and cost efficiency, while addressing limitations such as high throughput and low latency scenarios. He explains how Dragonfly DB solves operational complexities for users and delves into its technical aspects, including maintaining compatibility with Redis while innovating on memory efficiency. Roman discusses the importance of cost efficiency and operational simplicity in driving adoption and shares insights on the broader ecosystem of in-memory data stores, future directions like SSD tiering and vector search capabilities, and the lessons learned from building a new database engine.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Roman Gershman about building a high-speed in-memory database and the impact of the performance gains on data applications

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what DragonflyDB is and the story behind it?
What is the core problem/use case that is solved by making a "faster Redis"?
The other major player in the high performance key/value database space is Aerospike. What are the heuristics that an engineer should use to determine whether to use that vs. Dragonfly/Redis?
Common use cases for Redis involve application caches and queueing (e.g. Celery/RQ). What are some of the other applications that you have seen Redis/Dragonfly used for, particularly in data engineering use cases?
There is a piece of tribal wisdom that it takes 10 years for a database to iron out all of the kinks. At the same time, there have been substantial investments in commoditizing the underlying components of database engines. Can you describe how you approached the implementation of DragonflyDB to arive at a functional and reliable implementation?
What are the architectural elements that contribute to the performance and scalability benefits of Dragonfly?
- How have the design and goals of the system changed since you first started working on it?
For teams who migrate from Redis to Dragonfly, beyond the cost savings what are some of the ways that it changes the ways that they think about their overall system design?
What are the most interesting, innovative, or unexpected ways that you have seen Dragonfly used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on DragonflyDB?
When is DragonflyDB the wrong choice?
What do you have planned for the future of DragonflyDB?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links