Glauber Costa, co-founder and CEO of Turso, is reshaping database technology with Limbo, a transformative rewrite of SQLite in Rust. He delves into the limitations of SQLite's public domain status and the innovative LibSQL project designed for open contributions. Their use of Deterministic Simulation Testing ensures robust development, while engaging the community keeps the project dynamic. Glauber highlights the technical challenges of maintaining compatibility and introduces exciting enhancements like asynchronous I/O, all while paving the way for the future of database innovation.
Turso is creating a truly open-source alternative to SQLite by rewriting it in Rust, aiming for community involvement and contributions.
The Limbo project seeks to overcome SQLite's limitations by introducing asynchronous operations and enhanced support for complex queries and analytics.
Deterministic simulation testing is being implemented by Terso to ensure high reliability and robust performance for their database solution Limbo.
Deep dives
Introduction to Terso and Limbo
Terso, founded by Glabra Costa, is an open-source distributed database that seeks to replace SQLite. It is powered by a project called Limbo, which is essentially a rewrite of SQLite in Rust. Unlike SQLite, which is in the public domain but does not allow contributions, Terso aims to create a truly open-source database that encourages community involvement. The move to rewrite SQLite rather than simply forking it reflects the team's desire to innovate and create a more dynamic and contributory environment.
LibSQL and Challenges of SQLite
Terso started with the development of LibSQL, a fork of SQLite meant to allow contributions and facilitate certain operational enhancements. The creators encountered challenges with SQLite's limitations, such as its inability to accept external contributions and the proprietary nature of its testing suite. They recognized that while SQLite is an efficient database for certain simple use cases, it falls short for operational tasks like replication and analytics. These limitations prompted the team to consider a fundamental rewrite, leading to the Limbo project.
Testing with Deterministic Simulation Testing (DST)
To ensure the reliability and robustness of Limbo, the Terso team is implementing deterministic simulation testing (DST) as a key testing framework. This method allows the developers to simulate various operational scenarios systematically, identifying bugs and performance issues with high precision. By integrating DST from the start, the team aims to surpass the testing quality of SQLite, ensuring that Limbo not only functions well but also supports extremely high reliability. The use of DST will enable the team to detect and resolve potential issues early in the development process.
Asynchronous Operations and Compatibility Goals
Limbo aims to enhance the functionality of SQLite by introducing asynchronous operations, allowing greater responsiveness and efficiency in data handling. This change will enable Limbo to support more complex queries and serve better in environments like cloud or serverless setups where responsiveness is critical. While maintaining backward compatibility with SQLite, Limbo aspires to offer additional features such as faster schema changes and support for analytical workloads. The project intends to create not only a more capable database but also one that developers in the community can actively contribute to and help evolve.
Future of Terso and Community Contributions
Terso is committed to transforming the future of database management by inviting community contributions to Limbo, fostering an inclusive environment. As the project evolves, the team envisions situations where community members not only participate in discussions but also become maintainers, thus empowering developers to shape the project's direction. The strategic decision to separate the Terso Cloud server implementation from Limbo's client-side library also reflects their commitment to clear operational goals without compromising the openness of the client. Ultimately, Terso aims to achieve significant adoption within the database realm, aspiring for a future where Limbo serves as a trusted and widely-used alternative to SQLite.
Glauber Costa, co-founder and CEO of Turso, joins us to discuss libSQL, Limbo, and how they’re rewriting SQLite in Rust. We discuss their efforts with libSQL, the challenge of SQLite being in the public domain but not being open for contribution, their choice to rewrite everything with Limbo, how this all plays into the future of the Turso platform, how they test Limbo with Deterministic Simulation Testing (DST), and their plan to replace SQLite.
Changelog++ members save 6 minutes on this episode because they made the ads disappear. Join today!
Sponsors:
Retool – The low-code platform for developers to build internal tools — Some of the best teams out there trust Retool…Brex, Coinbase, Plaid, Doordash, LegalGenius, Amazon, Allbirds, Peloton, and so many more – the developers at these teams trust Retool as the platform to build their internal tools. Try it free at retool.com/changelog
Temporal – Build invincible applications. Manage failures, network outages, flaky endpoints, long-running processes and more, ensuring your workflows never fail. Register for Replay in London, March 3-5 to break free from the status quo.
Notion – Notion is a place where any team can write, plan, organize, and rediscover the joy of play. It’s a workspace designed not just for making progress, but getting inspired. Notion is for everyone — whether you’re a Fortune 500 company or freelance designer, starting a new startup or a student juggling classes and clubs.
Fly.io – The home of Changelog.com — Deploy your apps close to your users — global Anycast load-balancing, zero-configuration private networking, hardware isolation, and instant WireGuard VPN connections. Push-button deployments that scale to thousands of instances. Check out the speedrun to get started in minutes.
