How would you design a database on Object Storage?
Dec 2, 2024
auto_awesome
In this talk, Simon Hørup Eskildsen, an experienced software engineer and founder of Turbopuffer, shares insights from his vast experience in database scalability and object storage solutions. He dives into the challenges of building databases on object storage, discussing write-ahead logs, multi-tenancy issues, and the intricacies of handling multiple writers. Simon elaborates on trade-offs in write operations and the complexities of optimizing database read paths, offering practical strategies for performance enhancement. A must-listen for database enthusiasts!
The guest emphasizes the high cost of transitioning from relational to vector databases, significantly impacting financial viability in data management.
Key architectural considerations for building databases on object storage include balancing write latency with read efficiency and ensuring consistency in multi-writer scenarios.
Recent advancements in technology, such as NVMe SSDs and innovative storage solutions, have made object storage a competitive option for modern database frameworks.
Deep dives
Journey into Database Development
The guest shares his background starting from his work at Shopify, detailing his pivotal role in the company's data layer and infrastructure. He emphasizes the significance of databases in managing high traffic and stateful applications, reflecting on the impressive growth from processing under a thousand requests per second to over a million. This journey equipped him with comprehensive knowledge about database operations, including sharding and multi-tenancy protection, essential for scaling under pressure. Specifically, he highlights the critical balance between the ambitions of the product and the capabilities of the underlying infrastructure.
Challenges of Vector Workloads
A major challenge discussed is the steep cost associated with moving from a relational database like Postgres to a vector database for handling extensive workloads, particularly in article recommendation scenarios. The guest illustrates the surprising financial implications, revealing that transitioning would require escalating monthly expenditures from three thousand to thirty thousand dollars due to the increased demands of vector embeddings. Through experiments with embeddings similar to how Spotify recommends songs, he underscores the complexity and high costs associated with storing and processing vectorized data compared to traditional databases. This stark cost contrasts serve as a pivotal reason for exploring alternative architectures.
Designing a Database on Object Storage
The discussion delves into the design philosophy behind building a database atop object storage, focusing on the critical trade-offs related to write latency and read efficiency. The guest outlines a simplistic yet effective architecture where data is first written to a write-ahead log and later indexed to enable querying. This approach highlights the complexity of maintaining concurrency and ensuring order of operations when multiple writers are involved, addressing potential conflicts and the need for decision-making mechanisms. As he explains, the ability to maintain consistency while optimizing for performance and cost efficiency is crucial for a successful deployment.
Optimizing Read Performance
Strategies for enhancing query performance with this architecture are discussed, emphasizing the need for effective caching and derived indexing mechanisms. By introducing derived indexes alongside the write-ahead log, performance improvements are achieved during cold queries, where fetching data from object storage might otherwise lag. The conversation also covers the importance of designing the indexes smartly, potentially reducing round trips required for fetching data to enhance data retrieval speeds. Moreover, the necessity to persist these indexes within the object storage becomes apparent, allowing recovery from failures and scaling needs.
The Future of Database Innovation
In concluding remarks, the guest reflects on the recent advancements that have positioned object storage as a viable architecture for modern databases. He points out pivotal moments, such as the introduction of NVMe SSDs and consistent storage solutions in S3, marking foundational changes in what was technically feasible. These developments open the door for innovative database frameworks that leverage the cost-effectiveness and scalability of object storage. By harnessing these new tools and paradigms, developers can construct databases that strike a balance between performance and operating costs, revolutionizing data management strategies.
Join Kaivalya Apte and Simon Hørup Eskildsen from Turbopuffer as they talk about the complexities of building a database on top of object storage. Discover the key challenges, the nuances of various storage formats, and the critical trade-offs involved.
Learn from Simon's rich experience, from his time at Shopify to creating Turbopuffer. This episode covers everything—from approaches to write-ahead logs to multi-tenancy and object storage advancements. Perfect for database enthusiasts and those keen on first-principles thinking!
00:00 Introduction
00:17 Simon's Background and Journey to TurboBuffer
02:42 Challenges in Database Scalability
04:21 Experimenting with Vector Databases
05:02 Cost Implications of Vector Databases
05:52 Architectural Considerations for Search Workloads
07:39 Building a Database on Object Storage
16:14 Designing a Simple Database on Object Storage
26:01 Handling Multiple Writers and Consistency
31:26 Trade-offs in Write Operations
32:36 Optimizing MySQL Write Performance
34:03 Batching Writes in Object Storage
35:08 Time-Based vs Size-Based Batching
36:32 Understanding Amplification in Databases
42:26 Challenges with Cold Queries
44:02 Building and Persisting B-Trees
50:53 Separating Workloads in Databases
56:07 Multi-Tenancy Challenges
01:00:39 Choosing Storage Formats
01:06:10 Key Innovations in Object Storage Databases
Important links:
- https://github.com/sirupsen/napkin-math (numbers)
- https://turbopuffer.com/
- https://turbopuffer.com/architecture
- https://sirupsen.com/napkin/problem-10-mysql-transactions-per-second
- https://sirupsen.com (my blog, napkin math)
- https://sirupsen.com/subscribe (napkin math newsletter)
- https://github.com/rkyv/rkyv rkyv rust
Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me.
Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.
https://app.codecrafters.io/join?via=geeknarrator
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: https://youtu.be/yV_Zp0Mi3xs
Popular playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Stay Curios! Keep Learning!
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
