The GeekNarrator cover image

The GeekNarrator

How would you design a database on Object Storage?

Dec 2, 2024
In this talk, Simon Hørup Eskildsen, an experienced software engineer and founder of Turbopuffer, shares insights from his vast experience in database scalability and object storage solutions. He dives into the challenges of building databases on object storage, discussing write-ahead logs, multi-tenancy issues, and the intricacies of handling multiple writers. Simon elaborates on trade-offs in write operations and the complexities of optimizing database read paths, offering practical strategies for performance enhancement. A must-listen for database enthusiasts!
01:08:26

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

  • The guest emphasizes the high cost of transitioning from relational to vector databases, significantly impacting financial viability in data management.
  • Key architectural considerations for building databases on object storage include balancing write latency with read efficiency and ensuring consistency in multi-writer scenarios.

Deep dives

Journey into Database Development

The guest shares his background starting from his work at Shopify, detailing his pivotal role in the company's data layer and infrastructure. He emphasizes the significance of databases in managing high traffic and stateful applications, reflecting on the impressive growth from processing under a thousand requests per second to over a million. This journey equipped him with comprehensive knowledge about database operations, including sharding and multi-tenancy protection, essential for scaling under pressure. Specifically, he highlights the critical balance between the ambitions of the product and the capabilities of the underlying infrastructure.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode