The separation of compute and storage allows for the scaling up of storage and compute separately, enabling systems to process huge amounts of data using seemingly weak compute resources. This separation is achieved through a columnar format paired with an index, such as a vector index, allowing for fast random access and scan. The new columnar format like lands provides the ability for fast random access and fast scans, essential for interactive query performance. This separation is made possible by the underlying data architecture, which supports storing vectors and data. The data architecture relies on wide tables in columnar format and an index, while the connection between the index and columnar format enables fast random access and query performance. Additionally, the core of the data format and the vector database is implemented in Rust, which is also used to build embedded databases in JavaScript.
Prashanth Rao mentioned LanceDB as a stand out amongst the many vector DB options in episode #234. Now, Chang She (co-founder and CEO of LanceDB) joins us to talk through the specifics of their open source, on-disk, embedded vector search offering. We talk about how their unique columnar database structure enables serverless deployments and drastic savings (without performance hits) at scale. This one is super practical, so don’t miss it!
Leave us a comment
Changelog++ members save 1 minute on this episode because they made the ads disappear. Join today!
Sponsors:
- Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com
- Fly.io – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs.
- Typesense – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster!
Featuring:
Show Notes:
Something missing or broken? PRs welcome!