How AI Is Built  cover image

How AI Is Built

Building the database for AI, Multi-modal AI, Multi-modal Storage | S2 E10

Oct 23, 2024
Chang She, CEO of Lens and co-creator of the Pandas library, shares insights on building LanceDB for AI data management. He discusses how LanceDB tackles data bottlenecks and speeds up machine learning experiments with unstructured data. The conversation dives into the decision to use Rust for enhanced performance, achieving up to 1,000 times faster results than Parquet. Chang also explores multimodal AI's challenges, future applications of LanceDB in recommendation systems, and the vision for more composable data infrastructures.
44:54

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

  • LanceDB addresses data management complexities in AI by optimizing for large-scale vector storage, enabling rapid access and effective training.
  • The transition from C++ to Rust significantly enhanced LanceDB's development speed and safety, improving productivity in AI data infrastructure management.

Deep dives

Challenges of Multimodal Embeddings in AI

Working with multimodal embeddings in AI presents significant challenges, primarily related to storage and data access patterns. In enterprise settings, it's common to employ multiple storage solutions concurrently, such as using blob storage for source data and a vector database for embeddings, leading to complexities in data management. Additionally, effective training requires random data access, filtering, and even stratified sampling, which complicates the ability to manage evolving datasets. Solutions for these challenges focus on optimizing large-scale storage to meet the needs of AI applications, which often involve diverse access requirements.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode