The GeekNarrator cover image

The GeekNarrator

Modern OLAP Database System Design with FDAP (Andrew Lamb)

Jun 5, 2024
Andrew Lamb, Staff Software Engineer at InfluxDB and chair of the Apache Data Fusion project, shares his expertise on modern OLAP database design. He explains the power of the FDAP stack, highlighting how Apache Parquet and Arrow enhance data storage and retrieval efficiency. The conversation delves into the challenges of data immutability and management, while also discussing Flight's role in simplifying data transfer. Looking ahead, Andrew envisions evolving trends in database technologies, paving the way for innovative solutions in analytics.
56:48

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

  • The FDAP stack, integrating Apache Arrow, Parquet, Flight, and Data Fusion, revolutionizes modern OLAP database systems by enhancing performance and efficiency.
  • Data Fusion enables flexible SQL processing and custom query languages while optimizing execution through its extendable architecture for varied use cases.

Deep dives

Introduction to Data Fusion and the FDAP Stack

Data Fusion represents a significant advancement in database design by utilizing the FDAP stack, which comprises Apache Arrow, Apache Parquet, Apache Arrow Flight, and Data Fusion itself. This stack aims to improve analytical workloads by providing high performance and efficiency during data processing. The intention behind developing the FDAP stack was to address the limitations of traditional analytical systems, particularly their reliance on legacy database technologies that fall short in today's data-intensive environments. By integrating these components, developers can leverage a shared framework that streamlines both storage and processing functionalities essential for handling massive datasets.

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode