The GeekNarrator Modern OLAP Database System Design with FDAP (Andrew Lamb)
15 snips
Jun 5, 2024 
Andrew Lamb, Staff Software Engineer at InfluxDB and chair of the Apache Data Fusion project, shares his expertise on modern OLAP database design. He explains the power of the FDAP stack, highlighting how Apache Parquet and Arrow enhance data storage and retrieval efficiency. The conversation delves into the challenges of data immutability and management, while also discussing Flight's role in simplifying data transfer. Looking ahead, Andrew envisions evolving trends in database technologies, paving the way for innovative solutions in analytics.
AI Snips
Chapters
Transcript
Episode notes
Analytics Workload Focus
- Analytics workloads focus on throughput (rows per second) rather than individual transactions.
- They involve large-scale aggregations, statistics computation, and data slicing for various consumers.
Bottlenecks in Traditional Analytics
- Traditional analytical systems face bottlenecks due to increasing data volumes and velocity.
- FDAP aims to address these limitations by providing pre-built, optimized components.
Genesis of FDAP
- Paul Dix, seeking to rebuild InfluxDB, wanted an analytics engine with columnar storage and other features.
- He found that re-implementing these common components was expensive, leading to the creation of FDAP.