Developer Voices

How Apache Pinot Achieves 200,000 Queries per Second (with Tim Berglund)

Mar 20, 2024

Discover how Apache Pinot achieves an impressive 200,000 queries per second and the architectural decisions behind it. Tim Berglund explains the roles, optimization, and performance of Pinot, covering queries, data movement, and indexing strategies. Learn about the evolution of databases, query optimization, and the technical details of real-time data applications with Apache Pinot.

01:14:28

Creator website

Episode guests

Tim Berglund

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Scaling databases face challenges at each growth stage, requiring advanced clustering solutions.

Transactional and analytics-focused databases require specialized optimizations to handle different types of queries effectively.

Apache Pinot optimizes query processing through data partitioning, query distribution, and specialized indexing techniques.

Deep dives

Scalability of Database Architectures

As demands on a system increase exponentially, the architecture faces unique challenges at each power of 10 growth. The podcast discusses how databases handle varying levels of concurrent queries, highlighting the shift in strategies as query loads rise. For instance, while some relational databases can manage thousands of concurrent queries, scaling beyond certain thresholds may require advanced approaches like clustering multiple machines.

Introduction

3min

Evolution of Databases: From Batch Analytics to Interactive Capabilities

9min

Optimizing User Experience and Database Performance

2min

Apache Pinot Architecture and Query Optimization

17min

Optimizing Query Processing and Data Movement in a Server Cluster

11min

Exploring Technology Evolution and Resource Allocation

2min

Exploring Advanced Indexing Strategies in Apache Pinot

21min

Technical Discussion on Data Segments and Real-time Data Applications

3min

Efficient Data Queries and Caching in Apache Pinot

7min

The likes of LinkedIn and Uber use Pinot to power some astonishingly high-scale queries against realtime data. The numbers alone would make an impressive case-study. But behind the headline lies a fascinating set of architectural decisions and constraints to get there. So how does Pinot work? How does it process queries? How are the various roles split across a cluster? And equally important - what does it *not* try to achieve.

Joining me to go through the nuts and bolts of how Pinot handles SQL queries is Tim Berglund, veteran technology explainer of the realtime-data world. He takes us through Pinot step-by-step, covering the roles of brokers, servers, controllers and minions as we build up the picture of a query engine that's interesting in theory and massively performant in practice.

–

Apache Pinot: https://pinot.apache.org/

Apache Pinot Docs: https://docs.pinot.apache.org/

StarTree: https://startree.ai/

Event Driven Design episode with Bobby Calderwood: https://youtu.be/V7vhSHqMxus

Tim on Twitter: https://twitter.com/tlberglund

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #softwaredevelopment #apachepinot #database #dataengineering #sql