Optimizing Query Processing and Data Movement in a Server Cluster

The likes of LinkedIn and Uber use Pinot to power some astonishingly high-scale queries against realtime data. The numbers alone would make an impressive case-study. But behind the headline lies a fascinating set of architectural decisions and constraints to get there. So how does Pinot work? How does it process queries? How are the various roles split across a cluster? And equally important - what does it *not* try to achieve.

Joining me to go through the nuts and bolts of how Pinot handles SQL queries is Tim Berglund, veteran technology explainer of the realtime-data world. He takes us through Pinot step-by-step, covering the roles of brokers, servers, controllers and minions as we build up the picture of a query engine that's interesting in theory and massively performant in practice.

–

Apache Pinot: https://pinot.apache.org/

Apache Pinot Docs: https://docs.pinot.apache.org/

StarTree: https://startree.ai/

Event Driven Design episode with Bobby Calderwood: https://youtu.be/V7vhSHqMxus

Tim on Twitter: https://twitter.com/tlberglund

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

–

#podcast #softwaredevelopment #apachepinot #database #dataengineering #sql