StarRocks: Bridging Lakehouse and OLAP for High-Performance Analytics

9 snips

May 5, 2025

Sida Shen, a product manager at CelerData and a contributor to StarRocks, dives into the innovative world of high-performance analytical databases. He shares the origins of StarRocks, illustrating its evolution from Apache Doris into a robust Lakehouse query engine. Topics include handling high concurrency and low latency queries, bridging traditional OLAP with lakehouse architecture, and the importance of integration with formats like Apache Iceberg. Sida also emphasizes the challenges of denormalization and real-time data processing in modern analytics.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

StarRocks Lakehouse Innovation

StarRocks is a high-performance Lakehouse query engine forked from Apache Doris in 2020.
It supports on-the-fly joins and eliminates the need for denormalization pipelines for flexible, fast queries.

INSIGHT

Bridging Lakehouse and OLAP

StarRocks supports fast joins on the fly, solving denormalization challenges common in other OLAP systems.
It bridges Lakehouse open formats with low latency, high concurrency typical of data warehouses.

ADVICE

Simplify Scaling and Schema Design

Use automatic sharding in StarRocks to avoid manual data redistributions when scaling.
Opt for columnar format storage over indexes for better OLAP performance in most cases.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the Data Engineering Podcast Sida Shen, product manager at CelerData, talks about StarRocks, a high-performance analytical database. Sida discusses the inception of StarRocks, which was forked from Apache Doris in 2020 and evolved into a high-performance Lakehouse query engine. He explains the architectural design of StarRocks, highlighting its capabilities in handling high concurrency and low latency queries, and its integration with open table formats like Apache Iceberg, Delta Lake, and Apache Hudi. Sida also discusses how StarRocks differentiates itself from other query engines by supporting on-the-fly joins and eliminating the need for denormalization pipelines, and shares insights into its use cases, such as customer-facing analytics and real-time data processing, as well as future directions for the platform.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Sida Shen about StarRocks, a high performance analytical database supporting shared nothing and shared data patterns

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what StarRocks is and the story behind it?
There are numerous analytical databases on the market. What are the attributes of StarRocks that differentiate it from other options?
Can you describe the architecture of StarRocks?
- What are the "-ilities" that are foundational to the design of the system?
How have the design and focus of the project evolved since it was first created?
What are the tradeoffs involved in separating the communication layer from the data layers?
The tiered architecture enables the shared nothing and shared data behaviors, which allows for the implementation of lakehouse patterns. What are some of the patterns that are possible due to the single interface/dual pattern nature of StarRocks?
- The shared data implementation has cacheing built in to accelerate interaction with datasets. What are some of the limitations/edge cases that operators and consumers should be aware of?
StarRocks supports management of lakehouse tables (Iceberg, Delta, Hudi, etc.), which overlaps with use cases for Trino/Presto/Dremio/etc. What are the cases where StarRocks acts as a replacement for those systems vs. a supplement to them?
The other major category of engines that StarRocks overlaps with is OLAP databases (e.g. Clickhouse, Firebolt, etc.). Why might someone use StarRocks in addition to or in place of those techologies?
We would be remiss if we ignored the dominating trend of AI and the systems that support it. What is the role of StarRocks in the context of an AI application?
What are the most interesting, innovative, or unexpected ways that you have seen StarRocks used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on StarRocks?
When is StarRocks the wrong choice?
What do you have planned for the future of StarRocks?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA