StarRocks: Bridging Lakehouse and OLAP for High-Performance Analytics
Data Engineering Podcast
00:00
Challenges and Lessons in Building a Cost-Based Optimizer
This chapter explores the challenges faced in developing a cost-based optimizer and scaling it for production use. It underscores the importance of community feedback and open-source collaboration in creating an efficient and reliable system.
Transcript
Play full episode
Transcript
Episode notes
Summary
In this episode of the Data Engineering Podcast Sida Shen, product manager at CelerData, talks about StarRocks, a high-performance analytical database. Sida discusses the inception of StarRocks, which was forked from Apache Doris in 2020 and evolved into a high-performance Lakehouse query engine. He explains the architectural design of StarRocks, highlighting its capabilities in handling high concurrency and low latency queries, and its integration with open table formats like Apache Iceberg, Delta Lake, and Apache Hudi. Sida also discusses how StarRocks differentiates itself from other query engines by supporting on-the-fly joins and eliminating the need for denormalization pipelines, and shares insights into its use cases, such as customer-facing analytics and real-time data processing, as well as future directions for the platform.
Announcements
Parting Question
In this episode of the Data Engineering Podcast Sida Shen, product manager at CelerData, talks about StarRocks, a high-performance analytical database. Sida discusses the inception of StarRocks, which was forked from Apache Doris in 2020 and evolved into a high-performance Lakehouse query engine. He explains the architectural design of StarRocks, highlighting its capabilities in handling high concurrency and low latency queries, and its integration with open table formats like Apache Iceberg, Delta Lake, and Apache Hudi. Sida also discusses how StarRocks differentiates itself from other query engines by supporting on-the-fly joins and eliminating the need for denormalization pipelines, and shares insights into its use cases, such as customer-facing analytics and real-time data processing, as well as future directions for the platform.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- Your host is Tobias Macey and today I'm interviewing Sida Shen about StarRocks, a high performance analytical database supporting shared nothing and shared data patterns
- Introduction
- How did you get involved in the area of data management?
- Can you describe what StarRocks is and the story behind it?
- There are numerous analytical databases on the market. What are the attributes of StarRocks that differentiate it from other options?
- Can you describe the architecture of StarRocks?
- What are the "-ilities" that are foundational to the design of the system?
- How have the design and focus of the project evolved since it was first created?
- What are the tradeoffs involved in separating the communication layer from the data layers?
- The tiered architecture enables the shared nothing and shared data behaviors, which allows for the implementation of lakehouse patterns. What are some of the patterns that are possible due to the single interface/dual pattern nature of StarRocks?
- The shared data implementation has cacheing built in to accelerate interaction with datasets. What are some of the limitations/edge cases that operators and consumers should be aware of?
- StarRocks supports management of lakehouse tables (Iceberg, Delta, Hudi, etc.), which overlaps with use cases for Trino/Presto/Dremio/etc. What are the cases where StarRocks acts as a replacement for those systems vs. a supplement to them?
- The other major category of engines that StarRocks overlaps with is OLAP databases (e.g. Clickhouse, Firebolt, etc.). Why might someone use StarRocks in addition to or in place of those techologies?
- We would be remiss if we ignored the dominating trend of AI and the systems that support it. What is the role of StarRocks in the context of an AI application?
- What are the most interesting, innovative, or unexpected ways that you have seen StarRocks used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on StarRocks?
- When is StarRocks the wrong choice?
- What do you have planned for the future of StarRocks?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
- StarRocks
- CelerData
- Apache Doris
- SIMD == Single Instruction Multiple Data
- Apache Iceberg
- ClickHouse
- Druid
- Firebolt
- Snowflake
- BigQuery
- Trino
- Databricks
- Dremio
- Data Lakehouse
- Delta Lake
- Apache Hive
- C++
- Cost-Based Optimizer
- Iceberg Summit Tencent Games Presentation
- Apache Paimon
- Lance
- Delta Uniform
- Apache Arrow
- StarRocks Python UDF
- Debezium
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
