Data Engineering Podcast The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse
55:07
Trust SQL Semantics, Declare Configuration
- Rely on Iceberg's SQL behavior for consistent table operations without worrying about engine-specific quirks.
- Declare your table configuration once and let engines handle the operational details automatically.
Choose Streaming Strategy Carefully
- Understand different streaming insert approaches like copy-on-write, merge-on-read, and changelog tables.
- Choose the approach that fits your use case best and manage compaction accordingly for performance.
Select Engines and Optimize Smartly
- Choose query engines according to your workload needs; Trino excels for fast, interactive queries.
- Complement engines with appropriate Iceberg maintenance like running compactions to optimize table state.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right 15 chevron_right 16 chevron_right 17 chevron_right 18 chevron_right 19 chevron_right
Introduction
00:00 • 2min
What Is the Iceberg Table Format?
01:34 • 2min
Cloud Data Warehouses
03:14 • 4min
Data Warehouse - Column Projection
06:45 • 2min
The Iceberg Project and the Vendor Adoption
09:01 • 3min
I Don't Want You to Think About Iceberg, Right?
11:37 • 4min
What Are the Characteristics of Iceberg?
15:22 • 5min
Databricks and Delta - What Are the Edge Cases and Semantics?
20:23 • 3min
Is the Trino Engine the Best Engine for Iceberg?
23:00 • 3min
How Do I Maintain My Iceberg Tables?
25:33 • 3min
When Do You Run Vacuum on Postgres Tables?
28:45 • 4min
Is There a Business Model for Iceberg Maintenance?
32:45 • 3min
Iceberg
36:09 • 4min
Do You Have a Coordinated CDC Pattern?
39:59 • 2min
Can I Just Run CDC and Keep My Analytic Database Up to Date?
42:17 • 2min
Using Branching and Tacking in a Data Lake House Environment
43:49 • 2min
Building Iceberg, Building Tabular - What Are Some of the Most Interesting or Unexpected Lessons That You've Learned?
45:42 • 2min
Are You Using Iceberg or Tabular?
47:37 • 4min
The Biggest Gap in Data Management
51:19 • 4min
Summary
Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. Projects like Apache Iceberg provide a viable alternative in the form of data lakehouses that provide the scalability and flexibility of data lakes, combined with the ease of use and performance of data warehouses. Ryan Blue helped create the Iceberg project, and in this episode he rejoins the show to discuss how it has evolved and what he is doing in his new business Tabular to make it even easier to implement and maintain.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Hey there podcast listener, are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. It ends up being anything but that. Setting it up, integrating it, maintaining it—it’s all kind of a nightmare. And let's not even get started on all the extra tools you have to buy to get it to do its thing. But don't worry, there is a better way. TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. If you're fed up with the 'Modern Data Stack', give TimeXtender a try. Head over to timextender.com/dataengineering where you can do two things: watch us build a data estate in 15 minutes and start for free today.
- Your host is Tobias Macey and today I'm interviewing Ryan Blue about the evolution and applications of the Iceberg table format and how he is making it more accessible at Tabular
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Iceberg is and its position in the data lake/lakehouse ecosystem?
- Since it is a fundamentally a specification, how do you manage compatibility and consistency across implementations?
- What are the notable changes in the Iceberg project and its role in the ecosystem since our last conversation October of 2018?
- Around the time that Iceberg was first created at Netflix a number of alternative table formats were also being developed. What are the characteristics of Iceberg that lead teams to adopt it for their lakehouse projects?
- Given the constant evolution of the various table formats it can be difficult to determine an up-to-date comparison of their features, particularly earlier in their development. What are the aspects of this problem space that make it so challenging to establish unbiased and comprehensive comparisons?
- For someone who wants to manage their data in Iceberg tables, what does the implementation look like?
- How does that change based on the type of query/processing engine being used?
- Once a table has been created, what are the capabilities of Iceberg that help to support ongoing use and maintenance?
- What are the most interesting, innovative, or unexpected ways that you have seen Iceberg used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Iceberg/Tabular?
- When is Iceberg/Tabular the wrong choice?
- What do you have planned for the future of Iceberg/Tabular?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Iceberg
- Hadoop
- Data Lakehouse
- ACID == Atomic, Consistent, Isolated, Durable
- Apache Hive
- Apache Impala
- Bodo
- StarRocks
- Dremio
- DDL == Data Definition Language
- Trino
- PrestoDB
- Apache Hudi
- dbt
- Apache Flink
- TileDB
- CDC == Change Data Capture
- Substrait
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Acryl:  The modern data stack needs a reimagined metadata management platform. Acryl Data’s vision is to bring clarity to your data through its next generation multi-cloud metadata management platform. Founded by the leaders that created projects like LinkedIn DataHub and Airbnb Dataportal, Acryl Data enables delightful search and discovery, data observability, and federated governance across data ecosystems. Signup for the SaaS product today at [dataengineeringpodcast.com/acryl](https://www.dataengineeringpodcast.com/acryl)