Advanced Lakehouse Management With The LakeKeeper Iceberg REST Catalog
Data Engineering Podcast
Migrating Data Catalogs: Challenges and Solutions
This chapter explores the critical considerations for organizations migrating data catalogs to new systems, focusing on Iceberg and Lakekeeper. It outlines practical steps for migration, emphasizes the advantages of Iceberg, and discusses the complexities of table maintenance and optimization within data management.
00:00
Transcript
Play full episode
Transcript
Episode notes
Summary
In this episode of the Data Engineering Podcast Victor Kessler, co-founder of Vakama, talks about the architectural patterns in the lake house enabled by a fast and feature-rich Iceberg catalog. Victor shares his journey from data warehouses to developing the open-source project, Lakekeeper, an Apache Iceberg REST catalog written in Rust that facilitates building lake houses with essential components like storage, compute, and catalog management. He discusses the importance of metadata in making data actionable, the evolution of data catalogs, and the challenges and innovations in the space, including integration with OpenFGA for fine-grained access control and managing data across formats and compute engines.
Announcements
Parting Question
In this episode of the Data Engineering Podcast Victor Kessler, co-founder of Vakama, talks about the architectural patterns in the lake house enabled by a fast and feature-rich Iceberg catalog. Victor shares his journey from data warehouses to developing the open-source project, Lakekeeper, an Apache Iceberg REST catalog written in Rust that facilitates building lake houses with essential components like storage, compute, and catalog management. He discusses the importance of metadata in making data actionable, the evolution of data catalogs, and the challenges and innovations in the space, including integration with OpenFGA for fine-grained access control and managing data across formats and compute engines.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- Your host is Tobias Macey and today I'm interviewing Viktor Kessler about architectural patterns in the lakehouse that are unlocked by a fast and feature-rich Iceberg catalog
- Introduction
- How did you get involved in the area of data management?
- Can you describe what LakeKeeper is and the story behind it?
- What is the core of the problem that you are addressing?
- There has been a lot of activity in the catalog space recently. What are the driving forces that have highlighted the need for a better metadata catalog in the data lake/distributed data ecosystem?
- How would you characterize the feature sets/problem spaces that different entrants are focused on addressing?
- Iceberg as a table format has gained a lot of attention and adoption across the data ecosystem. The REST catalog format has opened the door for numerous implementations. What are the opportunities for innovation and improving user experience in that space?
- What is the role of the catalog in managing security and governance? (AuthZ, auditing, etc.)
- What are the channels for propagating identity and permissions to compute engines? (how do you avoid head-scratching about permission denied situations)
- Can you describe how LakeKeeper is implemented?
- How have the design and goals of the project changed since you first started working on it?
- For someone who has an existing set of Iceberg tables and catalog, what does the migration process look like?
- What new workflows or capabilities does LakeKeeper enable for data teams using Iceberg tables across one or more compute frameworks?
- What are the most interesting, innovative, or unexpected ways that you have seen LakeKeeper used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on LakeKeeper?
- When is LakeKeeper the wrong choice?
- What do you have planned for the future of LakeKeeper?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
- LakeKeeper
- SAP
- Microsoft Access
- Microsoft Excel
- Apache Iceberg
- Iceberg REST Catalog
- PyIceberg
- Spark
- Trino
- Dremio
- Hive Metastore
- Hadoop
- NATS
- Polars
- DuckDB
- DataFusion
- Atlan
- Open Metadata
- Apache Atlas
- OpenFGA
- Hudi
- Delta Lake
- Lance Table Format
- Unity Catalog
- Polaris Catalog
- Apache Gravitino
- Keycloak
- Open Policy Agent (OPA)
- Apache Ranger
- Apache NiFi
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
