Exploring Apache Iceberg: A Higher-Level Data Management Solution

9min Snip

00:00

Play full episode

Summary

Transcript

Episode notes

This chapter dives into Apache Iceberg, detailing its role as a layer above data storage systems for improved data management within a data lake. It contrasts Iceberg and Hive in the Hadoop ecosystem, highlighting the need for better schema management and database-like operations on raw data files. The chapter discusses the challenges faced by software engineers and the approach to abstracting complexities through principles like ACID transactions and SQL semantics for easier data processing.

Apache Iceberg is an open source high-performance format for huge data tables. Iceberg enables the use of SQL tables for big data, while making it possible for engines like Spark and Hive to safely work with the same tables, at the same time.

Iceberg was started at Netflix by Ryan Blue and Dan Weeks, and was open-sourced and donated to the Apache Software Foundation in November 2018. It has now been adopted at many other companies including Airbnb, Apple, and Lyft.

Ryan Blue joins the podcast to describe the origins of Iceberg, how it works, the problems it solves, collaborating with Apple and others to open-source it, and more.

This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments.

Lee is the host of his podcast, Modern Digital Business, an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com.

Please click here to see the transcript of this episode.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Iceberg at Netflix and Beyond with Ryan Blue appeared first on Software Engineering Daily.