AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Building compatibility for metadata layer and support for iceberg files
Developing compatibility with iceberg involves supporting the metadata layer and building Avro files that act as a tree structure, where the root sits in a database or catalog with nodes scattered across S3. These nodes are pointers to other nodes on the tree leading to the data node for query processing. Support for iceberg can range from using it as an external table or external engine, to building metadata on top of proprietary storage, to storing data in an interchange format, and finally storing it at the file level in a non-proprietary open file format like parquet.