Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

Git for Data: Managing Data like Code with lakeFS

9 snips

Jan 19, 2023

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Git-Like Versioning For Object Stores

lakeFS provides Git-like data versioning for object stores using branches, commits, and merges without copying all data.
It maps Git semantics onto petabyte-scale object stores by storing pointers and metadata rather than file contents.

INSIGHT

Merkle Trees Track Data State

lakeFS uses a Merkle (cryptographic) tree of pointers and hashes to detect changes without storing file contents inside the system.
The system updates the tree on copy-on-write operations and keeps objects in the underlying S3-compatible storage.

INSIGHT

Reproducibility Through Point-in-Time Versions

Versioning enables exact reproducibility and point-in-time recovery to find when data corruption occurred.
Engineers can roll back to a historical commit to reproduce and fix issues in data pipelines.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Is it possible to manage and test data like code? lakeFS is an open-source data version control tool that transforms object storage into Git-like repositories, offering teams a way to use the same workflows for code and data. In this episode, Kris sits down with guest Adi Polak, VP of DevX at Treeverse, to discuss how lakeFS can be used to facilitate better management and testing of data.

At its core, lakeFS provides teams with better data management. A theoretical data engineer on a large team runs a script to delete some data, but a bug in the script accidentally deletes a lot more data than intended. Application engineers can checkout the main branch, effectively erasing their mistakes, but without a tool like lakeFS, this data engineer would be in a lot of trouble.

Polak is quick to explain that lakeFS isn’t built on Git. The source code behind an application is usually a few dozen mega bytes, while lakeFS is designed to handle petabytes of data; however, it does use Git-like semantics to create and access versions so adoption is quick and simple.

Another big challenge that lakeFS helps teams tackle is reproducibility. Troubleshooting when and where a corruption in the data first appeared can be a tricky task for a data engineer, when data is constantly updating. With lakeFS, engineers can refer to snapshots to see where the product was corrupted, and rollback to that exact state.

lakeFS also assists teams with reprocessing of historical data. With lakeFS data can be reprocessed on an isolated branch, before merging, to ensure the reprocessed data is exposed atomically. It also makes it easier to access the different versions of reprocessed data using any tag or a historical commit ID.

Tune in to hear more about the benefits of lakeFS.

EPISODE LINKS

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo

🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
👍 If you enjoyed this, please leave us a rating.
🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.