Software Engineering Daily

Uber’s LedgerStore and its Trillions of Indexes with Kaushik Devarajaiah

Jul 30, 2024

00:00

Snipd AI

Kaushik Devarajaiah, Tech Lead for LedgerStore at Uber, dives into the complexities of managing vast amounts of data. He discusses the critical role of LedgerStore in ensuring data integrity and correct transaction records. The conversation highlights the integration of advanced technologies like Spark and Kafka in Uber’s scalable architecture. Kaushik reveals the challenges of performing dual database reads and writes to achieve low latency. He also shares future innovations planned for LedgerStore, emphasizing improving data visualization and consistency.

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Kaushik Devarajaiah emphasizes the critical importance of immutable storage solutions like LedgerStore in ensuring accuracy and integrity for Uber's financial transactions.

The challenges faced during the migration of 250 billion records highlight the intricate processes required to maintain data consistency and reliability at scale.

Deep dives

Scaling Data Infrastructure at Uber

Kaushik Devarojay discusses his journey at Uber, emphasizing the challenges of scaling the data infrastructure during the company's rapid growth. Initially focusing on large-scale data ingestion, he encountered unique issues due to the mutable nature of the data, such as real-time trip data that needed to be analyzed for decision-making purposes. This required building systems from the ground up, where he also contributed to the development of Apache Hootie, which facilitates snapshotting the states of data. The transition into storage infrastructure presented new challenges, particularly regarding availability and latency, which were crucial factors for Uber's operational success.

Intro

3min

Mastering Data at Scale: Uber's Evolving Architecture

15min

Managing Dual Database Systems for Optimal Performance

2min

Database Migration and Index Management

7min

Future Innovations and Adaptations in Ledger Store

3min

Challenges in Ledger Store Development

4min

Uber handles billions of trips and deliveries, and tens of billions of financial transactions across drivers, couriers, users, and merchants every quarter.

LedgerStore is an immutable storage solution at Uber that provides verifiable data completeness and correctness guarantees to ensure data integrity for its transactions.

Kaushik Devarajaiah is the Tech Lead for LedgerStore at Uber. He joins the show to talk about scaling Uber’s data and storage infrastructure.

Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer.