Scaling Derived Data for Planet-Scale Applications at Linkedin

Jun 5, 2024

Felix GV, Principal Staff Engineer at Linkedin, discusses the evolution from Voldemort to VeniceDB, challenges with derived data, planet-scale applications, writing and merging data, architectural components, fault tolerance, recommender systems, versioning, integration with Kafka, and future improvements.

Ask episode

Chapters

Transcript

Episode notes

Intro

00:00 • 2min

Evolution of VeniceDB at LinkedIn

01:33 • 6min

Exploring Derived Data and VeniceDB in Large-scale Applications

08:01 • 27min

Understanding Merging Processes and Timestamps in a Large-Scale System

34:42 • 6min

Architectural Components and Data Processing in LinkedIn

40:29 • 13min

Discussion on Lookup Requests, Fault Tolerance, and Recommender Systems in Distributed Data Systems

53:20 • 5min

Understanding Versioning and Time Travel Capabilities in a Data System

57:51 • 3min

Venice: The Integration with Kafka and Data Management at LinkedIn

01:01:08 • 12min

In this video I speak with Felix GV, who is a Principal Staff Engineer at Linkedin, and has done major contributions to the data infrastructure and Linkedin, including VeniceDB. This episode will give you a good understanding of why we need a new database for storing "Derived Data" in a low latency, high performance manner, which is very important for Machine Learning workloads. Chapters: 00:00 Introduction 01:42 The Evolution of LinkedIn's Databases 03:15 Challenges with Voldemort and the Birth of VeniceDB 08:42 Understanding Derived Data 13:33 Planet-Scale Applications and Multi-Region Support 17:40 Writing Data into VeniceDB 22:53 Merging Data in VeniceDB 40:31 Understanding the Architecture 40:47 Components of the Write Path 41:56 Leader and Follower Architecture 43:58 Partitioning and DaVinci Client 47:57 Read Patterns and Client Options 54:25 Fault Tolerance and Recommender Systems 01:01:19 Kafka Integration and Deployment 01:06:56 Roadmap and Future Improvements Important links: VeniceDB blog: https://www.linkedin.com/blog/engineering/open-source/open-sourcing-venice-linkedin-s-derived-data-platform VeniceDB docs: https://venicedb.org/ Qcon: https://youtu.be/pJeg4V3JgYo?si=vblGUxp5fNdKPHoC Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #kafka #linkedin #venicedb #Rocksdb