In this podcast, Felix GV discusses designing massive distributed systems at LinkedIn, including the Venice open-source project. He also shares his thoughts on designing software for a multi-planetary civilization, offering a unique perspective on the future of technology.
Read more
AI Summary
Highlights
AI Chapters
Episode notes
auto_awesome
Podcast summary created with Snipd AI
Quick takeaways
The Venice project at LinkedIn successfully transitioned batch processing system successes into stream processing, enhancing ML features for improved user experience.
Venice's future plans include decoupling from Kafka, advancing conflict resolution capabilities, and enhancing flexibility for diverse data processing needs.
Deep dives
Origin and Ideation of the Venice Project
The Venice project originated from the idea of mirroring the success of Voldemort, a batch processing system, into the stream processing domain. The project commenced in 2016 after facing initial delays due to competing priorities. By the end of 2016, Venice was fully operational, marking a year of intensive development and ensuring a smooth transition from Voldemort.
Venice's Impact on LinkedIn's Feed and People You May Know Features
Venice plays a crucial role at LinkedIn, notably in the feed algorithm that sorts posts based on ML features stored in Venice. Features like 'People You May Know' rely on Venice's ML data, filtering through vast networks to recommend second-degree connections efficiently. The system significantly contributes to LinkedIn's recommendation algorithms and overall user experience.
Scalability and Architecture of Venice for Lambda Architecture Concepts
Venice is adept at handling both batch and stream data while supporting Lambda architecture concepts. With a streaming input focus, the platform intertwines batch data operations with streaming inputs in a synchronized manner. The system manages batch processes using Hadoop infrastructure alongside stream processing, providing a cohesive approach for diverse data handling.
Future Developments and Enhancements in Venice's Pub-Sub System and Conflict Resolution
Future enhancements in Venice include decoupling the current Kafka dependency to allow for compatibility with various pub-sub systems like Pulsar. Furthermore, the team is advancing Venice's conflict resolution capabilities to support a broader array of operations and ensure efficient handling of out-of-order events across regions. These ongoing developments aim to bolster Venice's flexibility and robustness for diverse data processing needs.
Felix GV designs HUGE distributed systems as a principal staff engineer at LinkedIn. We talk about the Venice open-source project and the challenges of massively distributed data.
But Felix is also thinking about something WAY bigger - designing software for a multi-planetary civilization! This is a very interesting discussion that you won't usually come across on this planet, or others (yet).
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode