Tech on the Rocks

Unifying structured and unstructured data for AI: Rethinking ML infrastructure with Nikhil Simha and Varant Zanoyan

Aug 30, 2024

Nikhil Simha and Varant Zanoyan, both seasoned engineers with rich backgrounds in data systems and ML infrastructure, discuss the intricate balance of structured and unstructured data in AI. They delve into the challenges of merging real-time data with machine learning, emphasizing the importance of user-friendly APIs. The conversation touches on failures in data transformation and effective strategies for startups to engage users. They also introduce Cronon, an open-source platform, highlighting its potential to improve data orchestration and user experience.

01:01:45

Creator website

Episode guests

Nikhil Simha

Varant Zanoyan

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Nikhil and Varant emphasize the importance of merging real-time data processing with user-friendly design to improve AI infrastructure.

The conversation highlights the necessity of creating a unified data layer that efficiently handles both structured and unstructured data types.

Discussion around user experience underlines how simplifying data queries can significantly reduce errors and enhance machine learning operations.

Deep dives

Innovative Approaches to ML and AI Workflows

The episode highlights how the startup Zipline is redefining the integration of machine learning (ML) and artificial intelligence (AI) by simplifying the development of workflows. Founders Nikhil and Varanth discuss their background and how their experiences at companies like Instagram, Amazon, and Airbnb have influenced their vision for Zipline. They emphasize the need for real-time data processing integrated with user-friendly design to improve data infrastructure, a sentiment supported by their development of Cronon, an open-source data platform. Cronon demonstrates their commitment to creating a more manageable environment for data orchestration, thereby addressing one of the significant challenges in AI implementations today.

Intro

2min

Navigating Machine Learning Infrastructure

15min

Navigating Failure in Data Transformation

8min

Streamlining Data Computation with DSL

10min

Integrating Structured and Unstructured Data for AI

18min

Navigating Startup Beginnings

4min

The Art of Saying No in Sales and Entrepreneurship

5min

In this episode, we dive deep into the future of data infrastructure for AI and ML with Nikhil Simha and Varant Zanoyan, two seasoned engineers from Airbnb and Facebook. Nikhil and Varant share their journey from building real-time data systems and ML infrastructure at tech giants to launching their own venture.

The conversation explores the intricacies of designing developer-friendly APIs, the complexities of handling both batch and streaming data, and the delicate balance between customer needs and product vision in a startup environment.

Contacts & Links

Nikhil Simha
Varant Zanoyan
Chronon project

Chapters

00:00 Introduction and Past Experiences
04:38 The Challenges of Building Data Infrastructure for Machine Learning
08:01 Merging Real-Time Data Processing with Machine Learning
14:08 Backfilling New Features in Data Infrastructure
20:57 Defining Failure in Data Infrastructure
26:45 The Choice Between SQL and Data Frame APIs
34:31 The Vision for Future Improvements
38:17 Introduction to Chrono and Open Source
43:29 The Future of Chrono: New Computation Paradigms
48:38 Balancing Customer Needs and Vision
57:21 Engaging with Customers and the Open Source Community
01:01:26 Potential Use Cases and Future Directions

Click here to view the episode transcript.