AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
23min chapter
Why he created Pandas, the future of data systems, why he left his CTO role to become a chief architect - Wes McKinney - The Data Scientist Show #086
The Data Scientist Show
Evolving Data Processing Tools and the Rise of Modular Computing
The chapter explores the journey from creating Apache Arrow for efficient data sharing to establishing Voltron Data for modular computing engines and hardware acceleration, emphasizing the benefits of composable data processing tools. It discusses challenges in translating Pandas code across execution environments and the importance of API portability for supporting multiple use cases. The chapter delves into the evolution of tools like IBIS and SQL Glott for seamless interaction with SQL backends, the development of Lance file format for multi-modal AI data, and the future of data tools focusing on automation and modularity to address hardware heterogeneity.
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode