Why he created Pandas, the future of data systems, why he left his CTO role to become a chief architect - Wes McKinney - The Data Scientist Show #086
The Data Scientist Show - Daliana Liu
Evolving Data Processing Tools and the Rise of Modular Computing
The chapter explores the journey from creating Apache Arrow for efficient data sharing to establishing Voltron Data for modular computing engines and hardware acceleration, emphasizing the benefits of composable data processing tools. It discusses challenges in translating Pandas code across execution environments and the importance of API portability for supporting multiple use cases. The chapter delves into the evolution of tools like IBIS and SQL Glott for seamless interaction with SQL backends, the development of Lance file format for multi-modal AI data, and the future of data tools focusing on automation and modularity to address hardware heterogeneity.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
