The Joe Reis Show Wes McKinney
5 snips
Oct 30, 2024 Wes McKinney, the creator of Pandas and Apache Arrow, now works at Posit on Positron, a cutting-edge data science IDE. He delves into the innovative, React-based features of Positron and its integration with TypeScript and Jupyter. McKinney shares insights on optimizing data science with DuckDB and Wasm, enhancing workflows through AI, and navigating coding complexities. He reminisces about the creation of Pandas during financial turmoil and the evolution of Arrow for improved data processing. Outside of coding, he enjoys video gaming and language learning.
AI Snips
Chapters
Transcript
Episode notes
Pandas Origin Story
- Wes McKinney's work at AQR during the 2008 financial crisis led to Pandas' creation.
- Frustration with R and a colleague's introduction to Python spurred him to create a "derpy" data frame implementation.
Arrow's Purpose
- Data conversion inefficiencies, especially in Spark, were a key driver for Apache Arrow.
- Standardizing data interfaces with Arrow reduced complexity and improved performance.
Arrow-Native Processing
- Arrow-native processing engines are becoming increasingly important, enabling direct data manipulation.
- This represents a shift from retrofitting systems with Arrow to building native Arrow architectures.