Python Bytes

#496 A lake house in Seattle

Sep 15, 2026
Why settle for Pandas when Polars and DuckDB can tackle medium-sized data faster and with less memory? This conversation also explores unit-safe Pydantic models, running Rust inside Python with PyO3, Duck Lake’s open lakehouse vision, and AWS acquiring DuckDB Labs. Plus, UV’s signed binaries, macOS changes, charging gear, and the curious rise of “agentic” branding.
Ask episode
AI Snips
Chapters
Transcript
Episode notes
INSIGHT

Most Data Is Medium Data

  • Many teams mistake medium-sized datasets for big data because Pandas becomes slow too early.
  • Redshift telemetry shows most tables stay below 100GB, making single-machine tools more appropriate.
INSIGHT

DuckDB Crushes Pandas On Performance

  • DuckDB processed a comparable workload in about five seconds using 1.93GB of RAM, versus Pandas taking minutes and tens of gigabytes.
  • Polars and DuckDB also exploit multiple CPU cores more effectively.
ADVICE

Try Arrow Before Rewriting Everything

  • Try Polars or DuckDB before adopting distributed infrastructure for datasets that fit on one machine.
  • Use Apache Arrow to move between Pandas, Polars, and DuckDB without reloading or copying data.
Get the Snipd Podcast app to discover more snips from this episode
Get the app