How does a Python tool support all types of DataFrames and their various features? Could a lightweight library be used to add compatibility for newer formats like Polars or PyArrow? This week on the show, we speak with Marco Gorelli about his project, Narwhals.
Narwhals is a project aimed at library maintainers rather than end users. We discuss how the added compatibility benefits users by supporting modern features like lazy evaluation. We cover several projects Marco has been working with to implement Narwhals, including Altair, scikit-lego, and Ibis.
We also discuss how Marco started contributing to open-source projects. Marco has contributed to both pandas and Polars, which helps explain his interest in growing compatibility between libraries. He also offers advice on making your first contribution.
This episode is sponsored by CodeRabbit.
Topics:
- 00:00:00 – Introduction
- 00:02:02 – Euro SciPy 2024 and sprints
- 00:04:04 – How did you get involved in open source?
- 00:07:18 – Finding a good issue to get started
- 00:09:25 – Discord and open-source projects
- 00:11:12 – Who would you describe Narwhals?
- 00:16:47 – Working on Polars
- 00:19:17 – Apache Arrow and a data interchange protocol
- 00:22:55 – Sponsor: CodeRabbit
- 00:23:55 – Digging into eager vs lazy
- 00:27:04 – Ibis DataFrame library
- 00:28:57 – What do libraries need from Narwhals?
- 00:34:57 – The scikit-lego library
- 00:37:15 – Video Course Spotlight
- 00:38:45 – Other libraries interested in Narwhals
- 00:41:56 – Compatibility policy
- 00:45:18 – What should an end user expect?
- 00:46:32 – Have other projects that attempted this?
- 00:47:54 – Keeping the project light and pure Python
- 00:49:32 – Contributors and how to get involved
- 00:54:42 – What are you excited about in the world of Python?
- 00:57:18 – What do you want to learn next?
- 00:59:05 – How can people follow your work online?
- 00:59:27 – Thanks and goodbye
Show Links:
Level up your Python skills with our expert-led courses:
Support the podcast & join our community of Pythonistas
