The Test Set by Posit Marco Gorelli: Narwhals, ecosystem glue, and the value of boring work
You’ve probably used Narwhals without realizing it. It’s the compatibility layer helping apps and libraries like Plotly play nice with Pandas, Polars, Arrow, and more — while keeping computation native instead of converting everything to Pandas. In this episode, Marco Gorelli explains how his weekend experiment turned into essential ecosystem infrastructure and why data types, not APIs, are where interoperability gets tricky. Plus what it takes to build trust and community around an open-source project.
Episode Notes
Marco shares the Narwhals origin story (including the meme-powered name), the hard edge cases that live in data types and null semantics, and why he’s cautious about using AI for code generation when correctness hinges on tiny details. We also jam on proactive “GitHub surfing,” conference talks as trust-building exercises, celebrating contributors, and how early commit messages capture the genuine excitement of building something new.
What’s Inside
- Narwhals 101: You’ve probably used it (even if you didn’t know it)
- The real interoperability traps: data types, null semantics, and “looks-the-same” operations
- Why expression systems won, and how they shaped Marco’s approach — with nods to Ibis, Polars, and Pandas
- Open source as social work: proactive outreach, trust-building, and a Discord-powered community
- Extending Narwhals to new engines, starting with the Daft plugin