Ep. 22 - Navigating the Open Source Data Landscape
Some guests need a long introduction. Marco Gorelli is one of them, so I’ll keep it short: mathematician, software engineer at Quansight Labs, former pandas maintainer, Polars contributor, and the creator of Narwhals, a library that gives you one API on top of the major dataframe libraries.We first met at PyCon Italy, when Narwhals was only a few months old, and we’ve run into each other at conferences since. This time we sat down properly and talked about the stuff that usually only comes up in a conference hallway: why some APIs win, how you get anyone to notice a new open source project, and what maintainers are dealing with now that coding agents can open pull requests.Here are the highlights.From pure maths to pandas maintainerMarco studied mathematics, followed by a master’s in maths and the foundations of computer science. Despite the name, it involved no programming at all, “much to some recruiters’ disappointment” when they tried to place him in very applied jobs.He went into data science around 2017–2018, when the field was booming. He pointed out that the way people talked about data science back then sounds a lot like the way people talk about AI today, which in turn sounds like how they talked about big data a few years earlier. (When was the last time you heard someone say “big data”?)The way into open source was very ordinary. He hit a bug in pandas at work and thought it might be fun to fix it. Then COVID hit, he had extra time, and he kept going. Eventually he became a pandas maintainer, and that led to a job at Quansight Labs working on open source full time.Why Polars won by not copying pandasAt Quansight, Marco got into Rust and Polars, a dataframe library written in Rust that avoids many of pandas’ historical design decisions.It’s easy to forget how recently people “would look at me as if I had three heads” if he mentioned Polars alongside pandas. Early on, the critics said nobody would use a dataframe library without the pandas index or the pandas API. Polars was adopted widely anyway. Meanwhile, several projects that copied pandas exactly, only faster, have mostly disappeared.Marco’s explanation: Ritchie Vink, the creator of Polars, treated not following the pandas API as a feature and not a limitation.> “Yes, there is a learning curve, but once you’re over that learning curve, everything is very consistent. Once you get used to the Polars way of thinking, it is a pleasure to work with.”His practical advice: the best time to try Polars is when you’re starting a new project. Don’t try to recreate your pandas habits. Learn the Polars way of thinking from the start and you get the benefits.Why pandas struggles to keep upI asked whether many of the complaints about pandas come from the fact that it has been around so long, through data mining, big data, machine learning, deep learning and now generative AI.Marco agreed and pointed to hardware. pandas was designed around 2008, when 16-core CPUs weren’t something everyone had. Today they’re common and cheap to rent. The problem is that the pandas API doesn’t make it obvious what can run in parallel. People say “pandas should just do things in parallel”, but the API doesn’t make clear how that would work.Polars makes it explicit. Each expression inside with_columns runs independently, so they can all run multi-threaded.> “The world has certainly changed around pandas, but it’s not easy for pandas to change itself at the same pace.”What you gain by going from user to contributorMarco named two things.1. You appreciate the invisible work. pandas is run entirely by volunteers, with the occasional grant. Polars has a company behind it, but much of what goes into the free tool isn’t driven by money. It’s driven by people who want to make data work better.2. You learn API design the hard way. Users often think there should be a built-in function for their use case. But if a library added a function for every use case, the API would become impossible to navigate, and every function would have 20 parameters. As the old pandas maintainer joke goes:> “There is no problem that cannot be solved by adding an extra keyword argument to read_csv.”(It’s probably past 50 by now.) A good tool has to be fairly minimal. It can’t solve everyone’s problem, but it can give you the building blocks to solve your own.Narwhals: copying Polars, not pandasI’ve always found it fascinating that something as simple as mean behaves differently across libraries. Marco gave an example right away: missing values. Most tools skip nulls by default, but the way you configure that differs between PyArrow, pandas, Polars and SQL.That’s the kind of problem Narwhals solves. Marco started it about two and a half years ago as a unified dataframe API, aimed mainly at library maintainers who want to accept different kinds of dataframe inputs.He had three options: copy pandas, copy another library, or invent a new API. Copying pandas didn’t work. Its API suggests that some things should work, like in-place operations, that don’t fit the data models of other backends. Polars dataframes, for example, are typically immutable and built around method chaining and expressions.So he tried the opposite: follow the Polars API and get pandas to follow it too. To his surprise, it worked very well, as long as he defined an expression rigorously and stuck to that definition:> “A function from a dataframe to a collection of series.”Narwhals still differs from Polars in a few places, usually through an extra keyword argument or by disallowing certain operations. One reason is that Polars treats a dataframe as a collection of independent columns, while in SQL a table is an unordered collection of atomic rows. That’s powerful, but it can produce surprising results. Marco covered this in his PyData London talk, “The Polars vs SQL differences nobody is talking about.”The hard parts of translating between backendsWhat turned out harder than expected?* mode: it can return one or more values, which breaks the “preserve length or aggregate to one value” model. Narwhals adds a keep argument (e.g. keep the first occurrence, or any occurrence) and maps it back onto Polars.* Time zones: pandas and Polars allow a different time zone per column. In Spark, the time zone is set on the connection, so you can’t have a London column next to a Tokyo column.* Group by: pandas drops null groups by default.* Joins: how do you name duplicate columns? SQL makes you write an explicit SELECT. Polars adds _right, but the behavior differs between full outer joins and the other join types.Two things helped:* Hypothesis, the property-based testing library. Instead of writing “given / when / then” tests by hand, you describe the kind of input you want and assert invariants. It surfaced many inconsistencies.* Running downstream users’ test suites in Narwhals’ own CI. This caught regressions that passed every internal test, because real-world data is messier than toy examples.> “That’s one piece of advice for people writing tools for tool builders: the moment you start getting users, if you have any extra tests from your users that you can run, they can help suss out so many issues.”How do you get a new open source project noticed?Marco was clear that he was describing what worked for him, not a universal formula:* Go to conferences where your audience is. He was targeting library maintainers, and Python conferences are full of them. Even when his talks were about Polars, he made sure to mention Narwhals, and he talked about it in the hallways.* Use social media carefully. For him that has mostly meant LinkedIn. Don’t expect overnight success, and don’t annoy people.* Don’t spam Reddit. Communities like r/Python have strict self-promotion rules, and your post will probably be removed.* Pitch podcasts. He contacted Python Bytes and they featured it.Advice for your first pull requestIf you take one thing from this conversation, take this:* Read the contributing guide. Polars, for example, explicitly s… Get full access to My Data Guest at mydataguest.substack.com/subscribe