The setup
I used an AWS m6a.xlarge machine that has 4 vCores and 16GB RAM obtainable and utilized taskset to assign 1 vCore and a couple of vCores to the method at a time to simulate a machine with fewer vCores every time. For lib variations, I took essentially the most up-to-date secure releases obtainable on the time:pandas==2.2.2; polars=1.2.1
The info
The dataset was randomly generated to be made up of 1M rows and 5 columns, and is supposed to function a historical past of 100k consumer operations made in 10k periods inside a sure product:
user_id (int)
action_types (enum, can take the values in [“click”, “view”, “purchase”])
timestamp (datetime)
session_id (int)
session_duration (float)
The premise
Given the dataset, we wish to discover the highest 10% of most engaged customers, judging by their common session period. So, we’d first wish to calculate the typical session period per consumer (grouping and aggregation), discover the ninetieth quantile (quantile computation), choose all of the customers above the quantile (filtering), and ensure the checklist is ordered by the typical session period (sorting).
Testing
Every of the operations have been run 200 instances (utilizing timeit), taking the imply run time every time and the usual error to function the measurement error. The code can be found here.
A be aware on keen vs lazy analysis
One other distinction between pandas and Polars is that the previous makes use of keen execution (statements are executed as they’re written) by default and the latter makes use of lazy execution (statements are compiled and run solely when wanted). Polar’s lazy execution helps it optimize queries, which makes a really good characteristic in heavy knowledge evaluation duties. The selection to separate our process and have a look at 4 operations is made to get rid of this facet and give attention to evaluating extra primary efficiency points.
Group by + Mixture
We will see how pandas doesn’t scale with vCores — as anticipated. This pattern will stay all through our take a look at. I made a decision to maintain it within the plots, however we gained’t reference it once more.
polars’ outcomes are fairly spectacular right here — with a 1vCore setup it managed to complete quicker than pandas by a 3rd of the time, and as we scale to 2, 4 cores it finishes roughly 35% and 50% quicker respectively.
Quantile Computation
This one is fascinating. In all vCores setups, polars completed round 5x quicker than pandas. On the 1vCore setup, it measured 0.2ms on common, however with a major commonplace error (which means that the operation would typically end effectively after 0.2ms, and at different instances it might end effectively earlier than 0.2ms). When scaling to a number of cores we get stabler run instances — 2vCores at 0.21ms and 4vCores at 0.19 (round 10% quicker).
Filtering
In all circumstances, Polars finishes quicker than pandas (the more severe run time continues to be 2 instances quicker than pandas). Nevertheless, we are able to see a really uncommon pattern right here — the run time will increase with vCores (we’re anticipating it to lower). The run time of the operation with 4 vCores is roughly 35% slower than the run time with 1 vCore. Whereas parallelization provides you extra computing energy, it usually comes with some overhead — managing and orchestrating parallel processes is commonly very troublesome.
This Polars scaling challenge is perplexing — the implementation on my finish may be very easy, and I used to be not capable of finding a related open challenge on the Polars repo (there are at present over 1k open points there, although).
Do you’ve any concept as to why this might have occurred? Let me know within the feedback.
Sorting
After filtering, we’re left with round 13.5k rows.
On this one, we are able to see that the 1vCore Polars case is considerably slower than pandas (by round 45%). As we scale to 2vCores the run time turns into aggressive with pandas’, and by the point we scale to 4vCores Polars turns into considerably quicker than pandas. The doubtless state of affairs right here is that Polars makes use of a sorting algorithm that’s optimized for parallelization — such an algorithm could have poor efficiency on a single core.
Wanting extra carefully on the docs, I discovered that the type operation in Polars has a multithreaded parameter that controls whether or not a multi-threaded sorting algorithm is used or a single-threaded one.
Sorting (with multithreading=False)
This time, we are able to see far more constant run instances, which don’t scale with cores however do beat pandas.