If you want data manipulation library that’s both fast and memory-efficient, try Polars. Polars provides a high-level API similar to Pandas but with better performance for large datasets.

Performance Comparison: Polars vs Pandas

To compare the performance of these two libraries, create two Pandas DataFrames, each with 1 million rows.

import pandas as pd
import polars as pl
import numpy as np
import time

# Create two Pandas DataFrames with 1 million rows each
pandas_df1 = pd.DataFrame({
    'key': np.random.randint(0, 1000, size=1_000_000),
    'value1': np.random.rand(1_000_000)
})

pandas_df2 = pd.DataFrame({
    'key': np.random.randint(0, 1000, size=1_000_000),
    'value2': np.random.rand(1000000)
})

# Create two Polars DataFrames from the Pandas DataFrames
polars_df1 = pl.from_pandas(pandas_df1)
polars_df2 = pl.from_pandas(pandas_df2)

Next, we merge the two Pandas DataFrames on the ‘key’ column using the merge method. We also measure the execution time using the time library.

# Merge the two DataFrames on the 'key' column
start_time = time.time()
pandas_merged = pd.merge(pandas_df1, pandas_df2, on='key')
pandas_time = time.time() - start_time

Similarly, we merge the two Polars DataFrames on the ‘key’ column using the join method. We also measure the execution time using the time library.

start_time = time.time()
polars_merged = polars_df1.join(polars_df2, on='key')
polars_time = time.time() - start_time

Print the execution times for both Pandas and Polars:

print(f"Pandas time: {pandas_time:.6f} seconds")
print(f"Polars time: {polars_time:.6f} seconds")

On my test machine, the results were:

Pandas time: 127.604390 seconds
Polars time: 41.079080 seconds

This means that Polars is approximately 3.11 times faster than Pandas for this specific task.

Conclusion

Polars is a fast and memory-efficient data manipulation library that provides a high-level API similar to Pandas. With its ability to handle large datasets and perform complex operations quickly, Polars is an excellent choice for data scientists and analysts who need to work with big data.

Getting Started with Polars

To get started with Polars, simply install it using pip:

pip install polars

You can then import Polars in your Python code and start using its powerful features.

Learn More

To learn more about Polars and its features, check out the official documentation on GitHub.