Pandas Archives

DuckDB + PyArrow: 2900x Faster Than pandas for Large Dataset Processing

1 Comment / Delta Lake, DuckDB, Pandas, SQL / Khuyen Tran

DuckDB optimizes query execution with multiple optimizations, while PyArrow efficiently manages in-memory data processing and storage. Combining DuckDB and PyArrow allows you to efficiently process datasets larger than memory on a single machine.

In the code above, we convert a Delta Lake table with over 6 million rows to a pandas DataFrame and a PyArrow dataset, which are then used by DuckDB.

Running DuckDB on a PyArrow dataset is approximately 2906 times faster than running DuckDB on a pandas DataFrame.

DuckDB + PyArrow: 2900x Faster Than pandas for Large Dataset Processing Read More »

How to Load SQL Tables into Pandas DataFrames

Leave a Comment / Pandas, SQL / Khuyen Tran

Loading SQL tables into DataFrames allows you to analyze and preprocess the data using the rich functionality of pandas.

To read a SQL table into a pandas DataFrame, pass the database connection obtained from the SQLAlchemy Engine to the pandas.read_sql method.

How to Load SQL Tables into Pandas DataFrames Read More »

Great Tables: Create Scientific-Looking Tables in Python

Leave a Comment / Better Outputs, Pandas / Khuyen Tran

If you want to create visually appealing and scientific-looking tables in Python, use Great Tables. With Great Tables, you can customize your table by mixing and matching various elements such as headers, footers, and cell value formats.

Link to Great Tables.

Great Tables: Create Scientific-Looking Tables in Python Read More »

Use Resample to Alter Time-Series Data Frequency

Process Postgres Tables on Schedule with Kestra and Pandas

Leave a Comment / Pandas, SQL, Workflow Automation / Khuyen Tran

Imagine you have multiple tables from Postgres that you need to extract concurrently and process using Pandas on a schedule. How can you achieve this?

Kestra, an open-source orchestrator, enables you to effortlessly create this workflow using a YAML file.

Link to Kestra.

Process Postgres Tables on Schedule with Kestra and Pandas Read More »

Efficient String Data Handling in pandas 2.0 with PyArrow Arrays

Leave a Comment / Pandas, Python Tips / Khuyen Tran

As of pandas 2.0, data in pandas can be stored in PyArrow arrays in addition to NumPy arrays. PyArrow arrays provide a wide range of data types compared to NumPy.

One significant advantage of PyArrow arrays is their string datatype, which offers superior speed and memory efficiency than storing strings using object types.

Full code.

Efficient String Data Handling in pandas 2.0 with PyArrow Arrays Read More »

Enhancing Data Handling with scikit-learn’s DataFrame Support

Leave a Comment / Feature Engineer, Pandas / Khuyen Tran

By default, scikit-learn transformers return a NumPy array. This can pose a challenge if a pandas DataFrame is required for subsequent data processing steps.

Luckily, as of scikit-learn version 1.3.2, you can use the set_output method to obtain the results as a pandas DataFrame.

This method is not limited to individual transformers but can also be applied within a scikit-learn pipeline.

Enhancing Data Handling with scikit-learn’s DataFrame Support Read More »

Read HTML Tables Using Pandas

Say Goodbye to Data Type Conversion in pandas 2.0

4 Comments / Pandas / Khuyen Tran

Previously in pandas, if a Series had missing values, its data type would be converted to float, resulting in a potential loss of precision for the original data.

With the integration of Apache Arrow in pandas 2.0, this issue is solved.

Say Goodbye to Data Type Conversion in pandas 2.0 Read More »