Use the parse_dates parameter to specify datetime columns when creating a pandas DataFrame from a CSV, rather than converting columns to datetime post-creation. This keeps the code concise and easier to read.
Specify Datetime Columns with parse_dates Read More »
When dealing with Parquet files in pandas, it is common to first load the data into a pandas DataFrame and then apply filters.
To improve query execution speed, push down the filers to the PyArrow engine to leverage PyArrow’s processing optimizations.
In the code above, filtering a dataset of 100 million rows using PyArrow is approximately 113 times faster than filtering using pandas.
Leverage PyArrow for Efficient Parquet Data Filtering Read More »
When merging two DataFrames with overlapping columns, the default behavior is to add suffixes “_x” and “_y” to the column names. To improve readability, you can specify custom suffixes.
Enhance Readability in DataFrame Merging with Custom Suffixes Read More »
Use pandas’ vectorized operations instead of performing operations on each column individually.
This leverages pandas’ optimized C implementation for better performance, especially with large datasets.
Optimize Your Pandas Code with Vectorized Operations Read More »
Generating readable edge cases for unit tests can often be a challenging task. However, with the combined power of Pandera and Hypothesis, you can efficiently detect falsified examples and write cleaner tests.
Pandera allows you to define constraints for inputs and outputs, while Hypothesis automatically identifies edge cases that match the specified schema.
Hypothesis further simplifies complex examples until it finds a smaller example that still reproduces the issue.
Efficiently Generate Falsified Examples for Unit Tests with Pandera and Hypothesis Read More »
Large dataframes can consume a significant amount of memory. By processing data in smaller chunks, you can avoid running out of memory and access data faster.
In the code above, using chunksize=100000 is approximately 5495 times faster than not using chunksize.
Divide a Large pandas DataFrame into Chunks Read More »
To perform operations between two pandas objects, it’s often necessary to ensure that two pandas objects have the same row or column labels.
The df.align method allows you to align two pandas objects along specified axes.
Align Pandas Objects for Effective Data Manipulation Read More »
If you want to keep informed about the progress of a pandas apply operation, use tqdm.
tqdm: Add Progress Bar to Your Pandas Apply Read More »
Have you ever wanted to highlight your pandas DataFrame for easier analysis? For example, you might want positive values in green and negative ones in red.
That could be done with df.style.apply.
Highlight Your pandas DataFrame for Easier Analysis Read More »
To undo errors, avoid losing data, and reproduce results, it is crucial to implement a version control system for your data.
Delta Lake simplifies pandas DataFrame versioning and allows access to prior versions for auditing and debugging.
In the example above, Delta Lake creates two versions of a DataFrame. Version 0 contains the original data, while Version 1 includes the data that was appended.
Version Your Pandas DataFrame with Delta Lake Read More »