Apply Multiple Functions to a DataFrame with Pipe
To increase code readability when applying multiple functions to a DataFrame, use the pandas.DataFrame.pipe method.
Apply Multiple Functions to a DataFrame with Pipe Read More »
Pandas
Polars vs. Pandas for CSV Loading and Filtering
The read_csv method in Pandas loads all rows of the dataset into the DataFrame before filtering to remove all unwanted rows.
On the other hand, the scan_csv method in Polars delays execution and optimizes the operation until the collect method is called.
This approach accelerates code execution, particularly when handling large datasets.
Polars vs. Pandas for CSV Loading and Filtering Read More »
Seamless Tracking of Changes in Pandas DataFrame with Delta Lake
Maintaining a consistent record of database changes is crucial to recover data in the event of system failures or investigating security breaches.
Delta Lake enables seamless tracking of changes made to a pandas DataFrame such as creation time, size, and statistics.
Seamless Tracking of Changes in Pandas DataFrame with Delta Lake Read More »
Efficient Data Appending in Parquet Files: Delta Lake vs. Pandas
Appending data to an existing Parquet file using pandas involves loading the existing table and merging the new data with the existing table.
This process can be time-consuming and memory-intensive.
With Delta Lake, you can add, remove, or modify columns without the need to recreate the entire table.
Efficient Data Appending in Parquet Files: Delta Lake vs. Pandas Read More »
PandasAI: Gain Insights From Your pandas DataFrame With AI
If you want to quickly gain insights from your pandas DataFrame with AI, use PandasAI.
PandasAI serves as:
✅ A tool to analyze your DataFrame
❌ Not a tool to process your DataFrame
PandasAI: Gain Insights From Your pandas DataFrame With AI Read More »
Overwrite Partitions of a pandas DataFrame with Delta Lake
If you need to modify a specific subset of your pandas DataFrame, such as yesterday’s data, it is not possible to overwrite only that partition. Instead, you have to load the entire DataFrame into memory as a workaround solution.
Delta Lake makes it easy to overwrite partitions of a pandas DataFrame.
Overwrite Partitions of a pandas DataFrame with Delta Lake Read More »
Raise an Exception for a Chained Assignment in pandas
Pandas allows chained assignments, which involve performing multiple indexing operations in a single statement, but they can lead to unexpected results or errors.
The statement above fails to modify the values in df as intended, but it doesn’t throw an error.
Setting pd.options.mode.chained_assignment to 'raise' will cause pandas to raise an exception if a chained assignment occurs.
Raise an Exception for a Chained Assignment in pandas Read More »
Include All Rows When Merging Two DataFrames
df.merge only includes rows with matching values in both DataFrames. If you want to include all rows from both DataFrames, use how='outer'.My previous tips on pandas.
Include All Rows When Merging Two DataFrames Read More »
Optimizing Memory Usage in a pandas DataFrame with infer_objects
pandas DataFrames that contain columns of mixed data types are stored in a more general format (such as object), resulting in inefficient memory usage and slower computation times.
df.infer_objects() infers the true data types of columns in a DataFrame, which helps optimize memory usage in your code.
In the code above, df.infer_objects() converts the data type of “col1” from object to int64, saving approximately 27 MB of memory.
Optimizing Memory Usage in a pandas DataFrame with infer_objects Read More »
Stack Columns into Rows in Pandas
If you want to stack the columns into rows in pandas, use DataFrame.stack().
Stack Columns into Rows in Pandas Read More »