Large dataframes can consume a significant amount of memory. By processing data in smaller chunks, you can avoid running out of memory and access data faster.
In the code above, using chunksize=100000 is approximately 5495 times faster than not using chunksize.
Divide a Large pandas DataFrame into Chunks Read More »
Partitioning data allows queries to target specific segments rather than scanning the entire table, which speeds up data retrieval.
The code above uses Delta Lake to select partitions from a pandas DataFrame. Partitioned data loading is approximately 24.5 times faster than loading the complete dataset and then querying a particular subset.
Optimize Query Speed with Data Partitioning Read More »
If you want to run code checker and formatted such as ruff, isort, pyupgrade, mypy, pylint, flake8, black, and more on Jupyter Notebooks, use nbQA.
nbQA: Run Code Formatter and Code Checker on Jupyter Notebooks Read More »
It is a bad practice to use import * in Python because it is harder to track down the origin of variables and debug your code. However, writing numerous imports explicitly from a single module can be tedious.
removestar allows you to automate replacing star imports with explicit imports.
Link to removestar.
Favorite
removestar: Automate Replacing Start Imports with Explicit Imports Read More »
If you want to analyze the memory consumption of your Python code line-by-line, use memory_profiler. This package allows you to generate a full memory usage report of your executable and plot it.
Analyze the Memory Usage of Your Python Code Read More »
Have you ever wanted to see a timeline graph showing how the execution of your functions takes place in time? Prefect now allows you to do exactly that.
To use Prefect, simply add @task and @flow to your Python functions.
Timeline View of the Execution of Your Python Functions Read More »
If you want to compare the performance between different Python snippets and plot the results, use perfplot.
My previous tips on code review.
Favorite
perfplot: Performance Analysis for Python Snippets Read More »
Sometimes, it is useful to cancel a function call when the execution time is longer than expected.
In Prefect, you can limit the execution time of a Python function call with the decorators task(timeout_seconds=n) or flow(timeout_seconds=n).
If you are not aware of Prefect, checkout the getting started tutorials.
Favorite
Limit the Execution Time of a Function Call with Prefect Read More »
If you want to have some guidelines to improve and optimize your code, try Refurb.
Refurb is only available for Python 3.10 and above.
Link to Refurb.
Favorite
Refurb: Refurbish and Modernize Python Codebases Read More »
If you want to automatically rerun a process when a file changes, use watchfiles.
In the code above, the function train will run when the file process_data.py changes.
Link to watchfiles.
Favorite
watchfiles: Rerun Code When a File Changes Read More »
CodeCut is a platform that offers short and visually appealing code snippets related to data science, data analysis, data engineering, and Python programming.
Copyright © 2025 Code Cut - All rights reserved.