Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Filter by Categories
About Article
Analyze Data
Archive
Best Practices
Better Outputs
Blog
Code Optimization
Code Quality
Command Line
Course
Daily tips
Dashboard
Data Analysis & Manipulation
Data Engineer
Data Visualization
DataFrame
Delta Lake
DevOps
DuckDB
Environment Management
Feature Engineer
Git
Jupyter Notebook
LLM
LLM Tools
Machine Learning
Machine Learning & AI
Machine Learning Tools
Manage Data
MLOps
Natural Language Processing
Newsletter Archive
NumPy
Pandas
Polars
PySpark
Python Helpers
Python Tips
Python Utilities
Scrape Data
SQL
Testing
Time Series
Tools
Visualization
Visualization & Reporting
Workflow & Automation
Workflow Automation

Python Tips

Efficient String Data Handling in pandas 2.0 with PyArrow Arrays

As of pandas 2.0, data in pandas can be stored in PyArrow arrays in addition to NumPy arrays. PyArrow arrays provide a wide range of data types compared to NumPy.

One significant advantage of PyArrow arrays is their string datatype, which offers superior speed and memory efficiency than storing strings using object types.

Full code.

Efficient String Data Handling in pandas 2.0 with PyArrow Arrays Read More »

Save Disk Space on Large Datasets with Parquet

To save disk space on large datasets, use Parquet files instead of CSV. Because Parquet files are compressed, they take up less space on disk and in memory than uncompressed CSV files.

For a 1 million row, 10 column dataset, storing it as CSV takes about 189.59 MB, while storing it as Parquet takes around 78.96 MB, saving approximately 110.63 MB of storage.

Full code for the comparison.

Save Disk Space on Large Datasets with Parquet Read More »

Python 3.10’s strict=True: Preventing Data Loss in zip Operations

zip allows you to aggregate elements from each of the iterables. However, if the iterables are not the same length, the remaining elements of the longer iterables are not paired, potentially leading to unnoticed bugs.

Starting with Python 3.10, using the strict keyword argument with zip raises a ValueError if the lengths of the iterables are unequal.

Python 3.10’s strict=True: Preventing Data Loss in zip Operations Read More »

Use Mixins Over Inheritance for Enhanced Modularity

Use mixin instead of inheritance to add shared functionality without changing the primary structure of a class.

In the example above, the inheritance-based approach assumes that the PickleableModel class has a .model attribute that needs to be serialized. This assumption may not be true for all subclasses.

The PickleableMixin can be applied to any class that needs serialization, not just machine learning models.

Full code.

Use Mixins Over Inheritance for Enhanced Modularity Read More »

Scroll to Top

Work with Khuyen Tran

Work with Khuyen Tran