| 📅 Today’s Picks |
Faster Table Joins with Polars Multi-Threading
Problem:
pandas processes joins on a single CPU core, leaving other cores idle during large table operations.
Solution:
Polars distributes join operations across all available CPU cores, achieving significantly faster joins than pandas on large datasets.
What makes Polars fast:
- Processes rows in parallel batches
- Uses all available CPU cores
- Zero configuration required
| ⭐ Worth Revisiting |
Faster Polars Queries with Programmatic Expressions
Problem:
When you want to use for loops to apply similar transformations, each Polars with_columns() call processes sequentially.
This prevents the optimizer from seeing the full computation plan.
Solution:
Instead, generate all Polars expressions programmatically before applying them together.
This enables Polars to:
- See the complete computation plan upfront
- Optimize across all expressions simultaneously
- Parallelize operations across CPU cores
| ☕️ Weekly Finds |
Mole
Python Utils
Deep clean and optimize your Mac with a simple command-line tool.
marker
LLM
Convert PDF, DOCX, PPTX, and other documents to markdown with high speed and accuracy.
pathway
Data Engineer
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Looking for a specific tool?
Explore 70+ Python tools →
