| 📅 Today’s Picks |
Handle Large Data with Polars Streaming Mode
Problem:
In Polars, the .collect() method executes a lazy query and loads the entire dataset into memory. This works well for smaller data, but once the dataset grows beyond your available RAM, it can easily crash your process.
Solution:
Add engine=”streaming” to .collect() to process large datasets in small batches without running out of memory.
How it works:
- Breaks the dataset into smaller, memory-friendly chunks
- Processes one batch at a time while freeing memory as it goes
- Combines all partial results into a single DataFrame
Build Professional Python Packages with UV –package
Problem:
Python packages turn your code into reusable modules you can share across projects.
But building them requires complex setup with setuptools, managing build systems, and understanding distribution mechanics.
Solution:
UV, a fast Python package installer and resolver, reduces the entire process to 2 simple commands:
- uv init –package sets up your package structure instantly
- uv build and uv publish to create and distribute to PyPI
|
| ☕️ Weekly Finds |
whenever
Python Utils
Modern datetime library for Python that ensures correct and type-checked datetime manipulations. It is DST-safe and way faster than standard datetime libraries.
lancedb
MLOps
Developer-friendly, embedded retrieval database for AI/ML applications. The ultimate multimodal data platform designed for fast, scalable, and production-ready vector search.
grip
Python Utils
Preview GitHub README.md files locally before committing them. A command-line server that uses GitHub’s Markdown API to render local readme files.
