| 📅 Today’s Picks |
Split Large Parquet Files Automatically with Polars
Problem:
When writing large datasets to Parquet, you end up with either one massive file that is slow to read or must manually split data into smaller files.
Solution:
With Polars PartitionMaxSize, output is automatically broken into multiple Parquet files according to a defined size limit.
This enables:
- Parallel reads across multiple cores
- Faster, more reliable cloud storage transfers
Coiled: One Decorator Replaces Your Entire Docker Workflow (Sponsored)
Problem:
Have you ever had code work locally but fail on cloud VMs because of missing dependencies or version mismatches?
Docker solves this by freezing dependencies, but introduces friction: Dockerfiles, slow builds, registry pushes, and full redeploys for minor package changes.
Solution:
Coiled can remove Docker from the workflow entirely. With a single decorator, it automatically syncs your local environment to the cloud.
Key features:
- Exact dependency replication from local to cloud
- No need for container builds or registry management
- Compatible with pandas, Polars, DuckDB, Dask, and more
- Faster deployments through smart caching
|
| ☕️ Weekly Finds |
crewAI
LLM
Framework for orchestrating role-playing autonomous AI agents that work together to accomplish complex tasks
Ray
MLOps
Unified framework for scaling AI and Python applications from laptop to cluster with distributed runtime and ML libraries
Metabase
Data Viz
Open-source business intelligence tool that lets everyone visualize, analyze, and share data insights
Looking for a specific tool?
Explore 70+ Python tools →
