Newsletter Archive

📅
Today’s Picks

Query Nested JSON with DuckDB SQL Dot Notation

Problem:

Working with nested JSON structures requires complex normalization steps in pandas before analysis.

Solution:

DuckDB automatically flattens nested JSON files and allows direct querying of nested fields with dot notation.Other key benefits:
High-performance columnar engine for analytical workloads
Zero external dependencies – embedded database design
Native support for Parquet, CSV, JSON without data movement
Direct integration with pandas, NumPy, and Arrow format

Full Article:

A Deep Dive into DuckDB for Data Scientists

Run Code

View GitHub

Gradio: Turn Python Functions into Interactive AI Demos

Problem:

You built an AI model that works well for your use case in your notebook. But how do you demo it to stakeholders?Your stakeholders expect clickable demos, not code snippets, but building web interfaces requires frontend expertise you don’t have.

Solution:

With Gradio, you can create professional chat interfaces with just 10 lines of code.Key benefits:
Instant UI generation from Python functions
Zero frontend coding required
Share live demos with URL links without any deployment

Full Article:

Build a Complete RAG System with 5 Open-Source Tools

Run Code

View GitHub

☕️
Weekly Finds

presidio

Data Processing

Context aware, pluggable and customizable PII de-identification service for text and images

testcontainers-python

Python Utils

Python library providing a friendly API to run Docker containers for functional and integration testing

shapash

ML

Python library dedicated to the interpretability of Data Science models with explicit visualization labels

Favorite

📅
Today’s Picks

Query GitHub Issues with Natural Language Using LangChain

Problem:

Have you ever spent hours clicking through GitHub pages to understand project status, track bugs, or review recent changes? Manual repository analysis wastes development time that could be spent building features.

Solution:

LangChain’s GitHubIssuesLoader converts repository issues and PRs into searchable content that responds to natural language questions about bugs, features, and project status.This method integrates seamlessly with LangChain workflows.

Full Article:

Run Private AI Workflows with LangChain and Ollama

Run Code

View GitHub

Mock External APIs for Fast, Reliable Tests

Problem:

Testing with real APIs and databases is slow, expensive, and unreliable.External dependencies create flaky tests that can fail due to network issues, rate limits, or service downtime rather than code problems.

Solution:

The patch decorator replaces external calls with controllable mock objects for isolated testing.Key benefits:
Reproducible results across different machines
Fast, reliable tests that focus on your logic
Test edge cases and error conditions that are hard to trigger naturally
Test your data processing logic without waiting for external services or consuming API quotas.

Full Article:

Pytest for Data Scientists

Run Code

☕️
Weekly Finds

timesketch

Python Utils

Collaborative forensic timeline analysis tool for organizing and analyzing forensic timelines

ExtractThinker

LLM

AI-powered Document Intelligence library for LLMs, offering ORM-style interaction for flexible document workflows

ecco

ML

Explain, analyze, and visualize NLP language models with interactive visualizations in Jupyter notebooks

Favorite

📅
Today’s Picks

Delta Lake vs pandas: Stop Silent Data Corruption

Problem:

Pandas allows type coercion during DataFrame operations. A single string value can silently convert numeric columns to object dtype, breaking downstream systems and corrupting data integrity.

Solution:

Delta Lake prevents these issues through strict schema enforcement at write time, validating data types before ingestion to maintain table integrity.Other features of Delta Lake:
Time travel provides instant access to any historical data version
ACID transactions guarantee data consistency across all operations
Smart file skipping eliminates 95% of unnecessary data scanning
Incremental processing handles billion-row updates efficiently

Full Article:

Delta Lake: Transform pandas Prototypes into Production

Run Code

View GitHub

☕️
Weekly Finds

ZeroFS

Data Engineer

ZeroFS – The Filesystem That Makes S3 your Primary Storage. Provides file-level access via NFS and 9P and block-level access via NBD on S3 storage with encryption, caching, and high performance.

vicinity

ML

Lightweight Nearest Neighbors with Flexible Backends. Provides a unified interface for vector similarity search with support for multiple backends like HNSW, FAISS, Annoy, and more.

vec2text

LLM

Utilities for decoding deep representations (like sentence embeddings) back to text. Train models to reconstruct text sequences from embeddings and invert pre-trained embeddings.

⭐
Related Post

Delta Lake: Time Travel Your Data Pipeline

Problem:

Once data is overwritten in pandas, previous versions are lost forever.You can’t debug pipeline issues or rollback bad changes when your data history disappears.

Solution:

Delta Lake maintains version history allowing you to query any previous state of your data by timestamp or version number.Use cases:
Compare today’s sales data with yesterday’s to spot revenue anomalies
Recover accidentally deleted customer records from last week’s backup
Audit financial reports using data exactly as it existed at quarter-end

Full Article:

Delta Lake: Transform pandas Prototypes into Production

Run Code

View GitHub

Favorite

📅
Today’s Picks

Type-Safe Configuration Management with Hydra

Problem:

Configuration errors and type mismatches often go undetected until runtime, wasting time and computing resources.

Solution:

Hydra’s structured configurations with dataclasses validate types before your code runs, preventing configuration crashes.What Hydra adds to dataclasses:
Runtime parameter overrides from command line
Configuration composition and inheritance
Built-in experiment management and logging
Run multiple parameters in one command

Learn More:

Production-Ready Data Science: From Prototyping to Production with Python

Run Code

View GitHub

ChromaDB’s Automatic Indexing: Fast Vector Search Made Easy

Problem:

Why saving vector embeddings in a file is not enough?Basic file storage forces you to scan every single embedding for similarity search, creating massive performance bottlenecks as your dataset grows.

Solution:

ChromaDB provides persistent vector storage with automatic indexing and metadata filtering capabilities.Key benefits:
Find relevant content by meaning, not just keyword matching
Handle large datasets without memory crashes using efficient indexing
Complete toolkit included: similarity scoring, deduplication, search ranking, and more

Full Article:

Build a Complete RAG System with 5 Open-Source Tools

Run Code

View GitHub

☕️
Weekly Finds

wrapt

Python Utils

A Python module for decorators, wrappers and monkey patching

TabPFN

ML

A transformer-based foundation model for tabular data that outperforms traditional methods

superduperdb

Data Processing

A Python framework for integrating AI models, APIs, and vector search engines directly with your existing databases

Favorite

📅
Today’s Picks

DuckDB: Zero-Config SQL Database for DataFrames

Problem:

Setting up database servers for SQL operations requires complex configuration, service management, and credential setup.This creates barriers between data scientists and their analytical workflows.

Solution:

DuckDB provides an embedded SQL database with zero configuration required.Key benefits:
No server installation or management needed
Direct SQL operations on DataFrames and files
Compatible with pandas, Polars, and Arrow ecosystems
Fast analytical queries with columnar storage
Open-source with active development community
Query your data instantly without database administration overhead.

Full Article:

A Deep Dive into DuckDB for Data Scientists

Run Code

View GitHub

Build Dynamic AI Prompts with LangChain Templates

Problem:

Hard-coded prompts limit flexibility and make it difficult to adapt AI applications to different contexts or user inputs.Creating separate functions for each prompt variation leads to duplicate code with no reusability.

Solution:

LangChain’s PromptTemplate enables dynamic, reusable prompts with variable substitution.Create one template that adapts to multiple contexts:
Variable substitution with {topic}, {audience}, {examples}
Single template for unlimited prompt variations
Clean, maintainable code structure
Compatible with all major LLM providers
Transform repetitive hard-coded prompts into flexible, reusable templates that scale with your AI application needs.

Full Article:

Run Private AI Workflows with LangChain and Ollama

View GitHub

☕️
Weekly Finds

GHunt

Python Utils

Modulable OSINT tool designed to investigate Google accounts and objects using various techniques

nbQA

Python Utils

Run ruff, isort, pyupgrade, mypy, pylint, flake8, and more on Jupyter Notebooks

pg_vectorize

LLM

Postgres extension that automates the transformation and orchestration of text to embeddings for vector and semantic search

Favorite

📅
Today’s Picks

handcalcs: Generate LaTeX Step-by-Step Calculations from Python

Problem:

Showing the intermediate steps of the calculation is important for stakeholders to understand the calculation and verify the results.However, writing LaTeX for each calculation step is manual and time-consuming.

Solution:

handcalcs eliminates manual LaTeX writing by auto-generating mathematical documentation from your Python calculations.Perfect for engineering reports, data science documentation, and educational materials.

Full Article:

3 Tools That Automatically Convert Python Code to LaTeX Math

Run Code

View GitHub

☕️
Weekly Finds

nanoGPT

LLM

The simplest, fastest repository for training/finetuning medium-sized GPTs. A clean, minimal implementation of GPT in PyTorch.

GHunt

Python Utils

Modulable OSINT tool designed to evolve over the years, incorporates many techniques to investigate Google accounts.

beartype

Python Utils

Fast, efficient runtime type checking for Python. Open-source pure-Python runtime type checker emphasizing efficiency and portability.

⭐
Related Post

TinyDB: Python Databases Without SQL Complexity

Problem:

Databases provide essential persistence, queries, and data integrity that Python lists can’t match. However, setting up PostgreSQL or MySQL servers creates unnecessary complexity for small applications.

Solution:

TinyDB delivers these database capabilities through file-based JSON storage with simple Python dict-like operations.Key benefits:
No SQL syntax required – use familiar Python dictionary operations
Single JSON file storage – perfect for prototyping and small applications
Zero configuration setup – just import and start storing data
Pure Python implementation with no external dependencies
Start storing data with just three lines of code.

Run Code

View GitHub

Favorite

📅
Today’s Picks

LangChain: Smart Text Chunking Without Breaking Context

Problem:

RAG (Retrieval-Augmented Generation) applications require splitting documents into smaller chunks for processing.However, basic text splitting breaks semantic meaning, making your embeddings less effective for retrieval.

Solution:

LangChain’s RecursiveCharacterTextSplitter ensures your document chunks maintain meaning and context for better RAG performance.It intelligently splits text by trying these separators in order:
Double newlines (paragraphs)
Single newlines
Periods
Spaces
Individual characters (as last resort)
RecursiveCharacterTextSplitter also allows you to configure the chunk size and overlap to your specific use case.

Full Article:

Build a Complete RAG System with 5 Open-Source Tools

Run Code

View GitHub

Altair: Multi-Chart Filtering in Pure Python

Problem:

Static individual charts fail to show relationships between different data views and perspectives.Traditional dashboards require complex backend infrastructure for interactive filtering.

Solution:

Altair’s linked plots enable interactive selections that dynamically filter multiple connected visualizations.Other features of Altair:
Declarative syntax that makes visualization intuitive
Built-in data transformations and aggregations
Seamless chart composition and layering

Full Article:

Top 6 Python Libraries for Visualization: Which One to Use

Run Code

View GitHub

☕️
Weekly Finds

Boruta-Shap

ML

A Tree based feature selection algorithm which combines both the Boruta feature selection algorithm with Shapley values for interpretable feature importance

py-roughviz

Data Viz

A python visualization library for creating sketchy/hand-drawn styled charts that look fun and catchy compared to standard matplotlib graphs

prek

Python Utils

Better pre-commit re-engineered in Rust – automatically installs required Python versions and creates virtual environments with no hassle

Favorite

📅
Today’s Picks

Create Safe Temporary Files with Python tempfile

Problem:

Unit tests that create files for testing data processing functions often leave behind test artifacts or fail due to file conflicts.Running test suites in parallel or repeatedly creates naming conflicts and cluttered test environments.

Solution:

Python’s tempfile module ensures test isolation by creating unique temporary files that automatically cleanup after each test.Key benefits:
Automatic cleanup after test completion
Secure file creation with proper permissions
No naming conflicts between parallel tests
Production-safe workflows for processing large datasets
Use tempfile.NamedTemporaryFile() with context managers to process data in chunks without leaving artifacts behind.

Run Code

GLiNER: Zero-Shot Entity Recognition Without Retraining

Problem:

While spaCy provides excellent NER capabilities, its models need retraining for new entity types, which requires collecting training data, labeling examples, and running expensive model fine-tuning.This means weeks of model preparation before you can extract custom entities from your text data.

Solution:

GLiNER enables zero-shot entity recognition by accepting entity types as runtime parameters.With GLiNER, you can simply specify your desired entity types and get instant extraction results without any training.

Full Article:

langextract vs spaCy: AI-Powered vs Rule-Based Entity Extraction

Run Code

View GitHub

☕️
Weekly Finds

browser-use

LLM

Make websites accessible for AI agents. Automate tasks online with ease.

tiktoken

LLM

tiktoken is a fast BPE tokeniser for use with OpenAI’s models.

FuzzTypes

Python Utils

Pydantic extension for annotating autocorrecting fields.

Favorite

📅
Today’s Picks

Delta Lake: Time Travel Your Data Pipeline

Problem:

Once data is overwritten in pandas, previous versions are lost forever.You can’t debug pipeline issues or rollback bad changes when your data history disappears.

Solution:

Delta Lake maintains version history allowing you to query any previous state of your data by timestamp or version number.Use cases:
Compare today’s sales data with yesterday’s to spot revenue anomalies
Recover accidentally deleted customer records from last week’s backup
Audit financial reports using data exactly as it existed at quarter-end

Full Article:

Delta Lake: Transform pandas Prototypes into Production

Run Code

View GitHub

☕️
Weekly Finds

DALEX

ML

Model Agnostic Language for Exploration and eXplanation – helps explore and explain behavior of complex machine learning models

OpenBB

Data Processing

Investment Research for Everyone, Anywhere – free and open-source financial platform with analytics tools

fastlite

Python Utils

A bit of extra usability for sqlite – quality-of-life improvements for interactive use of sqlite-utils library

⭐
Related Post

Delta Lake: Never Lose Data to Failed Writes Again

Problem:

Have you ever had a pandas operation fail midway through writing data, leaving you with corrupted datasets?Partial writes create inconsistent data states that can break downstream analysis and reporting workflows.

Solution:

Delta Lake provides ACID transactions that guarantee all-or-nothing writes with automatic rollback on failures.ACID properties:
Atomicity: Complete transaction success or automatic rollback
Consistency: Data consistency guaranteed
Isolation: Safe concurrent operations
Durability: Version history with time travel

Full Article:

Delta Lake: Transform pandas Prototypes into Production

View GitHub

Favorite

📅
Today’s Picks

Build Dynamic Log Filters with Loguru Callables

Problem:

Logging is informative, but unnecessary logs can distract from the important ones. While you can filter by log level, sometimes you need to filter by some specific metric values.

Solution:

Loguru allows you to add a custom callable filter to your logger based on your specific criteria. This is significantly easier than setting up a custom filter class with standard logging.Other features of Loguru:
Beautiful logging output out of the box
Significantly simpler to use than standard logging
Rich exception tracebacks with variable values

Learn More:

Production-Ready Data Science: From Prototyping to Production with Python

Run Code

View GitHub

Whenever: Python DateTime Done Right

Problem:

Standard library datetime arithmetic ignores Daylight Saving Time (DST) transitions, producing incorrect results.Your time calculations can be off by an hour during DST changes.

Solution:

Whenever’s ZonedDateTime automatically accounts for Daylight Saving Time during time calculations.Why use Whenever:
Type-safe datetime operations prevent mixing errors
DST transitions handled automatically (no surprises)
Faster performance than standard library, Arrow and Pendulum
Drop-in replacement for standard library

Run Code

View GitHub

☕️
Weekly Finds

ExtractThinker

LLM

Document Intelligence library for LLMs offering ORM-style interaction for flexible and powerful document workflows

pytest-mock

Python Utils

Thin-wrapper around the mock package for easier use with pytest

ecco

LLM

Explain, analyze, and visualize NLP language models with interactive visualizations for Transformer models

Favorite