Newsletter Archive

📅
Today’s Picks

Query Nested JSON with DuckDB SQL Dot Notation

Problem:

Working with nested JSON structures requires complex normalization steps in pandas before analysis.

Solution:

DuckDB automatically flattens nested JSON files and allows direct querying of nested fields with dot notation.Other key benefits:
High-performance columnar engine for analytical workloads
Zero external dependencies – embedded database design
Native support for Parquet, CSV, JSON without data movement
Direct integration with pandas, NumPy, and Arrow format

Full Article:

A Deep Dive into DuckDB for Data Scientists

Gradio: Turn Python Functions into Interactive AI Demos

Problem:

You built an AI model that works well for your use case in your notebook. But how do you demo it to stakeholders?Your stakeholders expect clickable demos, not code snippets, but building web interfaces requires frontend expertise you don’t have.

Solution:

With Gradio, you can create professional chat interfaces with just 10 lines of code.Key benefits:
Instant UI generation from Python functions
Zero frontend coding required
Share live demos with URL links without any deployment

Full Article:

Build a Complete RAG System with 5 Open-Source Tools

☕️
Weekly Finds

presidio

Data Processing

Context aware, pluggable and customizable PII de-identification service for text and images

testcontainers-python

Python Utils

Python library providing a friendly API to run Docker containers for functional and integration testing

shapash

ML

Python library dedicated to the interpretability of Data Science models with explicit visualization labels

Favorite

📅
Today’s Picks

Query GitHub Issues with Natural Language Using LangChain

Problem:

Have you ever spent hours clicking through GitHub pages to understand project status, track bugs, or review recent changes? Manual repository analysis wastes development time that could be spent building features.

Solution:

LangChain’s GitHubIssuesLoader converts repository issues and PRs into searchable content that responds to natural language questions about bugs, features, and project status.This method integrates seamlessly with LangChain workflows.

Full Article:

Run Private AI Workflows with LangChain and Ollama

Mock External APIs for Fast, Reliable Tests

Problem:

Testing with real APIs and databases is slow, expensive, and unreliable.External dependencies create flaky tests that can fail due to network issues, rate limits, or service downtime rather than code problems.

Solution:

The patch decorator replaces external calls with controllable mock objects for isolated testing.Key benefits:
Reproducible results across different machines
Fast, reliable tests that focus on your logic
Test edge cases and error conditions that are hard to trigger naturally
Test your data processing logic without waiting for external services or consuming API quotas.

Full Article:

Pytest for Data Scientists

☕️
Weekly Finds

timesketch

Python Utils

Collaborative forensic timeline analysis tool for organizing and analyzing forensic timelines

ExtractThinker

LLM

AI-powered Document Intelligence library for LLMs, offering ORM-style interaction for flexible document workflows

ecco

ML

Explain, analyze, and visualize NLP language models with interactive visualizations in Jupyter notebooks

Favorite

📅
Today’s Picks

Delta Lake vs pandas: Stop Silent Data Corruption

Problem:

Pandas allows type coercion during DataFrame operations. A single string value can silently convert numeric columns to object dtype, breaking downstream systems and corrupting data integrity.

Solution:

Delta Lake prevents these issues through strict schema enforcement at write time, validating data types before ingestion to maintain table integrity.Other features of Delta Lake:
Time travel provides instant access to any historical data version
ACID transactions guarantee data consistency across all operations
Smart file skipping eliminates 95% of unnecessary data scanning
Incremental processing handles billion-row updates efficiently

Full Article:

Delta Lake: Transform pandas Prototypes into Production

☕️
Weekly Finds

ZeroFS

Data Engineer

ZeroFS – The Filesystem That Makes S3 your Primary Storage. Provides file-level access via NFS and 9P and block-level access via NBD on S3 storage with encryption, caching, and high performance.

vicinity

ML

Lightweight Nearest Neighbors with Flexible Backends. Provides a unified interface for vector similarity search with support for multiple backends like HNSW, FAISS, Annoy, and more.

vec2text

LLM

Utilities for decoding deep representations (like sentence embeddings) back to text. Train models to reconstruct text sequences from embeddings and invert pre-trained embeddings.

⭐
Related Post

Delta Lake: Time Travel Your Data Pipeline

Problem:

Once data is overwritten in pandas, previous versions are lost forever.You can’t debug pipeline issues or rollback bad changes when your data history disappears.

Solution:

Delta Lake maintains version history allowing you to query any previous state of your data by timestamp or version number.Use cases:
Compare today’s sales data with yesterday’s to spot revenue anomalies
Recover accidentally deleted customer records from last week’s backup
Audit financial reports using data exactly as it existed at quarter-end

Full Article:

Delta Lake: Transform pandas Prototypes into Production

Favorite

📅
Today’s Picks

Type-Safe Configuration Management with Hydra

Problem:

Configuration errors and type mismatches often go undetected until runtime, wasting time and computing resources.

Solution:

Hydra’s structured configurations with dataclasses validate types before your code runs, preventing configuration crashes.What Hydra adds to dataclasses:
Runtime parameter overrides from command line
Configuration composition and inheritance
Built-in experiment management and logging
Run multiple parameters in one command

Learn More:

Production-Ready Data Science: From Prototyping to Production with Python

ChromaDB’s Automatic Indexing: Fast Vector Search Made Easy

Problem:

Why saving vector embeddings in a file is not enough?Basic file storage forces you to scan every single embedding for similarity search, creating massive performance bottlenecks as your dataset grows.

Solution:

ChromaDB provides persistent vector storage with automatic indexing and metadata filtering capabilities.Key benefits:
Find relevant content by meaning, not just keyword matching
Handle large datasets without memory crashes using efficient indexing
Complete toolkit included: similarity scoring, deduplication, search ranking, and more

Full Article:

Build a Complete RAG System with 5 Open-Source Tools

☕️
Weekly Finds

wrapt

Python Utils

A Python module for decorators, wrappers and monkey patching

TabPFN

ML

A transformer-based foundation model for tabular data that outperforms traditional methods

superduperdb

Data Processing

A Python framework for integrating AI models, APIs, and vector search engines directly with your existing databases

Favorite

📅
Today’s Picks

DuckDB: Zero-Config SQL Database for DataFrames

Problem:

Setting up database servers for SQL operations requires complex configuration, service management, and credential setup.This creates barriers between data scientists and their analytical workflows.

Solution:

DuckDB provides an embedded SQL database with zero configuration required.Key benefits:
No server installation or management needed
Direct SQL operations on DataFrames and files
Compatible with pandas, Polars, and Arrow ecosystems
Fast analytical queries with columnar storage
Open-source with active development community
Query your data instantly without database administration overhead.

Full Article:

A Deep Dive into DuckDB for Data Scientists

Build Dynamic AI Prompts with LangChain Templates

Problem:

Hard-coded prompts limit flexibility and make it difficult to adapt AI applications to different contexts or user inputs.Creating separate functions for each prompt variation leads to duplicate code with no reusability.

Solution:

LangChain’s PromptTemplate enables dynamic, reusable prompts with variable substitution.Create one template that adapts to multiple contexts:
Variable substitution with {topic}, {audience}, {examples}
Single template for unlimited prompt variations
Clean, maintainable code structure
Compatible with all major LLM providers
Transform repetitive hard-coded prompts into flexible, reusable templates that scale with your AI application needs.

Full Article:

Run Private AI Workflows with LangChain and Ollama

☕️
Weekly Finds

GHunt

Python Utils

Modulable OSINT tool designed to investigate Google accounts and objects using various techniques

nbQA

Python Utils

Run ruff, isort, pyupgrade, mypy, pylint, flake8, and more on Jupyter Notebooks

pg_vectorize

LLM

Postgres extension that automates the transformation and orchestration of text to embeddings for vector and semantic search

Favorite