| 📅 Today’s Picks |
Automate LLM Evaluation at Scale with MLflow make_judge()
Problem:
When you ship LLM features without evaluating them, models might hallucinate, violate safety guidelines, or return incorrectly formatted responses.
Manual review doesn’t scale. Reviewers might miss subtle issues when evaluating thousands of outputs, and scoring standards often vary between people.
Solution:
MLflow make_judge() applies the same evaluation standards to every output, whether you’re checking 10 or 10,000 responses.
Key capabilities:
- Define evaluation criteria once, reuse everywhere
- Automatic rationale explaining each judgment
- Built-in judges for safety, toxicity, and hallucination detection
- Typed outputs that never return unexpected formats
| ⭐ Worth Revisiting |
LangChain v1.0: Auto-Protect Sensitive Data with PIIMiddleware
Problem:
User messages often contain sensitive information like emails and phone numbers.
Logging or storing this data without protection creates compliance and security risks.
Solution:
LangChain v1.0 introduces PIIMiddleware to automatically protect sensitive data before model processing.
PIIMiddleware supports multiple protection modes:
- 5 built-in detectors (email, credit card, IP, MAC, URL)
- Custom regex for any PII pattern
- Replace with [REDACTED], mask as ****1234, or block entirely
| ☕️ Weekly Finds |
litellm
LLM
Python SDK and Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI format with cost tracking, guardrails, and logging.
parlant
LLM
LLM agents built for control with behavioral guidelines, ensuring predictable and consistent agent behavior.
GLiNER2
ML
Unified schema-based information extraction for NER, text classification, and structured data parsing in one pass.
Looking for a specific tool?
Explore 70+ Python tools →
