Table of Contents

• Motivation
• Introduction to SmolVLM
• Getting Started
• Chart Analysis
• Document Understanding
• Image Content Summarization
• Automated Business Intelligence with Chart Analysis
• Building a Web Dashboard
• Conclusion

Motivation

Manually analyzing charts is time-consuming. Data teams spend hours examining dashboards, extracting insights from visualizations, and documenting findings – a process that doesn’t scale when dealing with dozens of reports daily.

# Current manual process: time-consuming chart analysis
import matplotlib.pyplot as plt
import pandas as pd

# Load sales data and create visualizations
df = pd.read_csv("quarterly_sales.csv")
charts = [
    df.groupby('region').sum().plot(kind='bar'),
    df.plot(x='date', y='revenue', kind='line'),
    df.corr().style.background_gradient()  # Correlation heatmap
]

# Manual analysis required for each chart:
# 1. Open and examine each visualization
# 2. Identify patterns and trends visually
# 3. Extract key insights manually
# 4. Document findings for stakeholders

Automated insight extraction with human oversight changes this dynamic. Use AI to quickly analyze visualizations and generate initial insights, then review and refine the output. This approach reduces analysis time from hours to minutes while maintaining accuracy through human validation.

Key Takeaways

Here’s what you’ll learn:

• Analyze charts and visualizations using natural language queries instead of manual pattern recognition
• Process documents, receipts, and forms automatically with SmolVLM’s multimodal capabilities
• Build interactive web dashboards with Gradio for non-technical stakeholders to analyze visualizations
• Reduce chart analysis time from hours to minutes while maintaining human oversight for accuracy
• Deploy lightweight 500M parameter models that run efficiently on standard hardware without GPU requirements

Introduction to SmolVLM

SmolVLM enables this workflow by combining image analysis with natural language processing. This lightweight multimodal AI model, hosted on Hugging Face, processes images and text together for visual question-answering tasks.

SmolVLM excels at:

• Chart and graph analysis
• Document understanding
• Visual question answering
• Image content summarization

Install the required packages:

pip install transformers>=4.45.0 pillow torch

💻 Get the Code: The complete source code and Jupyter notebook for this tutorial are available on GitHub. Clone it to follow along!

Getting Started

SmolVLM eliminates the complexity of traditional image analysis by enabling direct natural language queries about visual content.

Let’s start by loading the model:

import torch
from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image
import requests

# Load the processor and model
model_id = "HuggingFaceTB/SmolVLM-500M-Instruct"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForVision2Seq.from_pretrained(
    model_id,
    torch_dtype=torch.float16
)

# Move model to available device
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

SmolVLM setup explained:

• AutoProcessor: Converts images and text into format the model understands
• AutoModelForVision2Seq: Loads the actual SmolVLM neural network weights
• torch.float16: Uses half-precision to reduce memory usage by 50%
• Device detection: Automatically uses GPU if available for faster inference

Now, let’s create a helper function to analyze a single image with a natural language question:

def analyze_image_with_smolvlm(image, question, max_tokens=200):
    """Analyze an image with SmolVLM using a natural language question."""
    # Format input as chat conversation
    messages = [{
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": question}
        ]
    }]

    # Convert to model input format
    prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
    inputs = processor(text=prompt, images=[image], return_tensors="pt").to(device)

    # Generate response
    generated_ids = model.generate(
        **inputs, max_new_tokens=max_tokens, do_sample=True, temperature=0.3
    )

    # Extract and return the response text
    response = processor.batch_decode(
        generated_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
    )[0]

    return response

The function above does the following:

• apply_chat_template: Formats the conversation for SmolVLM’s expected input structure
• processor(): Tokenizes text and preprocesses images into tensors
• model.generate(): Runs the actual AI inference with configurable parameters
• batch_decode(): Converts model output tokens back to readable text

Next, create a helper function to analyze multiple questions:

def print_qa_results(questions, answers, separator_length=40):
    """Print question and answer pairs with formatted output."""
    for question, answer in zip(questions, answers):
        print(f"Question: {question}")
        print(f"Answer: {answer}")
        print("-" * separator_length)

Chart Analysis

Let’s put our helper function to work by analyzing a real heatmap using natural language queries.

We’ll analyze this correlation heatmap:

Here are the questions we’ll ask:

# Load a complex chart for analysis - financial correlation heatmap
image_url = (
    "https://eodhd.com/financial-academy/wp-content/uploads/2023/12/heatmap_sector.png"
)
image = Image.open(requests.get(image_url, stream=True).raw)

# Ask questions about the chart
questions = [
    "What type of chart is this?",
    "What are the main trends shown in this visualization?",
    "What insights can you derive from this data?",
]

answers = [analyze_image_with_smolvlm(image, q) for q in questions]
print_qa_results(questions, answers, separator_length=50)

Output:

Question: What type of chart is this?
Answer:  Heatmap.
--------------------------------------------------
Question: What are the main trends shown in this visualization?
Answer: This is a sector heat map showing percentage allocation across financial sectors. Consumer Staples has the highest allocation, followed by Energy and Industrials sectors.
--------------------------------------------------
Question: What insights can you derive from this data?
Answer: Key insight: Consumer-focused sectors (services and staples) dominate the market, with communication services also performing strongly.
--------------------------------------------------

The model provides direct answers about chart type, trends, and insights without requiring manual preprocessing or specialized analysis tools.

Document Understanding

SmolVLM excels at extracting information from documents, receipts, and forms. Let’s test this with a receipt:

and ask the following questions:

# Analyze a receipt or invoice
receipt_url = "https://raw.githubusercontent.com/mistralai/cookbook/main/mistral/ocr/receipt.png"
receipt_image = Image.open(requests.get(receipt_url, stream=True).raw)

# Document analysis questions
document_questions = [
    "What type of document is this?",
    "What is the total amount?",
    "What items can you identify?",
]

answers = [analyze_image_with_smolvlm(receipt_image, q, max_tokens=150) for q in document_questions]
print_qa_results(document_questions, answers)

Output:

Question: What type of document is this?
Answer:  This is an invoice or receipt for a parking permit.
----------------------------------------
Question: What is the total amount?
Answer:  The total amount is $15.00.
----------------------------------------
Question: What items can you identify?
Answer:  The text contains a list of items, but specific details about each item are not provided.
----------------------------------------

Image Content Summarization

We can also use SmolVLM to generate a summary of an image.

Let’s use the following image:

and ask the following questions:

# Analyze a street scene
street_url = "https://images.unsplash.com/photo-1449824913935-59a10b8d2000?w=500"
street_image = Image.open(requests.get(street_url, stream=True).raw)

# Summarization prompts
summary_prompts = [
    "What's the main activity happening here?",
    "Summarize the key elements of this scene"
]

answers = [analyze_image_with_smolvlm(street_image, q, max_tokens=250) for q in summary_prompts]
print_qa_results(summary_prompts, answers, separator_length=50)

Output:

Prompt: What's the main activity happening here?
Response:  There are many people on the road in the image.
--------------------------------------------------
Prompt: Summarize the key elements of this scene
Response:  A busy city street with a few people walking and a few cars.
--------------------------------------------------

Automated Business Intelligence with Chart Analysis

See how SmolVLM integrates into a real data analysis workflow. We’ll create a sales performance chart and then analyze it with natural language queries.

First, generate sample sales data and create the visualization:

# Generate quarterly sales data
import pandas as pd
import matplotlib.pyplot as plt

data = {
    'Quarter': ['Q1 2024', 'Q2 2024', 'Q3 2024', 'Q4 2024'],
    'Product A': [45000, 52000, 48000, 61000],
    'Product B': [38000, 41000, 39000, 44000],
    'Product C': [23000, 28000, 32000, 35000]
}

df = pd.DataFrame(data)
df.set_index('Quarter').plot(kind='bar', figsize=(10, 6))
plt.title('Quarterly Sales Performance by Product')
plt.ylabel('Sales ($)')
plt.xticks(rotation=45)
plt.savefig('sales_chart.png', dpi=300)
plt.show()

Now analyze the chart with SmolVLM using targeted business questions:

from PIL import Image

# Load the chart we just created
chart_image = Image.open('sales_chart.png')

questions = [
    "What quarter had the best overall performance?",
    "Are there any concerning trends I should investigate?"
]

for question in questions:
    answer = analyze_image_with_smolvlm(chart_image, question)
    print(f"Q: {question}")
    print(f"A: {answer}\n")

Output:

Q: What quarter had the best overall performance?
A:  Q4 2024.

Q: Are there any concerning trends I should investigate?
A:  The sales figures for Q1 2024 and Q4 2024 are significantly higher than those for Q2 2024 and Q3 2024.

This workflow demonstrates SmolVLM’s value in everyday data analysis: create visualizations with your preferred tools, then get instant insights through natural language queries.

Building a Web Dashboard

To make chart analysis accessible for non-technical stakeholders, you can create an interactive web dashboard using Gradio. This Python framework enables rapid deployment of machine learning applications with just a few lines of code.

To use Gradio, start by installing it:

pip install gradio

Next, create a dashboard function that combines image upload with question input. This function uses the SmolVLM helper we defined earlier:

def dashboard_analysis(image, question):
    if image is None:
        return "Please upload an image to analyze."

    # Use the analyze_image_with_smolvlm function from earlier in the article
    return analyze_image_with_smolvlm(image, question, max_tokens=200)

If running this dashboard code independently, make sure to include the model loading and helper function from the “Getting Started” section above.

The function handles image uploads and passes user questions directly to SmolVLM. Now build the Gradio interface with three components:

import gradio as gr

dashboard = gr.Interface(
    fn=dashboard_analysis,
    inputs=[
        gr.Image(type="pil", label="Upload Chart or Visualization"),
        gr.Textbox(value="What are the key trends in this chart?", label="Ask a Question")
    ],
    outputs=gr.Textbox(label="Analysis Results"),
    title="SmolVLM Chart Analysis Dashboard"
)

Launch the dashboard to make it accessible to your team:

dashboard.launch(share=True)  # share=True creates public link

The dashboard provides a clean interface where users upload charts and ask questions in natural language.

Conclusion

SmolVLM democratizes visual analysis for data scientists. No specialized computer vision expertise needed – just natural language queries on your existing charts and visualizations.

Practical applications using SmolVLM’s capabilities:

• Chart analysis: Upload any matplotlib/seaborn visualization and ask about patterns, outliers, and trends
• Document understanding: Extract information from scanned forms, receipts, and research papers automatically
• Image summarization: Generate descriptions of complex multi-panel figures from research publications
• Dashboard automation: Batch process weekly report screenshots for automated insight generation
• Visual validation: Ask “Does this chart support my hypothesis?” to strengthen analytical conclusions

Related Resources

Here are some related resources that you may find useful:

• Docling guide for transforming PDFs into searchable AI data
• MLflow guide for building quality metrics