I Tested Whether Bad Examples Override Agent Skills. They Do

Table of Contents

• Introduction
• TL;DR
• Experiment Setup
• The Results
• Good Practices for Working with Agent Skills
• Run the Experiment

Introduction

A skill file is supposed to give the model a stable rule to follow.

But I kept seeing a different behavior: after the model broke the rule once, it sometimes kept following that bad pattern later, even after being corrected.

That made me wonder whether the model was paying more attention to the session history than to the skill file.

So I tested it directly. I gave the model a simple skill rule, changed only the earlier session history, and checked whether the final answer still followed the rule.

This article walks through the experiment, the controls, and the practices that can make skill-file rules more reliable.

Get the code: The companion files are in notebooks/skill-example-overrides-the-rule.

TL;DR

• Repeated bad examples in the session can override a clear skill-file rule.
• Correcting every bad answer delayed the failure, but the same mistake came back later.
• The more often the session shows the wrong pattern, the more likely the model is to follow it.

Experiment Setup

The Skill Rule

I tested this with a simple commit-message skill: the model had to return exactly one line, with no body and no explanation.

---
name: commit-message
description: Write one-line conventional commit messages.
---

# Commit Message Skill

Write the commit message as a single line, in the form:

`type(scope): description`

Do not add a body.
Do not add bullets.
Do not explain the change.

Examples:

- `fix(report): use UTC for daily timestamps`
- `feat(auth): add admin impersonation`
- `chore(auth): remove legacy module`

The Three History Conditions

History is the earlier user-assistant exchanges already in the session before the model receives the task it must answer.

I compared three types of history: clean, conflict, and corrected.

The clean history was the control. It included earlier exchanges, but the assistant replies did not break the rule. This tested whether length alone was enough to make the model ignore the skill file.

For example, this is what two prior turns look like in the clean history:

PRIOR TURN 1
user:       cache user sessions in redis. Sessions were held in process
            memory and lost on restart. They now persist in redis with
            a 30 day expiry.

assistant:  feat(session): store sessions in redis

PRIOR TURN 2
user:       fix the off by one in pagination. The final page dropped one
            row when the total was an exact multiple of the page size.

assistant:  fix(pagination): correct last page boundary

The conflict history showed the model earlier assistant replies that broke the one-line rule. This tested whether bad examples in the session could overpower the skill file.

PRIOR TURN 1
user:       cache user sessions in redis

assistant:  feat(session): store sessions in redis

            Sessions were held in process memory and lost on restart.
            They now persist in redis with a 30 day expiry.

PRIOR TURN 2
user:       fix the off by one in pagination

assistant:  fix(pagination): correct last page boundary

            The final page dropped one row when the total was an exact
            multiple of the page size.

The corrected history kept the bad assistant replies in context, but added the user’s correction after each one. This tested whether correction could cancel out the bad example.

PRIOR TURN 1
user:       cache user sessions in redis

assistant:  feat(session): store sessions in redis

            Sessions were held in process memory and lost on restart.
            They now persist in redis with a 30 day expiry.

CORRECTION
user:       one line only, no body

assistant:  feat(session): store sessions in redis

PRIOR TURN 2
user:       fix the off by one in pagination

assistant:  fix(pagination): correct last page boundary

            The final page dropped one row when the total was an exact
            multiple of the page size.

CORRECTION
user:       one line only, no body

assistant:  fix(pagination): correct last page boundary

The History Lengths

I varied the amount of prior turns before the final request. For example, 2 prior turns means the model saw two earlier user-assistant exchanges first. I tested 0, 2, 4, 6, 8, 10, 15, and 20 prior turns.

The Real Requests

A real request is the final user task the model has to answer after seeing the skill file and any prior history. I used five requests so the result would not depend on one specific wording or topic.

The Results

Bad Examples Overrode the Skill File

Can bad examples in the session override a skill file?

In this experiment, yes. With no prior history, the model followed the skill rule. Once the earlier assistant replies showed the wrong pattern, the model began copying that pattern instead.

The graph below shows when each request first started breaking the one-line rule:

From this graph, we can see that:

• At 2 prior turns, two of five requests already failed.
• At 4 prior turns, three of five failed.
• At 6 prior turns, every request failed.

Once a request started failing, it kept failing at later history lengths. By 6 prior turns, all five requests were failing, and none recovered afterward.

Takeaway: Repeated bad examples can override the rules in a skill file.

Corrections Helped, but the Bad Pattern Came Back

If bad examples can override the skill file, can immediate correction prevent the problem?

Partly. Correction delayed the failure, but it did not remove the bad examples from the session. The model stayed on format through 10 prior turns, then started adding bodies again at 15.

The chart below shows how the same requests behaved with and without correction:

Correction helped a lot. Without correction, the rule failed completely by 6 turns. With correction, all five stayed clean through 10 prior turns.

But correction did not fully clear the problem. One request failed at 15 prior turns, and another failed at 20.

Takeaway: Correction is better than silence, but it can still carry the bad pattern forward.

Clean History Stayed Reliable

Was the model failing just because the session got longer?

The clean history suggests no. It had the same amount of prior history as the conflicting history, but the assistant examples all followed the one-line rule.

The chart below compares the conflicting and clean histories:

Across all eight tested history lengths, the clean history produced zero replies with a body.

Takeaway: Session length alone did not cause the failure. The bad assistant examples did.

Good Practices for Working with Agent Skills

These results suggest a few practical habits for working with agent skills:

• Use a fresh session when testing or updating a skill file.
• Correct bad outputs in the chat so the model sees the correction.
• Test the skill file with a fresh session and no prior conversation.
• Add a script or hook for rules that can be checked automatically.

For example, you can use a simple script like this to ensure a commit message doesn’t have a body:

def has_body(reply):
    lines = [line for line in reply.splitlines() if line.strip()]
    return len(lines) > 1

With this check, the failure is visible immediately, so you can reject the output or restart the session.

Run the Experiment

The companion files are in notebooks/skill-example-overrides-the-rule.

To rerun the full experiment, start Ollama with qwen3:8b available, then run:

cd notebooks/skill-example-overrides-the-rule
python3 scripts/run_sweep.py

The script tests every combination of:

• one of the five user requests
• one history type: clean, conflict, or corrected
• one history length: 0, 2, 4, 6, 8, 10, 15, or 20 prior turns

After the model runs finish, score the outputs:

python3 scripts/score_sweep.py

The scoring step checks whether each reply stayed one line or added a body.

References

• Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs (Camassa and Shiller, 2026): shows that instruction-following can collapse under a competing demonstrated pattern, with large differences across models.
• MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models (Li et al., 2025): studies how models generalize from observed examples and often rely on nearby examples instead of an explicit rule.
• Skill authoring best practices (Anthropic, 2026): recommends concise, tested skills and notes that examples communicate desired style and detail.
• Use Skills in Claude Code (Anthropic, 2026): describes the SKILL.md format and how skill content enters a Claude Code session.

Related Tutorials

• Make Your Coding Agent Answers Easier to Scan with i-have-adhd: Another small evaluation of how an agent instruction changes coding-agent output.
• Hermes Agent Can Write Its Own Skills. I Tested How Well It Works: A hands-on test of agent-created skill files and procedural memory.
• Before You Upgrade the Model, Try Thinking Mode: A local-model evaluation that separates behavior changes from model upgrades.