Detect breaking changes in LLM tool schemas

Contract guard for the tool/function schemas your LLM agent exposes to a model.

You rename a parameter, tighten a type, or make an optional field required. Your tests pass — the code is internally consistent. But the model was prompted with the old schema, so it keeps emitting the old argument shape, and the tool call fails at runtime. Often silently. Often only in production, on the calls you never wrote a test for.

Agent ToolSpec diffs the schemas you are about to ship against the baseline the model already knows, classifies every change as breaking or compatible, and cross-checks that a real handler exists for every tool you declare. It exits non-zero on breaking drift, so you can wire it into CI as a gate.

The promise (and its limits)

It does: deterministic, explainable schema diffing — parameter removals, type changes, new required arguments, narrowed enums, missing handlers, and handlers the model can never reach. Every verdict is derived only from the files you attach.

It does not: read your mind about semantics. If you keep a parameter's name and type but silently change what it means, that is a semantic break no schema diff can see. It also does not call your live model or replay traffic.

Who this is for

• Teams shipping an LLM agent whose tool definitions change between releases.
• Anyone running function calling / tool use where a bad arg fails quietly.
• Operators who want a cheap, offline pre-flight gate before a deploy.
• Builders who need a small, editable starting point for schema governance.

Quick start

bash install.sh                       # one-shot Debian install
python agent.py init-sample --dir examples
python agent.py audit \
    --baseline examples/baseline_tools.json \
    --current  examples/current_tools.json \
    --handlers examples/handlers.json \
    --deterministic
echo $?                               # 2 => breaking change found

The sample ships a deliberate break: search_docs.top_k flips from an optional integer to a required string. The agent flags it and blocks.

How it reads a change

   baseline_tools.json          current_tools.json         handlers.json
   (what the model knows)       (what you ship)            (what code accepts)
            |                          |                         |
            +------------+-------------+------------+------------+
                         |                          |
                    diff_parameters()         handler cross-check
                         |                          |
                         +-------------+------------+
                                       |
                                  classify()
                                       |
        +-------------+----------------+----------------+-------------+
        |             |                |                |             |
     stable    compatible_change  breaking_change  missing_handler  removed_tool
        |             |                |                |             |
        +------ ALLOW -+                +--------- BLOCK (exit 2) ----+

What counts as breaking

Set TOOLSPEC_STRICT_NEW_TOOLS=1 to also flag a brand-new tool that ships with a required argument.

Bring your own LLM (optional)

With claude, kimi-code, or codex on your PATH, the agent adds a written migration review through the Reviewer prompt in PROMPTS.md. No API key — your existing CLI subscription is the billing relationship. The LLM only comments; it never overrides a deterministic verdict.

python agent.py status                # which backend is connected, if any
python agent.py audit --baseline b.json --current c.json    # LLM review attached

Without a CLI, the deterministic core runs alone and the report says so.

CI gate

python agent.py audit --baseline baseline_tools.json --current current_tools.json \
    --handlers handlers.json --deterministic || exit 1

Commit baseline_tools.json as the schema your deployed prompt was built against. Refresh it only when you re-prompt or re-deploy the model.

Files

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.

License

Commercial pack. White-label friendly — rename, re-theme, and ship it.