Offline regression tests for LLM prompts and agents

Offline regression bench for prompts and autonomous-agent outputs.

Agent EvalBench gives an LLM operator a small, repeatable way to catch prompt regressions before they reach production. It reads evaluation cases, scores candidate outputs with deterministic checks, and can optionally ask a local BYO-LLM CLI for a structured review. No API key is required.

Who it is for

• Operators who maintain prompts, agent policies, support macros, or tool instructions.
• Teams that need a cheap "did this prompt get worse?" gate in CI.
• Solo builders who want a local benchmark before shipping a new model, prompt, or agent version.

Honest promise

This pack does not prove that an agent is safe or correct. It catches common regressions: missing required facts, forbidden claims, weak structure, over-short answers, and rubric drift. It is strongest when you add your own real cases.

Quick start

cd agent-evalbench-v1
python3 agent.py --help
python3 agent.py run --cases examples/cases.example.json --rubric examples/rubric.example.json --out eval_report.json
python3 agent.py summarize --report eval_report.json
bash smoke_test.sh

Optional BYO-LLM review:

PACK_LLM=codex python3 agent.py run --llm --cases examples/cases.example.json

If no Claude Code, Kimi Code, or Codex CLI is installed and logged in, the agent stays useful in deterministic mode.

Flow

                +----------------------+
                | cases.example.json   |
                | prompt + candidate   |
                +----------+-----------+
                           |
                           v
                +----------------------+
                | deterministic checks |
                | required/forbidden   |
                | length/structure     |
                +----------+-----------+
                           |
                optional   v
        +------------------+------------------+
        | BYO-LLM rubric judge from PROMPTS.md |
        +------------------+------------------+
                           |
                           v
                +----------------------+
                | JSON report + exit   |
                | pass/fail summary    |
                +----------------------+

What is included

• agent.py: CLI runner, deterministic scoring, optional LLM judge.
• PROMPTS.md: named prompts loaded by load_prompt().
• examples/cases.example.json: editable test cases.
• examples/rubric.example.json: configurable thresholds and dimensions.
• test_agent.py and smoke_test.sh: offline tests.
• install.sh: one-shot Debian installer with a systemd service.

Typical use

1. Save representative prompts and candidate outputs as JSON cases.
2. Define required keywords, forbidden phrases, minimum length, and rubric weights.
3. Run EvalBench before merging prompt changes.
4. Track the JSON report in CI artifacts or attach it to release notes.

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.