Offline regression bench for prompts and autonomous-agent outputs.

Agent EvalBench gives an LLM operator a small, repeatable way to catch prompt regressions before they reach production. It reads evaluation cases, scores candidate outputs with deterministic checks, and can optionally ask a local BYO-LLM CLI for a structured review. No API key is required.

Who it is for

Operators who maintain prompts, agent policies, support macros, or tool instructions.
Teams that need a cheap "did this prompt get worse?" gate in CI.
Solo builders who want a local benchmark before shipping a new model, prompt, or agent version.

Honest promise

This pack does not prove that an agent is safe or correct. It catches common regressions: missing required facts, forbidden claims, weak structure, over-short answers, and rubric drift. It is strongest when you add your own real cases.

Quick start

cd agent-evalbench-v1
python3 agent.py --help
python3 agent.py run --cases examples/cases.example.json --rubric examples/rubric.example.json --out eval_report.json
python3 agent.py summarize --report eval_report.json
bash smoke_test.sh

Optional BYO-LLM review:

PACK_LLM=codex python3 agent.py run --llm --cases examples/cases.example.json

If no Claude Code, Kimi Code, or Codex CLI is installed and logged in, the agent stays useful in deterministic mode.

Flow

                +----------------------+
                | cases.example.json   |
                | prompt + candidate   |
                +----------+-----------+
                           |
                           v
                +----------------------+
                | deterministic checks |
                | required/forbidden   |
                | length/structure     |
                +----------+-----------+
                           |
                optional   v
        +------------------+------------------+
        | BYO-LLM rubric judge from PROMPTS.md |
        +------------------+------------------+
                           |
                           v
                +----------------------+
                | JSON report + exit   |
                | pass/fail summary    |
                +----------------------+

What is included

agent.py: CLI runner, deterministic scoring, optional LLM judge.
PROMPTS.md: named prompts loaded by load_prompt().
examples/cases.example.json: editable test cases.
examples/rubric.example.json: configurable thresholds and dimensions.
test_agent.py and smoke_test.sh: offline tests.
install.sh: one-shot Debian installer with a systemd service.

Typical use

Save representative prompts and candidate outputs as JSON cases.
Define required keywords, forbidden phrases, minimum length, and rubric weights.
Run EvalBench before merging prompt changes.
Track the JSON report in CI artifacts or attach it to release notes.

Agent EvalBench Pack v1.0

What's inside