Diagnose failed LLM agent runs from your logs

Your agent runs do not fail because the model is dumb. They fail because a call timed out under a budget nobody re-measured, because a weekly rate-limit window is still closed, or because a 402 was filed as "the task failed" and burned the retry budget. This pack reads your own attempt logs and answers four questions deterministically, offline, with no provider call of its own.

Who it is for

Anyone running autonomous agents in production against a coding-agent CLI or an LLM API: multi-provider workers, long-horizon runs, nightly pipelines. If you already keep a log of "which provider, which tier, how long, which exit code", this pack turns it into a routing decision.

What it answers

Quick start

sudo bash install.sh --no-service
cp examples/run_ledger.json my_runs.json
./venv/bin/python agent.py --ledger my_runs.json audit
./venv/bin/python agent.py --ledger my_runs.json report --out reliability_report.md

Each attempt needs attempt_id, run_id, tier, provider, an ISO started_at, and whatever your logs already carry: duration_s, exit_code, http_status, error_text. Anything unreadable becomes a manual_review row with the source value echoed unchanged — never a guessed number.

run attempt ledger + routing config
        |
        v
classify each attempt      (provider signal BEFORE exit code)
        |
        +--> timeout budget vs longest successful call ----+
        +--> decode rate-limit windows (weekly / session) --+
        +--> 402 / 503 / login -> provider_unavailable -----+--> fallback plan
        +--> genuine failures -> task_failures              |        |
                                                            +--> re-dispatch queue
                                                                     |
                                              JSON / Markdown report + optional BYO-LLM brief

The three mistakes it refuses to make

1. A provider refusal is not a task failure. 402 (membership/credit), 503 (overload) and a missing login land in provider_unavailable with task_failed: false and a soft-defer. They never consume a run's retry budget as if the work had been wrong.
2. A reopened window is not an outage. A rate-limit window whose reset time is at or before the audit instant is reported expired: true, blocked: false and must not keep a healthy provider out of the fallback chain. A window whose reset cannot be decoded stays blocked — the audit fails closed, never open.
3. A run that already succeeded, or that ended on a real assertion failure, is never re-dispatched. The first would duplicate delivered work; the second would hide a defect behind an infinite retry.

Honest operating modes

No API key and no network call are required. The optional CLI uses the subscription already connected on the machine. This agent is read-only: it never calls a provider, never edits a routing config, and never re-dispatches anything itself — config_mutated, provider_calls_made and redispatch_executed are part of every output so you can prove it.

It also does not report money. Per-run spend is a different job; this pack is about time, windows, and order.

Commands

python agent.py --ledger my_runs.json audit
python agent.py --ledger my_runs.json plan
python agent.py --ledger my_runs.json --as-of 2026-07-30T12:00:00Z audit
python agent.py --ledger my_runs.json report --out reliability_report.md
python agent.py --ledger my_runs.json brief
python agent.py status
python test_agent.py

See CUSTOMIZE.md to connect your own logs and CHECKLIST.md before you change a timeout budget from the output.

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.