Fact-check LLM output against your sources

This pack is for operators who publish LLM-generated reports, support summaries, release notes, sales briefs, research notes, or executive decisions and need a repeatable proof trail. It checks each claim, confirms the referenced source is present and current, assigns a confidence score, and writes a JSON ledger that a human reviewer or downstream workflow can inspect.

It is not a legal certification tool and it does not prove that a source is true. It proves that a claim is connected to the sources you provided and flags missing, stale, weak, or unsupported evidence.

Quick Start

cd agent-provenance-v1
python3 agent.py --help
python3 agent.py init-sample --dir examples
python3 agent.py verify \
  --deterministic \
  --claims examples/claims.json \
  --sources examples/sources.json \
  --out out/ledger.json
python3 agent.py status --ledger out/ledger.json

One-shot Debian install:

sudo bash install.sh

Manual no-service install:

bash install.sh --no-service
venv/bin/python agent.py run --once --deterministic

What It Does

• Reads a claims file: claim id, text, risk, owner, artifact, and source ids.
• Reads a sources file: source id, URL, owner, verification timestamp, summary, and quotes.
• Scores each claim using deterministic token overlap, quote match, source age, and missing-source checks.
• Writes a ledger with statuses: verified, needs_review, unsupported, missing_source, stale, and invalid.
• Optionally asks your own local LLM CLI for a reviewer note. No API key is required by this pack.

Architecture

          claims.json                      sources.json
              |                                |
              v                                v
        +-------------+                +----------------+
        | Claim input |                | Source ledger  |
        +------+------+                +--------+-------+
               \                                /
                \                              /
                 v                            v
              +------------------------------------+
              | agent.py deterministic verifier    |
              | - source id resolution             |
              | - quote and token evidence score   |
              | - stale-source guard               |
              | - risk escalation flag             |
              +----------------+-------------------+
                               |
                               v
                       out/ledger.json
                               |
                               v
                 optional BYO-LLM reviewer note

Deterministic vs LLM

The LLM path uses llm_adapter.py to call a local CLI you already installed and logged in to: Claude Code, Kimi Code, or Codex. If none is present, the pack continues in deterministic mode.

Input Contract

claims.json can be a list or an object with a claims list:

{
  "claims": [
    {
      "claim_id": "c-refunds",
      "artifact": "weekly-report.md",
      "owner": "support-ops",
      "risk": "high",
      "text": "The refund queue has 18 open cases and needs triage today.",
      "source_ids": ["support-dashboard-2026-07-01"]
    }
  ]
}

sources.json can be a list or an object with a sources list:

{
  "sources": [
    {
      "source_id": "support-dashboard-2026-07-01",
      "title": "Support dashboard export",
      "url": "https://example.invalid/support/dashboard",
      "owner": "support-ops",
      "last_verified_at": "2026-07-01T09:00:00Z",
      "summary": "Daily support metrics export for refunds and triage.",
      "quotes": [
        {"text": "The refund queue has 18 open cases and needs triage today.", "locator": "row 14"}
      ]
    }
  ]
}

Useful Commands

# Generate sample input files
python3 agent.py init-sample --dir examples

# Run one deterministic audit
python3 agent.py verify --deterministic --claims examples/claims.json --sources examples/sources.json

# Inspect health
python3 agent.py status --ledger out/ledger.json

# Inspect one claim
python3 agent.py explain --claim-id c-refunds --ledger out/ledger.json

# Run the service loop once through environment defaults
PROVENANCE_CLAIMS_PATH=examples/claims.json \
PROVENANCE_SOURCES_PATH=examples/sources.json \
PROVENANCE_OUTPUT_PATH=out/ledger.json \
python3 agent.py run --once --deterministic

Expected Output

The ledger contains a summary and one entry per claim:

{
  "summary": {
    "claims_total": 2,
    "verified": 1,
    "unsupported": 1,
    "escalations": 0
  },
  "entries": [
    {
      "claim_id": "c-refunds",
      "status": "verified",
      "confidence": 1.0,
      "evidence": [{"source_id": "support-dashboard-2026-07-01"}]
    }
  ]
}

Operational Fit

Use this pack when the repeated pain is "the LLM produced a polished answer, but I cannot tell which facts are backed by which sources." It fits daily or hourly pipelines where claims are already extracted by another tool, or where a human exports claims to JSON before publication.

Do not use it as the only approval gate for regulated decisions. Treat it as a provenance and triage layer before expert review.

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.