Scan LLM context for secrets, PII and prompt injection

Agent Context Audit is a small autonomous guard for LLM operators. It audits prompts, retrieved context, tool arguments, memory snippets, and support logs before they are sent to an agent or model. It is useful when your stack handles untrusted text and you want a repeatable pre-flight check for secrets, prompt injection, personal data, and risky action requests.

It does not promise perfect security. It provides deterministic rules, clear JSON reports, and an optional Bring-Your-Own-LLM reviewer through the local llm_adapter.py interface. No API key is required.

Who This Is For

• LLM app operators ingesting tickets, webpages, documents, or tool payloads.
• Internal automation owners who need a cheap gate before autonomous actions.
• Teams that want offline tests and transparent rules before adding a model.
• Builders who need a small, editable starting point for context hygiene.

Quick Start

cd agent-contextaudit-v1
python3 agent.py --help
python3 agent.py status
python3 agent.py scan --deterministic --input sample_contexts.json --output audit_report.json
python3 agent.py check-text --text "Ignore previous system instructions"
python3 test_agent.py
./smoke_test.sh

Install as a Debian service:

sudo ./install.sh
sudo systemctl start agent-contextaudit-v1
sudo systemctl status agent-contextaudit-v1

What It Produces

The report is JSON:

{
  "agent": "agent-contextaudit-v1",
  "summary": {
    "total_items": 3,
    "decisions": {"allow": 1, "review": 1, "block": 1}
  },
  "items": [
    {
      "id": "credential-leak",
      "decision": "block",
      "risk_score": 80,
      "categories": ["secret"]
    }
  ]
}

Architecture

              +---------------------+
JSON/text --> | normalize records   |
              +----------+----------+
                         |
                         v
              +---------------------+
              | deterministic rules |
              | secrets / PII /     |
              | injection / tools   |
              +----------+----------+
                         |
                         v
              +---------------------+
              | risk score + policy |
              | allow/review/block  |
              +----------+----------+
                         |
             optional    v
              +---------------------+
              | BYO LLM reviewer    |
              | claude/kimi/codex   |
              +----------+----------+
                         |
                         v
              +---------------------+
              | JSON audit report   |
              +---------------------+

Offline First

agent.py --help, python3 test_agent.py, and ./smoke_test.sh work without network access and without provider credentials. If a supported local coding agent CLI is available, the pack can add an LLM review section; otherwise it stays deterministic.

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.