Data quality checks and data contracts for datasets

A dataset gate for LLM / RAG / analytics pipelines. It profiles a tabular dataset (CSV / JSONL / JSON), checks it against declarative rules and an optional baseline profile, then returns a machine-readable verdict — and a non-zero exit code when the data is bad enough to stop the pipeline.

The promise (and its limits)

Most pipeline incidents are not code bugs: an upstream extract silently returns 0 rows, a column is renamed, a NULL rate triples, prices arrive in cents instead of euros. The job downstream keeps running and quietly trains, indexes or bills on garbage.

This agent puts a gate in front of that job:

• It blocks on hard breaches: missing required column, wrong type, duplicate primary key, out-of-range value, empty extract, stale data.
• It warns on drift against a baseline: null-rate jumps, numeric mean shifts.
• It explains — if you connect a coding-agent CLI — why the data broke and which rule change is safe, via the Analyst prompt.

What it is not: a data catalog, a lineage tracker, or a transformation engine. It does not fix your data. It refuses to let bad data through and tells you what happened.

Who it is for

Anyone running a scheduled job over data they do not control: RAG ingestion, nightly analytics loads, ETL into a warehouse, billing exports, ML retraining.

Quick start

./install.sh                       # one-shot Debian install
python3 agent.py --help

# 1. Learn what "normal" looks like
python3 agent.py profile --input samples/orders_clean.csv --out baseline.json

# 2. Gate a dataset (exit code 1 = BLOCK)
python3 agent.py check --input samples/orders_broken.csv --rules rules.json
echo "exit code: $?"

# 3. Explain the failure (uses your LLM CLI if connected, deterministic otherwise)
python3 agent.py triage --report reports/latest.json

Use it in CI or cron exactly as you would any other gate:

python3 agent.py check --input daily_export.csv || exit 1   # pipeline stops here

Architecture

        ┌─────────────┐
        │  dataset    │  CSV / JSONL / JSON
        └──────┬──────┘
               │
        ┌──────▼──────┐     rules.json      ┌──────────────┐
        │   profile   │◄────────────────────│  baseline    │
        │  (no LLM)   │                     │  (optional)  │
        └──────┬──────┘                     └──────────────┘
               │ column stats, null rates, types
        ┌──────▼──────┐
        │    check    │  deterministic rules + drift
        │  (no LLM)   │
        └──────┬──────┘
               │ findings[] {rule, severity, message}
        ┌──────▼──────┐
        │   verdict   │  PASS → 0   WARN → 0   BLOCK → 1
        └──────┬──────┘
               │ reports/latest.json
        ┌──────▼──────┐
        │   triage    │  BYO-LLM: root cause + safe rule fixes
        │  (optional) │  falls back to deterministic summary
        └─────────────┘

The gate is deterministic end to end. The LLM only ever explains a report that was already produced without it — so a missing CLI, a rate limit or an offline box degrades the agent to "still blocks bad data, just less chatty".

Contents

Verdicts

Bring your own LLM

No API key ships with this pack and none is required. If a coding-agent CLI (Claude Code / Kimi Code / Codex) is on PATH, triage uses it. Otherwise every command still runs — see CUSTOMIZE.md.

Test offline

./smoke_test.sh    # deterministic, no network, no key

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.