Detect and quarantine flaky CI tests

Stop re-running CI and hoping. This agent watches your JUnit reports run after run, proves which tests are actually non-deterministic, groups them by shared root cause, and keeps a quarantine list that releases itself once a test is stable again.

The honest promise

What it does

• Ingests JUnit XML from any runner (pytest, jest-junit, go-junit-report, Maven Surefire).
• Keeps every (run, test) outcome in a local SQLite history — no service, no account.
• Separates flaky (same commit, different result) from broken (fails every time) and regressed (green, then red from one commit on). Only the first is a flake.
• Clusters failures by a normalised signature, so one infra hiccup that reddens 12 tests shows up as one problem, not twelve.
• Maintains a quarantine list with an automatic release rule, and exports it in the form your runner actually consumes (--deselect args, a conftest hook, jest config).
• Produces a markdown report for humans and a JSON report for pipelines.

What it does not do

• It does not run your tests, patch your code, or talk to GitHub/GitLab. It reads reports you already produce and writes files you choose to apply.
• It cannot detect a flake from a single run — except when your runner records a retry (rerunFailure / flakyFailure), which the agent does read as evidence.
• It never quarantines a consistently failing test. That would hide a real bug, and the classifier is built specifically to refuse it.
• The LLM part is optional. Every number above is computed deterministically.

Who it is for

• A team whose CI is red often enough that "just re-run it" became a habit.
• An LLM/agent operator running a test suite in a loop, who needs a machine-readable answer to "is this failure real?" before spending a coding cycle on it.
• Anyone who wants a flake budget they can point at in a retro, with evidence.

Quick start (60 seconds)

bash install.sh --no-service          # venv only; add sudo + drop the flag for systemd
venv/bin/python agent.py demo         # bundled 6-run history -> full report

demo materializes six real JUnit reports (from samples/sample_runs.json) into a temp directory and ingests them through the normal path. They cover every verdict the classifier can emit — flaky, broken, regressed, stable, retry-recovered, insufficient data — so you see the output shape before wiring your own CI. Add --dest ./samples-out to keep the generated XML and inspect it.

Then point it at your own reports:

export FLAKY_REPORTS_DIR=/var/ci/junit
venv/bin/python agent.py ingest --path "$FLAKY_REPORTS_DIR" --commit "$GIT_SHA" --branch main
venv/bin/python agent.py report --format md --out FLAKY_REPORT.md
venv/bin/python agent.py quarantine --sync --dry-run     # see the policy's decision
venv/bin/python agent.py quarantine --sync               # apply it
venv/bin/python agent.py quarantine --export pytest --out quarantine.args
pytest $(cat quarantine.args)                            # green pipeline, debt tracked

How it works

        CI job                         agent.py                      you
   ┌──────────────┐              ┌──────────────────┐         ┌──────────────┐
   │ pytest/jest  │  junit.xml   │  WATCHING        │         │              │
   │  --junitxml ─┼─────────────▶│    │ new file?   │         │              │
   └──────────────┘              │    ▼             │         │              │
                                 │  INGESTING       │         │              │
                                 │    │ parse+store │         │              │
   ┌──────────────┐              │    ▼             │         │              │
   │  SQLite      │◀────────────▶│  CORRELATING     │         │              │
   │  history     │  runs/results│    │ same commit?│         │              │
   │  (local)     │              │    │ flips? retry│         │              │
   └──────────────┘              │    ▼             │         │              │
                                 │  verdict: flaky  │         │              │
                                 │  broken │ regressed│       │              │
                                 │    │             │         │              │
                                 │    ├─▶ QUARANTINING ──────▶│ quarantine.args
                                 │    ├─▶ RELEASING  ────────▶│ (auto, 10 clean)
                                 │    └─▶ REPORTING  ────────▶│ FLAKY_REPORT.md
                                 └──────────────────┘         └──────────────┘
                                          │
                                   optional BYO-LLM
                                   (root-cause hypothesis)

The same-commit contradiction is the core signal: if commit a1b2c3d produced both a pass and a fail for the same test, the code is not the variable — the test is. Status flips and retry recoveries are the secondary signals, used when your CI does not report a commit SHA.

Commands

Bring your own LLM (optional)

If a coding-agent CLI is installed and logged in, explain upgrades from a keyword heuristic to a reasoned hypothesis using the Executor Prompt in PROMPTS.md. No API key is stored or required — your existing subscription is the billing relationship:

PACK_LLM=claude venv/bin/python agent.py explain --test "tests/test_checkout.py::test_checkout_completes"

Supported: Claude Code (claude), Kimi Code (kimi-code), Codex CLI (codex). None installed? Every command above still works — the reply just carries "mode": "deterministic".

Install as a service

sudo bash install.sh          # venv + systemd unit, watches FLAKY_REPORTS_DIR
systemctl status agent-flaky-tests-v1
journalctl -u agent-flaky-tests-v1 -f

Files

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.

Licence & support

Sold as-is for your own infrastructure. You own the database, the reports and the quarantine list — everything is a local file you can inspect, diff and revert.