Agent Flaky Tests v1.0

Stop re-running CI and hoping. This agent watches your JUnit reports run after run, proves which tests are actually non-deterministic, groups them by shared root cause, and keeps a quarantine list that releases itself once a test is stable again.

The honest promise

What it does

• Ingests JUnit XML from any runner (pytest, jest-junit, go-junit-report, Maven Surefire).
• Keeps every (run, test) outcome in a local SQLite history — no service, no account.
• Separates flaky (same commit, different result) from broken (fails every time) and regressed (green, then red from one commit on). Only the first is a flake.
• Clusters failures by a normalised signature, so one infra hiccup that reddens 12 tests shows up as one problem, not twelve.
• Maintains a quarantine list with an automatic release rule, and exports it in the form your runner actually consumes (--deselect args, a conftest hook, jest config).
• Produces a markdown report for humans and a JSON report for pipelines.

What it does not do

• It does not run your tests, patch your code, or talk to GitHub/GitLab. It reads reports you already produce and writes files you choose to apply.
• It cannot detect a flake from a single run — except when your runner records a retry (rerunFailure / flakyFailure), which the agent does read as evidence.
• It never quarantines a consistently failing test. That would hide a real bug, and the classifier is built specifically to refuse it.
• The LLM part is optional. Every number above is computed deterministically.

Who it is for

• A team whose CI is red often enough that "just re-run it" became a habit.
• An LLM/agent operator running a test suite in a loop, who needs a machine-readable answer to "is this failure real?" before spending a coding cycle on it.
• Anyone who wants a flake budget they can point at in a retro, with evidence.

Quick start (60 seconds)

bash install.sh --no-service          # venv only; add sudo + drop the flag for systemd
venv/bin/python agent.py demo         # bundled 6-run history -> full report

demo materializes six real JUnit reports (from samples/sample_runs.json) into a temp directory and ingests them through the normal path. They cover every verdict the classifier can emit — flaky, broken, regressed, stable, retry-recovered, insufficient data — so you see the output shape before wiring your own CI. Add --dest ./samples-out to keep the generated XML and inspect it.

Then point it at your own reports:

export FLAKY_REPORTS_DIR=/var/ci/junit
venv/bin/python agent.py ingest --path "$FLAKY_REPORTS_DIR" --commit "$GIT_SHA" --branch main
venv/bin/python agent.py report --format md --out FLAKY_REPORT.md
venv/bin/python agent.py quarantine --sync --dry-run     # see the policy's decision
venv/bin/python agent.py quarantine --sync               # apply it
venv/bin/python agent.py quarantine --export pytest --out quarantine.args
pytest $(cat quarantine.args)                            # green pipeline, debt tracked

How it works

        CI job                         agent.py                      you
   ┌──────────────┐              ┌──────────────────┐         ┌──────────────┐
   │ pytest/jest  │  junit.xml   │  WATCHING        │         │              │
   │  --junitxml ─┼─────────────▶│    │ new file?   │         │              │
   └──────────────┘              │    ▼             │         │              │
                                 │  INGESTING       │         │              │
                                 │    │ parse+store │         │              │
   ┌──────────────┐              │    ▼             │         │              │
   │  SQLite      │◀────────────▶│  CORRELATING     │         │              │
   │  history     │  runs/results│    │ same commit?│         │              │
   │  (local)     │              │    │ flips? retry│         │              │
   └──────────────┘              │    ▼             │         │              │
                                 │  verdict: flaky  │         │              │
                                 │  broken │ regressed│       │              │
                                 │    │             │         │              │
                                 │    ├─▶ QUARANTINING ──────▶│ quarantine.args
                                 │    ├─▶ RELEASING  ────────▶│ (auto, 10 clean)
                                 │    └─▶ REPORTING  ────────▶│ FLAKY_REPORT.md
                                 └──────────────────┘         └──────────────┘
                                          │
                                   optional BYO-LLM
                                   (root-cause hypothesis)

The same-commit contradiction is the core signal: if commit a1b2c3d produced both a pass and a fail for the same test, the code is not the variable — the test is. Status flips and retry recoveries are the secondary signals, used when your CI does not report a commit SHA.

Commands

Bring your own LLM (optional)

If a coding-agent CLI is installed and logged in, explain upgrades from a keyword heuristic to a reasoned hypothesis using the Executor Prompt in PROMPTS.md. No API key is stored or required — your existing subscription is the billing relationship:

PACK_LLM=claude venv/bin/python agent.py explain --test "tests/test_checkout.py::test_checkout_completes"

Supported: Claude Code (claude), Kimi Code (kimi-code), Codex CLI (codex). None installed? Every command above still works — the reply just carries "mode": "deterministic".

Install as a service

sudo bash install.sh          # venv + systemd unit, watches FLAKY_REPORTS_DIR
systemctl status agent-flaky-tests-v1
journalctl -u agent-flaky-tests-v1 -f

Files

Licence & support

Sold as-is for your own infrastructure. You own the database, the reports and the quarantine list — everything is a local file you can inspect, diff and revert.