How do I automate a release GO/NO-GO decision from CI evidence?

Write every release signal to a file (test results, open risks, owners, SLO status, license audit, eval scores), then run one deterministic rule set that turns those files into GO, NO-GO or GO with named, dated waivers, and store that decision as a release artifact. Keep language models out of the verdict: they can explain a decision, but rules make it. Before any of that, separate flaky tests from real failures, or the gate will be ignored within a month.

The problem and why it matters

Most release decisions are made in a chat thread. That works until something breaks and nobody can say who approved what, based on which evidence.

• Red builds get re-run until green. Once a team learns that failures are often noise, a real regression gets the same treatment.
• Some gates never run. License changes, tool schema changes for an LLM agent, and prompt regressions do not show up in unit tests.
• Risk acceptance is verbal. "We know about that one" is not an owner and a date.
• Release notes are written last, from memory, by whoever is around.

An automated gate does not replace judgment. It makes the judgment explicit, repeatable and checkable afterwards.

A practical approach you can apply today

1. Define evidence as files

Pick a directory per release and decide what goes in it:

• Changes: id, owner, whether it affects production, and a rollback plan.
• Tests: id, status, whether it is required, owner.
• Risks: severity, status, owner, and if accepted, who accepted it and when.
• SLO: healthy or not, and any approved waiver with a date.
• Owners: the list of valid owner ids.
• Side reports: license audit, eval report, tool schema diff.

Files beat API calls here: they can be committed, diffed and replayed.

2. Write the rules as code, and block by default

A starting rule set:

• A required test that did not pass: NO-GO.
• A production-impacting change with no known owner: NO-GO.
• An open risk with no named owner and dated acceptance: NO-GO.
• An unhealthy SLO with no dated, approved waiver: NO-GO.
• An empty rollback plan: NO-GO.
• A denied license, a breaking tool schema change, or an eval score under threshold: NO-GO.
• Everything passes, but some risk was formally accepted: GO with waivers, listing who signed.

3. Handle flaky tests with evidence, not reruns

Record the commit SHA with every JUnit report. Then:

• Same commit, both pass and fail: the test is flaky, not the code.
• Fails every run: broken. Never quarantine it; that hides a real bug.
• Green, then red from one commit on: regressed. Treat it as a real failure.

Quarantine only proven flakes, keep the list in version control, and release a test automatically after a run of clean results.

4. Make the decision replayable

Fingerprint the inputs, give each decision a record id, and store the full record. Later, anyone should be able to recompute it from the same files and get the same answer. If the stored record was edited, the recomputation should fail.

5. Use exit codes as the contract

Non-zero for block, zero otherwise. Keep "the tool crashed" distinguishable from "the gate said no".

6. Generate release notes from the same commits

Classify Conventional Commits by type and put breaking changes first. A model can improve wording, but should not add or drop entries.

How the featured packs implement this

All six packs are $3 one-time, run offline in their deterministic core, and need no API key. Where an LLM is used, it is your own Claude Code, Kimi Code or Codex CLI, or a local Ollama server, and it never decides a verdict.

Release Gate Agent: the decision

Release Gate Agent reads changes.json, tests.json, risks.json, slo.json and owners.json and returns GO, NO-GO or GO-with-waivers. Its rules match the list above: required tests must be passed, production-impacting changes need a listed owner, open risks need a named owner and dated acceptance, an unhealthy SLO needs a dated approved waiver, and a nonempty rollback plan is mandatory. The record contains blockers with owners, accepted residual risks, an evidence fingerprint, a record id and a replay command. evaluate exits 2 on NO-GO; verify recalculates the record and exits 2 if any field was changed, removed or added. It performs no network access, deployment or approval action.

python3 agent.py evaluate --input my-release-evidence --output decision.json
python3 agent.py verify --input my-release-evidence --record decision.json

Agent Flaky Tests: clean the test signal first

Agent Flaky Tests ingests JUnit XML (pytest, jest-junit, go-junit-report, Maven Surefire) into a local SQLite history and labels each test flaky, broken, regressed, stable, suspect or insufficient_data. It reads retry markers as flake evidence and clusters failures by a normalised message signature, so one infrastructure problem shows up once. Quarantine only admits proven flakes above a threshold, never auto-quarantines a broken test, releases a test after a configurable number of clean runs, and exports the list as pytest --deselect arguments, a pytest marker hook, jest config, text or JSON.

venv/bin/python agent.py ingest --path "$FLAKY_REPORTS_DIR" --commit "$GIT_SHA" --branch main
venv/bin/python agent.py quarantine --export pytest --out quarantine.args

Agent License Audit: the license gate

Agent License Audit reads requirements.txt and package.json, maps each dependency to an SPDX identifier from an extendable knowledge base, classifies it (permissive, public domain, weak copyleft, strong copyleft, network copyleft, proprietary, unknown), and grades it allow, warn or deny against your policy.json. Unknown licenses are flagged, never assumed.

python3 agent.py audit --manifest package.json --strict || echo "license gate failed"

Agent EvalBench: LLM regression tests

Agent EvalBench scores saved prompt and output cases with required keywords, forbidden keywords and a minimum length, weighted by a rubric. The suite score is the mean of case scores, and the process exits non-zero below --fail-under or the rubric's pass_threshold.

python3 agent.py run --cases examples/cases.example.json --rubric examples/rubric.example.json --out eval_report.json

Agent ToolSpec: tool schema drift

Agent ToolSpec diffs the tool schemas you are about to ship against the baseline your model was prompted with. Removed parameters, new required parameters, type changes, optional-to-required changes, removed enum values, removed tools and missing handlers are breaking and exit 2. It accepts OpenAI-style parameters and Anthropic-style input_schema definitions.

Agent Changelog: the release notes

Agent Changelog turns a list of commit messages into Keep a Changelog style Markdown, grouped by Conventional Commits type with breaking changes first. Jobs run through a SQLite queue with retries, a dead-letter table and an idempotency key so the same version is not generated twice. An optional LLM rewrites wording only and cannot add or remove a change.

Honest limits

• No built-in connectors. Release Gate Agent reads its five files only; you export them from your CI, tracker and monitoring. One way to fold in the other packs is to record each of their exit codes as a required entry in tests.json.
• Flaky detection needs history. Without --commit, the strongest proof is unavailable and you will see more suspect verdicts; a single run proves nothing unless your runner records retries.
• License audit covers declared dependencies. For transitive coverage, feed it a flattened or lock manifest, and extend the bundled knowledge base.
• EvalBench checks keywords and length. Its README says it does not prove an agent is safe or correct.
• ToolSpec cannot see semantic drift. Same name, same type, new meaning passes.
• Changelog quality follows commit quality. Non-conventional messages land under "Other".

Where to start

1. Write the five evidence files by hand for your last release and run Release Gate Agent on them. Disagreements with what actually happened are your missing rules.
2. Start ingesting JUnit reports with commit SHAs into Agent Flaky Tests now; it needs runs to build evidence.
3. Add Agent License Audit with --strict, then Agent EvalBench and Agent ToolSpec if you ship LLM features.

Browse all packs at /packs, or read the quickstart to see how a pack is installed and run.