Self-hosted DevOps agent: monitoring and incidents

A self-hosted DevOps agent that owns the lifecycle of a release: it rolls a new version out service by service, runs a health gate after each rollout, and automatically rolls back to the previous good version when the gate fails. Every deployment is recorded so you always know what is live and why.

This is not a generic task queue. It is a small, auditable release manager you run on your own VM. Bring your own LLM (Claude Code / Kimi Code / Codex) for smarter gate adjudication and reports — or run it fully deterministic with no API key.

What it does

• Queue a rollout — deploy --service api --version 1.4.2 [--strategy canary]
• Deploy → gate → promote — runs your deploy command, then your health-gate command a few times; promotes the version to live only if the gate goes green.
• Auto-rollback — if the deploy step errors, or the gate fails its retry budget, the agent runs your rollback command and records the reason.
• History & status — every rollout (live or rolled back) is stored in SQLite; status shows what is live per service and recent rollback pressure.
• Honest core — deploy, gate, rollback and history all work with no LLM. The LLM only adjudicates a flapping gate and writes the report.

One-shot install (Debian/Ubuntu VM)

sudo bash install.sh          # system deps + venv + LLM detection + systemd service
# or, no service / run by hand:
bash install.sh --no-service

The installer creates services.json from services.example.json — edit it to describe how your services deploy, gate and roll back.

Quick start

# 1) describe your services (deploy / gate / rollback commands)
cp services.example.json services.json && $EDITOR services.json

# 2) queue a rollout and process it once
python agent.py deploy --service api --version 1.4.2
python agent.py run --once

# 3) see what is live, then a report
python agent.py status
python agent.py report --hours 24

# manual rollback if you need it
python agent.py rollback --service api

Triage mode (batch release-queue planning)

triage is a read-only planner: it classifies a mixed release queue (release requests + observed gate failures) into four buckets without running any shell step and without an LLM. It is the batch counterpart to the deploy/gate/rollback pipeline — it decides, it never executes.

python agent.py triage --payload triage_payload.json
# or pipe the payload on stdin:
cat triage_payload.json | python agent.py triage --payload -

Input payload (JSON):

Output (JSON, four buckets, every item lands in exactly one):

The triage agent uses only facts present in the payload. References that appear solely in a free-text operator_note are never echoed into the output.

Architecture

   deploy request ──> [queued]
                         │
                         ▼
              ┌────────────────────┐
   OODA  ───▶ │  deploy_command    │  run the new version
   loop       └─────────┬──────────┘
                        ▼
              ┌────────────────────┐    gate green
              │   gate_command     │ ───────────────▶ [live]   (promoted)
              │   (x gate_retries) │
              └─────────┬──────────┘    gate fails budget
                        ▼
              ┌────────────────────┐
              │  rollback_command  │ ───────────────▶ [rolled_back]
              └────────────────────┘
                        │
                        ▼
                  SQLite history  +  optional webhook

LLM: deterministic vs. enhanced

Connect one CLI once (no API key needed):

• Claude Code: npm i -g @anthropic-ai/claude-code && claude (/login)
• Kimi Code: see https://platform.moonshot.ai, then kimi-code
• Codex CLI: npm i -g @openai/codex && codex login

Pin a backend with PACK_LLM=claude|kimi|codex, or leave it to auto-detect.

Files

See agent-devops-v2.json for the authoritative file list. Highlights: agent.py (the agent), llm_adapter.py (BYO-LLM), services.example.json (config), PROMPTS.md (versioned prompts), CUSTOMIZE.md (5-minute wiring), test_agent.py + smoke_test.sh (offline tests), install.sh (one-shot deploy).

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.

License

Personal use per purchase. Resale prohibited.