On-call alert triage agent with runbooks

Self-hosted agent pack for orchestrating an autonomous agent specialized in on-call. Version 1.0.0 — actionable kit with scripts, templates, named prompts and an extended checklist.

Overview

This pack provides an operational autonomous agent for the on-call domain. It includes a dedicated OODA loop, versioned prompts, runbook templates and a set of scripts for installation, testing and execution.

The core of the product is a local on-call engine. Each alert is handled according to an explicit flow: detection, acknowledgment, triage, runbook execution, silent escalation if the runbook fails, resolution and shift report. The agent does not lose an alert: unacknowledged alerts are re-raised, critical incidents are escalated and every decision is recorded in SQLite.

This pack targets teams running a 24/7 service with an SME (on-call engineer), a solo SRE or a small NOC team. It does not replace a large-scale PagerDuty platform and does not execute any arbitrary command supplied in an alert.

The pack is designed to be self-hosted (BYO-LLM): no API key is included. The LLM adapter detects a local or CLI provider (ollama, claude-code, kimi, codex).

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                    Alert source (webhook / API / file)               │
└──────────────────────────────────┬──────────────────────────────────┘
                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│  IDLE → MONITORING → TRIAGING → ACKNOWLEDGING → RUNBOOK → ESCALATING │
│  ↓                                                                   │
│  RESOLVED → HANDOFF → REPORTING                                      │
│                        OODA State Machine                            │
└──────────────────────────────────┬──────────────────────────────────┘
                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│  Named prompts → Deterministic handler → BYO-LLM Adapter             │
│  scripts/test_prompt.py    scripts/run_agent.py                      │
└──────────────────────────────────┬──────────────────────────────────┘
                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│  SQLite (alerts, incidents, escalation_log) → runbooks.yaml          │
│  → notification (webhook/simulated) / report                         │
└─────────────────────────────────────────────────────────────────────┘

Pack contents

Quick installation

cd /opt/autonomous-stack/agent-oncall-v1
bash scripts/install.sh

The installer checks for Python 3.10+, creates a venv, installs the dependencies, detects a local LLM provider and configures a systemd unit.

Immediate test

# Test the main prompt
python3 scripts/test_prompt.py --prompt triage_engine --input templates/alert_example.yaml

# Run the agent in dry-run mode
python3 scripts/run_agent.py --config scripts/config.yaml --dry-run

Execution contract

• pending: alert received, awaiting triage.
• acknowledged: alert acknowledged by the agent or a human.
• open: active incident linked to a critical alert.
• resolved: alert and incident closed with cause/resolution.
• escalated: runbook exhausted or critical severity, notification sent.
• closed: incident archived after automatic post-mortem.

Runbooks are defined in templates/runbooks.yaml. Each step carries an id, an allowed command (ping, curl, check_disk, check_service), a success condition and a maximum delay. The agent only executes the commands explicitly listed in the runbook.

Common operations

python3 agent.py alert --source monitoring --severity critical --message "disk full on db-01"
python3 agent.py run --once --deterministic
python3 agent.py status
python3 agent.py resolve --alert a-example
python3 agent.py handoff --shift night

The deterministic mode covers the included handlers without an LLM. The BYO-LLM mode is reserved for semantic triage and writing the shift report; its output remains a JSON object and any error follows the retry policy.

Customization

1. Copy scripts/config.yaml to config.oncall.yaml.
2. Adapt the templates in templates/.
3. Plug your own alert sources into run_agent.py.

Costs and limits

This pack favors deterministic logic for critical decisions. The LLM is used for semantic triage and report synthesis. Refer to SPEC.md for the operating cost estimate.

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.

License

Personal use per purchase. Resale prohibited.