Early-warning anomaly detection for slow drifts

Most monitoring waits for a hard line to break: latency > 500 ms, error rate > 5%, disk > 90%. But real incidents rarely announce themselves with one clean breach. They creep: latency drifts up 1% an hour, the error rate ticks from 0.8% to 1.4%, the queue grows slowly, a never-seen log signature appears. Every single number is still "green" — and yet something is off.

Intuition is the agent that feels that. It watches many signals at once, learns their normal rhythm, and raises a graded unease score (0-100) with a level — calm / uneasy / alarmed — the moment the combination of many small deviations becomes statistically improbable, even though no single threshold has been crossed.

Tagline: It smells trouble before your thresholds do.

The honest promise

The LLM is optional and is used for one thing only: to phrase a readable diagnosis of a signal the deterministic core already raised. No LLM, no API key → the agent is fully functional and honest about it. This is bring-your-own-LLM: if you already pay for Claude Code, Kimi Code, or Codex, the agent reasons with your subscription — no API key. See PROMPTS.md.

Why not just add more thresholds?

A threshold answers "has X broken?". Intuition answers a different question: "is the system behaving unlike itself?" — a question no single threshold can answer, because the whole point is that each metric is individually fine.

Concretely, the flair combines the tail probabilities of every metric's deviation with Fisher's method. Eight metrics each sitting at a boring p ≈ 0.2 are, taken together, jointly improbable. It also folds in slow drift (a fast baseline pulling away from a slow one) and pattern novelty (a signature never seen before). To avoid crying wolf on a single unlucky sample, the score rises only when the smoothed surprise stays elevated — a persistent unease, not a blip.

The 30-second example — "the thresholds saw nothing, the flair did"

$ python agent.py demo
{
  "calm_phase_signals": 0,          # 120 samples of normal traffic → total silence
  "drift_signal_raised": true,      # then a slow, correlated creep …
  "final_verdict": { "unease": 100.0, "level": "alarmed", ... }
}

During the drift phase, latency rose ~0.9 ms/sample, the error rate ~0.06%/sample, the queue ~0.35/sample — none of which would trip a classic threshold. The flair raised a signal well before any hard line would have.

And it grades its own hunches

A hunch you can't audit is a gimmick. Every signal the flair raises goes into a calibration ledger. Later, you annotate it — was it real trouble? — and the agent reports the precision and recall of its own intuition:

$ python agent.py calibrate --signal SIG-... --outcome true
$ python agent.py calibrate --missed          # log an incident the flair missed
$ python agent.py report
{ "calibration": { "precision": 0.83, "recall": 0.71, ... } }

That feedback loop is the difference between "a vibe" and "a measurable early-warning".

Quick start

# 1) install (Debian/Ubuntu): system deps + venv + optional systemd service
sudo bash install.sh          # or: bash install.sh --no-service

# 2) feed it observations (JSONL: one JSON object per line)
printf '%s\n' \
  '{"metrics": {"latency_ms": 102, "err_rate": 0.011}}' \
  '{"metrics": {"latency_ms": 140, "err_rate": 0.02}}' \
  | python agent.py ingest

# 3) ask how it feels
python agent.py score
python agent.py signals

# 4) get a diagnosis (LLM if a CLI is connected, deterministic otherwise)
python agent.py report

# offline self-test (no API key, exit 0 == green)
python agent.py demo
bash smoke_test.sh

Input shapes accepted by ingest (JSONL):

{"metric": "latency_ms", "value": 123, "ts": "2026-07-05T10:00:00Z"}
{"metrics": {"latency_ms": 123, "err_rate": 0.01}, "ts": "..."}
{"signature": "ERROR:db_connection_timeout"}

Who this is for

• Operators of agents and services that run 24/7 and need an early nudge, not just a pager that fires after the outage has already started.
• Teams drowning in dashboards where "everything is green" right up until it isn't.
• Anyone who wants an early-warning signal that proves its own hit-rate instead of asking for blind trust.

What it is NOT

• Not a replacement for hard thresholds and paging — it complements them by filling the gap between "nominal" and "breached".
• Not a black box: the score is a transparent statistical combination you can read in SPEC.md, and every signal lists its top contributors.
• Not a heavy install: pure Python standard library, no numpy/pandas.

See SPEC.md for the math and tunables, CUSTOMIZE.md for wiring your own data, and OPERATING_COST_ESTIMATE.md for costs.

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.