How do I set up self-hosted uptime monitoring with automatic restart on a Linux VM?

Use two layers. Let systemd restart a process the moment it exits, and run a small watcher on the same VM that probes what users actually hit (an HTTP endpoint, a TCP port) and restarts the service only after several failures in a row, with a cap on restarts per hour. Add one incident record per outage and a backup job you can verify, and you cover most single-VM failures without a full monitoring stack.

The problem

A service on a VM fails in more ways than "the process died":

• It crashes and exits. systemd can handle this on its own.
• It is alive but stuck. The port answers 502, or nothing answers at all. systemd still sees an active unit and does nothing.
• A dependency is down. The database port is closed or the disk is full. Restarting your app will not help.
• It degrades slowly. Latency creeps up, the error rate ticks up, but no alert threshold is crossed until users notice.

Hosted uptime checkers cannot restart anything on the box, and large monitoring stacks are a lot to run for one or two VMs. A small watcher on the machine itself sits in between.

A practical approach you can apply today

1. Let systemd handle crashes

A minimal unit for an app:

[Unit]
Description=My API
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=600
StartLimitBurst=5

[Service]
Type=simple
User=myapi
WorkingDirectory=/opt/myapi
ExecStart=/opt/myapi/venv/bin/python app.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

• Restart=on-failure restarts after a non-zero exit, a crash signal or a timeout. Restart=always also restarts after a clean exit.
• RestartSec=5 adds a pause so a crash loop does not burn CPU.
• StartLimitIntervalSec and StartLimitBurst (in [Unit]) are systemd's own loop guard: more than 5 starts within 10 minutes and systemd gives up and marks the unit failed. After fixing the cause, clear it with systemctl reset-failed myapi.

See the systemd.service and systemd.unit man pages.

2. Probe what users see

systemd only knows whether the process runs. Add probes:

• HTTP: request a health URL; treat a 5xx status or a timeout as a failure.
• TCP: open a connection to a port (database, cache, queue).
• Unit state: systemctl is-active for services you do not own.
• Custom command: anything that exits 0 when healthy, such as a free-space check or a file that must exist.

Run probes every 30 to 60 seconds with a short timeout.

3. Avoid restart loops and false alarms

• Require several consecutive failures before you call a target down. One slow response is not an outage.
• Cap restarts per target per hour. When the cap is reached, stop and leave the problem for a human.
• Do not restart your app when a dependency probe fails. Open a ticket instead.
• Log every restart with its time and exit code, so patterns show up later.

4. Write things down

Keep one incident per outage: open it when the target goes down, close it on recovery, and store the probe detail. Send a webhook (Slack, Discord, anything that accepts a JSON POST) on open and on close, not on every failed probe.

5. Watch for slow failures

Thresholds catch clean breaks. For drift, keep a rolling baseline per metric (moving average and variance) and raise a flag when several metrics move away from it at once, even if each one is still "green".

6. Back up, and check the backup exists

Back up config and data on a schedule, prune by a retention rule, check that each backup file is really where you expect it, and test a restore by hand from time to time.

How the packs implement this

The packs below are Python code with an install.sh for Debian/Ubuntu and an offline test suite, at $3 one-time each. Their core logic is deterministic and works without any LLM. If Claude Code, Kimi Code or Codex is logged in on the machine, or a local Ollama server is running, the pack uses it for the written parts (diagnosis, reports), with no API key.

SRE Agent Pack: probes, restarts and tickets

SRE Agent Pack covers steps 2 to 4. You list targets in targets.json with a type of http, tcp, systemd or command. Per target you can set:

• fail_threshold: consecutive failures before the target is down (default 3).
• restart_command: defaults to systemctl restart <unit> for systemd targets. Leave it out and the target is watch-only (ticket and alert, no restart).
• max_restarts_per_hour: the anti-flap budget (default 3). When it runs out, the agent logs budget_exhausted and keeps the incident open for a human.

Each outage gets one markdown ticket in tickets/, closed on recovery. Checks, streaks, incidents and restarts are stored in SQLite. If SRE_WEBHOOK_URL is set, it receives a JSON POST on incident_opened and incident_closed. With an LLM CLI connected, a Diagnostician prompt appends a root-cause diagnosis to each new ticket.

cp targets.example.json targets.json
sudo bash install.sh
python agent.py check
python agent.py status
python agent.py tickets --open
python agent.py report --hours 24

The installer also writes a systemd unit for the guardian itself (Restart=always, RestartSec=5), so the watcher is supervised too.

Agent On-Call Pack: what happens after the alert

Agent On-Call Pack handles alerts once they exist. An alert moves through triage, acknowledgment, runbook, escalation and resolution, and every step is recorded in SQLite (alerts, incidents, escalation_log). Runbooks live in templates/runbooks.yaml, and the agent only runs commands listed there (such as ping, curl, check_disk and check_service), never a command supplied inside an alert. Critical alerts and failed runbooks are escalated.

python3 agent.py alert --source monitoring --severity critical --message "disk full on db-01"
python3 agent.py run --once --deterministic
python3 agent.py handoff --shift night

Intuition Agent Pack: early warning on drift

Intuition Agent Pack covers step 5. You feed it metrics as JSONL, either with ingest or with run, which follows a growing JSONL file under systemd. Per metric it keeps an EWMA mean and variance, an hour-of-day baseline and a fast versus slow baseline for drift; it also tracks how rare each log signature is. It combines these deviations with Fisher's method into an unease score from 0 to 100, with levels calm, uneasy and alarmed. It is written in the Python standard library only.

python agent.py demo
python agent.py score
python agent.py signals
python agent.py calibrate --missed

You annotate raised signals with calibrate, and report then shows the precision and recall of its own signals.

Agent Backup Pack: queued backups with verification

Agent Backup Pack covers step 6 with handlers backup_file, backup_directory, prune_backups, verify_backup and list_backups, run from a SQLite queue with retry backoff (10 s, 60 s, 600 s) and a dead-letter queue you can replay. verify_backup tells apart a missing file (not_found), a refused listing (no_access) and a location it cannot check (unverified).

python agent.py enqueue --type backup_file --payload '{"source":"/etc/hosts","dest_dir":"/backups"}'
python agent.py enqueue --type prune_backups --payload '{"directory":"/backups","retention_days":30}'

Honest limits

• Same-VM blind spot. A watcher on the VM cannot alert you when the whole VM is down. Pair it with a simple external check from another machine.
• No built-in wiring between packs. To send SRE latency data into Intuition, you write a small export to JSONL yourself.
• Restarts fix crashes, not causes. A bad deploy, a full disk or a dead database will not be fixed by a restart; that is why the restart budget exists.
• Backups. The Backup pack does not provide off-site replication or encryption key management, and does not guarantee that an untested restore will work.
• Scale. The On-Call pack does not replace a large paging platform.
• Statistics. Intuition assumes the combined metrics are roughly independent, and its recall is only known if you log the incidents it missed.

Where to start

1. Add the systemd unit above, with start limits, to every service you run. It costs nothing.
2. If you want probes, tickets and a restart budget without writing them yourself, start with SRE Agent Pack and point it at one health URL.
3. Add Agent Backup Pack for your data, Agent On-Call Pack if alerts need runbooks and escalation, and Intuition Agent Pack once you collect metrics.
4. To see the pack format first, install the free Free Sample Agent using the quickstart, or browse all packs.