Self-hosted task queue worker with retries and cron

Self-hosted agent pack for orchestrating an autonomous agent specialized in worker. Version 2.0.2 — actionable kit with scripts, templates, named prompts and an extended checklist.

Overview

This pack provides an operational autonomous agent for the worker domain. It includes a dedicated OODA loop, versioned prompts, configuration templates and a set of installation, test and execution scripts.

The core of the product is a durable SQLite queue. Each task carries a type, a priority, a JSON payload, a maximum number of attempts, dependencies and an optional idempotency key. The worker never silently loses a task: a success is archived, a transient failure is rescheduled and a terminal failure goes to the dead-letter queue for inspection or replay.

This pack targets operators who run repeatable jobs such as report generation, file validation, internal synchronizations or batch processing. It does not replace a large-scale distributed scheduler and does not execute any arbitrary command supplied in the payload.

The pack is designed to be self-hosted (BYO-LLM): no API key is included. The LLM adapter detects a local or CLI provider (ollama, claude-code, kimi, codex).

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         Signal / External input                     │
└──────────────────────────────────┬──────────────────────────────────┘
                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│  IDLE → PICKING → EXECUTING → RETRYING → DEAD_LETTER → REPORTING  │
│                           OODA State Machine                         │
└──────────────────────────────────┬──────────────────────────────────┘
                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│  Named prompts  → Deterministic handler →  BYO-LLM Adapter          │
│  scripts/test_prompt.py    scripts/run_agent.py                     │
└──────────────────────────────────┬──────────────────────────────────┘
                                   ▼
┌─────────────────────────────────────────────────────────────────────┐
│  SQLite queue  →  templates/*.yaml  →  Action / Output / Archive    │
└─────────────────────────────────────────────────────────────────────┘

Pack contents

Quick installation

cd /opt/autonomous-stack/agent-worker-v2
bash scripts/install.sh

The installer checks for Python 3.10+, creates a venv, installs the dependencies, detects a local LLM provider and configures a systemd unit.

Immediate test

# Test the main prompt
python3 scripts/test_prompt.py --prompt task_dispatcher --input templates/worker_queue.yaml

# Run the agent in dry-run mode
python3 scripts/run_agent.py --config scripts/config.yaml --dry-run

Execution contract

• pending: task recorded and available at its scheduled date.
• running: task reserved by the current cycle.
• completed: JSON result stored successfully.
• blocked: a dependency is missing or has failed permanently.
• dead: all attempts are used up; manual replay possible.

The idempotency key prevents double processing when the producer retries. Dependencies are checked before reservation. The priorities HIGH, NORMAL and LOW are stable, then tasks are ordered by creation date.

Common operations

python3 agent.py enqueue --type send_report --payload '{"report_id":"weekly"}'
python3 agent.py run --once --deterministic
python3 agent.py status
python3 agent.py replay --task t-example

Deterministic mode covers the included handlers without an LLM. BYO-LLM mode is reserved for tasks that require textual interpretation; its output must remain a JSON object and any error follows the same retry policy.

The multi_task_dispatch handler dispatches batch tasks only to the to field of each task. Unverified operator notes, additional recipients, webhooks and external references are never evidence of rejection or escalation.

Customization

1. Copy scripts/config.yaml to config.local.yaml.
2. Adapt the templates in templates/.
3. Plug your own signal/task/data sources into run_agent.py.

Costs and limits

This pack favors deterministic logic for critical decisions. The LLM is used for ranking, summarization and report generation. See SPEC.md for the operating cost estimate.

Use a local model (Ollama)

The optional LLM step can also run on your own machine through Ollama, so no text leaves your server. No account and no API key.

ollama pull <model>               # any chat model you like
export PACK_LLM=ollama            # prefer Ollama even if a coding-agent CLI is installed
export PACK_OLLAMA_MODEL=<model>  # optional: default is the first chat model the server lists
python3 -c "import llm_adapter; print(llm_adapter.detect())"   # prints: ollama

• PACK_OLLAMA_HOST points to another server (default http://127.0.0.1:11434); OLLAMA_HOST is also read when PACK_LLM=ollama.
• PACK_OLLAMA_NUM_CTX sets the context size if your reports or prompts are long.
• Without PACK_LLM, a local Ollama server is used only when no Claude Code, Kimi Code or Codex CLI is found.
• If the server is down or the model is missing, the pack keeps working in its deterministic mode.

Learning memory (local and private)

Every LLM call this pack makes is journaled in ./.pack_memory/ next to the code, so the pack can improve for you. Nothing is ever sent anywhere.

• What is kept: the task name, the masked input, the masked answer, status and latency. Keys, tokens, e-mails, phone numbers, IP addresses, card and IBAN numbers and personal paths are masked before writing; the system prompt is never stored.
• Teach it: python3 pack_memory.py feedback last ok confirms a result, python3 pack_memory.py feedback last bad --fix "the right answer" corrects it. Confirmed and corrected runs are reused as validated examples when a similar input comes in. Unrated runs are never injected into a prompt.
• Find similar cases: python3 pack_memory.py similar "some text" and python3 pack_memory.py clusters (groups of near-identical failures) work in plain Python, no model needed. Optional meaning-based matching uses an embedding model you already run locally: python3 pack_memory.py enable-semantic (Ollama on this machine only).
• Share, only if you want: python3 pack_memory.py export writes an anonymised summary (counts, statuses, size buckets, no text) for you to review.
• Control: PACK_MEMORY=0 turns it off, PACK_MEMORY_STORE_TEXT=0 keeps fingerprints only, python3 pack_memory.py purge --yes wipes it.

License

Personal use per purchase. Resale prohibited.