diff --git a/docs/superpowers/specs/2026-10-12-master-agent-design.md b/docs/superpowers/specs/2026-10-12-master-agent-design.md new file mode 100644 index 0000000..ce50afc --- /dev/null +++ b/docs/superpowers/specs/2026-10-12-master-agent-design.md @@ -0,0 +1,373 @@ +# Master/Agent Architecture — Design + +**Date:** 2026-10-12 +**Status:** Approved in brainstorming, pending written-spec review + +## Problem + +package-updater today is one instance per Docker daemon: a webhook (`POST /update`) +finds opted-in containers by label and runs `docker compose` or `docker service update`. +It is stateless and has no UI. + +The goal now is a tool that replaces Watchtower and Shepherd across the whole fleet: + +- One overview of every service on every machine — Swarm services and Compose stacks, + which can coexist on the same machine. +- Per-service choice, made in the UI, between **manual** (show that an update exists, + a human clicks) and **auto** (apply as soon as a new version is detected). +- Detect new versions both from CI (webhook) and by polling registries, so images we do + not build ourselves are covered too. +- See available and completed updates, pick any of the recently seen versions + (which also gives rollback). +- Live updates in the UI. + +Building this ourselves instead of adopting Komodo or Portainer is a deliberate choice: +it is a hobby project, and owning it means it works exactly the way we want. + +## Environment and assumptions + +- Several machines on one internal network, reachable from each other by IP. +- One machine (`swarm-mgr`) is a Swarm manager; most services are Swarm stacks + (`docker-stack.yml`). At least one Compose stack exists (`slusk`). +- A central Postgres already runs and will be used. +- Authentik runs and is used per application via OIDC (not via the reverse proxy). +- A reverse proxy fronts the web UI. +- Stack files are version-controlled. Image references are ad hoc: a mix of `repo:tag`, + `repo:tag@digest` and bare `repo@digest`. Many digests were written by a Claude instance + on the servers, not by an automated process. +- Shepherd and Watchtower currently run and will be removed (see Migration). + +## Decisions + +### Topology: one master, one agent per machine + +- **Master** (one instance): Postgres, web UI, OIDC login, webhook, registry polling, + job decisions. Runs as a Swarm service on `swarm-mgr`. +- **Agent** (one per machine): owns that machine's Docker daemon via `docker.sock`. + Reports inventory and runs update jobs. Runs via plain `docker compose` on each host + (not a global Swarm service), because each agent has its own token and a global + service would hand every node the same environment. + The agent mounts `docker.sock` and `STACKS_ROOT` **read-write** (today it is `:ro`), + because it writes `compose.override.yaml` files. + +### Connection: agent dials master over WebSocket, live + +The agent opens a long-lived WebSocket to the master and keeps it alive with reconnect +and backoff. Inventory changes are pushed live from Docker events; the master forwards +them to browsers. Chosen over the master polling agents over HTTP because live updates +are an explicit goal. + +Messages are JSON in a common envelope `{type, id, payload}` with one Go type per +message kind, shared in `internal/protocol`. JSON over WebSocket rather than gRPC: +same tooling as the browser side, debuggable with `websocat`, no protobuf toolchain. + +### Trust in both directions + +Commands flow from master to agent and can restart anything on that machine, so the +agent must authenticate the master, not only the other way round. + +- On first start the master generates a key pair and stores it in Postgres + (`master_identity`), so it survives container recreation and is covered by DB backups. +- The master serves the agent endpoint on its own port with TLS, using a self-signed + certificate for that key. No reverse proxy, no CA. +- The agent is configured with `MASTER_URL`, `MASTER_KEY` and `AGENT_TOKEN`. + `MASTER_KEY` is the SHA-256 fingerprint of the master's public key. The agent skips + normal chain verification and instead accepts the connection only if the + certificate's public key hashes to `MASTER_KEY` (same idea as SSH `known_hosts`). + The TLS handshake proves the master holds the private key. +- `AGENT_TOKEN` is per agent, created in the UI ("Add agent", which shows a ready + compose snippet including token and `MASTER_KEY`). Only its hash is stored. + Agents can be revoked individually. A leaked token lets someone impersonate an agent + (feed false inventory), not command machines. +- Losing the master key means every agent needs a new `MASTER_KEY`. + +Plain `ws://` with a signed handshake was rejected: anyone able to hijack the TCP +stream after the handshake could inject commands. + +### State: the master remembers, agents report reality + +Agents are the truth for what is running. Postgres holds what the master must remember: +desired version, update mode, version history and job history. This deliberately +retires the "stateless" property for the master; agents stay stateless. + +### Security gate moves from labels to the UI + +The opt-in label no longer decides what may be updated; the per-service mode in the UI +does. Consequences: + +- Every newly discovered service starts as `manual`. Nothing is ever `auto` by default. +- Access to the UI is equivalent to being able to restart everything, so login is the + most important control (see Authentication). +- The `STACKS_ROOT` path check stays on the agent as an independent guard for Compose, + and matters more now that the agent needs write access there (override files). +- The master cannot reliably tell which service is its own Postgres or Authentik, so + they are not locked automatically. Putting them on `auto` is the operator's risk and + is documented. + +### Changing the version of a service + +| Kind | How the version is changed | What can revert it | +|---|---|---| +| Compose | Agent writes the service's image into `compose.override.yaml` next to the stack's compose file (the agent owns this file, preserving other services' entries in it), then `docker compose pull ` + `up -d `. `docker compose` loads the override automatically, so manual `docker compose up -d` keeps the new version. | Deleting the override file. | +| Swarm | `service update` via the Docker API with `repo:tag@digest` (the agent resolves the digest; without it Swarm does not re-pull when the tag is unchanged). | Re-running `docker stack deploy` with a file holding an older reference. | + +The updater always writes `repo:tag@digest`. The stack's own files in git are never +edited. Writing version bumps back to git (GitOps style) was considered and deferred: +it needs repo write access, YAML rewriting and conflict handling — a different kind of +tool. It can be added later as a step after a successful update. + +### Drift + +A service drifts when `actual_ref` differs from `desired_ref`. A service the updater has +never touched has no `desired_ref` and cannot drift. Drift is **never corrected +automatically**, not even in `auto` mode, because a manual `stack deploy` with a new tag +may be intentional. The UI shows the drift and offers "restore desired" or +"accept running". + +### Service identity and moves + +A service row has its own ID. Location is an attribute with a unique index: +Compose = agent + project + service; Swarm = cluster ID + service name (independent of +which manager agent reports it); plain container = agent + container name. + +Identity without the agent was rejected: the same stack legitimately runs on several +machines (the agent itself, node exporters), and they would be merged into one history. +Automatic merging was rejected for the same reason. + +When a Compose service disappears on one agent and exactly one service with the same +project and service name appears on another, the UI suggests "Moved from A?" with a +**Merge** action: the old row is moved to the new location with its history, mode and +`desired_ref`, and the duplicate is removed. Merging can also be done by hand. + +### Versions: what the system has seen + +Every version the system sees is recorded in `image_versions`, from three sources: +webhook (every build CI announces), polling (every new digest under a followed tag) and +inventory (whatever is running). The service page lists the 10 most recent versions for +the service's repo; picking one creates a manual job to exactly that reference. Picking +an older one is a rollback. + +`available_*` on the service remains as "the newest suggestion", which drives the +"update available" badge and `auto` mode. + +| Reference format | Polling | Webhook | Pick a version | +|---|---|---|---| +| `repo:tag` | yes | yes | yes | +| `repo:tag@digest` | yes (follows the tag) | yes | yes | +| `repo@digest` | no — nothing to follow | yes (own images) | yes (seen versions) | + +Services without a tag are flagged "follows no tag" in the UI. Recommended convention +for stack files: `repo:tag@digest`, tag = what to follow. + +## Data model (Postgres) + +**`master_identity`** — single row: private key, created_at. + +**`agents`** — `id`, `name` (set when adding), `token_hash`, `hostname`, +`agent_version`, `is_swarm_manager` (reported), `last_seen_at`, `revoked_at`, `created_at`. +Online/offline is held in memory only. + +**`services`** +- `id` +- `kind`: `compose` | `swarm` | `container` (plain containers are shown but cannot be + updated in v1) +- location: `agent_id`, `compose_project`, `compose_service`, `swarm_cluster_id`, + `swarm_service`, `container_name` (populated per kind; unique index per kind) +- `image_repo` — normalised, used to match webhooks and polling +- `update_mode`: `manual` (default) | `auto` +- `desired_ref` +- `actual_ref`, `actual_digest`, `status` — last known, kept so the UI has something to + show while an agent is offline +- `available_ref`, `available_digest`, `available_source` (`webhook` | `poll`), + `available_at` +- `last_poll_error` +- `first_seen_at`, `removed_at` — disappeared services are marked, not deleted + +**`image_versions`** — `repo`, `tag`, `digest`, `source` (`webhook` | `poll` | `running`), +`first_seen_at`; unique on (repo, tag, digest). + +**`update_jobs`** — `id`, `service_id`, `from_ref`, `to_ref`, +`trigger` (`manual` | `webhook` | `poll`), `requested_by` (OIDC subject for manual), +`status` (`queued` | `running` | `succeeded` | `failed`), `output`, `error`, +`created_at`, `started_at`, `finished_at`. + +**`sessions`** — web sessions for OIDC logins. + +## Flows + +### Inventory + +1. On connect the agent sends `hello` (hostname, version, Swarm manager or not, cluster + ID) and a full `snapshot`. +2. The master diffs the snapshot against that agent's rows: new services are inserted as + `manual`, missing ones get `removed_at`, changed ones get new `actual_*`. Running + references are recorded in `image_versions` with source `running`. +3. The agent subscribes to Docker events (container start/stop/die/destroy, service + update), debounces bursts (a Compose restart emits many events), re-inspects the + affected item and sends `upsert` or `remove`. +4. Swarm services are reported only by agents on manager nodes. Agents skip containers + that belong to Swarm tasks, so a service is not listed once per node. +5. The master pushes changes to browsers over SSE. + +### Update + +1. Triggers: a click in the UI, the webhook, or polling. Webhook and polling always set + `available_*` and record the version; they create a job only if the service is `auto`. +2. The master inserts an `update_jobs` row and sends `job.start` to the owning agent + (for Swarm, a connected manager agent). +3. The agent puts the job in its FIFO queue (single worker, as today) and executes it as + described under "Changing the version of a service". +4. The agent sends `job.progress` and `job.result` with output. On success the master + sets `desired_ref` and clears `available_*` if that version is now running. + +### Failure handling + +| Situation | Behaviour | +|---|---| +| Agent offline when a job is created | Job fails immediately with "agent offline". It is not stored for later — a job that runs hours later surprises more than it helps. `available_*` remains, so the user can retry. | +| Connection drops mid-job | The agent finishes the job (its queue is local) and keeps results in memory until the master acknowledges them. They are resent after reconnect along with the snapshot. | +| Master restarts mid-job | Jobs in `running` are reconciled when the agent reconnects; jobs the agent does not know are marked `failed` ("lost"). | +| New trigger while a job for the same service is queued or running | No new job; only `available_*` is updated. Two jobs never race on one service. | +| Job exceeds `UPDATE_TIMEOUT` | The agent cancels it, as today. | +| Polling fails for one image | Error stored in `last_poll_error` and shown; other images continue. | + +### Self-protection + +The agent reports its own container and the master knows its own service. Both are shown +as "updated manually outside the system", and the master refuses to create jobs for +them. An agent recreating itself would kill the job it is running. + +## Registry polling + +- The master polls, not the agents: credentials live in one place and identical + references are polled once. +- Every distinct `repo:tag` in use is polled, including `repo:tag@digest`. Bare + `repo@digest` and locally built images without a registry digest are skipped and + shown as such. +- Mechanism: `HEAD /v2//manifests/`; the returned digest is compared with + `actual_digest`. Use `go-containerregistry` — it handles Docker Hub's token auth and + multi-arch indexes. +- Interval `POLL_INTERVAL`, default 6h. Believed not to count against Docker Hub pull + limits for HEAD requests — verify during implementation. +- Credentials: a Docker `config.json` (as written by `docker login`) mounted into the + master. No credential UI, no encryption in Postgres. + +## Webhook + +- `POST /update` with `{image, tag}` and a bearer token — the same contract as today, so + the existing Gitea action works unchanged; only its `endpoint` moves to the master. +- `image` is matched against `image_repo` across all agents using today's normalisation + (`NormaliseImage`). Each match gets `available_ref = repo:tag` (source `webhook`) and + the version is recorded; `auto` services get a job. +- The response waits for the `auto` jobs to finish (bounded by `UPDATE_TIMEOUT`), and + returns 5xx if any failed or timed out, so CI still surfaces a broken deploy. `manual` matches are reported as "marked", not as + errors. +- No match returns **404**, which catches typos in CI configuration. +- The webhook never creates services. The first deployment of a stack is done by hand. + +## Authentication and UI + +### Authentication + +Three separate mechanisms that never mix: + +| Who | How | Where | +|---|---|---| +| People | OIDC authorization code flow with PKCE against Authentik (`coreos/go-oidc`, `x/oauth2`) | Web UI, via reverse proxy | +| CI | Shared bearer token | `POST /update`, via reverse proxy | +| Agents | Per-agent token + pinned master key over TLS | Dedicated agent port, internal network | + +- Sessions are stored in Postgres; the cookie holds a session ID (`HttpOnly`, `Secure`, + `SameSite=Lax`). +- The master also requires membership of `ALLOWED_GROUP` from the token's `groups` + claim. Restricting the application in Authentik alone is not enough: one misconfigured + binding would hand every Authentik user the ability to restart the fleet. +- All state-changing UI requests carry a CSRF token. +- `/healthz`, `/metrics` and `/version` stay unauthenticated, as today. + +### UI + +Server-rendered Go templates with htmx; live updates via htmx's SSE support. Embedded in +the master binary with `go:embed`. No Node build step. Visual design is drafted +separately; this spec defines what each page contains. + +1. **Overview** — all services grouped per machine (Swarm services per cluster). Each + shows status, running version, the manual/auto toggle, and badges for + *update available*, *drifted* and *follows no tag*. Live. +2. **Service** — the 10 most recent versions with an *Update* action, job history with + logs, drift actions (restore / accept), and *Merge* for moves. +3. **Agents** — online/offline, version. *Add agent* produces a compose snippet with + token and `MASTER_KEY`. *Revoke*. +4. **Jobs** — full history; running jobs stream their log live. + +## Code layout + +- `cmd/master`, `cmd/agent` — replace `cmd/server`. +- `internal/protocol` — message types shared by master and agent. +- `internal/agent/...` — today's `discovery` and `updater` (queue, Compose and Swarm + executors) move here, plus inventory, Docker events and the connection loop. +- `internal/master/...` — agent hub, jobs, polling, webhook, OIDC, storage. +- `web/` — templates and static assets. + +Existing conventions continue: interfaces defined at the consumer, hand-written fakes, +nil-able `*Metrics`, Conventional Commits. + +## Testing + +- `go test ./...` must keep passing without Docker or Postgres. +- Master logic (mode decisions, drift, merge, webhook matching, job reconciliation) is + tested against a storage interface with a hand-written fake. +- Postgres storage is tested by integration tests behind the `integration` build tag, + requiring `DATABASE_URL`. +- Master and agent are tested together in-process: an `httptest` TLS server, a real + WebSocket client and a fake Docker. Covers the `MASTER_KEY` handshake (including a + wrong key being rejected), reconnect with snapshot, and jobs completed while + disconnected. +- Table-driven tests with testify. + +## Migration from today's deployment + +1. Create the database and user in Postgres; register the application and group in + Authentik. +2. Deploy the master. +3. **Remove Shepherd and Watchtower.** Running alongside, they would update services + behind the master's back and every such update would show up as drift. +4. Add one agent per machine through the UI. Inventory fills in; everything is `manual`. +5. Point the Gitea action's `endpoint` at the master. Contract unchanged. +6. Remove today's package-updater instance. +7. Convert bare `repo@digest` references to `repo:tag@digest`, and set `auto` where + wanted. Recommended: record the convention in the self-hosted repo's `CLAUDE.md` + so Claude instances on the servers stop writing bare digests. + +`CLAUDE.md` in this repo is updated as part of the implementation: the "stateless" and +"token AND opt-in label AND STACKS_ROOT" design-intent entries no longer describe the +system, and the `MODE` entry is replaced by the agent model. + +## Implementation phases + +Each phase ends with something that works on its own. + +1. **Protocol, agent and master with inventory.** Pinned-key TLS, per-agent tokens, + snapshot and live events, Postgres storage, OIDC login, Overview and Agents pages. + The overview works; nothing can be updated yet. +2. **Jobs and manual update.** Job protocol, agent queue and executors (Compose override + file, Swarm with digest), job history, Service and Jobs pages, self-protection. +3. **Webhook and auto mode.** `POST /update` on the master, matching, waiting for + auto jobs, mode toggle. +4. **Polling and version history.** `image_versions`, registry polling, version picker + and rollback. +5. **Drift and merge.** Drift detection and actions, move suggestion and merge. + +## Out of scope (later specs) + +- Detecting newer version *tags* for images we do not build (e.g. `gitea:1.27.3` → + `1.27.4`, `postgres:17.10`, `loki:3.4.2`). Requires listing tags and ordering them; + major upgrades such as Postgres 17 → 18 must never be automatic. Revisit after + checking the other two servers' stack files. +- Notifications. +- Export/import of a fleet configuration to restore a setup. The `services` table + already holds what an export needs. +- Writing version bumps back to git. +- Self-update of master and agents (`internal/selfupdate` is the starting point). +- Updating plain (non-Compose, non-Swarm) containers. +- Credential management UI for registries.