docs: implementation plan for master/agent phase 1 (inventory)

Adds MODE=compose|swarm. Each instance owns one Docker daemon and runs
one paradigm; see docs/superpowers/specs/2026-08-04-multi-server-fanout-design.md
for why this exclusivity is the intended design.

docs/superpowers/specs/2026-10-12-master-agent-design.md

@@ -0,0 +1,373 @@
+# Master/Agent Architecture — Design
+
+**Date:** 2026-10-12
+**Status:** Approved in brainstorming, pending written-spec review
+
+## Problem
+
+package-updater today is one instance per Docker daemon: a webhook (`POST /update`)
+finds opted-in containers by label and runs `docker compose` or `docker service update`.
+It is stateless and has no UI.
+
+The goal now is a tool that replaces Watchtower and Shepherd across the whole fleet:
+
+- One overview of every service on every machine — Swarm services and Compose stacks,
+  which can coexist on the same machine.
+- Per-service choice, made in the UI, between **manual** (show that an update exists,
+  a human clicks) and **auto** (apply as soon as a new version is detected).
+- Detect new versions both from CI (webhook) and by polling registries, so images we do
+  not build ourselves are covered too.
+- See available and completed updates, pick any of the recently seen versions
+  (which also gives rollback).
+- Live updates in the UI.
+
+Building this ourselves instead of adopting Komodo or Portainer is a deliberate choice:
+it is a hobby project, and owning it means it works exactly the way we want.
+
+## Environment and assumptions
+
+- Several machines on one internal network, reachable from each other by IP.
+- One machine (`swarm-mgr`) is a Swarm manager; most services are Swarm stacks
+  (`docker-stack.yml`). At least one Compose stack exists (`slusk`).
+- A central Postgres already runs and will be used.
+- Authentik runs and is used per application via OIDC (not via the reverse proxy).
+- A reverse proxy fronts the web UI.
+- Stack files are version-controlled. Image references are ad hoc: a mix of `repo:tag`,
+  `repo:tag@digest` and bare `repo@digest`. Many digests were written by a Claude instance
+  on the servers, not by an automated process.
+- Shepherd and Watchtower currently run and will be removed (see Migration).
+
+## Decisions
+
+### Topology: one master, one agent per machine
+
+- **Master** (one instance): Postgres, web UI, OIDC login, webhook, registry polling,
+  job decisions. Runs as a Swarm service on `swarm-mgr`.
+- **Agent** (one per machine): owns that machine's Docker daemon via `docker.sock`.
+  Reports inventory and runs update jobs. Runs via plain `docker compose` on each host
+  (not a global Swarm service), because each agent has its own token and a global
+  service would hand every node the same environment.
+  The agent mounts `docker.sock` and `STACKS_ROOT` **read-write** (today it is `:ro`),
+  because it writes `compose.override.yaml` files.
+
+### Connection: agent dials master over WebSocket, live
+
+The agent opens a long-lived WebSocket to the master and keeps it alive with reconnect
+and backoff. Inventory changes are pushed live from Docker events; the master forwards
+them to browsers. Chosen over the master polling agents over HTTP because live updates
+are an explicit goal.
+
+Messages are JSON in a common envelope `{type, id, payload}` with one Go type per
+message kind, shared in `internal/protocol`. JSON over WebSocket rather than gRPC:
+same tooling as the browser side, debuggable with `websocat`, no protobuf toolchain.
+
+### Trust in both directions
+
+Commands flow from master to agent and can restart anything on that machine, so the
+agent must authenticate the master, not only the other way round.
+
+- On first start the master generates a key pair and stores it in Postgres
+  (`master_identity`), so it survives container recreation and is covered by DB backups.
+- The master serves the agent endpoint on its own port with TLS, using a self-signed
+  certificate for that key. No reverse proxy, no CA.
+- The agent is configured with `MASTER_URL`, `MASTER_KEY` and `AGENT_TOKEN`.
+  `MASTER_KEY` is the SHA-256 fingerprint of the master's public key. The agent skips
+  normal chain verification and instead accepts the connection only if the
+  certificate's public key hashes to `MASTER_KEY` (same idea as SSH `known_hosts`).
+  The TLS handshake proves the master holds the private key.
+- `AGENT_TOKEN` is per agent, created in the UI ("Add agent", which shows a ready
+  compose snippet including token and `MASTER_KEY`). Only its hash is stored.
+  Agents can be revoked individually. A leaked token lets someone impersonate an agent
+  (feed false inventory), not command machines.
+- Losing the master key means every agent needs a new `MASTER_KEY`.
+
+Plain `ws://` with a signed handshake was rejected: anyone able to hijack the TCP
+stream after the handshake could inject commands.
+
+### State: the master remembers, agents report reality
+
+Agents are the truth for what is running. Postgres holds what the master must remember:
+desired version, update mode, version history and job history. This deliberately
+retires the "stateless" property for the master; agents stay stateless.
+
+### Security gate moves from labels to the UI
+
+The opt-in label no longer decides what may be updated; the per-service mode in the UI
+does. Consequences:
+
+- Every newly discovered service starts as `manual`. Nothing is ever `auto` by default.
+- Access to the UI is equivalent to being able to restart everything, so login is the
+  most important control (see Authentication).
+- The `STACKS_ROOT` path check stays on the agent as an independent guard for Compose,
+  and matters more now that the agent needs write access there (override files).
+- The master cannot reliably tell which service is its own Postgres or Authentik, so
+  they are not locked automatically. Putting them on `auto` is the operator's risk and
+  is documented.
+
+### Changing the version of a service
+
+| Kind | How the version is changed | What can revert it |
+|---|---|---|
+| Compose | Agent writes the service's image into `compose.override.yaml` next to the stack's compose file (the agent owns this file, preserving other services' entries in it), then `docker compose pull <svc>` + `up -d <svc>`. `docker compose` loads the override automatically, so manual `docker compose up -d` keeps the new version. | Deleting the override file. |
+| Swarm | `service update` via the Docker API with `repo:tag@digest` (the agent resolves the digest; without it Swarm does not re-pull when the tag is unchanged). | Re-running `docker stack deploy` with a file holding an older reference. |
+
+The updater always writes `repo:tag@digest`. The stack's own files in git are never
+edited. Writing version bumps back to git (GitOps style) was considered and deferred:
+it needs repo write access, YAML rewriting and conflict handling — a different kind of
+tool. It can be added later as a step after a successful update.
+
+### Drift
+
+A service drifts when `actual_ref` differs from `desired_ref`. A service the updater has
+never touched has no `desired_ref` and cannot drift. Drift is **never corrected
+automatically**, not even in `auto` mode, because a manual `stack deploy` with a new tag
+may be intentional. The UI shows the drift and offers "restore desired" or
+"accept running".
+
+### Service identity and moves
+
+A service row has its own ID. Location is an attribute with a unique index:
+Compose = agent + project + service; Swarm = cluster ID + service name (independent of
+which manager agent reports it); plain container = agent + container name.
+
+Identity without the agent was rejected: the same stack legitimately runs on several
+machines (the agent itself, node exporters), and they would be merged into one history.
+Automatic merging was rejected for the same reason.
+
+When a Compose service disappears on one agent and exactly one service with the same
+project and service name appears on another, the UI suggests "Moved from A?" with a
+**Merge** action: the old row is moved to the new location with its history, mode and
+`desired_ref`, and the duplicate is removed. Merging can also be done by hand.
+
+### Versions: what the system has seen
+
+Every version the system sees is recorded in `image_versions`, from three sources:
+webhook (every build CI announces), polling (every new digest under a followed tag) and
+inventory (whatever is running). The service page lists the 10 most recent versions for
+the service's repo; picking one creates a manual job to exactly that reference. Picking
+an older one is a rollback.
+
+`available_*` on the service remains as "the newest suggestion", which drives the
+"update available" badge and `auto` mode.
+
+| Reference format | Polling | Webhook | Pick a version |
+|---|---|---|---|
+| `repo:tag` | yes | yes | yes |
+| `repo:tag@digest` | yes (follows the tag) | yes | yes |
+| `repo@digest` | no — nothing to follow | yes (own images) | yes (seen versions) |
+
+Services without a tag are flagged "follows no tag" in the UI. Recommended convention
+for stack files: `repo:tag@digest`, tag = what to follow.
+
+## Data model (Postgres)
+
+**`master_identity`** — single row: private key, created_at.
+
+**`agents`** — `id`, `name` (set when adding), `token_hash`, `hostname`,
+`agent_version`, `is_swarm_manager` (reported), `last_seen_at`, `revoked_at`, `created_at`.
+Online/offline is held in memory only.
+
+**`services`**
+- `id`
+- `kind`: `compose` | `swarm` | `container` (plain containers are shown but cannot be
+  updated in v1)
+- location: `agent_id`, `compose_project`, `compose_service`, `swarm_cluster_id`,
+  `swarm_service`, `container_name` (populated per kind; unique index per kind)
+- `image_repo` — normalised, used to match webhooks and polling
+- `update_mode`: `manual` (default) | `auto`
+- `desired_ref`
+- `actual_ref`, `actual_digest`, `status` — last known, kept so the UI has something to
+  show while an agent is offline
+- `available_ref`, `available_digest`, `available_source` (`webhook` | `poll`),
+  `available_at`
+- `last_poll_error`
+- `first_seen_at`, `removed_at` — disappeared services are marked, not deleted
+
+**`image_versions`** — `repo`, `tag`, `digest`, `source` (`webhook` | `poll` | `running`),
+`first_seen_at`; unique on (repo, tag, digest).
+
+**`update_jobs`** — `id`, `service_id`, `from_ref`, `to_ref`,
+`trigger` (`manual` | `webhook` | `poll`), `requested_by` (OIDC subject for manual),
+`status` (`queued` | `running` | `succeeded` | `failed`), `output`, `error`,
+`created_at`, `started_at`, `finished_at`.
+
+**`sessions`** — web sessions for OIDC logins.
+
+## Flows
+
+### Inventory
+
+1. On connect the agent sends `hello` (hostname, version, Swarm manager or not, cluster
+   ID) and a full `snapshot`.
+2. The master diffs the snapshot against that agent's rows: new services are inserted as
+   `manual`, missing ones get `removed_at`, changed ones get new `actual_*`. Running
+   references are recorded in `image_versions` with source `running`.
+3. The agent subscribes to Docker events (container start/stop/die/destroy, service
+   update), debounces bursts (a Compose restart emits many events), re-inspects the
+   affected item and sends `upsert` or `remove`.
+4. Swarm services are reported only by agents on manager nodes. Agents skip containers
+   that belong to Swarm tasks, so a service is not listed once per node.
+5. The master pushes changes to browsers over SSE.
+
+### Update
+
+1. Triggers: a click in the UI, the webhook, or polling. Webhook and polling always set
+   `available_*` and record the version; they create a job only if the service is `auto`.
+2. The master inserts an `update_jobs` row and sends `job.start` to the owning agent
+   (for Swarm, a connected manager agent).
+3. The agent puts the job in its FIFO queue (single worker, as today) and executes it as
+   described under "Changing the version of a service".
+4. The agent sends `job.progress` and `job.result` with output. On success the master
+   sets `desired_ref` and clears `available_*` if that version is now running.
+
+### Failure handling
+
+| Situation | Behaviour |
+|---|---|
+| Agent offline when a job is created | Job fails immediately with "agent offline". It is not stored for later — a job that runs hours later surprises more than it helps. `available_*` remains, so the user can retry. |
+| Connection drops mid-job | The agent finishes the job (its queue is local) and keeps results in memory until the master acknowledges them. They are resent after reconnect along with the snapshot. |
+| Master restarts mid-job | Jobs in `running` are reconciled when the agent reconnects; jobs the agent does not know are marked `failed` ("lost"). |
+| New trigger while a job for the same service is queued or running | No new job; only `available_*` is updated. Two jobs never race on one service. |
+| Job exceeds `UPDATE_TIMEOUT` | The agent cancels it, as today. |
+| Polling fails for one image | Error stored in `last_poll_error` and shown; other images continue. |
+
+### Self-protection
+
+The agent reports its own container and the master knows its own service. Both are shown
+as "updated manually outside the system", and the master refuses to create jobs for
+them. An agent recreating itself would kill the job it is running.
+
+## Registry polling
+
+- The master polls, not the agents: credentials live in one place and identical
+  references are polled once.
+- Every distinct `repo:tag` in use is polled, including `repo:tag@digest`. Bare
+  `repo@digest` and locally built images without a registry digest are skipped and
+  shown as such.
+- Mechanism: `HEAD /v2/<repo>/manifests/<tag>`; the returned digest is compared with
+  `actual_digest`. Use `go-containerregistry` — it handles Docker Hub's token auth and
+  multi-arch indexes.
+- Interval `POLL_INTERVAL`, default 6h. Believed not to count against Docker Hub pull
+  limits for HEAD requests — verify during implementation.
+- Credentials: a Docker `config.json` (as written by `docker login`) mounted into the
+  master. No credential UI, no encryption in Postgres.
+
+## Webhook
+
+- `POST /update` with `{image, tag}` and a bearer token — the same contract as today, so
+  the existing Gitea action works unchanged; only its `endpoint` moves to the master.
+- `image` is matched against `image_repo` across all agents using today's normalisation
+  (`NormaliseImage`). Each match gets `available_ref = repo:tag` (source `webhook`) and
+  the version is recorded; `auto` services get a job.
+- The response waits for the `auto` jobs to finish (bounded by `UPDATE_TIMEOUT`), and
+  returns 5xx if any failed or timed out, so CI still surfaces a broken deploy. `manual` matches are reported as "marked", not as
+  errors.
+- No match returns **404**, which catches typos in CI configuration.
+- The webhook never creates services. The first deployment of a stack is done by hand.
+
+## Authentication and UI
+
+### Authentication
+
+Three separate mechanisms that never mix:
+
+| Who | How | Where |
+|---|---|---|
+| People | OIDC authorization code flow with PKCE against Authentik (`coreos/go-oidc`, `x/oauth2`) | Web UI, via reverse proxy |
+| CI | Shared bearer token | `POST /update`, via reverse proxy |
+| Agents | Per-agent token + pinned master key over TLS | Dedicated agent port, internal network |
+
+- Sessions are stored in Postgres; the cookie holds a session ID (`HttpOnly`, `Secure`,
+  `SameSite=Lax`).
+- The master also requires membership of `ALLOWED_GROUP` from the token's `groups`
+  claim. Restricting the application in Authentik alone is not enough: one misconfigured
+  binding would hand every Authentik user the ability to restart the fleet.
+- All state-changing UI requests carry a CSRF token.
+- `/healthz`, `/metrics` and `/version` stay unauthenticated, as today.
+
+### UI
+
+Server-rendered Go templates with htmx; live updates via htmx's SSE support. Embedded in
+the master binary with `go:embed`. No Node build step. Visual design is drafted
+separately; this spec defines what each page contains.
+
+1. **Overview** — all services grouped per machine (Swarm services per cluster). Each
+   shows status, running version, the manual/auto toggle, and badges for
+   *update available*, *drifted* and *follows no tag*. Live.
+2. **Service** — the 10 most recent versions with an *Update* action, job history with
+   logs, drift actions (restore / accept), and *Merge* for moves.
+3. **Agents** — online/offline, version. *Add agent* produces a compose snippet with
+   token and `MASTER_KEY`. *Revoke*.
+4. **Jobs** — full history; running jobs stream their log live.
+
+## Code layout
+
+- `cmd/master`, `cmd/agent` — replace `cmd/server`.
+- `internal/protocol` — message types shared by master and agent.
+- `internal/agent/...` — today's `discovery` and `updater` (queue, Compose and Swarm
+  executors) move here, plus inventory, Docker events and the connection loop.
+- `internal/master/...` — agent hub, jobs, polling, webhook, OIDC, storage.
+- `web/` — templates and static assets.
+
+Existing conventions continue: interfaces defined at the consumer, hand-written fakes,
+nil-able `*Metrics`, Conventional Commits.
+
+## Testing
+
+- `go test ./...` must keep passing without Docker or Postgres.
+- Master logic (mode decisions, drift, merge, webhook matching, job reconciliation) is
+  tested against a storage interface with a hand-written fake.
+- Postgres storage is tested by integration tests behind the `integration` build tag,
+  requiring `DATABASE_URL`.
+- Master and agent are tested together in-process: an `httptest` TLS server, a real
+  WebSocket client and a fake Docker. Covers the `MASTER_KEY` handshake (including a
+  wrong key being rejected), reconnect with snapshot, and jobs completed while
+  disconnected.
+- Table-driven tests with testify.
+
+## Migration from today's deployment
+
+1. Create the database and user in Postgres; register the application and group in
+   Authentik.
+2. Deploy the master.
+3. **Remove Shepherd and Watchtower.** Running alongside, they would update services
+   behind the master's back and every such update would show up as drift.
+4. Add one agent per machine through the UI. Inventory fills in; everything is `manual`.
+5. Point the Gitea action's `endpoint` at the master. Contract unchanged.
+6. Remove today's package-updater instance.
+7. Convert bare `repo@digest` references to `repo:tag@digest`, and set `auto` where
+   wanted. Recommended: record the convention in the self-hosted repo's `CLAUDE.md`
+   so Claude instances on the servers stop writing bare digests.
+
+`CLAUDE.md` in this repo is updated as part of the implementation: the "stateless" and
+"token AND opt-in label AND STACKS_ROOT" design-intent entries no longer describe the
+system, and the `MODE` entry is replaced by the agent model.
+
+## Implementation phases
+
+Each phase ends with something that works on its own.
+
+1. **Protocol, agent and master with inventory.** Pinned-key TLS, per-agent tokens,
+   snapshot and live events, Postgres storage, OIDC login, Overview and Agents pages.
+   The overview works; nothing can be updated yet.
+2. **Jobs and manual update.** Job protocol, agent queue and executors (Compose override
+   file, Swarm with digest), job history, Service and Jobs pages, self-protection.
+3. **Webhook and auto mode.** `POST /update` on the master, matching, waiting for
+   auto jobs, mode toggle.
+4. **Polling and version history.** `image_versions`, registry polling, version picker
+   and rollback.
+5. **Drift and merge.** Drift detection and actions, move suggestion and merge.
+
+## Out of scope (later specs)
+
+- Detecting newer version *tags* for images we do not build (e.g. `gitea:1.27.3` →
+  `1.27.4`, `postgres:17.10`, `loki:3.4.2`). Requires listing tags and ordering them;
+  major upgrades such as Postgres 17 → 18 must never be automatic. Revisit after
+  checking the other two servers' stack files.
+- Notifications.
+- Export/import of a fleet configuration to restore a setup. The `services` table
+  already holds what an export needs.
+- Writing version bumps back to git.
+- Self-update of master and agents (`internal/selfupdate` is the starting point).
+- Updating plain (non-Compose, non-Swarm) containers.
+- Credential management UI for registries.