package-updater/docs/superpowers/specs/2026-10-12-master-agent-design.md

# Master/Agent Architecture — Design

**Date:** 2026-10-12
**Status:** Approved in brainstorming, pending written-spec review

## Problem

package-updater today is one instance per Docker daemon: a webhook (`POST /update`)
finds opted-in containers by label and runs `docker compose` or `docker service update`.
It is stateless and has no UI.

The goal now is a tool that replaces Watchtower and Shepherd across the whole fleet:

- One overview of every service on every machine — Swarm services and Compose stacks,
  which can coexist on the same machine.
- Per-service choice, made in the UI, between **manual** (show that an update exists,
  a human clicks) and **auto** (apply as soon as a new version is detected).
- Detect new versions both from CI (webhook) and by polling registries, so images we do
  not build ourselves are covered too.
- See available and completed updates, pick any of the recently seen versions
  (which also gives rollback).
- Live updates in the UI.

Building this ourselves instead of adopting Komodo or Portainer is a deliberate choice:
it is a hobby project, and owning it means it works exactly the way we want.

## Environment and assumptions

- Several machines on one internal network, reachable from each other by IP.
- One machine (`swarm-mgr`) is a Swarm manager; most services are Swarm stacks
  (`docker-stack.yml`). At least one Compose stack exists (`slusk`).
- A central Postgres already runs and will be used.
- Authentik runs and is used per application via OIDC (not via the reverse proxy).
- A reverse proxy fronts the web UI.
- Stack files are version-controlled. Image references are ad hoc: a mix of `repo:tag`,
  `repo:tag@digest` and bare `repo@digest`. Many digests were written by a Claude instance
  on the servers, not by an automated process.
- Shepherd and Watchtower currently run and will be removed (see Migration).

## Decisions

### Topology: one master, one agent per machine

- **Master** (one instance): Postgres, web UI, OIDC login, webhook, registry polling,
  job decisions. Runs as a Swarm service on `swarm-mgr`.
- **Agent** (one per machine): owns that machine's Docker daemon via `docker.sock`.
  Reports inventory and runs update jobs. Runs via plain `docker compose` on each host
  (not a global Swarm service), because each agent has its own token and a global
  service would hand every node the same environment.
  The agent mounts `docker.sock` and `STACKS_ROOT` **read-write** (today it is `:ro`),
  because it writes `compose.override.yaml` files.

### Connection: agent dials master over WebSocket, live

The agent opens a long-lived WebSocket to the master and keeps it alive with reconnect
and backoff. Inventory changes are pushed live from Docker events; the master forwards
them to browsers. Chosen over the master polling agents over HTTP because live updates
are an explicit goal.

Messages are JSON in a common envelope `{type, id, payload}` with one Go type per
message kind, shared in `internal/protocol`. JSON over WebSocket rather than gRPC:
same tooling as the browser side, debuggable with `websocat`, no protobuf toolchain.

### Trust in both directions

Commands flow from master to agent and can restart anything on that machine, so the
agent must authenticate the master, not only the other way round.

- On first start the master generates a key pair and stores it in Postgres
  (`master_identity`), so it survives container recreation and is covered by DB backups.
- The master serves the agent endpoint on its own port with TLS, using a self-signed
  certificate for that key. No reverse proxy, no CA.
- The agent is configured with `MASTER_URL`, `MASTER_KEY` and `AGENT_TOKEN`.
  `MASTER_KEY` is the SHA-256 fingerprint of the master's public key. The agent skips
  normal chain verification and instead accepts the connection only if the
  certificate's public key hashes to `MASTER_KEY` (same idea as SSH `known_hosts`).
  The TLS handshake proves the master holds the private key.
- `AGENT_TOKEN` is per agent, created in the UI ("Add agent", which shows a ready
  compose snippet including token and `MASTER_KEY`). Only its hash is stored.
  Agents can be revoked individually. A leaked token lets someone impersonate an agent
  (feed false inventory), not command machines.
- Losing the master key means every agent needs a new `MASTER_KEY`.

Plain `ws://` with a signed handshake was rejected: anyone able to hijack the TCP
stream after the handshake could inject commands.

### State: the master remembers, agents report reality

Agents are the truth for what is running. Postgres holds what the master must remember:
desired version, update mode, version history and job history. This deliberately
retires the "stateless" property for the master; agents stay stateless.

### Security gate moves from labels to the UI

The opt-in label no longer decides what may be updated; the per-service mode in the UI
does. Consequences:

- Every newly discovered service starts as `manual`. Nothing is ever `auto` by default.
- Access to the UI is equivalent to being able to restart everything, so login is the
  most important control (see Authentication).
- The `STACKS_ROOT` path check stays on the agent as an independent guard for Compose,
  and matters more now that the agent needs write access there (override files).
- The master cannot reliably tell which service is its own Postgres or Authentik, so
  they are not locked automatically. Putting them on `auto` is the operator's risk and
  is documented.

### Changing the version of a service

| Kind | How the version is changed | What can revert it |
|---|---|---|
| Compose | Agent writes the service's image into `compose.override.yaml` next to the stack's compose file (the agent owns this file, preserving other services' entries in it), then `docker compose pull <svc>` + `up -d <svc>`. `docker compose` loads the override automatically, so manual `docker compose up -d` keeps the new version. | Deleting the override file. |
| Swarm | `service update` via the Docker API with `repo:tag@digest` (the agent resolves the digest; without it Swarm does not re-pull when the tag is unchanged). | Re-running `docker stack deploy` with a file holding an older reference. |

The updater always writes `repo:tag@digest`. The stack's own files in git are never
edited. Writing version bumps back to git (GitOps style) was considered and deferred:
it needs repo write access, YAML rewriting and conflict handling — a different kind of
tool. It can be added later as a step after a successful update.

### Drift

A service drifts when `actual_ref` differs from `desired_ref`. A service the updater has
never touched has no `desired_ref` and cannot drift. Drift is **never corrected
automatically**, not even in `auto` mode, because a manual `stack deploy` with a new tag
may be intentional. The UI shows the drift and offers "restore desired" or
"accept running".

### Service identity and moves

A service row has its own ID. Location is an attribute with a unique index:
Compose = agent + project + service; Swarm = cluster ID + service name (independent of
which manager agent reports it); plain container = agent + container name.

Identity without the agent was rejected: the same stack legitimately runs on several
machines (the agent itself, node exporters), and they would be merged into one history.
Automatic merging was rejected for the same reason.

When a Compose service disappears on one agent and exactly one service with the same
project and service name appears on another, the UI suggests "Moved from A?" with a
**Merge** action: the old row is moved to the new location with its history, mode and
`desired_ref`, and the duplicate is removed. Merging can also be done by hand.

### Versions: what the system has seen

Every version the system sees is recorded in `image_versions`, from three sources:
webhook (every build CI announces), polling (every new digest under a followed tag) and
inventory (whatever is running). The service page lists the 10 most recent versions for
the service's repo; picking one creates a manual job to exactly that reference. Picking
an older one is a rollback.

`available_*` on the service remains as "the newest suggestion", which drives the
"update available" badge and `auto` mode.

| Reference format | Polling | Webhook | Pick a version |
|---|---|---|---|
| `repo:tag` | yes | yes | yes |
| `repo:tag@digest` | yes (follows the tag) | yes | yes |
| `repo@digest` | no — nothing to follow | yes (own images) | yes (seen versions) |

Services without a tag are flagged "follows no tag" in the UI. Recommended convention
for stack files: `repo:tag@digest`, tag = what to follow.

## Data model (Postgres)

**`master_identity`** — single row: private key, created_at.

**`agents`** — `id`, `name` (set when adding), `token_hash`, `hostname`,
`agent_version`, `is_swarm_manager` (reported), `last_seen_at`, `revoked_at`, `created_at`.
Online/offline is held in memory only.

**`services`**
- `id`
- `kind`: `compose` | `swarm` | `container` (plain containers are shown but cannot be
  updated in v1)
- location: `agent_id`, `compose_project`, `compose_service`, `swarm_cluster_id`,
  `swarm_service`, `container_name` (populated per kind; unique index per kind)
- `image_repo` — normalised, used to match webhooks and polling
- `update_mode`: `manual` (default) | `auto`
- `desired_ref`
- `actual_ref`, `actual_digest`, `status` — last known, kept so the UI has something to
  show while an agent is offline
- `available_ref`, `available_digest`, `available_source` (`webhook` | `poll`),
  `available_at`
- `last_poll_error`
- `first_seen_at`, `removed_at` — disappeared services are marked, not deleted

**`image_versions`** — `repo`, `tag`, `digest`, `source` (`webhook` | `poll` | `running`),
`first_seen_at`; unique on (repo, tag, digest).

**`update_jobs`** — `id`, `service_id`, `from_ref`, `to_ref`,
`trigger` (`manual` | `webhook` | `poll`), `requested_by` (OIDC subject for manual),
`status` (`queued` | `running` | `succeeded` | `failed`), `output`, `error`,
`created_at`, `started_at`, `finished_at`.

**`sessions`** — web sessions for OIDC logins.

## Flows

### Inventory

1. On connect the agent sends `hello` (hostname, version, Swarm manager or not, cluster
   ID) and a full `snapshot`.
2. The master diffs the snapshot against that agent's rows: new services are inserted as
   `manual`, missing ones get `removed_at`, changed ones get new `actual_*`. Running
   references are recorded in `image_versions` with source `running`.
3. The agent subscribes to Docker events (container start/stop/die/destroy, service
   update), debounces bursts (a Compose restart emits many events), re-inspects the
   affected item and sends `upsert` or `remove`.
4. Swarm services are reported only by agents on manager nodes. Agents skip containers
   that belong to Swarm tasks, so a service is not listed once per node.
5. The master pushes changes to browsers over SSE.

### Update

1. Triggers: a click in the UI, the webhook, or polling. Webhook and polling always set
   `available_*` and record the version; they create a job only if the service is `auto`.
2. The master inserts an `update_jobs` row and sends `job.start` to the owning agent
   (for Swarm, a connected manager agent).
3. The agent puts the job in its FIFO queue (single worker, as today) and executes it as
   described under "Changing the version of a service".
4. The agent sends `job.progress` and `job.result` with output. On success the master
   sets `desired_ref` and clears `available_*` if that version is now running.

### Failure handling

| Situation | Behaviour |
|---|---|
| Agent offline when a job is created | Job fails immediately with "agent offline". It is not stored for later — a job that runs hours later surprises more than it helps. `available_*` remains, so the user can retry. |
| Connection drops mid-job | The agent finishes the job (its queue is local) and keeps results in memory until the master acknowledges them. They are resent after reconnect along with the snapshot. |
| Master restarts mid-job | Jobs in `running` are reconciled when the agent reconnects; jobs the agent does not know are marked `failed` ("lost"). |
| New trigger while a job for the same service is queued or running | No new job; only `available_*` is updated. Two jobs never race on one service. |
| Job exceeds `UPDATE_TIMEOUT` | The agent cancels it, as today. |
| Agent updates itself | The old agent runs `up -d` in a helper container and waits. If the helper exits while the old agent still runs, the old agent reports the outcome. If the helper stops the old agent, the old agent records nothing; the new agent finds the helper by its labels before it connects, so its hello lists the job, and it reports the helper's outcome. If no new agent resumes the job, it is marked lost when the agent reconnects without it. If `UPDATE_TIMEOUT` passes first, the old agent reports a timeout and leaves the helper running; a later success is then not recorded. |
| Polling fails for one image | Error stored in `last_poll_error` and shown; other images continue. |

### Self-update

The agent reports its own container with `self`; the overview shows "This agent". The
master's own Compose service and every Compose agent's own service are normal update
targets. A Swarm agent's own service is refused ("an agent's own Swarm service cannot
update itself").

- **Master.** No special execution. The agent on the master's host runs the job; the
  recreate drops its connection, and it sends the result again after the new master is up
  (unacknowledged results are resent).
- **Agent.** The old agent does the usual checks, writes the override and pulls. Only
  `up -d <service>` runs in a helper container made from the agent's own image, with the
  agent's own bind and volume mounts and no Compose labels. The helper's labels
  (`se.shcizo.package-updater.self-update.` plus `job`, `ref`, `started`, `project`,
  `service`, `output`) hold what the new agent needs: the job ID, the pinned ref, the
  start time and the output so far (tail, 32 KiB). If the helper exits while the old agent
  still runs, the old agent copies its logs into the job output, removes it and reports
  the outcome itself.
- **Resume.** At startup, before the runner and the connection start, the agent lists
  helpers whose project and service labels match its own container and registers each as
  a job at the front of its queue. The job is then in `known_jobs` of the first hello.
  Its result is a normal `job.result`: output, helper logs, pinned ref on exit 0,
  otherwise an error. A helper with unusable labels fails the job with a message saying so.
- **Stopping.** When the agent is stopped (SIGTERM) while a job runs, the runner drops the
  job and records no result, so a "context canceled" result cannot overtake the new
  agent's.
- **Order.** `job.start` carries `self`; the agent runs a job for its own service only
  when no other job is queued (the master's job on a shared host has to finish first).
  The master starts such jobs after the other jobs of the same poll cycle or webhook
  request; messages to one agent arrive in order, so the others are queued first. A job
  that reaches an agent while it replaces itself is marked lost when the new agent
  connects without it. Whether to use a helper is decided by the agent from its own
  container ID, not from the flag.
- **Protocol 4.** The master routes a job for an agent's own service, and any job for a
  service with the `AGENT_IMAGE` repo (an older agent does not report itself), only to an
  agent speaking protocol 4 (`agent too old to update itself`). The helper runs on the
  default network, so `DOCKER_HOST` must be a `docker.sock` bind mount.

## Registry polling

- The master polls, not the agents: credentials live in one place and identical
  references are polled once.
- Every distinct `repo:tag` in use is polled, including `repo:tag@digest`. Bare
  `repo@digest` and locally built images without a registry digest are skipped and
  shown as such.
- Mechanism: `HEAD /v2/<repo>/manifests/<tag>`; the returned digest is compared with
  `actual_digest`. Use `go-containerregistry` — it handles Docker Hub's token auth and
  multi-arch indexes.
- Interval `POLL_INTERVAL`, default 6h. Docker Hub documents that HEAD requests do not
  count against the pull limit ("Using GET emulates a real pull and counts towards the
  limit. Using HEAD won't."; verified against
  https://docs.docker.com/docker-hub/usage/pulls/). The same page mentions a separate,
  unquantified abuse rate limit for many pulls from one IP address, so the poller stays
  HEAD-only and never downloads a manifest.
- Credentials: a Docker `config.json` (as written by `docker login`) mounted into the
  master. No credential UI, no encryption in Postgres.

## Webhook

- `POST /update` with `{image, tag}` and a bearer token — the same contract as today, so
  the existing Gitea action works unchanged; only its `endpoint` moves to the master.
- `image` is matched against `image_repo` across all agents using today's normalisation
  (`NormaliseImage`). Each match gets `available_ref = repo:tag` (source `webhook`) and
  the version is recorded; `auto` services get a job.
- The response waits for the `auto` jobs to finish (bounded by `UPDATE_TIMEOUT`), and
  returns 5xx if any failed or timed out, so CI still surfaces a broken deploy. `manual` matches are reported as "marked", not as
  errors.
- No match returns **404**, which catches typos in CI configuration.
- The webhook never creates services. The first deployment of a stack is done by hand.

## Authentication and UI

### Authentication

Three separate mechanisms that never mix:

| Who | How | Where |
|---|---|---|
| People | OIDC authorization code flow with PKCE against Authentik (`coreos/go-oidc`, `x/oauth2`) | Web UI, via reverse proxy |
| CI | Shared bearer token | `POST /update`, via reverse proxy |
| Agents | Per-agent token + pinned master key over TLS | Dedicated agent port, internal network |

- Sessions are stored in Postgres; the cookie holds a session ID (`HttpOnly`, `Secure`,
  `SameSite=Lax`).
- The master also requires membership of `ALLOWED_GROUP` from the token's `groups`
  claim. Restricting the application in Authentik alone is not enough: one misconfigured
  binding would hand every Authentik user the ability to restart the fleet.
- All state-changing UI requests carry a CSRF token.
- `/healthz`, `/metrics` and `/version` stay unauthenticated, as today.

### UI

Server-rendered Go templates with htmx; live updates via htmx's SSE support. Embedded in
the master binary with `go:embed`. No Node build step. Visual design is drafted
separately; this spec defines what each page contains.

1. **Overview** — all services grouped per machine (Swarm services per cluster). Each
   shows status, running version, the manual/auto toggle, and badges for
   *update available*, *drifted* and *follows no tag*. Live.
2. **Service** — the 10 most recent versions with an *Update* action, job history with
   logs, drift actions (restore / accept), and *Merge* for moves.
3. **Agents** — online/offline, version. *Add agent* produces a compose snippet with
   token and `MASTER_KEY`. *Revoke*.
4. **Jobs** — full history; running jobs stream their log live.

## Code layout

- `cmd/master`, `cmd/agent` — replace `cmd/server`.
- `internal/protocol` — message types shared by master and agent.
- `internal/agent/...` — today's `discovery` and `updater` (queue, Compose and Swarm
  executors) move here, plus inventory, Docker events and the connection loop.
- `internal/master/...` — agent hub, jobs, polling, webhook, OIDC, storage.
- `web/` — templates and static assets.

Existing conventions continue: interfaces defined at the consumer, hand-written fakes,
nil-able `*Metrics`, Conventional Commits.

## Testing

- `go test ./...` must keep passing without Docker or Postgres.
- Master logic (mode decisions, drift, merge, webhook matching, job reconciliation) is
  tested against a storage interface with a hand-written fake.
- Postgres storage is tested by integration tests behind the `integration` build tag,
  requiring `DATABASE_URL`.
- Master and agent are tested together in-process: an `httptest` TLS server, a real
  WebSocket client and a fake Docker. Covers the `MASTER_KEY` handshake (including a
  wrong key being rejected), reconnect with snapshot, and jobs completed while
  disconnected.
- Table-driven tests with testify.

## Migration from today's deployment

1. Create the database and user in Postgres; register the application and group in
   Authentik.
2. Deploy the master.
3. **Remove Shepherd and Watchtower.** Running alongside, they would update services
   behind the master's back and every such update would show up as drift.
4. Add one agent per machine through the UI. Inventory fills in; everything is `manual`.
5. Point the Gitea action's `endpoint` at the master. Contract unchanged.
6. Remove today's package-updater instance.
7. Convert bare `repo@digest` references to `repo:tag@digest`, and set `auto` where
   wanted. Recommended: record the convention in the self-hosted repo's `CLAUDE.md`
   so Claude instances on the servers stop writing bare digests.

`CLAUDE.md` in this repo is updated as part of the implementation: the "stateless" and
"token AND opt-in label AND STACKS_ROOT" design-intent entries no longer describe the
system, and the `MODE` entry is replaced by the agent model.

## Implementation phases

Each phase ends with something that works on its own.

1. **Protocol, agent and master with inventory.** Pinned-key TLS, per-agent tokens,
   snapshot and live events, Postgres storage, OIDC login, Overview and Agents pages.
   The overview works; nothing can be updated yet.
2. **Jobs and manual update.** Job protocol, agent queue and executors (Compose override
   file, Swarm with digest), job history, Service and Jobs pages.
3. **Webhook and auto mode.** `POST /update` on the master, matching, waiting for
   auto jobs, mode toggle.
4. **Polling and version history.** `image_versions`, registry polling, version picker
   and rollback.
5. **Drift and merge.** Drift detection and actions, move suggestion and merge.

## Out of scope (later specs)

- Detecting newer version *tags* for images we do not build (e.g. `gitea:1.27.3` →
  `1.27.4`, `postgres:17.10`, `loki:3.4.2`). Requires listing tags and ordering them;
  major upgrades such as Postgres 17 → 18 must never be automatic. Revisit after
  checking the other two servers' stack files.
- Notifications.
- Export/import of a fleet configuration to restore a setup. The `services` table
  already holds what an export needs.
- Writing version bumps back to git.
- Updating plain (non-Compose, non-Swarm) containers.
- Credential management UI for registries.