package-updater/docs/superpowers/specs/2026-10-12-master-agent-design.md

Master/Agent Architecture — Design

Date: 2026-10-12 Status: Approved in brainstorming, pending written-spec review

Problem

package-updater today is one instance per Docker daemon: a webhook (POST /update) finds opted-in containers by label and runs docker compose or docker service update. It is stateless and has no UI.

The goal now is a tool that replaces Watchtower and Shepherd across the whole fleet:

• One overview of every service on every machine — Swarm services and Compose stacks, which can coexist on the same machine.
• Per-service choice, made in the UI, between manual (show that an update exists, a human clicks) and auto (apply as soon as a new version is detected).
• Detect new versions both from CI (webhook) and by polling registries, so images we do not build ourselves are covered too.
• See available and completed updates, pick any of the recently seen versions (which also gives rollback).
• Live updates in the UI.

Building this ourselves instead of adopting Komodo or Portainer is a deliberate choice: it is a hobby project, and owning it means it works exactly the way we want.

Environment and assumptions

• Several machines on one internal network, reachable from each other by IP.
• One machine (swarm-mgr) is a Swarm manager; most services are Swarm stacks (docker-stack.yml). At least one Compose stack exists (slusk).
• A central Postgres already runs and will be used.
• Authentik runs and is used per application via OIDC (not via the reverse proxy).
• A reverse proxy fronts the web UI.
• Stack files are version-controlled. Image references are ad hoc: a mix of repo:tag, repo:tag@digest and bare repo@digest. Many digests were written by a Claude instance on the servers, not by an automated process.
• Shepherd and Watchtower currently run and will be removed (see Migration).

Decisions

Topology: one master, one agent per machine

• Master (one instance): Postgres, web UI, OIDC login, webhook, registry polling, job decisions. Runs as a Swarm service on swarm-mgr.
• Agent (one per machine): owns that machine's Docker daemon via docker.sock. Reports inventory and runs update jobs. Runs via plain docker compose on each host (not a global Swarm service), because each agent has its own token and a global service would hand every node the same environment. The agent mounts docker.sock and STACKS_ROOT read-write (today it is :ro), because it writes compose.override.yaml files.

Connection: agent dials master over WebSocket, live

The agent opens a long-lived WebSocket to the master and keeps it alive with reconnect and backoff. Inventory changes are pushed live from Docker events; the master forwards them to browsers. Chosen over the master polling agents over HTTP because live updates are an explicit goal.

Messages are JSON in a common envelope {type, id, payload} with one Go type per message kind, shared in internal/protocol. JSON over WebSocket rather than gRPC: same tooling as the browser side, debuggable with websocat, no protobuf toolchain.

Trust in both directions

Commands flow from master to agent and can restart anything on that machine, so the agent must authenticate the master, not only the other way round.

• On first start the master generates a key pair and stores it in Postgres (master_identity), so it survives container recreation and is covered by DB backups.
• The master serves the agent endpoint on its own port with TLS, using a self-signed certificate for that key. No reverse proxy, no CA.
• The agent is configured with MASTER_URL, MASTER_KEY and AGENT_TOKEN. MASTER_KEY is the SHA-256 fingerprint of the master's public key. The agent skips normal chain verification and instead accepts the connection only if the certificate's public key hashes to MASTER_KEY (same idea as SSH known_hosts). The TLS handshake proves the master holds the private key.
• AGENT_TOKEN is per agent, created in the UI ("Add agent", which shows a ready compose snippet including token and MASTER_KEY). Only its hash is stored. Agents can be revoked individually. A leaked token lets someone impersonate an agent (feed false inventory), not command machines.
• Losing the master key means every agent needs a new MASTER_KEY.

Plain ws:// with a signed handshake was rejected: anyone able to hijack the TCP stream after the handshake could inject commands.

State: the master remembers, agents report reality

Agents are the truth for what is running. Postgres holds what the master must remember: desired version, update mode, version history and job history. This deliberately retires the "stateless" property for the master; agents stay stateless.

Security gate moves from labels to the UI

The opt-in label no longer decides what may be updated; the per-service mode in the UI does. Consequences:

• Every newly discovered service starts as manual. Nothing is ever auto by default.
• Access to the UI is equivalent to being able to restart everything, so login is the most important control (see Authentication).
• The STACKS_ROOT path check stays on the agent as an independent guard for Compose, and matters more now that the agent needs write access there (override files).
• The master cannot reliably tell which service is its own Postgres or Authentik, so they are not locked automatically. Putting them on auto is the operator's risk and is documented.

Changing the version of a service

Kind	How the version is changed	What can revert it
Compose	Agent writes the service's image into compose.override.yaml next to the stack's compose file (the agent owns this file, preserving other services' entries in it), then docker compose pull <svc> + up -d <svc>. docker compose loads the override automatically, so manual docker compose up -d keeps the new version.	Deleting the override file.
Swarm	service update via the Docker API with repo:tag@digest (the agent resolves the digest; without it Swarm does not re-pull when the tag is unchanged).	Re-running docker stack deploy with a file holding an older reference.

The updater always writes repo:tag@digest. The stack's own files in git are never edited. Writing version bumps back to git (GitOps style) was considered and deferred: it needs repo write access, YAML rewriting and conflict handling — a different kind of tool. It can be added later as a step after a successful update.

Drift

A service drifts when actual_ref differs from desired_ref. A service the updater has never touched has no desired_ref and cannot drift. Drift is never corrected automatically, not even in auto mode, because a manual stack deploy with a new tag may be intentional. The UI shows the drift and offers "restore desired" or "accept running".

Service identity and moves

A service row has its own ID. Location is an attribute with a unique index: Compose = agent + project + service; Swarm = cluster ID + service name (independent of which manager agent reports it); plain container = agent + container name.

Identity without the agent was rejected: the same stack legitimately runs on several machines (the agent itself, node exporters), and they would be merged into one history. Automatic merging was rejected for the same reason.

When a Compose service disappears on one agent and exactly one service with the same project and service name appears on another, the UI suggests "Moved from A?" with a Merge action: the old row is moved to the new location with its history, mode and desired_ref, and the duplicate is removed. Merging can also be done by hand.

Versions: what the system has seen

Every version the system sees is recorded in image_versions, from three sources: webhook (every build CI announces), polling (every new digest under a followed tag) and inventory (whatever is running). The service page lists the 10 most recent versions for the service's repo; picking one creates a manual job to exactly that reference. Picking an older one is a rollback.

available_* on the service remains as "the newest suggestion", which drives the "update available" badge and auto mode.

Reference format	Polling	Webhook	Pick a version
repo:tag	yes	yes	yes
repo:tag@digest	yes (follows the tag)	yes	yes
repo@digest	no — nothing to follow	yes (own images)	yes (seen versions)

Services without a tag are flagged "follows no tag" in the UI. Recommended convention for stack files: repo:tag@digest, tag = what to follow.

Data model (Postgres)

master_identity — single row: private key, created_at.

agents — id, name (set when adding), token_hash, hostname, agent_version, is_swarm_manager (reported), last_seen_at, revoked_at, created_at. Online/offline is held in memory only.

services

• id
• kind: compose | swarm | container (plain containers are shown but cannot be updated in v1)
• location: agent_id, compose_project, compose_service, swarm_cluster_id, swarm_service, container_name (populated per kind; unique index per kind)
• image_repo — normalised, used to match webhooks and polling
• update_mode: manual (default) | auto
• desired_ref
• actual_ref, actual_digest, status — last known, kept so the UI has something to show while an agent is offline
• available_ref, available_digest, available_source (webhook | poll), available_at
• last_poll_error
• first_seen_at, removed_at — disappeared services are marked, not deleted

image_versions — repo, tag, digest, source (webhook | poll | running), first_seen_at; unique on (repo, tag, digest).

update_jobs — id, service_id, from_ref, to_ref, trigger (manual | webhook | poll), requested_by (OIDC subject for manual), status (queued | running | succeeded | failed), output, error, created_at, started_at, finished_at.

sessions — web sessions for OIDC logins.

Flows

Inventory

1. On connect the agent sends hello (hostname, version, Swarm manager or not, cluster ID) and a full snapshot.
2. The master diffs the snapshot against that agent's rows: new services are inserted as manual, missing ones get removed_at, changed ones get new actual_*. Running references are recorded in image_versions with source running.
3. The agent subscribes to Docker events (container start/stop/die/destroy, service update), debounces bursts (a Compose restart emits many events), re-inspects the affected item and sends upsert or remove.
4. Swarm services are reported only by agents on manager nodes. Agents skip containers that belong to Swarm tasks, so a service is not listed once per node.
5. The master pushes changes to browsers over SSE.

Update

1. Triggers: a click in the UI, the webhook, or polling. Webhook and polling always set available_* and record the version; they create a job only if the service is auto.
2. The master inserts an update_jobs row and sends job.start to the owning agent (for Swarm, a connected manager agent).
3. The agent puts the job in its FIFO queue (single worker, as today) and executes it as described under "Changing the version of a service".
4. The agent sends job.progress and job.result with output. On success the master sets desired_ref and clears available_* if that version is now running.

Failure handling

Situation	Behaviour
Agent offline when a job is created	Job fails immediately with "agent offline". It is not stored for later — a job that runs hours later surprises more than it helps. available_* remains, so the user can retry.
Connection drops mid-job	The agent finishes the job (its queue is local) and keeps results in memory until the master acknowledges them. They are resent after reconnect along with the snapshot.
Master restarts mid-job	Jobs in running are reconciled when the agent reconnects; jobs the agent does not know are marked failed ("lost").
New trigger while a job for the same service is queued or running	No new job; only available_* is updated. Two jobs never race on one service.
Job exceeds UPDATE_TIMEOUT	The agent cancels it, as today.
Polling fails for one image	Error stored in last_poll_error and shown; other images continue.

Self-protection

The agent reports its own container and the master knows its own service. Both are shown as "updated manually outside the system", and the master refuses to create jobs for them. An agent recreating itself would kill the job it is running.

Registry polling

• The master polls, not the agents: credentials live in one place and identical references are polled once.
• Every distinct repo:tag in use is polled, including repo:tag@digest. Bare repo@digest and locally built images without a registry digest are skipped and shown as such.
• Mechanism: HEAD /v2/<repo>/manifests/<tag>; the returned digest is compared with actual_digest. Use go-containerregistry — it handles Docker Hub's token auth and multi-arch indexes.
• Interval POLL_INTERVAL, default 6h. Docker Hub documents that HEAD requests do not count against the pull limit ("Using GET emulates a real pull and counts towards the limit. Using HEAD won't."; verified against https://docs.docker.com/docker-hub/usage/pulls/). The same page mentions a separate, unquantified abuse rate limit for many pulls from one IP address, so the poller stays HEAD-only and never downloads a manifest.
• Credentials: a Docker config.json (as written by docker login) mounted into the master. No credential UI, no encryption in Postgres.

Webhook

• POST /update with {image, tag} and a bearer token — the same contract as today, so the existing Gitea action works unchanged; only its endpoint moves to the master.
• image is matched against image_repo across all agents using today's normalisation (NormaliseImage). Each match gets available_ref = repo:tag (source webhook) and the version is recorded; auto services get a job.
• The response waits for the auto jobs to finish (bounded by UPDATE_TIMEOUT), and returns 5xx if any failed or timed out, so CI still surfaces a broken deploy. manual matches are reported as "marked", not as errors.
• No match returns 404, which catches typos in CI configuration.
• The webhook never creates services. The first deployment of a stack is done by hand.

Authentication and UI

Authentication

Three separate mechanisms that never mix:

Who	How	Where
People	OIDC authorization code flow with PKCE against Authentik (coreos/go-oidc, x/oauth2)	Web UI, via reverse proxy
CI	Shared bearer token	POST /update, via reverse proxy
Agents	Per-agent token + pinned master key over TLS	Dedicated agent port, internal network

• Sessions are stored in Postgres; the cookie holds a session ID (HttpOnly, Secure, SameSite=Lax).
• The master also requires membership of ALLOWED_GROUP from the token's groups claim. Restricting the application in Authentik alone is not enough: one misconfigured binding would hand every Authentik user the ability to restart the fleet.
• All state-changing UI requests carry a CSRF token.
• /healthz, /metrics and /version stay unauthenticated, as today.

UI

Server-rendered Go templates with htmx; live updates via htmx's SSE support. Embedded in the master binary with go:embed. No Node build step. Visual design is drafted separately; this spec defines what each page contains.

1. Overview — all services grouped per machine (Swarm services per cluster). Each shows status, running version, the manual/auto toggle, and badges for update available, drifted and follows no tag. Live.
2. Service — the 10 most recent versions with an Update action, job history with logs, drift actions (restore / accept), and Merge for moves.
3. Agents — online/offline, version. Add agent produces a compose snippet with token and MASTER_KEY. Revoke.
4. Jobs — full history; running jobs stream their log live.

Code layout

• cmd/master, cmd/agent — replace cmd/server.
• internal/protocol — message types shared by master and agent.
• internal/agent/... — today's discovery and updater (queue, Compose and Swarm executors) move here, plus inventory, Docker events and the connection loop.
• internal/master/... — agent hub, jobs, polling, webhook, OIDC, storage.
• web/ — templates and static assets.

Existing conventions continue: interfaces defined at the consumer, hand-written fakes, nil-able *Metrics, Conventional Commits.

Testing

• go test ./... must keep passing without Docker or Postgres.
• Master logic (mode decisions, drift, merge, webhook matching, job reconciliation) is tested against a storage interface with a hand-written fake.
• Postgres storage is tested by integration tests behind the integration build tag, requiring DATABASE_URL.
• Master and agent are tested together in-process: an httptest TLS server, a real WebSocket client and a fake Docker. Covers the MASTER_KEY handshake (including a wrong key being rejected), reconnect with snapshot, and jobs completed while disconnected.
• Table-driven tests with testify.

Migration from today's deployment

1. Create the database and user in Postgres; register the application and group in Authentik.
2. Deploy the master.
3. Remove Shepherd and Watchtower. Running alongside, they would update services behind the master's back and every such update would show up as drift.
4. Add one agent per machine through the UI. Inventory fills in; everything is manual.
5. Point the Gitea action's endpoint at the master. Contract unchanged.
6. Remove today's package-updater instance.
7. Convert bare repo@digest references to repo:tag@digest, and set auto where wanted. Recommended: record the convention in the self-hosted repo's CLAUDE.md so Claude instances on the servers stop writing bare digests.

CLAUDE.md in this repo is updated as part of the implementation: the "stateless" and "token AND opt-in label AND STACKS_ROOT" design-intent entries no longer describe the system, and the MODE entry is replaced by the agent model.

Implementation phases

Each phase ends with something that works on its own.

1. Protocol, agent and master with inventory. Pinned-key TLS, per-agent tokens, snapshot and live events, Postgres storage, OIDC login, Overview and Agents pages. The overview works; nothing can be updated yet.
2. Jobs and manual update. Job protocol, agent queue and executors (Compose override file, Swarm with digest), job history, Service and Jobs pages, self-protection.
3. Webhook and auto mode. POST /update on the master, matching, waiting for auto jobs, mode toggle.
4. Polling and version history. image_versions, registry polling, version picker and rollback.
5. Drift and merge. Drift detection and actions, move suggestion and merge.

Out of scope (later specs)

• Detecting newer version tags for images we do not build (e.g. gitea:1.27.3 → 1.27.4, postgres:17.10, loki:3.4.2). Requires listing tags and ordering them; major upgrades such as Postgres 17 → 18 must never be automatic. Revisit after checking the other two servers' stack files.
• Notifications.
• Export/import of a fleet configuration to restore a setup. The services table already holds what an export needs.
• Writing version bumps back to git.
• Self-update of master and agents (internal/selfupdate is the starting point).
• Updating plain (non-Compose, non-Swarm) containers.
• Credential management UI for registries.