package-updater/CLAUDE.md

package-updater — Claude context

Master/agent fleet updater. A master (UI behind OIDC, Postgres) shows every Compose stack, Swarm service and container on every machine and starts updates; one agent per machine (/var/run/docker.sock mounted) reports its inventory and runs the jobs. Both are in one image: updater-master and updater-agent.

See README.md for user-facing docs. This file is for working ON the code.

Commands

go test ./...                          # all tests, fast (no docker, no postgres)
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
go test -run TestFoo ./internal/master/hub    # single test
go build ./cmd/master ./cmd/agent      # produce ./master and ./agent
docker build -t package-updater:dev .  # multi-stage, builds golang:1.26-alpine

# Postgres tests (store), against any throwaway database:
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/

Architecture (one-line per package)

• cmd/master — wiring only: config → store → hub → jobs → web + webhook, two listeners (UI, agents over pinned TLS)
• cmd/agent — wiring only: config → docker client → inventory → job runner → master connection
• internal/protocol — the WebSocket messages both sides speak (Version, JobStart, JobResult, ...)
• internal/pin — master key pair, certificate and fingerprint pinning
• internal/agenttoken — agent token generation and hashing
• internal/imageref — image reference parsing (repo, tag, digest)
• internal/logging — slog JSON, request-id context propagation
• internal/agent/config — env-var loading; fails fast on missing MASTER_URL/MASTER_KEY/AGENT_TOKEN
• internal/agent/inventory — reads Compose, Swarm and plain containers from the Docker daemon
• internal/agent/conn — dial, hello, snapshot, job messages, reconnect with backoff; one writer goroutine per session
• internal/agent/updater — job Runner (single FIFO worker, outbox until acked), Compose and Swarm executors, override file, mount-root check, registry auth
• internal/master/config — env-var loading
• internal/master/store — Postgres access and migrations
• internal/master/hub — agent connections: auth, per-connection writer, routing, job reconcile on hello
• internal/master/registry — HEAD-only digest lookup and tag listing against a registry (go-containerregistry, DOCKER_CONFIG credentials)
• internal/master/tagver — which tag is a newer version of a followed tag (pure functions)
• internal/master/poller — every POLL_INTERVAL: one lookup per followed repo:tag, sets the available version, starts auto jobs; one tag list per repo, sets the newer tag suggestion
• internal/master/jobs — decides whether a service may be updated and starts a job on the right agent
• internal/master/webhook — POST /update: records the available version, starts jobs for auto services, waits for them
• internal/master/web/icons — embedded Simple Icons path data; slug derived from the image ref; regenerate with go generate ./internal/master/web/icons
• internal/master/web, webauth — HTMX UI behind OIDC login, SSE signals
• internal/master/events — in-process publish/subscribe for UI refresh signals
• internal/master/model — domain types shared by the above

Design intent (do not break without discussion)

• Single FIFO worker per agent by design (internal/agent/updater/runner.go). Two docker compose calls against the same stack race.
• One queued or running job per service, enforced by the unique index update_jobs_one_active (not by application code).
• The agent writes exactly one file, <stack dir>/compose.override.yaml, and only if it is absent or starts with the header line. It refuses the job when any other auto-loaded override name exists.
• Every config file and the override path must be inside a read-write bind mount of the agent's own container whose source equals its destination. Mounts are read per job (ContainerInspect(SELF_ID)) and checked after symlink resolution. This replaces the old STACKS_ROOT.
• Self-update (an agent's own Compose service): only up -d runs in the helper container; the new agent resumes the job (updater.ResumeSelfUpdates) before the runner and connection start; the old agent records no result once SIGTERM cancels it. Jobs for the agent's own service run last on the agent and the master starts them last in a batch. An agent's own Swarm service is refused (ReasonSelfSwarm). Protocol 4 is needed, also for any service with the AGENT_IMAGE repo.
• Unknown message types are logged and ignored; malformed messages close the connection.
• Reconcile before routable: the hub fails a reconnecting agent's jobs it does not list in known_jobs, and only then accepts jobs for that connection.
• The webhook never creates services; new services always start manual. The upsert does not touch update_mode.
• Drift is never corrected automatically; poll/webhook jobs are refused for a drifted service (jobs.Start, ReasonDrifted). Only a manual job, RestoreDesired or an accept gets it out.
• Merging is never automatic; merge is the only path that deletes a services row (store.MergeService). The old row keeps its id and takes the duplicate's location.
• Auth: agents by token over pinned TLS; people by OIDC; CI by the UPDATER_API_KEY bearer token on POST /update only. Every state-changing UI request is a POST with a CSRF token.

Gotchas

• Registry manifest access is HEAD only (internal/master/registry). Docker Hub counts a manifest GET as a pull; the client never GETs /manifests/, and a test asserts this. The only GETs are the /v2/ ping and tags/list (newer-tag detection, once per repo per cycle).
• services.newer_ref is a suggestion only: no auto jobs, not even for auto services. The poller writes it only on change; a failed tag list keeps the old value and never sets last_poll_error. The UI re-checks it at render time with tagver.Newer, so a stale value is hidden until the next cycle; it is also hidden when its repo is not the one the service follows. The suggestion is computed from the service list read at the start of the cycle, so if the service moved on meanwhile, the render-time check hides it. Tags with one numeric component or no digits (7, pg16, latest) never get suggestions.
• image_versions uses '' instead of NULL for a missing tag or digest, so UNIQUE (repo, tag, digest) deduplicates. The webhook records (repo, tag, ''); that row is hidden from RecentVersions when a row with the same tag and a digest was first seen at or after it.
• The UI picks a version by image_versions.id, never by a ref from the form, and the row must belong to the service's image_repo. A digest-pinned or tagless pick needs agent protocol 3 (protocol.MinPinnedRefVersion); Hub.Route takes the minimum version.
• Picking a tagless version (repo@digest) leaves the service following no tag: it is no longer polled and the plain Update button is refused.
• services.last_tag_digest is set by the poller when it starts an auto job and when it sees the service already running the tag's digest, and by FinishJob after any successful plain tag job (not pinned/tagless picks; the digest is the one the agent pinned). The poller starts no job for that digest, nor for one desired_ref (ref@digest) already names, and re-reads the service before Start; so a rollback of an auto service holds until the registry shows a different digest. ErrJobActive and NotUpdatableError leave it unset so the next cycle retries.
• A bare image's running version is recorded with tag latest (like jobs.FollowedRef); repo@digest is recorded with tag ''. Only sha256:<64 hex> digests are recorded.
• Image matching is case-sensitive. The tricky case is localhost:5000/foo:v1 where the first colon is a port, not a tag. See internal/imageref.
• Path safety uses filepath.Rel + ".." prefix check, NOT strings.HasPrefix. Prevents the /foo-evil vs /foo confusion. See internal/agent/updater/pathcheck.go.
• Go 1.26.3 (go.mod). Dockerfile pins golang:1.26-alpine. Bumping one without the other has caused a fix commit already.
• <summary> C# convention from global CLAUDE.md does not apply here — this is Go. Use idiomatic GoDoc (// FuncName does X.).
• Compose loads only one override file. Of compose.override.yml, compose.override.yaml, docker-compose.override.yml, docker-compose.override.yaml only the first that exists is read. Hence the refusal described above.
• Stacks started with explicit -f do not load the override on a manual up. The agent's own jobs always pass it explicitly.
• Override entries are never pruned. A service deleted from the stack file keeps its image: line in compose.override.yaml.
• The override file gets the owner of the existing override, else of the stack's first compose file. The exception is a root-owned existing override (written before this fix): then the compose file's owner is used, so it stops being root-owned. The agent runs as root, so WriteOverride chowns the open temp file before the rename (never the path afterwards: a symlink could be swapped in). A failure is a warning: line in the job output, never a failed job.
• There is no STACKS_ROOT. Stack directories must be mounted read-write at the host path (/path:/path), and SELF_ID must resolve to the agent's own container. On Docker Desktop/OrbStack bind sources are translated, so Compose jobs fail there.
• job.start carries no paths. The agent reads working dir and config files from its own Docker daemon at execution time.
• protocol.Service.ConfigFiles is the raw comma-joined label string, so Service stays comparable with == (inventory.Diff depends on it).
• Helper containers carry se.shcizo.package-updater.self-update.* labels and are not inventoried (the collector skips any container with the .job label). The label strings are constants in internal/agent/updater/selfupdate.go, and the .job one is repeated in internal/agent/inventory.
• Swarm "succeeded" means the daemon accepted the new spec, not that tasks converged.
• Swarm updates need manager-node API access for the agent doing the update.
• The image's default entrypoint is updater-agent. The master overrides it with updater-master.
• The example stack runs the master as nobody (user: "65534:65534"). Never add USER to the Dockerfile: the agent uses the same image and needs root for the Docker socket.
• The webhook waits via a broker signal plus a store re-read, with a fallback poll (the broker drops signals for slow subscribers). Marking and starting jobs runs on context.WithoutCancel, so a client that disconnects cannot leave a queued job no agent knows about; only the waiting stops. The master never fails or cancels a timed-out job.
• A webhook job's target is <image_repo>:<request tag>, built from the service's own repo, never from the request's image string. With an explicit target the "follows no tag" refusal does not apply; removed, built, container and an agent's own Swarm service still do.
• available_* is cleared by FinishJob on success only when the job's to_ref (as created, without @digest) equals available_ref, available_at <= job.created_at (that version is now running) and, when the job's to_ref carries a digest, that digest equals available_digest. A rollback to an older digest of the tag therefore keeps the newer suggestion, and a digest-pinned pick does not clear a digest-less (webhook) one.
• Job output is capped at protocol.MaxJobOutput (128 KiB, tail). NUL bytes are stripped by the hub because Postgres text rejects them.
• Drift compares by digest, not by string (model.RunsRef): repo:tag@D is run by repo:tag reporting digest D. Same digest under another tag is not drift. Different repo strings (nginx vs docker.io/library/nginx) are.
• services.awaiting_inventory is true from a successful FinishJob until the next inventory write for the row; Service.Drifted() is false meanwhile, since desired_ref is ahead of actual_ref for a debounce interval after every job.
• Accept never touches last_tag_digest, available_* or update_mode, and the actual form field (Service.RunningRef(), ref plus digest) is only a guard against a changed service; the value written comes from the row. RestoreDesired sets last_tag_digest to the drifted digest when it is another digest of the desired tag and last_tag_digest is empty or equals the desired digest (otherwise it holds back a newer digest and stays).
• awaiting_inventory can hide real drift until the next inventory change when a job result arrives after the snapshot that shows the drift (an agent reconnect resends its outbox after the snapshot), or when Swarm coalesces an update and a rollback into one debounce interval. This is accepted.
• Merge lock order is agent, then service rows by id (as in inventory writes), and the duplicate row is deleted before the old row is updated, because location_key is unique and not deferrable. A removal racing a merge can be lost; the next snapshot fixes it.
• The SSE stream sends a named ping event every 25 s; the sidebar Live indicator relies on it.
• Agents persist swarm_cluster_id. Only managers report it, so workers show no role.

References

• Master/agent spec: docs/superpowers/specs/2026-10-12-master-agent-design.md (authoritative)
• Phase 1 plan: docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md
• Phase 2 plan: docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md
• Phase 3 plan: docs/superpowers/plans/2026-10-12-master-agent-phase-3-webhook.md
• Phase 4 plan: docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md
• Phase 5 plan: docs/superpowers/plans/2026-10-13-master-agent-phase-5-drift-merge.md
• Original single-instance design (historical): docs/superpowers/specs/2026-05-22-package-updater-design.md
• Consumer-side CI integration: gitea-action/ (calls the master's POST /update)

Conventions

• Conventional Commits (feat:, fix:, docs:, ci:, build:, chore:).
• One package = one responsibility; interfaces defined at consumer site (jobs.Store, jobs.Hub, conn.Jobs, hub.Store) for testability.
• Table-driven tests with stretchr/testify. No mocks beyond hand-written fakes.