204 lines
15 KiB
Markdown
204 lines
15 KiB
Markdown
# package-updater — Claude context
|
|
|
|
Master/agent fleet updater. A master (UI behind OIDC, Postgres) shows every Compose
|
|
stack, Swarm service and container on every machine and starts updates; one agent per
|
|
machine (`/var/run/docker.sock` mounted) reports its inventory and runs the jobs.
|
|
Both are in one image: `updater-master` and `updater-agent`.
|
|
|
|
See `README.md` for user-facing docs. This file is for working ON the code.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
go test ./... # all tests, fast (no docker, no postgres)
|
|
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
|
|
go test -run TestFoo ./internal/master/hub # single test
|
|
go build ./cmd/master ./cmd/agent # produce ./master and ./agent
|
|
docker build -t package-updater:dev . # multi-stage, builds golang:1.26-alpine
|
|
|
|
# Postgres tests (store), against any throwaway database:
|
|
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
|
|
```
|
|
|
|
## Architecture (one-line per package)
|
|
|
|
- `cmd/master` — wiring only: config → store → hub → jobs → web + webhook, two listeners (UI, agents over pinned TLS)
|
|
- `cmd/agent` — wiring only: config → docker client → inventory → job runner → master connection
|
|
- `internal/protocol` — the WebSocket messages both sides speak (`Version`, `JobStart`, `JobResult`, ...)
|
|
- `internal/pin` — master key pair, certificate and fingerprint pinning
|
|
- `internal/agenttoken` — agent token generation and hashing
|
|
- `internal/imageref` — image reference parsing (repo, tag, digest)
|
|
- `internal/logging` — slog JSON, request-id context propagation
|
|
- `internal/agent/config` — env-var loading; fails fast on missing `MASTER_URL`/`MASTER_KEY`/`AGENT_TOKEN`
|
|
- `internal/agent/inventory` — reads Compose, Swarm and plain containers from the Docker daemon
|
|
- `internal/agent/conn` — dial, hello, snapshot, job messages, reconnect with backoff; one writer goroutine per session
|
|
- `internal/agent/updater` — job `Runner` (single FIFO worker, outbox until acked), Compose and Swarm executors, override file, mount-root check, registry auth
|
|
- `internal/master/config` — env-var loading
|
|
- `internal/master/store` — Postgres access and migrations
|
|
- `internal/master/hub` — agent connections: auth, per-connection writer, routing, job reconcile on hello
|
|
- `internal/master/registry` — HEAD-only digest lookup and tag listing against a registry (go-containerregistry, `DOCKER_CONFIG` credentials)
|
|
- `internal/master/tagver` — which tag is a newer version of a followed tag (pure functions)
|
|
- `internal/master/poller` — every `POLL_INTERVAL`: one lookup per followed `repo:tag`, sets the available version, starts auto jobs; one tag list per repo, sets the newer tag suggestion
|
|
- `internal/master/jobs` — decides whether a service may be updated and starts a job on the right agent
|
|
- `internal/master/webhook` — `POST /update`: records the available version, starts jobs for auto services, waits for them
|
|
- `internal/master/web/icons` — embedded Simple Icons path data; slug derived from the image ref; regenerate with `go generate ./internal/master/web/icons`
|
|
- `internal/master/web`, `webauth` — HTMX UI behind OIDC login, SSE signals
|
|
- `internal/master/events` — in-process publish/subscribe for UI refresh signals
|
|
- `internal/master/model` — domain types shared by the above
|
|
|
|
## Design intent (do not break without discussion)
|
|
|
|
- **Single FIFO worker per agent by design** (`internal/agent/updater/runner.go`).
|
|
Two `docker compose` calls against the same stack race.
|
|
- **One queued or running job per service**, enforced by the unique index
|
|
`update_jobs_one_active` (not by application code).
|
|
- **The agent writes exactly one file**, `<stack dir>/compose.override.yaml`, and only
|
|
if it is absent or starts with the header line. It refuses the job when any other
|
|
auto-loaded override name exists.
|
|
- **Every config file and the override path must be inside a read-write bind mount of
|
|
the agent's own container whose source equals its destination.** Mounts are read per
|
|
job (`ContainerInspect(SELF_ID)`) and checked after symlink resolution. This replaces
|
|
the old `STACKS_ROOT`.
|
|
- **Self-update** (an agent's own Compose service): only `up -d` runs in the helper
|
|
container; the new agent resumes the job (`updater.ResumeSelfUpdates`) before the runner
|
|
and connection start; the old agent records no result once SIGTERM cancels it. Jobs for
|
|
the agent's own service run last on the agent and the master starts them last in a
|
|
batch. An agent's own Swarm service is refused (`ReasonSelfSwarm`). Protocol 4 is needed,
|
|
also for any service with the `AGENT_IMAGE` repo.
|
|
- **Unknown message types are logged and ignored; malformed messages close the connection.**
|
|
- **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not
|
|
list in `known_jobs`, and only then accepts jobs for that connection.
|
|
- **The webhook never creates services; new services always start `manual`.** The upsert
|
|
does not touch `update_mode`.
|
|
- **Drift is never corrected automatically; poll/webhook jobs are refused for a drifted
|
|
service** (`jobs.Start`, `ReasonDrifted`). Only a manual job, `RestoreDesired` or an
|
|
accept gets it out.
|
|
- **Merging is never automatic; merge is the only path that deletes a `services` row**
|
|
(`store.MergeService`). The old row keeps its id and takes the duplicate's location.
|
|
- **Auth**: agents by token over pinned TLS; people by OIDC; CI by the `UPDATER_API_KEY`
|
|
bearer token on `POST /update` only. Every state-changing UI request is a POST with a
|
|
CSRF token.
|
|
|
|
## Gotchas
|
|
|
|
- **Registry manifest access is HEAD only** (`internal/master/registry`). Docker Hub counts a
|
|
manifest GET as a pull; the client never GETs `/manifests/`, and a test asserts this. The
|
|
only GETs are the `/v2/` ping and `tags/list` (newer-tag detection, once per repo per cycle).
|
|
- **`services.newer_ref` is a suggestion only**: no auto jobs, not even for auto services. The
|
|
poller writes it only on change; a failed tag list keeps the old value and never sets
|
|
`last_poll_error`. The UI re-checks it at render time with `tagver.Newer`, so a stale value
|
|
is hidden until the next cycle; it is also hidden when its repo is not the one the service
|
|
follows. The suggestion is computed from the service list read at the start of the cycle, so
|
|
if the service moved on meanwhile, the render-time check hides it. Tags with one numeric component or no digits (`7`, `pg16`,
|
|
`latest`) never get suggestions.
|
|
- **`image_versions` uses `''` instead of NULL** for a missing tag or digest, so
|
|
`UNIQUE (repo, tag, digest)` deduplicates. The webhook records `(repo, tag, '')`; that
|
|
row is hidden from `RecentVersions` when a row with the same tag and a digest was first
|
|
seen at or after it.
|
|
- **The UI picks a version by `image_versions.id`, never by a ref from the form**, and the row
|
|
must belong to the service's `image_repo`. A digest-pinned or tagless pick needs agent
|
|
protocol 3 (`protocol.MinPinnedRefVersion`); `Hub.Route` takes the minimum version.
|
|
- **Picking a tagless version (`repo@digest`) leaves the service following no tag:** it is no
|
|
longer polled and the plain Update button is refused.
|
|
- **`services.last_tag_digest`** is set by the poller when it starts an auto job and when it
|
|
sees the service already running the tag's digest, and by `FinishJob` after any successful
|
|
plain tag job (not pinned/tagless picks; the digest is the one the agent pinned). The poller
|
|
starts no job for that digest, nor for one `desired_ref` (`ref@digest`) already names, and
|
|
re-reads the service before `Start`; so a rollback of an auto service holds until the
|
|
registry shows a different digest. `ErrJobActive` and `NotUpdatableError` leave it unset so
|
|
the next cycle retries.
|
|
- **A bare image's running version is recorded with tag `latest`** (like `jobs.FollowedRef`);
|
|
`repo@digest` is recorded with tag `''`. Only `sha256:<64 hex>` digests are recorded.
|
|
- **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where
|
|
the first colon is a port, not a tag. See `internal/imageref`.
|
|
- **Path safety uses `filepath.Rel` + ".." prefix check**, NOT `strings.HasPrefix`.
|
|
Prevents the `/foo-evil` vs `/foo` confusion. See `internal/agent/updater/pathcheck.go`.
|
|
- **Go 1.26.3** (`go.mod`). Dockerfile pins `golang:1.26-alpine`. Bumping one
|
|
without the other has caused a fix commit already.
|
|
- **`<summary>` C# convention from global CLAUDE.md does not apply here** — this
|
|
is Go. Use idiomatic GoDoc (`// FuncName does X.`).
|
|
- **Compose loads only one override file.** Of `compose.override.yml`,
|
|
`compose.override.yaml`, `docker-compose.override.yml`, `docker-compose.override.yaml`
|
|
only the first that exists is read. Hence the refusal described above.
|
|
- **Stacks started with explicit `-f` do not load the override on a manual `up`.** The
|
|
agent's own jobs always pass it explicitly.
|
|
- **Override entries are never pruned.** A service deleted from the stack file keeps its
|
|
`image:` line in `compose.override.yaml`.
|
|
- **The override file gets the owner of the existing override, else of the stack's first
|
|
compose file.** The exception is a root-owned existing override (written before this
|
|
fix): then the compose file's owner is used, so it stops being root-owned. The agent runs
|
|
as root, so `WriteOverride` chowns the open temp file before the rename (never the path
|
|
afterwards: a symlink could be swapped in). A failure is a `warning:` line in the job
|
|
output, never a failed job.
|
|
- **There is no `STACKS_ROOT`.** Stack directories must be mounted read-write at the host
|
|
path (`/path:/path`), and `SELF_ID` must resolve to the agent's own container. On Docker
|
|
Desktop/OrbStack bind sources are translated, so Compose jobs fail there.
|
|
- **`job.start` carries no paths.** The agent reads working dir and config files from its
|
|
own Docker daemon at execution time.
|
|
- **`protocol.Service.ConfigFiles` is the raw comma-joined label string**, so `Service`
|
|
stays comparable with `==` (`inventory.Diff` depends on it).
|
|
- **Helper containers carry `se.shcizo.package-updater.self-update.*` labels** and are
|
|
not inventoried (the collector skips any container with the `.job` label). The label
|
|
strings are constants in `internal/agent/updater/selfupdate.go`, and the `.job` one is
|
|
repeated in `internal/agent/inventory`.
|
|
- **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged.
|
|
- **Swarm updates need manager-node API access** for the agent doing the update.
|
|
- **The image's default entrypoint is `updater-agent`.** The master overrides it with
|
|
`updater-master`.
|
|
- **The example stack runs the master as nobody (`user: "65534:65534"`).** Never add `USER`
|
|
to the Dockerfile: the agent uses the same image and needs root for the Docker socket.
|
|
- **The webhook waits via a broker signal plus a store re-read, with a fallback poll**
|
|
(the broker drops signals for slow subscribers). Marking and starting jobs runs on
|
|
`context.WithoutCancel`, so a client that disconnects cannot leave a queued job no agent
|
|
knows about; only the waiting stops. The master never fails or cancels a timed-out job.
|
|
- **A webhook job's target is `<image_repo>:<request tag>`**, built from the service's own
|
|
repo, never from the request's image string. With an explicit target the "follows no tag"
|
|
refusal does not apply; removed, built, container and an agent's own Swarm service still do.
|
|
- **`available_*` is cleared by `FinishJob`** on success only when the job's `to_ref`
|
|
(as created, without `@digest`) equals `available_ref`, `available_at <= job.created_at`
|
|
(that version is now running) and, when the job's `to_ref` carries a digest, that digest
|
|
equals `available_digest`. A rollback to an older digest of the tag therefore keeps the
|
|
newer suggestion, and a digest-pinned pick does not clear a digest-less (webhook) one.
|
|
- **Job output is capped at `protocol.MaxJobOutput` (128 KiB, tail).** NUL bytes are
|
|
stripped by the hub because Postgres text rejects them.
|
|
- **Drift compares by digest, not by string** (`model.RunsRef`): `repo:tag@D` is run by
|
|
`repo:tag` reporting digest D. Same digest under another tag is not drift. Different
|
|
repo strings (`nginx` vs `docker.io/library/nginx`) are.
|
|
- **`services.awaiting_inventory`** is true from a successful `FinishJob` until the next
|
|
inventory write for the row; `Service.Drifted()` is false meanwhile, since `desired_ref`
|
|
is ahead of `actual_ref` for a debounce interval after every job.
|
|
- **Accept never touches `last_tag_digest`, `available_*` or `update_mode`**, and the
|
|
`actual` form field (`Service.RunningRef()`, ref plus digest) is only a guard against a
|
|
changed service; the value written comes from the row. `RestoreDesired` sets
|
|
`last_tag_digest` to the drifted digest when it is another digest of the desired tag and
|
|
`last_tag_digest` is empty or equals the desired digest (otherwise it holds back a newer
|
|
digest and stays).
|
|
- **`awaiting_inventory` can hide real drift** until the next inventory change when a job
|
|
result arrives after the snapshot that shows the drift (an agent reconnect resends its
|
|
outbox after the snapshot), or when Swarm coalesces an update and a rollback into one
|
|
debounce interval. This is accepted.
|
|
- **Merge lock order is agent, then service rows by id** (as in inventory writes), and the
|
|
duplicate row is deleted before the old row is updated, because `location_key` is unique
|
|
and not deferrable. A removal racing a merge can be lost; the next snapshot fixes it.
|
|
- **The SSE stream sends a named `ping` event every 25 s**; the sidebar Live indicator relies on it.
|
|
- **Agents persist `swarm_cluster_id`.** Only managers report it, so workers show no role.
|
|
|
|
## References
|
|
|
|
- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md` (authoritative)
|
|
- Phase 1 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md`
|
|
- Phase 2 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md`
|
|
- Phase 3 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-3-webhook.md`
|
|
- Phase 4 plan: `docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md`
|
|
- Phase 5 plan: `docs/superpowers/plans/2026-10-13-master-agent-phase-5-drift-merge.md`
|
|
- Original single-instance design (historical):
|
|
`docs/superpowers/specs/2026-05-22-package-updater-design.md`
|
|
- Consumer-side CI integration: `gitea-action/` (calls the master's `POST /update`)
|
|
|
|
## Conventions
|
|
|
|
- Conventional Commits (`feat:`, `fix:`, `docs:`, `ci:`, `build:`, `chore:`).
|
|
- One package = one responsibility; interfaces defined at consumer site
|
|
(`jobs.Store`, `jobs.Hub`, `conn.Jobs`, `hub.Store`) for testability.
|
|
- Table-driven tests with `stretchr/testify`. No mocks beyond hand-written fakes.
|