diff --git a/CLAUDE.md b/CLAUDE.md index 8c526cc..47e7d27 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,104 +1,111 @@ # package-updater — Claude context -Webhook-driven Docker Compose updater. Go service, single binary, deployed as -a container with `/var/run/docker.sock` mounted. +Master/agent fleet updater. A master (UI behind OIDC, Postgres) shows every Compose +stack, Swarm service and container on every machine and starts updates; one agent per +machine (`/var/run/docker.sock` mounted) reports its inventory and runs the jobs. +Both are in one image: `updater-master` and `updater-agent`. See `README.md` for user-facing docs. This file is for working ON the code. ## Commands ```bash -go test ./... # all tests, fast (no docker required) -go test -run TestFoo ./internal/api # single test -go build ./cmd/server # produces ./server -go build ./cmd/master ./cmd/agent # new master/agent binaries (phase 1: inventory) -DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/ # Postgres tests +go test ./... # all tests, fast (no docker, no postgres) +go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/ +go test -run TestFoo ./internal/master/hub # single test +go build ./cmd/master ./cmd/agent # produce ./master and ./agent docker build -t package-updater:dev . # multi-stage, builds golang:1.26-alpine -# Local smoke test (builds from Dockerfile, runs against /var/run/docker.sock): -cp env.sample .env && $EDITOR .env # fill UPDATER_API_KEY -docker compose -f docker-compose.local.yml up --build -curl -sH "Authorization: Bearer $UPDATER_API_KEY" \ - -d '{"image":"foo"}' http://localhost:8080/update | jq +# Postgres tests (store), against any throwaway database: +DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/ ``` ## Architecture (one-line per package) -- `cmd/server` — wiring only: config → docker client → discovery → queue → http -- `internal/api` — HTTP handlers, bearer-token auth, request-id + access-log middleware -- `internal/config` — env-var loading; fails fast if `UPDATER_API_KEY` missing -- `internal/discovery`— given an image, return Compose `Job`s to run (label parsing, path check, dedup) -- `internal/discovery/swarm.go` — SwarmDiscovery: same FindJobs signature, lists `docker service ls`, gates on service-level opt-in label instead of STACKS_ROOT path-check -- `internal/updater` — FIFO queue + single worker + `docker compose` subprocess executor -- `internal/updater/swarm_executor.go` — SwarmExecutor: `docker service update --image` via the Docker API instead of a `docker compose` subprocess -- `internal/selfupdate` — flush-then-exec wrapper for updating ourselves (NOT wired in live, see gotcha) -- `internal/metrics` — Prometheus collectors +- `cmd/master` — wiring only: config → store → hub → jobs → web, two listeners (UI, agents over pinned TLS) +- `cmd/agent` — wiring only: config → docker client → inventory → job runner → master connection +- `internal/protocol` — the WebSocket messages both sides speak (`Version`, `JobStart`, `JobResult`, ...) +- `internal/pin` — master key pair, certificate and fingerprint pinning +- `internal/agenttoken` — agent token generation and hashing +- `internal/imageref` — image reference parsing (repo, tag, digest) - `internal/logging` — slog JSON, request-id context propagation -- `cmd/master`, `cmd/agent` — master/agent architecture (spec `2026-10-12-master-agent-design.md`), phase 1 = inventory only. `cmd/server` keeps doing updates until phase 3. -- `internal/protocol` — agent↔master WebSocket messages -- `internal/pin` — master key pair, self-signed cert, fingerprint-pinned TLS client -- `internal/agenttoken`— per-agent tokens (`pua_` prefix, only the SHA-256 hash is stored) -- `internal/imageref` — split `repo:tag@digest` -- `internal/agent/...` — inventory collection, Docker event debounce, connection loop -- `internal/master/...`— Postgres store, agent hub, change broker, OIDC auth, htmx UI +- `internal/agent/config` — env-var loading; fails fast on missing `MASTER_URL`/`MASTER_KEY`/`AGENT_TOKEN` +- `internal/agent/inventory` — reads Compose, Swarm and plain containers from the Docker daemon +- `internal/agent/conn` — dial, hello, snapshot, job messages, reconnect with backoff; one writer goroutine per session +- `internal/agent/updater` — job `Runner` (single FIFO worker, outbox until acked), Compose and Swarm executors, override file, mount-root check, registry auth +- `internal/master/config` — env-var loading +- `internal/master/store` — Postgres access and migrations +- `internal/master/hub` — agent connections: auth, per-connection writer, routing, job reconcile on hello +- `internal/master/jobs` — decides whether a service may be updated and starts a job on the right agent +- `internal/master/web`, `webauth` — HTMX UI behind OIDC login, SSE signals +- `internal/master/events` — in-process publish/subscribe for UI refresh signals +- `internal/master/model` — domain types shared by the above ## Design intent (do not break without discussion) -- **Single FIFO worker by design** (`internal/updater/queue.go`). Spec §5.7. Never - parallelise the queue — two `docker compose` calls against the same stack race. -- **Defense in depth: token AND opt-in label AND STACKS_ROOT prefix** must all - hold before a container is touched. Weakening any of these breaks the security - model — discuss before changing. -- **Auth on `/update` only**. `/healthz`, `/metrics`, `/version` are intentionally - unauthenticated (internal network, scraper/healthcheck need them). See - `cmd/server/main.go` `routeAuth`. -- **`metrics *Metrics` parameters may be nil**; constructors and call sites check. - Tests rely on this — don't drop the nil-check. -- **Stateless**: no DB, no config file, no on-disk audit log. Docker daemon is - the source of truth. -- **Compose mode and Swarm mode are selected once per deployment via `MODE`**, never - mixed at request time. Swarm mode's security gate is opt-in label only — there is - no STACKS_ROOT-equivalent path check, since Swarm services have no local compose - file. Don't add one; don't weaken Compose mode's three-factor gate to match. +- **Single FIFO worker per agent by design** (`internal/agent/updater/runner.go`). + Two `docker compose` calls against the same stack race. +- **One queued or running job per service**, enforced by the unique index + `update_jobs_one_active` (not by application code). +- **The agent writes exactly one file**, `/compose.override.yaml`, and only + if it is absent or starts with the header line. It refuses the job when any other + auto-loaded override name exists. +- **Every config file and the override path must be inside a read-write bind mount of + the agent's own container whose source equals its destination.** Mounts are read per + job (`ContainerInspect(SELF_ID)`) and checked after symlink resolution. This replaces + the old `STACKS_ROOT`. +- **Self-protection on the master**: a service is refused when its image repo equals the + repo of `AGENT_IMAGE` or the row has the `self` flag. The agent does not re-check. +- **Unknown message types are logged and ignored; malformed messages close the connection.** +- **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not + list in `known_jobs`, and only then accepts jobs for that connection. +- **Auth**: agents by token over pinned TLS; people by OIDC. Every state-changing UI + request is a POST with a CSRF token. ## Gotchas -- **`internal/selfupdate.Wrapped` is implemented and unit-tested but NOT wired - into the live queue.** Spec §15 calls this out as a v1 gap. If you "fix" this, - read the package doc — flush-before-exec ordering is subtle. -- **Image matching is case-sensitive and tag/digest-agnostic.** The tricky case - is `localhost:5000/foo:v1` where the first colon is a port, not a tag. See - `internal/discovery/matching.go` `NormaliseImage`. +- **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where + the first colon is a port, not a tag. See `internal/imageref`. - **Path safety uses `filepath.Rel` + ".." prefix check**, NOT `strings.HasPrefix`. - Prevents the `/foo-evil` vs `/foo` confusion. See `internal/discovery/pathcheck.go`. -- **`STACKS_ROOT` defaults to `/home/shcizo/self-hosted`** in prod but `env.sample` - suggests `/tmp` for smoke tests. The default in `config.go` is the prod value - — set explicitly in test/dev envs. + Prevents the `/foo-evil` vs `/foo` confusion. See `internal/agent/updater/pathcheck.go`. - **Go 1.26.3** (`go.mod`). Dockerfile pins `golang:1.26-alpine`. Bumping one without the other has caused a fix commit already. - **`` C# convention from global CLAUDE.md does not apply here** — this is Go. Use idiomatic GoDoc (`// FuncName does X.`). -- **Swarm mode requires manager-node API access.** `docker service update` fails - with a permission error against a worker-only node. This is an operator/deployment - concern (point `DOCKER_HOST` at a manager, or schedule the updater on a manager), - not something the code can detect or work around. -- **WebSocket read limit is raised to 1 MiB on both sides.** coder/websocket defaults to 32 KiB, which a full snapshot from a busy machine exceeds — symptom is an agent stuck in a reconnect loop. -- **The master's key lives in Postgres (`master_identity`).** Wiping the database means every agent needs a new `MASTER_KEY`. -- **SSE responses must flush through every wrapper.** `web.events` uses `http.ResponseController`; a middleware that wraps `ResponseWriter` without `Unwrap()` silently breaks live updates. Nginx needs `X-Accel-Buffering: no` (set by the handler). -- **The "stateless" and "label gate" design intents above describe `cmd/server` only.** The master is stateful by design and gates updates per service in the UI (spec 2026-10-12). -- **The master does OIDC discovery once at startup and exits if the issuer is unreachable.** It does not retry; Swarm restarts the container, so a short Authentik outage during deploy resolves itself. +- **Compose loads only one override file.** Of `compose.override.yml`, + `compose.override.yaml`, `docker-compose.override.yml`, `docker-compose.override.yaml` + only the first that exists is read. Hence the refusal described above. +- **Stacks started with explicit `-f` do not load the override on a manual `up`.** The + agent's own jobs always pass it explicitly. +- **Override entries are never pruned.** A service deleted from the stack file keeps its + `image:` line in `compose.override.yaml`. +- **There is no `STACKS_ROOT`.** Stack directories must be mounted read-write at the host + path (`/path:/path`), and `SELF_ID` must resolve to the agent's own container. On Docker + Desktop/OrbStack bind sources are translated, so Compose jobs fail there. +- **`job.start` carries no paths.** The agent reads working dir and config files from its + own Docker daemon at execution time. +- **`protocol.Service.ConfigFiles` is the raw comma-joined label string**, so `Service` + stays comparable with `==` (`inventory.Diff` depends on it). +- **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged. +- **Swarm updates need manager-node API access** for the agent doing the update. +- **The image's default entrypoint is `updater-agent`.** The master overrides it with + `updater-master`. There is no `POST /update` until phase 3. +- **Job output is capped at `protocol.MaxJobOutput` (128 KiB, tail).** NUL bytes are + stripped by the hub because Postgres text rejects them. ## References -- Design spec: `docs/superpowers/specs/2026-05-22-package-updater-design.md` (489 lines, authoritative) -- Implementation plan: `docs/superpowers/plans/2026-05-22-package-updater-implementation.md` -- Consumer-side CI integration: `gitea-action/` -- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md` +- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md` (authoritative) - Phase 1 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md` +- Phase 2 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md` +- Original single-instance design (historical, its webhook is gone until phase 3): + `docs/superpowers/specs/2026-05-22-package-updater-design.md` +- Consumer-side CI integration: `gitea-action/` (calls `POST /update`, which no longer + exists until phase 3) ## Conventions - Conventional Commits (`feat:`, `fix:`, `docs:`, `ci:`, `build:`, `chore:`). - One package = one responsibility; interfaces defined at consumer site - (`api.Finder`, `api.Submitter`, `api.Pinger`) for testability. + (`jobs.Store`, `jobs.Hub`, `conn.Jobs`, `hub.Store`) for testability. - Table-driven tests with `stretchr/testify`. No mocks beyond hand-written fakes. diff --git a/README.md b/README.md index d967afd..20f359a 100644 --- a/README.md +++ b/README.md @@ -1,121 +1,110 @@ # package-updater -Webhook-driven Docker Compose service updater. Fills the gap between Watchtower (polling, no CI integration) and full GitOps (Argo CD, Flux) for a self-hosted, single-host environment. +Updates self-hosted Docker Compose and Docker Swarm services from one place. A +**master** (web UI behind OIDC login, state in Postgres) shows every Compose stack, +Swarm service and container on every machine, and starts updates when someone clicks +*Update*. One **agent** per machine reports its inventory and runs the jobs. Both are in +the same image: `updater-master` and `updater-agent` (the default entrypoint). -**Trigger flow:** +Design: [master/agent spec](docs/superpowers/specs/2026-10-12-master-agent-design.md). +Phase 2 plan: [jobs and manual update](docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md). -1. Gitea workflow builds and pushes a new image to your registry. -2. Workflow calls `POST /update` on this service with the image name. -3. Service finds the matching Compose-managed container(s) on the host via Docker labels. -4. Runs `docker compose pull` + `up -d` for the relevant service(s). +> **Webhook:** the old `POST /update` endpoint is gone. It returns in phase 3 on the +> master; until then the Gitea action in `gitea-action/` has nothing to call. -See [design spec](docs/superpowers/specs/2026-05-22-package-updater-design.md) and [implementation plan](docs/superpowers/plans/2026-05-22-package-updater-implementation.md) for full design and rationale. +## What an update does -## How it finds the right stack +Clicking *Update* on a service re-pulls the service's **current tag** (`repo:tag`). The +agent asks its Docker daemon for the tag's digest and deploys `repo:tag@digest`, so what +runs is exactly what was pulled. The log streams to the browser while it runs; job +history is kept in Postgres. A service without a tag (`repo@digest`), a plain container, +and the updater's own services cannot be updated, and the UI says why. One update at a +time per service, and one job at a time per agent. -The service queries the Docker socket and reads the labels Compose itself attaches to every container: +## Deploying the master -- `com.docker.compose.project` -- `com.docker.compose.service` -- `com.docker.compose.project.working_dir` -- `com.docker.compose.project.config_files` +Use [`deploy/master.stack.yml`](deploy/master.stack.yml) as a starting point (Swarm stack; +needs Postgres and an OIDC provider). The master listens for the UI on `PORT` (default +`8080`, put it behind your reverse proxy) and for agents on `AGENT_PORT` (default `8090`, +pinned TLS, reached directly on the internal network). -A container is eligible for update only if it has **both**: +## Adding agents -- An image name matching the request (tag-agnostic), AND -- The opt-in label `se.shcizo.auto-update=true`. - -Defense in depth: a valid bearer token AND the opt-in label must both be present before any container is touched. - -## Swarm mode - -Set `MODE=swarm` to update Docker Swarm services instead of Compose stacks. The -service runs `docker service update --image` via the Docker API instead of -shelling out to `docker compose`. - -Two things to get right when running in Swarm mode: - -- **The opt-in label goes on the service, not the container/task.** Swarm mode - reads `Service.Spec.Labels`, so add it with - `docker service update --label-add se.shcizo.auto-update=true ` or - set it under `deploy.labels` in the stack file — a plain `labels:` entry on - the service (container-level) is not visible to Swarm mode's discovery. -- **The updater must talk to a Swarm manager.** `docker service update` - requires manager API access, so either point `DOCKER_HOST` at a manager node - or schedule the updater container itself on a manager with the socket - mounted. This is a deployment concern the service cannot detect or work - around. - -## Master/agent (in progress) - -A master with one agent per machine is replacing the single-instance webhook -model; see the [design spec](docs/superpowers/specs/2026-10-12-master-agent-design.md). -Phase 1 delivers a live inventory: every Compose stack, Swarm service and -container on every machine, behind Authentik login. Updates still go through -the existing `/update` webhook until phase 3. - -- Master: `updater-master` in the same image, example stack in `deploy/master.stack.yml`. - At startup it does OIDC discovery against `OIDC_ISSUER` and exits if the issuer - is unreachable; Swarm then restarts it. -- Agents: add one per machine on the master's **Agents** page; it shows a - ready-to-run compose file with the agent's token and the master's key - fingerprint. - -## Quick start - -1. Build and push the image (e.g. via your own CI). -2. Copy `docker-compose.example.yml` to `/home/shcizo/self-hosted/package-updater/docker-compose.yml`. -3. Create `.env` next to it: `UPDATER_API_KEY=$(openssl rand -hex 32)`. -4. Point your reverse proxy (NPM/Traefik/Caddy) at `package-updater:8080`. NPM should handle TLS. -5. `docker compose up -d`. -6. Add the opt-in label `se.shcizo.auto-update: "true"` to each service you want auto-updated. -7. Use the [Gitea composite action](gitea-action/README.md) in your repos to call `/update` after a build. - -## Configuration - -All via environment variables. +Open the master's **Agents** page and add one. It shows a ready-to-run compose file with +the agent's token (shown once) and the master's key fingerprint; the same layout is in +[`deploy/agent.compose.yml`](deploy/agent.compose.yml). | Variable | Required | Default | Purpose | |---|---|---|---| -| `UPDATER_API_KEY` | **yes** | — | Bearer token. Service refuses to start without it. | -| `STACKS_ROOT` | no | `/home/shcizo/self-hosted` | Required parent for any stack eligible to update. | -| `PORT` | no | `8080` | HTTP listen port. | -| `LOG_LEVEL` | no | `info` | `debug` / `info` / `warn` / `error`. | +| `MASTER_URL` | yes | — | `wss://host:port/agent` | +| `MASTER_KEY` | yes | — | Master key fingerprint (`sha256:...`) shown in the UI. | +| `AGENT_TOKEN` | yes | — | Token shown once when the agent was added. | +| `SELF_ID` | no | hostname | Container ID or name of the agent itself; used to read its mounts. | | `UPDATE_TIMEOUT` | no | `5m` | Per-job timeout (Go duration). | -| `OPT_IN_LABEL` | no | `se.shcizo.auto-update` | Label name to check; value must equal `"true"`. | -| `MODE` | no | `compose` | `compose` or `swarm`. Selects the update mechanism for the whole deployment; not mixed per-request. | +| `DOCKER_CONFIG` | no | — | Directory with a `config.json` holding registry credentials. Only inline `auths` entries work; `credsStore`/`credHelpers` are not available in the container. | +| `LOG_LEVEL` | no | `info` | `debug` / `info` / `warn` / `error`. | -## Endpoints +### Mounting stack directories -| Endpoint | Auth | Purpose | -|---|---|---| -| `POST /update` | Bearer token | Trigger pull + restart for matching services | -| `GET /healthz` | none | Liveness + Docker socket reachability | -| `GET /version` | none | Build info | -| `GET /metrics` | none | Prometheus exposition | +There is no `STACKS_ROOT`. The agent may only touch Compose stacks in directories that +are bind-mounted into its own container **read-write at the same path as on the host** +(Compose labels hold host paths). Add one line per directory: -`/healthz`, `/version`, and `/metrics` are intentionally unauthenticated — they're internal-network only behind the reverse proxy. +```yaml +volumes: + - /var/run/docker.sock:/var/run/docker.sock + - /home/shcizo/self-hosted:/home/shcizo/self-hosted +``` -## Observability +Read-only mounts and mounts whose source differs from the destination are refused with a +clear message. On Docker Desktop and OrbStack bind sources are translated host paths, so +Compose jobs fail there; that is a development-only limitation. -- **Logs**: JSON to stdout, picked up by Promtail/Alloy → Loki. -- **Metrics**: Prometheus exposition on `/metrics`. Notable: `package_updater_update_jobs_total{project,service,status}`, `package_updater_last_update_timestamp{project,service}`, `package_updater_docker_ping_up`. +## How Compose updates work, and their limits + +The agent never edits your files. It writes one file next to the stack's base file, +`compose.override.yaml`, with a header line and an `image:` entry per updated service +(`repo:tag@digest`), then runs `docker compose pull` and `up -d` for that service with +the override passed explicitly. + +- The agent only writes `compose.override.yaml` if it is absent or starts with its header + line, and **refuses the job** if any other auto-loaded override exists + (`compose.override.yml`, `docker-compose.override.yml`, `docker-compose.override.yaml`): + Compose loads only one of them, so ours would silently disable yours. +- Stacks you start with explicit `-f` files do not load the override on a manual + `docker compose up`; the agent's own jobs always pass it. +- **The override is never pruned.** If you delete a service from the stack file, its + `image:` entry stays in `compose.override.yaml`, and a manual `docker compose up -d` + would recreate it. Remove the entry by hand. + +**Check before enabling.** List the config files of every running stack to spot existing +override files and stacks started with explicit `-f`: + +```bash +docker ps --format '{{.Label "com.docker.compose.project.config_files"}}' | sort -u +``` + +## Swarm updates + +The agent updates the service with `ServiceUpdate` and the pinned image. "Succeeded" +means the daemon accepted the new spec, not that the tasks converged; the inventory shows +`degraded` until they do. The agent doing the update must talk to a manager node (run it +on a manager, or point `DOCKER_HOST` at one). + +## Upgrading from the single-instance updater + +This release removes `cmd/server` and changes the image's default entrypoint to +`updater-agent`. A running old-style updater (default entrypoint, `UPDATER_API_KEY`) will +crash-loop the next time its image is pulled. Pin it to the last pre-phase-2 tag first. +Master and agents must be on this release together: a phase-1 agent connects but every +job sent to it fails "agent too old for updates". ## Development ```bash go test ./... -go build ./cmd/server +go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/ +go build ./cmd/master ./cmd/agent docker build -t package-updater:dev . +DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/ ``` - -## Known v1 gaps - -These are tracked in the spec's section 2 and section 15 as deliberate out-of-scope: - -- **Self-update wiring**: `internal/selfupdate.Wrapped` exists and is unit-tested but is not wired into the live queue. The HTTP response flush ordering for self-replacement is a future enhancement; for now, expect to manually rerun `docker compose up -d` on the host if pushing a new image of `package-updater` itself causes a mid-response interruption. -- **No rollback**: Compose's "keep old container if new fails to start" is the only safety net. -- **Single host only in Compose mode**. Swarm mode (`MODE=swarm`) is the - multi-node path, but only from a manager node's point of view — the updater - itself still needs manager API access (see "Swarm mode" above). -- **No per-repo API keys**: a single shared bearer token is used. diff --git a/deploy/master.stack.yml b/deploy/master.stack.yml index e4aaa63..e4f52e7 100644 --- a/deploy/master.stack.yml +++ b/deploy/master.stack.yml @@ -1,4 +1,4 @@ -# Example Swarm stack for the package-updater master (phase 1: inventory only). +# Example Swarm stack for the package-updater master (phase 2: inventory and manual updates). # Deploy on the manager: docker stack deploy -c master.stack.yml package-updater-master # # Before deploying: