docs: master/agent phase 2 — jobs and manual update (#7)

CLAUDE.md

@@ -1,104 +1,111 @@
 # package-updater — Claude context
 
-Webhook-driven Docker Compose updater. Go service, single binary, deployed as
-a container with `/var/run/docker.sock` mounted.
+Master/agent fleet updater. A master (UI behind OIDC, Postgres) shows every Compose
+stack, Swarm service and container on every machine and starts updates; one agent per
+machine (`/var/run/docker.sock` mounted) reports its inventory and runs the jobs.
+Both are in one image: `updater-master` and `updater-agent`.
 
 See `README.md` for user-facing docs. This file is for working ON the code.
 
 ## Commands
 
 ```bash
-go test ./...                          # all tests, fast (no docker required)
-go test -run TestFoo ./internal/api    # single test
-go build ./cmd/server                  # produces ./server
-go build ./cmd/master ./cmd/agent      # new master/agent binaries (phase 1: inventory)
-DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/   # Postgres tests
+go test ./...                          # all tests, fast (no docker, no postgres)
+go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
+go test -run TestFoo ./internal/master/hub    # single test
+go build ./cmd/master ./cmd/agent      # produce ./master and ./agent
 docker build -t package-updater:dev .  # multi-stage, builds golang:1.26-alpine
 
-# Local smoke test (builds from Dockerfile, runs against /var/run/docker.sock):
-cp env.sample .env && $EDITOR .env     # fill UPDATER_API_KEY
-docker compose -f docker-compose.local.yml up --build
-curl -sH "Authorization: Bearer $UPDATER_API_KEY" \
-  -d '{"image":"foo"}' http://localhost:8080/update | jq
+# Postgres tests (store), against any throwaway database:
+DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
 ```
 
 ## Architecture (one-line per package)
 
-- `cmd/server`        — wiring only: config → docker client → discovery → queue → http
-- `internal/api`      — HTTP handlers, bearer-token auth, request-id + access-log middleware
-- `internal/config`   — env-var loading; fails fast if `UPDATER_API_KEY` missing
-- `internal/discovery`— given an image, return Compose `Job`s to run (label parsing, path check, dedup)
-- `internal/discovery/swarm.go`  — SwarmDiscovery: same FindJobs signature, lists `docker service ls`, gates on service-level opt-in label instead of STACKS_ROOT path-check
-- `internal/updater`  — FIFO queue + single worker + `docker compose` subprocess executor
-- `internal/updater/swarm_executor.go` — SwarmExecutor: `docker service update --image` via the Docker API instead of a `docker compose` subprocess
-- `internal/selfupdate` — flush-then-exec wrapper for updating ourselves (NOT wired in live, see gotcha)
-- `internal/metrics`  — Prometheus collectors
+- `cmd/master`        — wiring only: config → store → hub → jobs → web, two listeners (UI, agents over pinned TLS)
+- `cmd/agent`         — wiring only: config → docker client → inventory → job runner → master connection
+- `internal/protocol` — the WebSocket messages both sides speak (`Version`, `JobStart`, `JobResult`, ...)
+- `internal/pin`      — master key pair, certificate and fingerprint pinning
+- `internal/agenttoken` — agent token generation and hashing
+- `internal/imageref` — image reference parsing (repo, tag, digest)
 - `internal/logging`  — slog JSON, request-id context propagation
-- `cmd/master`, `cmd/agent` — master/agent architecture (spec `2026-10-12-master-agent-design.md`), phase 1 = inventory only. `cmd/server` keeps doing updates until phase 3.
-- `internal/protocol`  — agent↔master WebSocket messages
-- `internal/pin`       — master key pair, self-signed cert, fingerprint-pinned TLS client
-- `internal/agenttoken`— per-agent tokens (`pua_` prefix, only the SHA-256 hash is stored)
-- `internal/imageref`  — split `repo:tag@digest`
-- `internal/agent/...` — inventory collection, Docker event debounce, connection loop
-- `internal/master/...`— Postgres store, agent hub, change broker, OIDC auth, htmx UI
+- `internal/agent/config`    — env-var loading; fails fast on missing `MASTER_URL`/`MASTER_KEY`/`AGENT_TOKEN`
+- `internal/agent/inventory` — reads Compose, Swarm and plain containers from the Docker daemon
+- `internal/agent/conn`      — dial, hello, snapshot, job messages, reconnect with backoff; one writer goroutine per session
+- `internal/agent/updater`   — job `Runner` (single FIFO worker, outbox until acked), Compose and Swarm executors, override file, mount-root check, registry auth
+- `internal/master/config`   — env-var loading
+- `internal/master/store`    — Postgres access and migrations
+- `internal/master/hub`      — agent connections: auth, per-connection writer, routing, job reconcile on hello
+- `internal/master/jobs`     — decides whether a service may be updated and starts a job on the right agent
+- `internal/master/web`, `webauth` — HTMX UI behind OIDC login, SSE signals
+- `internal/master/events`   — in-process publish/subscribe for UI refresh signals
+- `internal/master/model`    — domain types shared by the above
 
 ## Design intent (do not break without discussion)
 
-- **Single FIFO worker by design** (`internal/updater/queue.go`). Spec §5.7. Never
-  parallelise the queue — two `docker compose` calls against the same stack race.
-- **Defense in depth: token AND opt-in label AND STACKS_ROOT prefix** must all
-  hold before a container is touched. Weakening any of these breaks the security
-  model — discuss before changing.
-- **Auth on `/update` only**. `/healthz`, `/metrics`, `/version` are intentionally
-  unauthenticated (internal network, scraper/healthcheck need them). See
-  `cmd/server/main.go` `routeAuth`.
-- **`metrics *Metrics` parameters may be nil**; constructors and call sites check.
-  Tests rely on this — don't drop the nil-check.
-- **Stateless**: no DB, no config file, no on-disk audit log. Docker daemon is
-  the source of truth.
-- **Compose mode and Swarm mode are selected once per deployment via `MODE`**, never
-  mixed at request time. Swarm mode's security gate is opt-in label only — there is
-  no STACKS_ROOT-equivalent path check, since Swarm services have no local compose
-  file. Don't add one; don't weaken Compose mode's three-factor gate to match.
+- **Single FIFO worker per agent by design** (`internal/agent/updater/runner.go`).
+  Two `docker compose` calls against the same stack race.
+- **One queued or running job per service**, enforced by the unique index
+  `update_jobs_one_active` (not by application code).
+- **The agent writes exactly one file**, `<stack dir>/compose.override.yaml`, and only
+  if it is absent or starts with the header line. It refuses the job when any other
+  auto-loaded override name exists.
+- **Every config file and the override path must be inside a read-write bind mount of
+  the agent's own container whose source equals its destination.** Mounts are read per
+  job (`ContainerInspect(SELF_ID)`) and checked after symlink resolution. This replaces
+  the old `STACKS_ROOT`.
+- **Self-protection on the master**: a service is refused when its image repo equals the
+  repo of `AGENT_IMAGE` or the row has the `self` flag. The agent does not re-check.
+- **Unknown message types are logged and ignored; malformed messages close the connection.**
+- **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not
+  list in `known_jobs`, and only then accepts jobs for that connection.
+- **Auth**: agents by token over pinned TLS; people by OIDC. Every state-changing UI
+  request is a POST with a CSRF token.
 
 ## Gotchas
 
-- **`internal/selfupdate.Wrapped` is implemented and unit-tested but NOT wired
-  into the live queue.** Spec §15 calls this out as a v1 gap. If you "fix" this,
-  read the package doc — flush-before-exec ordering is subtle.
-- **Image matching is case-sensitive and tag/digest-agnostic.** The tricky case
-  is `localhost:5000/foo:v1` where the first colon is a port, not a tag. See
-  `internal/discovery/matching.go` `NormaliseImage`.
+- **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where
+  the first colon is a port, not a tag. See `internal/imageref`.
 - **Path safety uses `filepath.Rel` + ".." prefix check**, NOT `strings.HasPrefix`.
-  Prevents the `/foo-evil` vs `/foo` confusion. See `internal/discovery/pathcheck.go`.
-- **`STACKS_ROOT` defaults to `/home/shcizo/self-hosted`** in prod but `env.sample`
-  suggests `/tmp` for smoke tests. The default in `config.go` is the prod value
-  — set explicitly in test/dev envs.
+  Prevents the `/foo-evil` vs `/foo` confusion. See `internal/agent/updater/pathcheck.go`.
 - **Go 1.26.3** (`go.mod`). Dockerfile pins `golang:1.26-alpine`. Bumping one
   without the other has caused a fix commit already.
 - **`<summary>` C# convention from global CLAUDE.md does not apply here** — this
   is Go. Use idiomatic GoDoc (`// FuncName does X.`).
-- **Swarm mode requires manager-node API access.** `docker service update` fails
-  with a permission error against a worker-only node. This is an operator/deployment
-  concern (point `DOCKER_HOST` at a manager, or schedule the updater on a manager),
-  not something the code can detect or work around.
-- **WebSocket read limit is raised to 1 MiB on both sides.** coder/websocket defaults to 32 KiB, which a full snapshot from a busy machine exceeds — symptom is an agent stuck in a reconnect loop.
-- **The master's key lives in Postgres (`master_identity`).** Wiping the database means every agent needs a new `MASTER_KEY`.
-- **SSE responses must flush through every wrapper.** `web.events` uses `http.ResponseController`; a middleware that wraps `ResponseWriter` without `Unwrap()` silently breaks live updates. Nginx needs `X-Accel-Buffering: no` (set by the handler).
-- **The "stateless" and "label gate" design intents above describe `cmd/server` only.** The master is stateful by design and gates updates per service in the UI (spec 2026-10-12).
-- **The master does OIDC discovery once at startup and exits if the issuer is unreachable.** It does not retry; Swarm restarts the container, so a short Authentik outage during deploy resolves itself.
+- **Compose loads only one override file.** Of `compose.override.yml`,
+  `compose.override.yaml`, `docker-compose.override.yml`, `docker-compose.override.yaml`
+  only the first that exists is read. Hence the refusal described above.
+- **Stacks started with explicit `-f` do not load the override on a manual `up`.** The
+  agent's own jobs always pass it explicitly.
+- **Override entries are never pruned.** A service deleted from the stack file keeps its
+  `image:` line in `compose.override.yaml`.
+- **There is no `STACKS_ROOT`.** Stack directories must be mounted read-write at the host
+  path (`/path:/path`), and `SELF_ID` must resolve to the agent's own container. On Docker
+  Desktop/OrbStack bind sources are translated, so Compose jobs fail there.
+- **`job.start` carries no paths.** The agent reads working dir and config files from its
+  own Docker daemon at execution time.
+- **`protocol.Service.ConfigFiles` is the raw comma-joined label string**, so `Service`
+  stays comparable with `==` (`inventory.Diff` depends on it).
+- **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged.
+- **Swarm updates need manager-node API access** for the agent doing the update.
+- **The image's default entrypoint is `updater-agent`.** The master overrides it with
+  `updater-master`. There is no `POST /update` until phase 3.
+- **Job output is capped at `protocol.MaxJobOutput` (128 KiB, tail).** NUL bytes are
+  stripped by the hub because Postgres text rejects them.
 
 ## References
 
-- Design spec: `docs/superpowers/specs/2026-05-22-package-updater-design.md` (489 lines, authoritative)
-- Implementation plan: `docs/superpowers/plans/2026-05-22-package-updater-implementation.md`
-- Consumer-side CI integration: `gitea-action/`
-- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md`
+- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md` (authoritative)
 - Phase 1 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md`
+- Phase 2 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md`
+- Original single-instance design (historical, its webhook is gone until phase 3):
+  `docs/superpowers/specs/2026-05-22-package-updater-design.md`
+- Consumer-side CI integration: `gitea-action/` (calls `POST /update`, which no longer
+  exists until phase 3)
 
 ## Conventions
 
 - Conventional Commits (`feat:`, `fix:`, `docs:`, `ci:`, `build:`, `chore:`).
 - One package = one responsibility; interfaces defined at consumer site
-  (`api.Finder`, `api.Submitter`, `api.Pinger`) for testability.
+  (`jobs.Store`, `jobs.Hub`, `conn.Jobs`, `hub.Store`) for testability.
 - Table-driven tests with `stretchr/testify`. No mocks beyond hand-written fakes.