docs: master/agent phase 2 — jobs and manual update (#7)

CLAUDE.md

@@ -1,104 +1,111 @@
 # package-updater — Claude context
 
-Webhook-driven Docker Compose updater. Go service, single binary, deployed as
-a container with `/var/run/docker.sock` mounted.
+Master/agent fleet updater. A master (UI behind OIDC, Postgres) shows every Compose
+stack, Swarm service and container on every machine and starts updates; one agent per
+machine (`/var/run/docker.sock` mounted) reports its inventory and runs the jobs.
+Both are in one image: `updater-master` and `updater-agent`.
 
 See `README.md` for user-facing docs. This file is for working ON the code.
 
 ## Commands
 
 ```bash
-go test ./...                          # all tests, fast (no docker required)
-go test -run TestFoo ./internal/api    # single test
-go build ./cmd/server                  # produces ./server
-go build ./cmd/master ./cmd/agent      # new master/agent binaries (phase 1: inventory)
-DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/   # Postgres tests
+go test ./...                          # all tests, fast (no docker, no postgres)
+go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
+go test -run TestFoo ./internal/master/hub    # single test
+go build ./cmd/master ./cmd/agent      # produce ./master and ./agent
 docker build -t package-updater:dev .  # multi-stage, builds golang:1.26-alpine
 
-# Local smoke test (builds from Dockerfile, runs against /var/run/docker.sock):
-cp env.sample .env && $EDITOR .env     # fill UPDATER_API_KEY
-docker compose -f docker-compose.local.yml up --build
-curl -sH "Authorization: Bearer $UPDATER_API_KEY" \
-  -d '{"image":"foo"}' http://localhost:8080/update | jq
+# Postgres tests (store), against any throwaway database:
+DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
 ```
 
 ## Architecture (one-line per package)
 
-- `cmd/server`        — wiring only: config → docker client → discovery → queue → http
-- `internal/api`      — HTTP handlers, bearer-token auth, request-id + access-log middleware
-- `internal/config`   — env-var loading; fails fast if `UPDATER_API_KEY` missing
-- `internal/discovery`— given an image, return Compose `Job`s to run (label parsing, path check, dedup)
-- `internal/discovery/swarm.go`  — SwarmDiscovery: same FindJobs signature, lists `docker service ls`, gates on service-level opt-in label instead of STACKS_ROOT path-check
-- `internal/updater`  — FIFO queue + single worker + `docker compose` subprocess executor
-- `internal/updater/swarm_executor.go` — SwarmExecutor: `docker service update --image` via the Docker API instead of a `docker compose` subprocess
-- `internal/selfupdate` — flush-then-exec wrapper for updating ourselves (NOT wired in live, see gotcha)
-- `internal/metrics`  — Prometheus collectors
+- `cmd/master`        — wiring only: config → store → hub → jobs → web, two listeners (UI, agents over pinned TLS)
+- `cmd/agent`         — wiring only: config → docker client → inventory → job runner → master connection
+- `internal/protocol` — the WebSocket messages both sides speak (`Version`, `JobStart`, `JobResult`, ...)
+- `internal/pin`      — master key pair, certificate and fingerprint pinning
+- `internal/agenttoken` — agent token generation and hashing
+- `internal/imageref` — image reference parsing (repo, tag, digest)
 - `internal/logging`  — slog JSON, request-id context propagation
-- `cmd/master`, `cmd/agent` — master/agent architecture (spec `2026-10-12-master-agent-design.md`), phase 1 = inventory only. `cmd/server` keeps doing updates until phase 3.
-- `internal/protocol`  — agent↔master WebSocket messages
-- `internal/pin`       — master key pair, self-signed cert, fingerprint-pinned TLS client
-- `internal/agenttoken`— per-agent tokens (`pua_` prefix, only the SHA-256 hash is stored)
-- `internal/imageref`  — split `repo:tag@digest`
-- `internal/agent/...` — inventory collection, Docker event debounce, connection loop
-- `internal/master/...`— Postgres store, agent hub, change broker, OIDC auth, htmx UI
+- `internal/agent/config`    — env-var loading; fails fast on missing `MASTER_URL`/`MASTER_KEY`/`AGENT_TOKEN`
+- `internal/agent/inventory` — reads Compose, Swarm and plain containers from the Docker daemon
+- `internal/agent/conn`      — dial, hello, snapshot, job messages, reconnect with backoff; one writer goroutine per session
+- `internal/agent/updater`   — job `Runner` (single FIFO worker, outbox until acked), Compose and Swarm executors, override file, mount-root check, registry auth
+- `internal/master/config`   — env-var loading
+- `internal/master/store`    — Postgres access and migrations
+- `internal/master/hub`      — agent connections: auth, per-connection writer, routing, job reconcile on hello
+- `internal/master/jobs`     — decides whether a service may be updated and starts a job on the right agent
+- `internal/master/web`, `webauth` — HTMX UI behind OIDC login, SSE signals
+- `internal/master/events`   — in-process publish/subscribe for UI refresh signals
+- `internal/master/model`    — domain types shared by the above
 
 ## Design intent (do not break without discussion)
 
-- **Single FIFO worker by design** (`internal/updater/queue.go`). Spec §5.7. Never
-  parallelise the queue — two `docker compose` calls against the same stack race.
-- **Defense in depth: token AND opt-in label AND STACKS_ROOT prefix** must all
-  hold before a container is touched. Weakening any of these breaks the security
-  model — discuss before changing.
-- **Auth on `/update` only**. `/healthz`, `/metrics`, `/version` are intentionally
-  unauthenticated (internal network, scraper/healthcheck need them). See
-  `cmd/server/main.go` `routeAuth`.
-- **`metrics *Metrics` parameters may be nil**; constructors and call sites check.
-  Tests rely on this — don't drop the nil-check.
-- **Stateless**: no DB, no config file, no on-disk audit log. Docker daemon is
-  the source of truth.
-- **Compose mode and Swarm mode are selected once per deployment via `MODE`**, never
-  mixed at request time. Swarm mode's security gate is opt-in label only — there is
-  no STACKS_ROOT-equivalent path check, since Swarm services have no local compose
-  file. Don't add one; don't weaken Compose mode's three-factor gate to match.
+- **Single FIFO worker per agent by design** (`internal/agent/updater/runner.go`).
+  Two `docker compose` calls against the same stack race.
+- **One queued or running job per service**, enforced by the unique index
+  `update_jobs_one_active` (not by application code).
+- **The agent writes exactly one file**, `<stack dir>/compose.override.yaml`, and only
+  if it is absent or starts with the header line. It refuses the job when any other
+  auto-loaded override name exists.
+- **Every config file and the override path must be inside a read-write bind mount of
+  the agent's own container whose source equals its destination.** Mounts are read per
+  job (`ContainerInspect(SELF_ID)`) and checked after symlink resolution. This replaces
+  the old `STACKS_ROOT`.
+- **Self-protection on the master**: a service is refused when its image repo equals the
+  repo of `AGENT_IMAGE` or the row has the `self` flag. The agent does not re-check.
+- **Unknown message types are logged and ignored; malformed messages close the connection.**
+- **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not
+  list in `known_jobs`, and only then accepts jobs for that connection.
+- **Auth**: agents by token over pinned TLS; people by OIDC. Every state-changing UI
+  request is a POST with a CSRF token.
 
 ## Gotchas
 
-- **`internal/selfupdate.Wrapped` is implemented and unit-tested but NOT wired
-  into the live queue.** Spec §15 calls this out as a v1 gap. If you "fix" this,
-  read the package doc — flush-before-exec ordering is subtle.
-- **Image matching is case-sensitive and tag/digest-agnostic.** The tricky case
-  is `localhost:5000/foo:v1` where the first colon is a port, not a tag. See
-  `internal/discovery/matching.go` `NormaliseImage`.
+- **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where
+  the first colon is a port, not a tag. See `internal/imageref`.
 - **Path safety uses `filepath.Rel` + ".." prefix check**, NOT `strings.HasPrefix`.
-  Prevents the `/foo-evil` vs `/foo` confusion. See `internal/discovery/pathcheck.go`.
-- **`STACKS_ROOT` defaults to `/home/shcizo/self-hosted`** in prod but `env.sample`
-  suggests `/tmp` for smoke tests. The default in `config.go` is the prod value
-  — set explicitly in test/dev envs.
+  Prevents the `/foo-evil` vs `/foo` confusion. See `internal/agent/updater/pathcheck.go`.
 - **Go 1.26.3** (`go.mod`). Dockerfile pins `golang:1.26-alpine`. Bumping one
   without the other has caused a fix commit already.
 - **`<summary>` C# convention from global CLAUDE.md does not apply here** — this
   is Go. Use idiomatic GoDoc (`// FuncName does X.`).
-- **Swarm mode requires manager-node API access.** `docker service update` fails
-  with a permission error against a worker-only node. This is an operator/deployment
-  concern (point `DOCKER_HOST` at a manager, or schedule the updater on a manager),
-  not something the code can detect or work around.
-- **WebSocket read limit is raised to 1 MiB on both sides.** coder/websocket defaults to 32 KiB, which a full snapshot from a busy machine exceeds — symptom is an agent stuck in a reconnect loop.
-- **The master's key lives in Postgres (`master_identity`).** Wiping the database means every agent needs a new `MASTER_KEY`.
-- **SSE responses must flush through every wrapper.** `web.events` uses `http.ResponseController`; a middleware that wraps `ResponseWriter` without `Unwrap()` silently breaks live updates. Nginx needs `X-Accel-Buffering: no` (set by the handler).
-- **The "stateless" and "label gate" design intents above describe `cmd/server` only.** The master is stateful by design and gates updates per service in the UI (spec 2026-10-12).
-- **The master does OIDC discovery once at startup and exits if the issuer is unreachable.** It does not retry; Swarm restarts the container, so a short Authentik outage during deploy resolves itself.
+- **Compose loads only one override file.** Of `compose.override.yml`,
+  `compose.override.yaml`, `docker-compose.override.yml`, `docker-compose.override.yaml`
+  only the first that exists is read. Hence the refusal described above.
+- **Stacks started with explicit `-f` do not load the override on a manual `up`.** The
+  agent's own jobs always pass it explicitly.
+- **Override entries are never pruned.** A service deleted from the stack file keeps its
+  `image:` line in `compose.override.yaml`.
+- **There is no `STACKS_ROOT`.** Stack directories must be mounted read-write at the host
+  path (`/path:/path`), and `SELF_ID` must resolve to the agent's own container. On Docker
+  Desktop/OrbStack bind sources are translated, so Compose jobs fail there.
+- **`job.start` carries no paths.** The agent reads working dir and config files from its
+  own Docker daemon at execution time.
+- **`protocol.Service.ConfigFiles` is the raw comma-joined label string**, so `Service`
+  stays comparable with `==` (`inventory.Diff` depends on it).
+- **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged.
+- **Swarm updates need manager-node API access** for the agent doing the update.
+- **The image's default entrypoint is `updater-agent`.** The master overrides it with
+  `updater-master`. There is no `POST /update` until phase 3.
+- **Job output is capped at `protocol.MaxJobOutput` (128 KiB, tail).** NUL bytes are
+  stripped by the hub because Postgres text rejects them.
 
 ## References
 
-- Design spec: `docs/superpowers/specs/2026-05-22-package-updater-design.md` (489 lines, authoritative)
-- Implementation plan: `docs/superpowers/plans/2026-05-22-package-updater-implementation.md`
-- Consumer-side CI integration: `gitea-action/`
-- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md`
+- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md` (authoritative)
 - Phase 1 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md`
+- Phase 2 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md`
+- Original single-instance design (historical, its webhook is gone until phase 3):
+  `docs/superpowers/specs/2026-05-22-package-updater-design.md`
+- Consumer-side CI integration: `gitea-action/` (calls `POST /update`, which no longer
+  exists until phase 3)
 
 ## Conventions
 
 - Conventional Commits (`feat:`, `fix:`, `docs:`, `ci:`, `build:`, `chore:`).
 - One package = one responsibility; interfaces defined at consumer site
-  (`api.Finder`, `api.Submitter`, `api.Pinger`) for testability.
+  (`jobs.Store`, `jobs.Hub`, `conn.Jobs`, `hub.Store`) for testability.
 - Table-driven tests with `stretchr/testify`. No mocks beyond hand-written fakes.

README.md

@@ -1,121 +1,110 @@
 # package-updater
 
-Webhook-driven Docker Compose service updater. Fills the gap between Watchtower (polling, no CI integration) and full GitOps (Argo CD, Flux) for a self-hosted, single-host environment.
+Updates self-hosted Docker Compose and Docker Swarm services from one place. A
+**master** (web UI behind OIDC login, state in Postgres) shows every Compose stack,
+Swarm service and container on every machine, and starts updates when someone clicks
+*Update*. One **agent** per machine reports its inventory and runs the jobs. Both are in
+the same image: `updater-master` and `updater-agent` (the default entrypoint).
 
-**Trigger flow:**
+Design: [master/agent spec](docs/superpowers/specs/2026-10-12-master-agent-design.md).
+Phase 2 plan: [jobs and manual update](docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md).
 
-1. Gitea workflow builds and pushes a new image to your registry.
-2. Workflow calls `POST /update` on this service with the image name.
-3. Service finds the matching Compose-managed container(s) on the host via Docker labels.
-4. Runs `docker compose pull` + `up -d` for the relevant service(s).
+> **Webhook:** the old `POST /update` endpoint is gone. It returns in phase 3 on the
+> master; until then the Gitea action in `gitea-action/` has nothing to call.
 
-See [design spec](docs/superpowers/specs/2026-05-22-package-updater-design.md) and [implementation plan](docs/superpowers/plans/2026-05-22-package-updater-implementation.md) for full design and rationale.
+## What an update does
 
-## How it finds the right stack
+Clicking *Update* on a service re-pulls the service's **current tag** (`repo:tag`). The
+agent asks its Docker daemon for the tag's digest and deploys `repo:tag@digest`, so what
+runs is exactly what was pulled. The log streams to the browser while it runs; job
+history is kept in Postgres. A service without a tag (`repo@digest`), a plain container,
+and the updater's own services cannot be updated, and the UI says why. One update at a
+time per service, and one job at a time per agent.
 
-The service queries the Docker socket and reads the labels Compose itself attaches to every container:
+## Deploying the master
 
-- `com.docker.compose.project`
-- `com.docker.compose.service`
-- `com.docker.compose.project.working_dir`
-- `com.docker.compose.project.config_files`
+Use [`deploy/master.stack.yml`](deploy/master.stack.yml) as a starting point (Swarm stack;
+needs Postgres and an OIDC provider). The master listens for the UI on `PORT` (default
+`8080`, put it behind your reverse proxy) and for agents on `AGENT_PORT` (default `8090`,
+pinned TLS, reached directly on the internal network).
 
-A container is eligible for update only if it has **both**:
+## Adding agents
 
-- An image name matching the request (tag-agnostic), AND
-- The opt-in label `se.shcizo.auto-update=true`.
-
-Defense in depth: a valid bearer token AND the opt-in label must both be present before any container is touched.
-
-## Swarm mode
-
-Set `MODE=swarm` to update Docker Swarm services instead of Compose stacks. The
-service runs `docker service update --image` via the Docker API instead of
-shelling out to `docker compose`.
-
-Two things to get right when running in Swarm mode:
-
-- **The opt-in label goes on the service, not the container/task.** Swarm mode
-  reads `Service.Spec.Labels`, so add it with
-  `docker service update --label-add se.shcizo.auto-update=true <service>` or
-  set it under `deploy.labels` in the stack file — a plain `labels:` entry on
-  the service (container-level) is not visible to Swarm mode's discovery.
-- **The updater must talk to a Swarm manager.** `docker service update`
-  requires manager API access, so either point `DOCKER_HOST` at a manager node
-  or schedule the updater container itself on a manager with the socket
-  mounted. This is a deployment concern the service cannot detect or work
-  around.
-
-## Master/agent (in progress)
-
-A master with one agent per machine is replacing the single-instance webhook
-model; see the [design spec](docs/superpowers/specs/2026-10-12-master-agent-design.md).
-Phase 1 delivers a live inventory: every Compose stack, Swarm service and
-container on every machine, behind Authentik login. Updates still go through
-the existing `/update` webhook until phase 3.
-
-- Master: `updater-master` in the same image, example stack in `deploy/master.stack.yml`.
-  At startup it does OIDC discovery against `OIDC_ISSUER` and exits if the issuer
-  is unreachable; Swarm then restarts it.
-- Agents: add one per machine on the master's **Agents** page; it shows a
-  ready-to-run compose file with the agent's token and the master's key
-  fingerprint.
-
-## Quick start
-
-1. Build and push the image (e.g. via your own CI).
-2. Copy `docker-compose.example.yml` to `/home/shcizo/self-hosted/package-updater/docker-compose.yml`.
-3. Create `.env` next to it: `UPDATER_API_KEY=$(openssl rand -hex 32)`.
-4. Point your reverse proxy (NPM/Traefik/Caddy) at `package-updater:8080`. NPM should handle TLS.
-5. `docker compose up -d`.
-6. Add the opt-in label `se.shcizo.auto-update: "true"` to each service you want auto-updated.
-7. Use the [Gitea composite action](gitea-action/README.md) in your repos to call `/update` after a build.
-
-## Configuration
-
-All via environment variables.
+Open the master's **Agents** page and add one. It shows a ready-to-run compose file with
+the agent's token (shown once) and the master's key fingerprint; the same layout is in
+[`deploy/agent.compose.yml`](deploy/agent.compose.yml).
 
 | Variable | Required | Default | Purpose |
 |---|---|---|---|
-| `UPDATER_API_KEY` | **yes** | — | Bearer token. Service refuses to start without it. |
-| `STACKS_ROOT` | no | `/home/shcizo/self-hosted` | Required parent for any stack eligible to update. |
-| `PORT` | no | `8080` | HTTP listen port. |
-| `LOG_LEVEL` | no | `info` | `debug` / `info` / `warn` / `error`. |
+| `MASTER_URL` | yes | — | `wss://host:port/agent` |
+| `MASTER_KEY` | yes | — | Master key fingerprint (`sha256:...`) shown in the UI. |
+| `AGENT_TOKEN` | yes | — | Token shown once when the agent was added. |
+| `SELF_ID` | no | hostname | Container ID or name of the agent itself; used to read its mounts. |
 | `UPDATE_TIMEOUT` | no | `5m` | Per-job timeout (Go duration). |
-| `OPT_IN_LABEL` | no | `se.shcizo.auto-update` | Label name to check; value must equal `"true"`. |
-| `MODE` | no | `compose` | `compose` or `swarm`. Selects the update mechanism for the whole deployment; not mixed per-request. |
+| `DOCKER_CONFIG` | no | — | Directory with a `config.json` holding registry credentials. Only inline `auths` entries work; `credsStore`/`credHelpers` are not available in the container. |
+| `LOG_LEVEL` | no | `info` | `debug` / `info` / `warn` / `error`. |
 
-## Endpoints
+### Mounting stack directories
 
-| Endpoint | Auth | Purpose |
-|---|---|---|
-| `POST /update` | Bearer token | Trigger pull + restart for matching services |
-| `GET /healthz` | none | Liveness + Docker socket reachability |
-| `GET /version` | none | Build info |
-| `GET /metrics` | none | Prometheus exposition |
+There is no `STACKS_ROOT`. The agent may only touch Compose stacks in directories that
+are bind-mounted into its own container **read-write at the same path as on the host**
+(Compose labels hold host paths). Add one line per directory:
 
-`/healthz`, `/version`, and `/metrics` are intentionally unauthenticated — they're internal-network only behind the reverse proxy.
+```yaml
+volumes:
+  - /var/run/docker.sock:/var/run/docker.sock
+  - /home/shcizo/self-hosted:/home/shcizo/self-hosted
+```
 
-## Observability
+Read-only mounts and mounts whose source differs from the destination are refused with a
+clear message. On Docker Desktop and OrbStack bind sources are translated host paths, so
+Compose jobs fail there; that is a development-only limitation.
 
-- **Logs**: JSON to stdout, picked up by Promtail/Alloy → Loki.
-- **Metrics**: Prometheus exposition on `/metrics`. Notable: `package_updater_update_jobs_total{project,service,status}`, `package_updater_last_update_timestamp{project,service}`, `package_updater_docker_ping_up`.
+## How Compose updates work, and their limits
+
+The agent never edits your files. It writes one file next to the stack's base file,
+`compose.override.yaml`, with a header line and an `image:` entry per updated service
+(`repo:tag@digest`), then runs `docker compose pull` and `up -d` for that service with
+the override passed explicitly.
+
+- The agent only writes `compose.override.yaml` if it is absent or starts with its header
+  line, and **refuses the job** if any other auto-loaded override exists
+  (`compose.override.yml`, `docker-compose.override.yml`, `docker-compose.override.yaml`):
+  Compose loads only one of them, so ours would silently disable yours.
+- Stacks you start with explicit `-f` files do not load the override on a manual
+  `docker compose up`; the agent's own jobs always pass it.
+- **The override is never pruned.** If you delete a service from the stack file, its
+  `image:` entry stays in `compose.override.yaml`, and a manual `docker compose up -d`
+  would recreate it. Remove the entry by hand.
+
+**Check before enabling.** List the config files of every running stack to spot existing
+override files and stacks started with explicit `-f`:
+
+```bash
+docker ps --format '{{.Label "com.docker.compose.project.config_files"}}' | sort -u
+```
+
+## Swarm updates
+
+The agent updates the service with `ServiceUpdate` and the pinned image. "Succeeded"
+means the daemon accepted the new spec, not that the tasks converged; the inventory shows
+`degraded` until they do. The agent doing the update must talk to a manager node (run it
+on a manager, or point `DOCKER_HOST` at one).
+
+## Upgrading from the single-instance updater
+
+This release removes `cmd/server` and changes the image's default entrypoint to
+`updater-agent`. A running old-style updater (default entrypoint, `UPDATER_API_KEY`) will
+crash-loop the next time its image is pulled. Pin it to the last pre-phase-2 tag first.
+Master and agents must be on this release together: a phase-1 agent connects but every
+job sent to it fails "agent too old for updates".
 
 ## Development
 
 ```bash
 go test ./...
-go build ./cmd/server
+go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
+go build ./cmd/master ./cmd/agent
 docker build -t package-updater:dev .
+DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
 ```
-
-## Known v1 gaps
-
-These are tracked in the spec's section 2 and section 15 as deliberate out-of-scope:
-
-- **Self-update wiring**: `internal/selfupdate.Wrapped` exists and is unit-tested but is not wired into the live queue. The HTTP response flush ordering for self-replacement is a future enhancement; for now, expect to manually rerun `docker compose up -d` on the host if pushing a new image of `package-updater` itself causes a mid-response interruption.
-- **No rollback**: Compose's "keep old container if new fails to start" is the only safety net.
-- **Single host only in Compose mode**. Swarm mode (`MODE=swarm`) is the
-  multi-node path, but only from a manager node's point of view — the updater
-  itself still needs manager API access (see "Swarm mode" above).
-- **No per-repo API keys**: a single shared bearer token is used.

deploy/master.stack.yml

@@ -1,4 +1,4 @@
-# Example Swarm stack for the package-updater master (phase 1: inventory only).
+# Example Swarm stack for the package-updater master (phase 2: inventory and manual updates).
 # Deploy on the manager: docker stack deploy -c master.stack.yml package-updater-master
 #
 # Before deploying: