feat: master/agent phase 2 — jobs and manual update (#7) #20

2026-10-12T17:48:30Z

shcizo commented

2026-10-12 17:48:30 +00:00

Master/agent phase 2: click Update on a service in the master UI; the owning agent re-pulls the service's current tag, pins it as repo:tag@digest and redeploys, streaming its log live to the browser. Job history is kept in Postgres.

Closes #7

Production risk — read before merging or tagging a release. This PR removes cmd/server and changes the image's default ENTRYPOINT to updater-agent. If the currently running single-instance updater uses :latest, the default entrypoint and its own se.shcizo.auto-update=true label, the next time anything pulls :latest for it (its own /update webhook, Watchtower, Shepherd) it restarts as an agent without MASTER_URL and crash-loops, and /update (called by the Gitea action) disappears until phase 3. Before the next v* tag: pin the old updater to the last pre-phase-2 tag and remove its opt-in label, or do not release until phase 3 moves POST /update to the master. Master and agents must be on this release together; a phase-1 agent gets every job refused ("agent too old for updates").

Merge with squash. Two intermediate commits (e2b73df, c9c4e0c) do not build — parallel implementers shared one git index. Every later commit builds.

Summary

Agent (internal/agent/...)

internal/updater moved to internal/agent/updater: single-worker FIFO Runner with per-job UPDATE_TIMEOUT, results kept until the master's job.ack and resent after reconnect.
Compose: writes an agent-owned compose.override.yaml (header-marked; refuses if any other override name exists), then docker compose -p … --project-directory … [--env-file …] -f <config_files…> -f <override> pull <svc> + up -d <svc>, with a minimal environment (PATH, HOME, DOCKER_*) so the agent's env never leaks into stack interpolation.
Allowed stack locations come from the agent's own read-write same-path bind mounts (replaces STACKS_ROOT); every config file, env file and the override are checked after symlink resolution.
Services built from source (build:, detected via docker compose --profile '*' config) are reported in the inventory and refused at job time; a service missing from the config output fails closed.
Swarm: verifies the job's cluster ID against its own daemon, resolves the digest, inspects by name and updates by ID + version. Optional registry auth from a mounted config.json.
Protocol v2: envelope id, job.start / job.progress / job.result / job.ack, Hello.ProtocolVersion and known job IDs; unknown message types are logged and ignored.

Master (internal/master/...)

Migration 0002_jobs.sql: update_jobs (one active job per service via partial unique index), services.desired_ref, config_files, built.
Hub: one writer goroutine per connection, routing (Compose/containers only to the reporting agent; Swarm to a manager of the same cluster), reconcile on hello (unknown running jobs → "lost"), job progress/result/ack; an agent can only progress/finish its own jobs.
internal/master/jobs: self-protection (same image repo as AGENT_IMAGE, and rows flagged self), offline / too-old / no-tag / built-from-source refusals; a bare image (postgres) is treated as :latest.
Revoking an agent fails its active jobs; CreateJob refuses revoked agents under a FOR SHARE lock.

Web (internal/master/web)

Service page with Update (POST + CSRF, requested_by = OIDC subject) or the reason it is unavailable; Jobs list; job page with live log (SSE job-<id> plus a 2s poll while active, re-read from Postgres each time).
Agent snippet: same-path read-write stacks mount, UPDATE_TIMEOUT, optional DOCKER_CONFIG.

Removed: cmd/server, internal/api, internal/config, internal/discovery, internal/selfupdate, internal/metrics, old compose examples. Docs: README.md, CLAUDE.md, deploy/, and the plan docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md.

Known limitations (documented)

The override file is never pruned; a service removed from the stack file can be recreated by a manual docker compose up -d.
Stacks started with explicit -f do not load the override on a manual up.
Swarm jobs are routed on the cluster ID an agent reports (accepted residual risk: a leaked token can receive jobs and report false results, but cannot command a machine — the agent re-verifies the cluster against its own daemon).
An agent revoked mid-dispatch may still run the job; its result is ignored.
Swarm "succeeded" means the daemon accepted the spec, not that tasks converged.
Built in the inventory can be stale after edits to extends:/include:/.env files; the job-time check is always fresh.

Test plan

go vet ./... && go test -race -count=1 ./... (incl. internal/e2e: job flow, result resent after reconnect, master restart, agent restart → lost, phase-1 agent refused)
Store integration tests (-tags integration -race) against Postgres 17, incl. concurrent revoke vs. CreateJob
Mutation-checked tests for the Review Focus items, env allowlist, profiles, symlinks, revoke lock
docker build; image exits with "MASTER_URL is required"
Browser verification (Playwright) with Postgres, mock OIDC, master and agent container: login, add agent, Update → live log → succeeded without reload, polling stops, override written with header and digest, double-click gives one job, disabled reasons (build:, self, plain container), revoke flips an open job page to failed; no console errors besides a missing favicon; 375 and 1440 widths
Manual smoke test on a Linux host with a real stack (run the config_files check from the README first)
Swarm update on a real cluster

Master/agent phase 2: click **Update** on a service in the master UI; the owning agent re-pulls the service's current tag, pins it as `repo:tag@digest` and redeploys, streaming its log live to the browser. Job history is kept in Postgres. Closes #7 > **Production risk — read before merging or tagging a release.** This PR removes `cmd/server` and changes the image's default `ENTRYPOINT` to `updater-agent`. If the currently running single-instance updater uses `:latest`, the default entrypoint and its own `se.shcizo.auto-update=true` label, the next time anything pulls `:latest` for it (its own `/update` webhook, Watchtower, Shepherd) it restarts as an agent without `MASTER_URL` and crash-loops, and `/update` (called by the Gitea action) disappears until phase 3. Before the next `v*` tag: pin the old updater to the last pre-phase-2 tag **and** remove its opt-in label, or do not release until phase 3 moves `POST /update` to the master. Master and agents must be on this release together; a phase-1 agent gets every job refused ("agent too old for updates"). > > **Merge with squash.** Two intermediate commits (`e2b73df`, `c9c4e0c`) do not build — parallel implementers shared one git index. Every later commit builds. ## Summary **Agent** (`internal/agent/...`) - `internal/updater` moved to `internal/agent/updater`: single-worker FIFO `Runner` with per-job `UPDATE_TIMEOUT`, results kept until the master's `job.ack` and resent after reconnect. - Compose: writes an agent-owned `compose.override.yaml` (header-marked; refuses if any other override name exists), then `docker compose -p … --project-directory … [--env-file …] -f <config_files…> -f <override> pull <svc>` + `up -d <svc>`, with a minimal environment (`PATH`, `HOME`, `DOCKER_*`) so the agent's env never leaks into stack interpolation. - Allowed stack locations come from the agent's own read-write same-path bind mounts (replaces `STACKS_ROOT`); every config file, env file and the override are checked after symlink resolution. - Services built from source (`build:`, detected via `docker compose --profile '*' config`) are reported in the inventory and refused at job time; a service missing from the config output fails closed. - Swarm: verifies the job's cluster ID against its own daemon, resolves the digest, inspects by name and updates by ID + version. Optional registry auth from a mounted `config.json`. - Protocol v2: envelope `id`, `job.start` / `job.progress` / `job.result` / `job.ack`, `Hello.ProtocolVersion` and known job IDs; unknown message types are logged and ignored. **Master** (`internal/master/...`) - Migration `0002_jobs.sql`: `update_jobs` (one active job per service via partial unique index), `services.desired_ref`, `config_files`, `built`. - Hub: one writer goroutine per connection, routing (Compose/containers only to the reporting agent; Swarm to a manager of the same cluster), reconcile on hello (unknown running jobs → "lost"), job progress/result/ack; an agent can only progress/finish its own jobs. - `internal/master/jobs`: self-protection (same image repo as `AGENT_IMAGE`, and rows flagged `self`), offline / too-old / no-tag / built-from-source refusals; a bare image (`postgres`) is treated as `:latest`. - Revoking an agent fails its active jobs; `CreateJob` refuses revoked agents under a `FOR SHARE` lock. **Web** (`internal/master/web`) - Service page with Update (POST + CSRF, `requested_by` = OIDC subject) or the reason it is unavailable; Jobs list; job page with live log (SSE `job-<id>` plus a 2s poll while active, re-read from Postgres each time). - Agent snippet: same-path read-write stacks mount, `UPDATE_TIMEOUT`, optional `DOCKER_CONFIG`. **Removed:** `cmd/server`, `internal/api`, `internal/config`, `internal/discovery`, `internal/selfupdate`, `internal/metrics`, old compose examples. Docs: `README.md`, `CLAUDE.md`, `deploy/`, and the plan `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md`. ## Known limitations (documented) - The override file is never pruned; a service removed from the stack file can be recreated by a manual `docker compose up -d`. - Stacks started with explicit `-f` do not load the override on a manual `up`. - Swarm jobs are routed on the cluster ID an agent reports (accepted residual risk: a leaked token can receive jobs and report false results, but cannot command a machine — the agent re-verifies the cluster against its own daemon). - An agent revoked mid-dispatch may still run the job; its result is ignored. - Swarm "succeeded" means the daemon accepted the spec, not that tasks converged. - `Built` in the inventory can be stale after edits to `extends:`/`include:`/`.env` files; the job-time check is always fresh. ## Test plan - [x] `go vet ./... && go test -race -count=1 ./...` (incl. `internal/e2e`: job flow, result resent after reconnect, master restart, agent restart → lost, phase-1 agent refused) - [x] Store integration tests (`-tags integration -race`) against Postgres 17, incl. concurrent revoke vs. CreateJob - [x] Mutation-checked tests for the Review Focus items, env allowlist, profiles, symlinks, revoke lock - [x] `docker build`; image exits with "MASTER_URL is required" - [x] Browser verification (Playwright) with Postgres, mock OIDC, master and agent container: login, add agent, Update → live log → succeeded without reload, polling stops, override written with header and digest, double-click gives one job, disabled reasons (build:, self, plain container), revoke flips an open job page to failed; no console errors besides a missing favicon; 375 and 1440 widths - [ ] Manual smoke test on a Linux host with a real stack (run the `config_files` check from the README first) - [ ] Swarm update on a real cluster

shcizo added 29 commits 2026-10-12 17:48:31 +00:00

docs: implementation plan for master/agent phase 2 (jobs) (#7 ) 7e55ec3b33

docs: plan revisions after review (#7 ) eb41967cd1

feat(protocol): job messages, protocol version and job model (#7 ) 9e09de0a16

refactor(web): web.New takes the jobs service (#7 ) c9c4e0cadb

feat(agent): UPDATE_TIMEOUT and DOCKER_CONFIG settings (#7 ) e2b73dfd81

feat(master): store update jobs (#7 ) 9d3a6f808c

feat(web): service page, manual update, jobs list and live job log (#7 ) 574f425267

Also publishes the jobs topic on agent revoke and updates the agent
snippet to mount stacks read-write at the same path.

feat(master): route jobs to agents and reconcile them on reconnect (#7 ) 68e5c33722

feat(master): start manual update jobs with self-protection (#7 ) 02de0b8cf3

feat(master): wire jobs into the master (#7 ) 32c30194c4

feat(agent): job runner, compose override executor and swarm executor (#7 ) e313365686

feat(agent): run jobs from the master and resend results until acked (#7 ) 601c6da0d4

build: agent is the image's default entrypoint; drop the old server examples (#7 ) 55f8021b0a

test: jobs end to end over pinned TLS (#7 ) b392e11b37

docs: master/agent phase 2 — jobs and manual update (#7 ) b7e25bb1fb

docs: describe Swarm registry auth (#7 ) 88d5507217

fix(web): job log polls while active, one service history limit (#7 ) 8b2c10cc05

fix(agent): minimal env and explicit project dir for compose, resolve every config file's symlinks (#7 ) 93ba6915b9

fix(master): refuse jobs for revoked agents, signal revoked jobs, agent start time wins (#7 ) c09e774d74

fix(master): treat a bare image as :latest and handle a revoked agent in Trigger (#7 ) caa6c81716

test(master): wait for the replacement connection to register (#7 ) 961ddd275c

docs: spell out the upgrade risk and the bare-image rule (#7 ) 7e23e1843f

feat(protocol): Service.Built marks Compose services built from source (#7 ) 91cb027ef6

test(master): CreateJob waits for a concurrent revoke (#7 ) c3bd4ca1ed

feat(master): show built-from-source services but never update them (#7 ) 3d3dfb47e6

docs: compose environment and built-from-source services (#7 ) 69ef55837e

feat(agent): report and refuse Compose services built from source; honour --env-file (#7 ) 27c4200dda

fix(agent): list all compose profiles and fail closed on unknown services (#7 ) c5dba96cf8

fix(agent): reset per-service warnings after a successful build check (#7 ) ec96d5687d

shcizo merged commit 417ef315c2 into main

2026-10-12 17:58:54 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: shcizo/package-updater#20