feat(master): check the registry now from the service page (#35) (#36)

## Summary

- Rebuilds the master's **Services** and **Agents** pages per the "Ledger" design: sidebar shell with live nav counts, machine filter and Live indicator; dark/light theme (follows the OS, toggle overrides); embedded Schibsted Grotesk and JetBrains Mono (fontsource 5.3.0, OFL).
- Services: Swarm grouped per cluster with its machines under it, offline banner ("Agent offline since HH:MM", browser-local), Simple Icons derived from the image ref (letter circle fallback), existing badges and mode switch restyled, client-side search, machine filter and Icon / Icon + text / Text toggle.
- Agents: Online / Offline / Never connected / Revoked, master version in the header with outdated agent versions marked, revoke via popover instead of `confirm()`, "Add agent" as a three-step panel that completes when the agent says hello (SSE `agents`).
- Store: migration `0005_agent_cluster.sql` persists each agent's `swarm_cluster_id` (needed to list a cluster's managers and its offline state after a master restart).
- SSE keepalive is now a named `ping` event so the Live indicator also shows the connection is alive. `revokeAgent` now publishes `agents`.

Closes #18

## Notes

- Workers never report a cluster id (Docker only sets `Swarm.Cluster` on managers), so their Swarm role shows "—". Fixing that needs a new hello field.
- Master binary grows by about 2.16 MB, almost all Simple Icons 16.35.0 path data (gzipped JSON, regenerated with `go generate ./internal/master/web/icons`). The agent binary does not import it.
- Jobs, Job and Service detail pages only get the new shell; their own artboards are out of scope, as are Update buttons, Update all, header stats and the drift badge.
- Unchanged: CSRF on every POST, `Cache-Control: no-store` on the token response, 401 + `HX-Redirect` without a session, no `template.HTML` on agent- or user-controlled values, no Node build step.

## Test plan

- [x] `go test ./...`, `go test -race ./internal/master/...`
- [x] `go test -tags integration ./internal/master/store/` against Postgres 17
- [x] `docker build`
- [x] Browser check on fake data: both themes, 1440 px and 390 px, add-agent flow through connect
- [ ] Services grouping and icons against real stacks and a real Swarm cluster
- [ ] "Agent offline since" after stopping an agent
- [ ] Theme toggle, search and machine filter across reloads and mode changes
- [ ] Add agent end to end with a real agent, Copy buttons
- [ ] Phone at 390 px

https://claude.ai/code/session_01GAZsp3P7Qto6JnhipxWUJQ
Reviewed-on: #29

CLAUDE.md

@@ -36,10 +36,12 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
 - `internal/master/config`   — env-var loading
 - `internal/master/store`    — Postgres access and migrations
 - `internal/master/hub`      — agent connections: auth, per-connection writer, routing, job reconcile on hello
-- `internal/master/registry` — HEAD-only digest lookup against a registry (go-containerregistry, `DOCKER_CONFIG` credentials)
+- `internal/master/registry` — HEAD-only digest lookup and tag listing against a registry (go-containerregistry, `DOCKER_CONFIG` credentials)
-- `internal/master/poller`   — every `POLL_INTERVAL`: one lookup per followed `repo:tag`, sets the available version, starts auto jobs
+- `internal/master/tagver`   — which tag is a newer version of a followed tag (pure functions)
+- `internal/master/poller`   — every `POLL_INTERVAL`: one lookup per followed `repo:tag`, sets the available version, starts auto jobs; one tag list per repo, sets the newer tag suggestion; `PollService` does the same for one service (UI Check now)
 - `internal/master/jobs`     — decides whether a service may be updated and starts a job on the right agent
 - `internal/master/webhook`  — `POST /update`: records the available version, starts jobs for auto services, waits for them
+- `internal/master/web/icons` — embedded Simple Icons path data; slug derived from the image ref; regenerate with `go generate ./internal/master/web/icons`
 - `internal/master/web`, `webauth` — HTMX UI behind OIDC login, SSE signals
 - `internal/master/events`   — in-process publish/subscribe for UI refresh signals
 - `internal/master/model`    — domain types shared by the above
@@ -57,21 +59,38 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
   the agent's own container whose source equals its destination.** Mounts are read per
   job (`ContainerInspect(SELF_ID)`) and checked after symlink resolution. This replaces
   the old `STACKS_ROOT`.
-- **Self-protection on the master**: a service is refused when its image repo equals the
+- **Self-update** (an agent's own Compose service): only `up -d` runs in the helper
-  repo of `AGENT_IMAGE` or the row has the `self` flag. The agent does not re-check.
+  container; the new agent resumes the job (`updater.ResumeSelfUpdates`) before the runner
+  and connection start; the old agent records no result once SIGTERM cancels it. Jobs for
+  the agent's own service run last on the agent and the master starts them last in a
+  batch. An agent's own Swarm service is refused (`ReasonSelfSwarm`). Protocol 4 is needed,
+  also for any service with the `AGENT_IMAGE` repo.
 - **Unknown message types are logged and ignored; malformed messages close the connection.**
 - **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not
   list in `known_jobs`, and only then accepts jobs for that connection.
 - **The webhook never creates services; new services always start `manual`.** The upsert
   does not touch `update_mode`.
+- **Drift is never corrected automatically; poll/webhook jobs are refused for a drifted
+  service** (`jobs.Start`, `ReasonDrifted`). Only a manual job, `RestoreDesired` or an
+  accept gets it out.
+- **Merging is never automatic; merge is the only path that deletes a `services` row**
+  (`store.MergeService`). The old row keeps its id and takes the duplicate's location.
 - **Auth**: agents by token over pinned TLS; people by OIDC; CI by the `UPDATER_API_KEY`
   bearer token on `POST /update` only. Every state-changing UI request is a POST with a
   CSRF token.
 
 ## Gotchas
 
-- **Registry access is HEAD only** (`internal/master/registry`). Docker Hub counts a manifest
+- **Registry manifest access is HEAD only** (`internal/master/registry`). Docker Hub counts a
-  GET as a pull; the client never falls back to GET, and a test asserts this.
+  manifest GET as a pull; the client never GETs `/manifests/`, and a test asserts this. The
+  only GETs are the `/v2/` ping and `tags/list` (newer-tag detection, once per repo per cycle).
+- **`services.newer_ref` is a suggestion only**: no auto jobs, not even for auto services. The
+  poller writes it only on change; a failed tag list keeps the old value and never sets
+  `last_poll_error`. The UI re-checks it at render time with `tagver.Newer`, so a stale value
+  is hidden until the next cycle; it is also hidden when its repo is not the one the service
+  follows. The suggestion is computed from the service list read at the start of the cycle, so
+  if the service moved on meanwhile, the render-time check hides it. Tags with one numeric component or no digits (`7`, `pg16`,
+  `latest`) never get suggestions.
 - **`image_versions` uses `''` instead of NULL** for a missing tag or digest, so
   `UNIQUE (repo, tag, digest)` deduplicates. The webhook records `(repo, tag, '')`; that
   row is hidden from `RecentVersions` when a row with the same tag and a digest was first
@@ -118,17 +137,23 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
   own Docker daemon at execution time.
 - **`protocol.Service.ConfigFiles` is the raw comma-joined label string**, so `Service`
   stays comparable with `==` (`inventory.Diff` depends on it).
+- **Helper containers carry `se.shcizo.package-updater.self-update.*` labels** and are
+  not inventoried (the collector skips any container with the `.job` label). The label
+  strings are constants in `internal/agent/updater/selfupdate.go`, and the `.job` one is
+  repeated in `internal/agent/inventory`.
 - **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged.
 - **Swarm updates need manager-node API access** for the agent doing the update.
 - **The image's default entrypoint is `updater-agent`.** The master overrides it with
   `updater-master`.
+- **The example stack runs the master as nobody (`user: "65534:65534"`).** Never add `USER`
+  to the Dockerfile: the agent uses the same image and needs root for the Docker socket.
 - **The webhook waits via a broker signal plus a store re-read, with a fallback poll**
   (the broker drops signals for slow subscribers). Marking and starting jobs runs on
   `context.WithoutCancel`, so a client that disconnects cannot leave a queued job no agent
   knows about; only the waiting stops. The master never fails or cancels a timed-out job.
 - **A webhook job's target is `<image_repo>:<request tag>`**, built from the service's own
   repo, never from the request's image string. With an explicit target the "follows no tag"
-  refusal does not apply; removed, self, built and container still do.
+  refusal does not apply; removed, built, container and an agent's own Swarm service still do.
 - **`available_*` is cleared by `FinishJob`** on success only when the job's `to_ref`
   (as created, without `@digest`) equals `available_ref`, `available_at <= job.created_at`
   (that version is now running) and, when the job's `to_ref` carries a digest, that digest
@@ -136,6 +161,27 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
   newer suggestion, and a digest-pinned pick does not clear a digest-less (webhook) one.
 - **Job output is capped at `protocol.MaxJobOutput` (128 KiB, tail).** NUL bytes are
   stripped by the hub because Postgres text rejects them.
+- **Drift compares by digest, not by string** (`model.RunsRef`): `repo:tag@D` is run by
+  `repo:tag` reporting digest D. Same digest under another tag is not drift. Different
+  repo strings (`nginx` vs `docker.io/library/nginx`) are.
+- **`services.awaiting_inventory`** is true from a successful `FinishJob` until the next
+  inventory write for the row; `Service.Drifted()` is false meanwhile, since `desired_ref`
+  is ahead of `actual_ref` for a debounce interval after every job.
+- **Accept never touches `last_tag_digest`, `available_*` or `update_mode`**, and the
+  `actual` form field (`Service.RunningRef()`, ref plus digest) is only a guard against a
+  changed service; the value written comes from the row. `RestoreDesired` sets
+  `last_tag_digest` to the drifted digest when it is another digest of the desired tag and
+  `last_tag_digest` is empty or equals the desired digest (otherwise it holds back a newer
+  digest and stays).
+- **`awaiting_inventory` can hide real drift** until the next inventory change when a job
+  result arrives after the snapshot that shows the drift (an agent reconnect resends its
+  outbox after the snapshot), or when Swarm coalesces an update and a rollback into one
+  debounce interval. This is accepted.
+- **Merge lock order is agent, then service rows by id** (as in inventory writes), and the
+  duplicate row is deleted before the old row is updated, because `location_key` is unique
+  and not deferrable. A removal racing a merge can be lost; the next snapshot fixes it.
+- **The SSE stream sends a named `ping` event every 25 s**; the sidebar Live indicator relies on it.
+- **Agents persist `swarm_cluster_id`.** Only managers report it, so workers show no role.
 
 ## References
 
@@ -144,6 +190,7 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
 - Phase 2 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md`
 - Phase 3 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-3-webhook.md`
 - Phase 4 plan: `docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md`
+- Phase 5 plan: `docs/superpowers/plans/2026-10-13-master-agent-phase-5-drift-merge.md`
 - Original single-instance design (historical):
   `docs/superpowers/specs/2026-05-22-package-updater-design.md`
 - Consumer-side CI integration: `gitea-action/` (calls the master's `POST /update`)

README.md

@@ -10,6 +10,7 @@ Design: [master/agent spec](docs/superpowers/specs/2026-10-12-master-agent-desig
 Phase 2 plan: [jobs and manual update](docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md).
 Phase 3 plan: [webhook and auto mode](docs/superpowers/plans/2026-10-12-master-agent-phase-3-webhook.md).
 Phase 4 plan: [registry polling and version history](docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md).
+Phase 5 plan: [drift and merge](docs/superpowers/plans/2026-10-13-master-agent-phase-5-drift-merge.md).
 
 ## What an update does
 
@@ -17,7 +18,8 @@ Clicking *Update* on a service re-pulls the service's **current tag** (`repo:tag
 agent asks its Docker daemon for the tag's digest and deploys `repo:tag@digest`, so what
 runs is exactly what was pulled. The log streams to the browser while it runs; job
 history is kept in Postgres. A service without a tag (`repo@digest`), a plain container,
-and the updater's own services cannot be updated, and the UI says why. A bare image
+and a Swarm agent's own service cannot be updated, and the UI says why (see
+[Self-update](#self-update) for the master and Compose agents). A bare image
 (`image: postgres`, no tag and no digest) counts as `postgres:latest`; only a reference
 with a digest and no tag (`repo@sha256:...`) "follows no tag". Compose services with a
 `build:` section are shown but never updated ("built from source (build:)"). One update at
@@ -28,14 +30,16 @@ a time per service, and one job at a time per agent.
 Use [`deploy/master.stack.yml`](deploy/master.stack.yml) as a starting point (Swarm stack;
 needs Postgres and an OIDC provider). The master listens for the UI on `PORT` (default
 `8080`, put it behind your reverse proxy) and for agents on `AGENT_PORT` (default `8090`,
-pinned TLS, reached directly on the internal network).
+pinned TLS, reached directly on the internal network). The example runs the master as
+`nobody` (`65534`); it needs no root. The agent runs as root because access to the Docker
+socket is root on the host anyway.
 
 | Variable | Required | Default | Purpose |
 |---|---|---|---|
 | `UPDATER_API_KEY` | no | — | Bearer token for `POST /update`. Without it the webhook answers `503` and the master logs `webhook_disabled`. |
 | `UPDATE_TIMEOUT` | no | `5m` | How long one `POST /update` waits for its auto jobs (Go duration). |
 | `POLL_INTERVAL` | no | `6h` | Time between registry polling cycles (Go duration, must be positive). The first cycle runs one minute after start. |
-| `DOCKER_CONFIG` | no | `~/.docker` | Directory with a `config.json` holding registry credentials for polling. Only inline `auths` entries work; see [Registry polling](#registry-polling). |
+| `DOCKER_CONFIG` | no | `~/.docker` | Directory with a `config.json` holding registry credentials for polling, readable by the master's user. Only inline `auths` entries work; see [Registry polling](#registry-polling). |
 
 The master's other settings (`DATABASE_URL`, `PUBLIC_URL`, `AGENT_URL`, `OIDC_*`,
 `ALLOWED_GROUP`, `AGENT_IMAGE`, `PORT`, `AGENT_PORT`, `LOG_LEVEL`) are shown in
@@ -58,7 +62,7 @@ Content-Type: application/json
 in the request are ignored for matching; matching is case-sensitive). `tag` is optional.
 For every match the master records the version as *available* (the overview shows an
 "update available" badge), including services that cannot be updated automatically
-(updater's own, `build:` and plain containers), so the badge shows for them too. Services
+(`build:` services, plain containers and an agent's own Swarm service), so the badge shows for them too. Services
 in **manual** mode stop there. Services in **auto**
 mode also get an update job, and the request waits for those jobs to finish, at most
 `UPDATE_TIMEOUT`.
@@ -70,8 +74,8 @@ one that has been pushed. Compose services get pinned to it through
 already follows (a bare image counts as `:latest`); a service that follows no tag
 (`repo@digest`) is then not touched and is reported as `refused` ("no tag given and the
 service follows no tag"), manual or auto.
-With a `tag`, the "follows no tag" refusal does not apply; the other refusals (the
+With a `tag`, the "follows no tag" refusal does not apply; the other refusals (`build:`
-updater's own services, `build:` services, plain containers) still do.
+services, plain containers, a Swarm agent's own service) still do.
 
 Response (`200` or `500`):
 
@@ -103,10 +107,13 @@ Response (`200` or `500`):
 | `500` | At least one result is `failed` or `timeout`. |
 | `503` | `UPDATER_API_KEY` is not set on the master. |
 
+A service that has **drifted** (see [Drift and moved services](#drift-and-moved-services))
+is refused with its reason, in auto mode too, until someone restores or accepts it.
+
 The webhook never creates services: the first deployment of a stack is done by hand. A
 new service always starts in **manual** mode; switch it with the *Make auto* button on the
-overview or the service page. Auto is refused for the updater's own services, `build:`
+overview or the service page. Auto is refused for `build:` services, plain containers and
-services and plain containers.
+a Swarm agent's own service.
 
 > **Warning:** auto mode deploys whatever CI pushes, without anyone looking at it. Using
 > it on critical services is your own risk.
@@ -137,20 +144,34 @@ multi-arch image the digest is the index digest, the same one `docker inspect` s
   it) and starts no job for that digest again, whether the job
   failed or the user rolled back afterwards. If the service already had a job when the
   digest was found, the next cycle tries again.
+- **Newer tags.** For a service that follows a version-like tag (`1.27.3`, `v1.2.3`,
+  `11.0-latest`; at least two numeric components), the master also lists the repo's tags
+  once per cycle (`GET /v2/<repo>/tags/list`, which Docker Hub does not count as a pull) and
+  suggests the highest tag with the same prefix, suffix and number of components and the
+  same major number, for example `gitea:1.27.3` gives `1.28.0` but never `2.0.0`. The
+  overview shows a "Newer tag" badge and the service page lists the tag under Versions, from
+  where you deploy it. It is a suggestion only: nothing is deployed automatically, not even
+  for **auto** services. Tags such as `latest`, `7` or `pg16` get no suggestion. A failed tag
+  list keeps the old suggestion and is only logged (`poll_tags_failed`).
 - **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and
   services without a registry digest (locally built images). The service page says which.
-  The updater's own services and plain containers are polled, so their badge works, but
+  Plain containers are polled, so their badge works, but jobs for them are refused.
-  jobs for them are still refused.
+- **Check now.** The service page's *Check now* button runs the same check immediately for that
+  service only (digest, auto job, newer tag); other services that follow the same tag are
+  updated at the next cycle.
 - **Errors.** A failed lookup is stored for the services that follow that tag and shown as a
   "poll failed" badge on the overview and in the "Polling" row of the service page; other
   tags continue. A later success clears it.
 - **Credentials.** The master reads `$DOCKER_CONFIG/config.json` (a file written by
   `docker login`), mounted read-only into the container; without it registries are accessed
-  anonymously. Only inline `auths` entries are supported. A `credsStore` or `credHelpers`
+  anonymously. The directory and file must be readable by the master's user (`65534` in the
+  example stack) without being world-readable: `chown -R 65534:65534` the directory and keep
+  `0700`/`0600`. Only inline `auths` entries are supported. A `credsStore` or `credHelpers`
   entry makes every lookup fail with `error getting credentials ... executable file not found
   in $PATH`, because the container has no credential helper binaries. The master logs
-  `registry_credentials` at startup with the directory and whether `config.json` exists,
+  `registry_credentials` at startup with the directory, `config_json_found` and
-  never its contents.
+  `config_json_readable`, never the contents. If the file exists but cannot be opened it also
+  logs a `registry_credentials_unreadable` warning with the error.
 - **Requests go wherever the image name points.** The master sends the lookup to the
   registry host in the image name, including a `GET /v2/` ping that follows redirects.
   Anyone who can start containers on a managed host can therefore make the master send
@@ -186,6 +207,42 @@ that differs from the newest one the master deployed. The newer version it was r
 from stays marked as available. A webhook that names a tag still starts a job for an auto
 service every time, so a CI push moves it forward again.
 
+## Drift and moved services
+
+A service has **drifted** when it runs something other than the version the last update
+job deployed. A service the updater never touched has nothing to drift from. Drift is
+decided by what runs, not by how the reference is written: the digest decides when both
+sides know one, so `repo:tag@sha256:…` is run by a service reporting `repo:tag` with that
+digest, and a different tag with the same digest is not drift. (The design spec compares
+the two reference strings; this is deliberately looser.) Between a successful job and the
+agent's next inventory report the service is not counted as drifted, because the report
+has not arrived yet.
+
+The master never corrects drift on its own, because a manual `docker stack deploy` of a
+newer tag may be intentional. Polling and the webhook start no job for a drifted
+service, in auto mode too; a person can still start one by hand. There are two ways out:
+
+- **Restore desired** deploys the recorded version again (a manual job to the exact
+  `repo:tag@digest`; an agent speaking protocol 3 is needed). If the service had drifted
+  to another digest of the same tag, that digest is remembered, so polling does not
+  deploy it again until the registry shows a newer one (unless `last_tag_digest` already
+  holds back a newer digest, which then stays).
+- **Accept running** records what runs now as the desired version. It starts no job and
+  leaves the available suggestion and the update mode alone.
+
+When a Compose service disappears from one agent and exactly one service with the same
+project and service name appears on another, and the removed one has history (a desired
+version, auto mode or jobs) while the new one has none, the service page suggests that it
+moved. **Merge** moves the old row, with its history, mode and desired version, to the new
+location and deletes the duplicate. Merging is never automatic. It can also be done by
+hand for any removed service of the same kind (Compose or Swarm, names may differ), but
+not for plain containers, an agent's own service, or a service with a queued or running job.
+A merge keeps the old row's mode and desired version, so a manually merged target with its
+own history may show drift right after the merge and can be accepted.
+
+The restore, accept and merge buttons arrive with the UI follow-up (issue #28); the endpoints
+(`POST /services/{id}/restore`, `/accept`, `/merge`) and the data for them are in place.
+
 ## Adding agents
 
 Open the master's **Agents** page and add one. It shows a ready-to-run compose file with
@@ -231,6 +288,39 @@ Read-only mounts and mounts whose source differs from the destination are refuse
 clear message. On Docker Desktop and OrbStack bind sources are translated host paths, so
 Compose jobs fail there; that is a development-only limitation.
 
+## Self-update
+
+The master's own Compose service and each Compose agent's own service are updated like
+any other service; the overview marks the agent's row "This agent".
+
+- **Master.** The agent on the master's host runs the job. The `up -d` recreates the
+  master, which ends the connection and any webhook request that waits for the job. The
+  agent keeps the result and sends it again when the new master is up, so the job is
+  recorded as finished. A client that waited gets an error and has to look at the job
+  afterwards.
+- **Agent.** `up -d` would stop the agent that runs it, so the agent starts a short-lived
+  helper container from its own image (same mounts) and the helper runs `docker compose
+  up -d <service>`. The helper carries `se.shcizo.package-updater.self-update.*` labels
+  with the job ID, the pinned ref and the output so far. When the new agent starts it
+  finds the helper by those labels, waits for it, adds its logs to the job output, removes
+  it and reports the result under the old job ID. The old agent reports nothing after it
+  is stopped.
+- **Requirements.** The agent's own stack directory has to be mounted read-write at the
+  same path (see below), like any stack it updates, and the agent must speak protocol 4
+  (`agent too old to update itself` otherwise). The master and the agent need to be
+  upgraded to a version with protocol 4 by hand once.
+- **Order.** A job for an agent's own service runs after every other queued job on that
+  agent. A master and an agent on the same host share an image, so one push starts both
+  jobs, and the master's job has to finish before the agent replaces itself. The master
+  also starts a job for an agent's own service after the other jobs of the same poll
+  cycle or webhook request. Any other job that reaches an agent while it replaces itself
+  is marked lost when the new agent connects without it.
+- **Docker socket.** The helper runs on the default network, so `DOCKER_HOST` only works
+  with a `docker.sock` bind mount. A TCP socket proxy that is reachable only on a Compose
+  network does not work for self-update.
+- **Swarm.** An agent's own Swarm service is refused: the agent cannot update the service
+  it runs in.
+
 ## How Compose updates work, and their limits
 
 The agent never edits your files. It writes one file next to the stack's base file,

cmd/agent/main.go

@@ -64,6 +64,11 @@ func run() error {
 	defer stop()
 
 	logMounts(ctx, dockerCli, cfg.SelfID, logger)
+	// Before the runner and the connection start: the first hello must already
+	// list the jobs an earlier agent process began.
+	if err := updater.ResumeSelfUpdates(ctx, dockerCli, cfg.SelfID, runner, logger); err != nil {
+		logger.Warn("self_update_resume_failed", "error", err)
+	}
 	go runner.Run(ctx)
 	go watchLoop(ctx, dockerCli, agent, logger)
 	logger.Info("agent_starting", "version", version, "master_url", cfg.MasterURL,

cmd/master/main.go

@@ -48,19 +48,29 @@ const pollFirstDelay = time.Minute
 const pollWorkers = 4
 
 // registryCredentialsDir returns the directory the registry client reads
-// credentials from (DOCKER_CONFIG, else ~/.docker) and whether a config.json
+// credentials from (DOCKER_CONFIG, else ~/.docker), whether a config.json is
-// is in it. Only the path is logged, never the file.
+// in it, and the error from opening it when it exists but cannot be read (for
-func registryCredentialsDir() (dir string, found bool) {
+// example because the master runs as another user than the file owner). The
+// file is opened and closed, never read; only the path is logged.
+func registryCredentialsDir() (dir string, found bool, readErr error) {
 	dir = os.Getenv("DOCKER_CONFIG")
 	if dir == "" {
 		home, err := os.UserHomeDir()
 		if err != nil {
-			return "", false
+			return "", false, nil
 		}
 		dir = filepath.Join(home, ".docker")
 	}
-	_, err := os.Stat(filepath.Join(dir, "config.json"))
+	path := filepath.Join(dir, "config.json")
-	return dir, err == nil
+	if _, err := os.Stat(path); err != nil {
+		return dir, false, nil
+	}
+	f, err := os.Open(path)
+	if err != nil {
+		return dir, true, err
+	}
+	_ = f.Close()
+	return dir, true, nil
 }
 
 func run() error {
@@ -105,9 +115,19 @@ func run() error {
 	}
 	auth := webauth.New(provider, st, cfg.AllowedGroup, strings.HasPrefix(cfg.PublicURL, "https://"), logger)
 
-	ui, err := web.New(st, agentHub, jobService, broker, web.Config{
+	credDir, credFound, credReadErr := registryCredentialsDir()
+	logger.Info("registry_credentials", "docker_config", credDir, "config_json_found", credFound,
+		"config_json_readable", credFound && credReadErr == nil)
+	if credReadErr != nil {
+		logger.Warn("registry_credentials_unreadable", "docker_config", credDir, "error", credReadErr)
+	}
+	registryPoller := poller.New(st, jobService, registry.New("package-updater-master/"+version, logger), broker, poller.Config{
+		Interval: cfg.PollInterval, FirstDelay: pollFirstDelay, Workers: pollWorkers,
+	}, logger)
+
+	ui, err := web.New(st, agentHub, jobService, broker, registryPoller, web.Config{
 		AgentURL: cfg.AgentURL, MasterKey: fingerprint, AgentImage: cfg.AgentImage,
-		PollInterval: cfg.PollInterval,
+		PollInterval: cfg.PollInterval, Version: version,
 	}, logger)
 	if err != nil {
 		return err
@@ -154,13 +174,8 @@ func run() error {
 		BaseContext: baseContext,
 	}
 
-	credDir, credFound := registryCredentialsDir()
-	logger.Info("registry_credentials", "docker_config", credDir, "config_json_found", credFound)
 	pollCtx, stopPolling := context.WithCancel(ctx)
 	defer stopPolling()
-	registryPoller := poller.New(st, jobService, registry.New("package-updater-master/"+version, logger), broker, poller.Config{
-		Interval: cfg.PollInterval, FirstDelay: pollFirstDelay, Workers: pollWorkers,
-	}, logger)
 	pollDone := make(chan struct{})
 	go func() {
 		defer close(pollDone)

cmd/master/main_test.go

@@ -0,0 +1,42 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestRegistryCredentialsDir(t *testing.T) {
+	t.Run("missing file", func(t *testing.T) {
+		dir := t.TempDir()
+		t.Setenv("DOCKER_CONFIG", dir)
+		got, found, readErr := registryCredentialsDir()
+		assert.Equal(t, dir, got)
+		assert.False(t, found)
+		assert.NoError(t, readErr)
+	})
+
+	t.Run("readable file", func(t *testing.T) {
+		dir := t.TempDir()
+		require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o600))
+		t.Setenv("DOCKER_CONFIG", dir)
+		_, found, readErr := registryCredentialsDir()
+		assert.True(t, found)
+		assert.NoError(t, readErr)
+	})
+
+	t.Run("unreadable file", func(t *testing.T) {
+		if os.Geteuid() == 0 {
+			t.Skip("root can read any file")
+		}
+		dir := t.TempDir()
+		require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o000))
+		t.Setenv("DOCKER_CONFIG", dir)
+		_, found, readErr := registryCredentialsDir()
+		assert.True(t, found)
+		assert.Error(t, readErr)
+	})
+}

deploy/agent.compose.yml

@@ -16,4 +16,5 @@ services:
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
       # - /path/to/stacks:/path/to/stacks  # mount each stacks directory at the same path, read-write
+      #   (also this agent's own stack directory, or it cannot update itself)
       # - /root/.docker/config.json:/docker-config/config.json:ro

deploy/master.stack.yml

@@ -12,12 +12,17 @@
 #      The reverse proxy must allow a read timeout longer than UPDATE_TIMEOUT (default 5m).
 #   5. Optional, for registry polling of private images: put a config.json written by
 #      `docker login` (inline "auths" only; no credsStore) in /path/to/docker-config and
-#      uncomment DOCKER_CONFIG and the volumes entry below. The directory must exist on every
+#      uncomment DOCKER_CONFIG and the volumes entry below. The master runs as 65534:65534, so
-#      manager node (the master is placed on a manager), or Swarm rejects the task. Without
+#      the directory and config.json must be readable by that user while staying private:
-#      it registries are accessed anonymously.
+#      `chown -R 65534:65534 /path/to/docker-config` (keep 0700/0600), or set `user:` below
+#      to the owner's uid:gid. Do not make config.json world-readable; it holds registry
+#      passwords. The directory must exist on every manager node (the master is placed on a
+#      manager), or Swarm rejects the task. Without it registries are accessed anonymously.
 services:
   master:
     image: gitea.shcizo.se/shcizo/package-updater:latest
+    # Runs as nobody: the master needs no root (the agent does, for the Docker socket).
+    user: "65534:65534"
     entrypoint: ["/bin/sh", "-c", "DATABASE_URL=$$(cat /run/secrets/pu_database_url) OIDC_CLIENT_SECRET=$$(cat /run/secrets/pu_oidc_client_secret) UPDATER_API_KEY=$$(cat /run/secrets/pu_updater_api_key) exec /usr/local/bin/updater-master"]
     environment:
       PUBLIC_URL: https://updater.example.com

docs/superpowers/plans/2026-10-13-master-agent-phase-5-drift-merge.md

@@ -0,0 +1,104 @@
+# Master/Agent Phase 5 — Drift and Merge Implementation Plan
+
+> This plan documents what was built for issue #10 (backend only) and where it deviates from the spec. Code in "Frozen contracts" is final; tasks give the contract and the test cases.
+
+**Goal:** The master knows when a service runs something other than the version the last job deployed (drift), never corrects it by itself, and offers two ways out: restore the desired version or accept the running one. When a Compose service disappears from one agent and appears on another, the master suggests that it moved, and a merge moves the old row (history, mode, desired ref) to the new location. Merging can also be done by hand.
+
+**Architecture:**
+- Model: `Service.Drifted()` plus `RunsRef` and `RunningRef` decide drift and what accept writes. `Move` carries a suggestion.
+- Store: migration 0005 adds `services.awaiting_inventory`; `FinishJob` sets it, every inventory upsert clears it. New `AcceptRunning`, `MoveCandidates`, `MergeSources`, `MergeService` (the only code that deletes a `services` row).
+- Jobs: `Start` refuses poll and webhook jobs for a drifted service. `Restorable` and `RestoreDesired` deploy the recorded ref again as a manual job.
+- Web: three POST routes (`/services/{id}/restore`, `/accept`, `/merge`) and new `pageData` fields. No template or CSS changes: issue #18 (the Ledger rewrite of the Services and Agents pages) is a separate UI change; the restore, accept and merge buttons arrive with the UI follow-up, issue #28.
+
+**Tech Stack:** no new dependencies.
+
+**Spec:** `docs/superpowers/specs/2026-10-12-master-agent-design.md`, sections "Drift" and "Service identity and moves". Phase 4 plan conventions still apply.
+
+## Global Constraints
+
+- `go test ./...` passes with no Docker and no Postgres. Postgres tests carry `//go:build integration`.
+- No changes to `internal/master/web/templates/*`, `static/*`, `layout.html` or `groupServices`.
+- Interfaces declared at the consumer (`jobs.Store`, `web.Store`, `web.Jobs`). Hand-written fakes and testify, table-driven where there is more than one case.
+- GoDoc on every exported identifier. Conventional Commits, each ending with ` (#10)`.
+- Every state-changing request is a POST with a CSRF token. Refusals render the service page with the reason (409), not a bare error.
+- The invariants of earlier phases hold: one active job per service (`update_jobs_one_active`), the webhook and poller never create services, the updater's own services are never updated or merged.
+
+## Deliberate deviations from the spec in this phase
+
+1. **Drift compares what runs, not reference strings.** The spec says a service drifts when `actual_ref` differs from `desired_ref`. `desired_ref` is always `repo:tag@digest` while `actual_ref` is the raw image string, so a string comparison would flag every healthy service. `RunsRef` compares the repo exactly (no `docker.io` normalisation), then the digest when both sides know one, else the followed tag. The same digest under a different tag is therefore not drift.
+2. **`awaiting_inventory`.** The agent's upsert arrives a debounce interval after `job.result`; until then `desired_ref` is newer than `actual_ref`. The flag, set by `FinishJob` and cleared by the next upsert of the row, keeps that window from counting as drift.
+3. **Poll and webhook jobs are refused for a drifted service** (`ReasonDrifted`). Manual jobs are allowed, so a person can still update or roll back.
+4. **Move suggestion is stricter than "exactly one appears".** Besides one removed and one live row on each side, the live row must be newer, have no desired ref and no jobs, and the removed row must have something worth moving (desired ref, auto mode or jobs). Otherwise every restarted stack would produce a suggestion.
+5. **Manual merge is allowed for Compose and Swarm of the same kind**, names may differ, same agent allowed. Plain containers and the updater itself are refused.
+6. **Accept does not clear `available_*`**, and the form's `actual` field (`Service.RunningRef()`) is only a guard; the value written comes from the row.
+7. **The overview does not show moves or drift yet.** `MoveCandidates` is only read on the service page. `Service.Drifted()` is a model method, so the UI follow-up needs no further wiring.
+8. **`CreateJob` maps the foreign key violation on `update_jobs_service_id_fkey` to `ErrNotFound`**, so a job for a service that a merge just deleted fails cleanly.
+
+## Review Focus
+
+| Failure case | Test that covers it |
+|---|---|
+| Drift by digest, not string: tag+digest vs tag and reported digest, bare image, tagless, port colon, other repo, case. | model `TestRunsRef`, `TestDrifted` |
+| Accept writes `repo:tag@digest` from the running version. | model `TestRunningRef` |
+| A fresh job result is not drift until the inventory arrives. | store `TestFinishJob_SetsAwaitingInventoryUntilTheNextUpsert`; e2e `TestNoDriftAfterASuccessfulJob` |
+| Accept is guarded by the running version, refuses removed/never-updated, leaves `last_tag_digest`, `available_*`, mode alone. | store `TestAcceptRunning_*` |
+| Poll and webhook start no job for a drifted service; manual does. | jobs `TestStart_RefusesPollAndWebhookWhileDrifted`; e2e `TestDriftedAutoServiceIsNotUpdatedByPollOrWebhook` |
+| Restore deploys the recorded ref and remembers the drifted digest of the same tag. | jobs `TestRestoreDesired_*`; e2e `TestRestoreDeploysThePinnedRefOnTheAgent` |
+| Move suggestion only for a unique, new, history-less target and a removed source with history. | store `TestMoveCandidates` |
+| Merge keeps id, mode, desired ref, first seen; takes location and inventory; moves jobs; deletes the duplicate. | store `TestMergeService_MovesHistoryModeAndDesiredToTheNewLocation` |
+| Merge refusals change nothing (active job, kinds, container, self, live old, removed new, revoked agent, unknown). | store `TestMergeService_Refusals` |
+| Inventory after a merge updates the survivor; the old agent returning creates a new row. | store `TestMergeService_InventoryAfterTheMerge` |
+| A job for a deleted service is `ErrNotFound`. | store `TestCreateJob_UnknownServiceIsNotFound` |
+| Handlers: redirects, 400/404/409, signals published, survivor id in the redirect. | web `TestRestoreDesired`, `TestAcceptRunning_*`, `TestMergeService_*`, `TestServiceDetail_DriftAndMoveData` |
+
+## Frozen contracts
+
+### C1. Migration `0005_drift_merge.sql`
+`services.awaiting_inventory boolean NOT NULL DEFAULT false`; index `services_by_compose_name` on `(compose_project, compose_service) WHERE kind = 'compose'`.
+
+### C2. `internal/master/model`
+`Service.AwaitingInventory`; `Service.Drifted()`; `RunsRef(desired, actualRef, actualDigest)`; `RunningRef(actualRef, actualDigest)`; `Move{FromID, ToID, FromAgent, ToAgent}` (FromID is the removed row). `ReasonDrifted`, `ReasonNotDrifted`; `MergeRefusedError{Reason}` with the `ReasonMerge*` constants (`ReasonSelf` is reused).
+
+### C3. Store
+- `upsertService` sets `awaiting_inventory = false`; `FinishJob` success sets it to true.
+- `AcceptRunning(ctx, id, actualRef, actualDigest, desired)`: updates only a live, previously updated row that still reports that actual ref and digest. `ErrNotFound` for unknown or removed, `ErrConflict` otherwise.
+- `MoveCandidates(ctx) ([]model.Move, error)`: Compose only, not self, other agent, unique on both sides before the extra filters (live row newer, no desired ref, no jobs; removed row has a desired ref, auto mode or jobs). Agent names by `LEFT JOIN`, so a revoked agent keeps its name.
+- `MergeSources(ctx, toID)`: removed rows of the same kind (not container, not self), same name first, then `removed_at DESC`, at most 50.
+- `MergeService(ctx, fromID, toID)`: one transaction. Lock order: the live row's agent (`FOR SHARE`, `ErrRevoked`), then both rows `ORDER BY id FOR UPDATE`. Validates (`MergeRefusedError`), refuses an active job on either row (`ErrJobActive`), moves the duplicate's jobs, reads its location and inventory, **deletes it**, then updates the old row (keeps id, kind, `first_seen_at`, `update_mode`, `desired_ref`; takes agent, location, inventory columns, available suggestion and poll error; `last_tag_digest` only if the old row has none; `removed_at = NULL`, `awaiting_inventory = false`). The delete comes first because `location_key` is unique and not deferrable.
+- `CreateJob`: foreign key violation `update_jobs_service_id_fkey` becomes `ErrNotFound`.
+
+### C4. Jobs
+`Start` returns `NotUpdatableError{ReasonDrifted}` for a non-manual trigger on a drifted service. `Restorable(svc)`: drifted, then `updatable(svc, true)`, and the desired ref's repo is not the agent image repo. `RestoreDesired(ctx, id, by)`: `Start` with `Ref: svc.DesiredRef`, manual. After a created job, `SetLastTagDigest` to the drifted digest when the actual tag equals the desired tag, the digest is well formed, differs from the desired digest and from `last_tag_digest`, and `last_tag_digest` is empty or equals the desired digest (otherwise it holds back a newer digest and stays).
+
+### C5. Web
+Routes `POST /services/{id}/restore`, `/accept` (form `actual` required, the `Service.RunningRef()` the user saw, compared with the row's; optional `return=overview`), `/merge` (form `from` = removed row, `{id}` = live row; redirects to the surviving id). `pageData` gains `RestoreBlocked`, `MoveFrom`, `MoveTo`, `MergeSources`. Log events `drift_accepted` and `service_merged`.
+
+### C6. Concurrency
+An upsert for the live row's location waits on its lock, then updates the survivor. A snapshot after the merge lists the survivor's key, so it is not marked removed. A removal racing a merge can be lost; the next snapshot fixes it. `CreateJob` on the deleted row gets `ErrNotFound`; the poller and webhook re-read and log it.
+
+## Tasks
+
+### Task 1: Store and model (C1, C2, C3)
+Migration, `imageref.IsDigest` (replaces the local pattern in the store), model types and functions, store methods. Tests: model tables; integration tests listed in Review Focus.
+
+### Task 2: Jobs (C4)
+`Store.SetLastTagDigest` on the consumer interface, `Start` refusal, `Restorable`, `RestoreDesired`. Tests: refusal table, manual job to the desired ref with protocol 3, active job, old agent, last-tag-digest table.
+
+### Task 3: Web (C5)
+Handlers, interfaces, `pageData`. Tests with fakes.
+
+### Task 4: End-to-end
+`memStore.FinishJob` sets `AwaitingInventory`. `internal/e2e/drift_test.go`.
+
+### Task 5: Documentation
+README section "Drift and moved services", webhook note, CLAUDE.md design intent and gotchas, this plan.
+
+## Final verification
+
+```bash
+gofmt -l . ; go vet ./...
+go build ./cmd/master ./cmd/agent && rm -f master agent
+go test ./...
+go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
+DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
+```

docs/superpowers/specs/2026-10-12-master-agent-design.md

@@ -229,13 +229,46 @@ Online/offline is held in memory only.
 | Master restarts mid-job | Jobs in `running` are reconciled when the agent reconnects; jobs the agent does not know are marked `failed` ("lost"). |
 | New trigger while a job for the same service is queued or running | No new job; only `available_*` is updated. Two jobs never race on one service. |
 | Job exceeds `UPDATE_TIMEOUT` | The agent cancels it, as today. |
+| Agent updates itself | The old agent runs `up -d` in a helper container and waits. If the helper exits while the old agent still runs, the old agent reports the outcome. If the helper stops the old agent, the old agent records nothing; the new agent finds the helper by its labels before it connects, so its hello lists the job, and it reports the helper's outcome. If no new agent resumes the job, it is marked lost when the agent reconnects without it. If `UPDATE_TIMEOUT` passes first, the old agent reports a timeout and leaves the helper running; a later success is then not recorded. |
 | Polling fails for one image | Error stored in `last_poll_error` and shown; other images continue. |
 
-### Self-protection
+### Self-update
 
-The agent reports its own container and the master knows its own service. Both are shown
+The agent reports its own container with `self`; the overview shows "This agent". The
-as "updated manually outside the system", and the master refuses to create jobs for
+master's own Compose service and every Compose agent's own service are normal update
-them. An agent recreating itself would kill the job it is running.
+targets. A Swarm agent's own service is refused ("an agent's own Swarm service cannot
+update itself").
+
+- **Master.** No special execution. The agent on the master's host runs the job; the
+  recreate drops its connection, and it sends the result again after the new master is up
+  (unacknowledged results are resent).
+- **Agent.** The old agent does the usual checks, writes the override and pulls. Only
+  `up -d <service>` runs in a helper container made from the agent's own image, with the
+  agent's own bind and volume mounts and no Compose labels. The helper's labels
+  (`se.shcizo.package-updater.self-update.` plus `job`, `ref`, `started`, `project`,
+  `service`, `output`) hold what the new agent needs: the job ID, the pinned ref, the
+  start time and the output so far (tail, 32 KiB). If the helper exits while the old agent
+  still runs, the old agent copies its logs into the job output, removes it and reports
+  the outcome itself.
+- **Resume.** At startup, before the runner and the connection start, the agent lists
+  helpers whose project and service labels match its own container and registers each as
+  a job at the front of its queue. The job is then in `known_jobs` of the first hello.
+  Its result is a normal `job.result`: output, helper logs, pinned ref on exit 0,
+  otherwise an error. A helper with unusable labels fails the job with a message saying so.
+- **Stopping.** When the agent is stopped (SIGTERM) while a job runs, the runner drops the
+  job and records no result, so a "context canceled" result cannot overtake the new
+  agent's.
+- **Order.** `job.start` carries `self`; the agent runs a job for its own service only
+  when no other job is queued (the master's job on a shared host has to finish first).
+  The master starts such jobs after the other jobs of the same poll cycle or webhook
+  request; messages to one agent arrive in order, so the others are queued first. A job
+  that reaches an agent while it replaces itself is marked lost when the new agent
+  connects without it. Whether to use a helper is decided by the agent from its own
+  container ID, not from the flag.
+- **Protocol 4.** The master routes a job for an agent's own service, and any job for a
+  service with the `AGENT_IMAGE` repo (an older agent does not report itself), only to an
+  agent speaking protocol 4 (`agent too old to update itself`). The helper runs on the
+  default network, so `DOCKER_HOST` must be a `docker.sock` bind mount.
 
 ## Registry polling
 
@@ -255,6 +288,11 @@ them. An agent recreating itself would kill the job it is running.
   HEAD-only and never downloads a manifest.
 - Credentials: a Docker `config.json` (as written by `docker login`) mounted into the
   master. No credential UI, no encryption in Postgres.
+- Newer version tags (e.g. `gitea:1.27.3` → `1.27.4`): for a followed tag with at least
+  two numeric components the poller lists the repo's tags once per cycle
+  (`GET /v2/<repo>/tags/list`) and stores the highest tag with the same prefix, suffix,
+  component count and major number in `services.newer_ref`. A suggestion only, never
+  deployed automatically; deploying it is the existing version pick.
 
 ## Webhook
 
@@ -355,7 +393,7 @@ Each phase ends with something that works on its own.
    snapshot and live events, Postgres storage, OIDC login, Overview and Agents pages.
    The overview works; nothing can be updated yet.
 2. **Jobs and manual update.** Job protocol, agent queue and executors (Compose override
-   file, Swarm with digest), job history, Service and Jobs pages, self-protection.
+   file, Swarm with digest), job history, Service and Jobs pages.
 3. **Webhook and auto mode.** `POST /update` on the master, matching, waiting for
    auto jobs, mode toggle.
 4. **Polling and version history.** `image_versions`, registry polling, version picker
@@ -364,14 +402,9 @@ Each phase ends with something that works on its own.
 
 ## Out of scope (later specs)
 
-- Detecting newer version *tags* for images we do not build (e.g. `gitea:1.27.3` →
-  `1.27.4`, `postgres:17.10`, `loki:3.4.2`). Requires listing tags and ordering them;
-  major upgrades such as Postgres 17 → 18 must never be automatic. Revisit after
-  checking the other two servers' stack files.
 - Notifications.
 - Export/import of a fleet configuration to restore a setup. The `services` table
   already holds what an export needs.
 - Writing version bumps back to git.
-- Self-update of master and agents (`internal/selfupdate` is the starting point).
 - Updating plain (non-Compose, non-Swarm) containers.
 - Credential management UI for registries.

go.mod

@@ -10,6 +10,7 @@ require (
 	github.com/google/go-containerregistry v0.22.1
 	github.com/jackc/pgx/v5 v5.10.0
 	github.com/opencontainers/go-digest v1.0.0
+	github.com/opencontainers/image-spec v1.1.1
 	github.com/stretchr/testify v1.11.1
 	golang.org/x/oauth2 v0.36.0
 	gopkg.in/yaml.v3 v3.0.1
@@ -38,7 +39,6 @@ require (
 	github.com/moby/sys/atomicwriter v0.1.0 // indirect
 	github.com/moby/term v0.5.2 // indirect
 	github.com/morikuni/aec v1.1.0 // indirect
-	github.com/opencontainers/image-spec v1.1.1 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/sirupsen/logrus v1.9.4 // indirect

internal/agent/inventory/collector.go

@@ -22,12 +22,14 @@ import (
 )
 
 const (
-	labelProject     = "com.docker.compose.project"
+	labelProject       = "com.docker.compose.project"
-	labelService     = "com.docker.compose.service"
+	labelService       = "com.docker.compose.service"
-	labelWorkingDir  = "com.docker.compose.project.working_dir"
+	labelWorkingDir    = "com.docker.compose.project.working_dir"
-	labelConfigFiles = "com.docker.compose.project.config_files"
+	labelConfigFiles   = "com.docker.compose.project.config_files"
-	labelEnvFile     = "com.docker.compose.project.environment_file"
+	labelEnvFile       = "com.docker.compose.project.environment_file"
-	labelSwarmTask   = "com.docker.swarm.service.id"
+	labelSwarmTask     = "com.docker.swarm.service.id"
+	labelSwarmName     = "com.docker.swarm.service.name"
+	labelSelfUpdateJob = "se.shcizo.package-updater.self-update.job"
 )
 
 // DockerClient is the subset of the Docker SDK the collector needs.
@@ -61,6 +63,7 @@ type Collector struct {
 	now   func() time.Time
 
 	mu     sync.Mutex
+	self   *selfInfo             // the agent's own container, once inspected
 	cache  map[string]buildEntry // by Compose project
 	warned map[string]bool       // logged warnings: "<project>" (check failed) and "<project>/<service>" (service not in config)
 }
@@ -77,8 +80,15 @@ type buildEntry struct {
 // stackLabels are the Compose labels a build check needs.
 type stackLabels struct{ workingDir, configFiles, envFile string }
 
-// New returns a Collector. selfID is a prefix of the agent's own container
+// selfInfo is what the inspect of the agent's own container tells.
-// ID (inside a container, the hostname); that container is marked Self.
+type selfInfo struct {
+	id           string // full container ID
+	swarmService string // name of the Swarm service the container is a task of, if any
+}
+
+// New returns a Collector. selfID is the agent's own container: a prefix of
+// its ID (inside a container, the hostname) or its name. That container is
+// marked Self, and so is the Swarm service it is a task of.
 func New(cli DockerClient, selfID string) *Collector {
 	return &Collector{cli: cli, selfID: selfID, now: time.Now, cache: map[string]buildEntry{}, warned: map[string]bool{}}
 }
@@ -92,6 +102,29 @@ func (c *Collector) WithBuildCheck(check BuildCheck, log *slog.Logger) *Collecto
 	return c
 }
 
+// resolveSelf inspects the agent's own container once and remembers the
+// answer. It returns nil while that fails (or selfID is empty), and the
+// caller then compares container IDs with the selfID prefix.
+func (c *Collector) resolveSelf(ctx context.Context) *selfInfo {
+	if c.selfID == "" {
+		return nil
+	}
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if c.self != nil {
+		return c.self
+	}
+	insp, err := c.cli.ContainerInspect(ctx, c.selfID)
+	if err != nil || insp.ContainerJSONBase == nil || insp.ID == "" {
+		return nil
+	}
+	c.self = &selfInfo{id: insp.ID}
+	if insp.Config != nil {
+		c.self.swarmService = insp.Config.Labels[labelSwarmName]
+	}
+	return c.self
+}
+
 // Hello returns the machine facts sent on connect. AgentVersion is left for
 // the caller to fill in.
 func (c *Collector) Hello(ctx context.Context) (protocol.Hello, error) {
@@ -125,17 +158,21 @@ func (c *Collector) Snapshot(ctx context.Context) ([]protocol.Service, error) {
 		repoDigests[img.ID] = img.RepoDigests
 	}
 
+	self := c.resolveSelf(ctx)
 	byKey := make(map[string]protocol.Service)
 	stacks := make(map[string]stackLabels)
 	for _, ct := range containers {
 		if _, isTask := ct.Labels[labelSwarmTask]; isTask {
 			continue
 		}
+		if _, isHelper := ct.Labels[labelSelfUpdateJob]; isHelper {
+			continue
+		}
 		ref, err := c.imageRef(ctx, ct)
 		if err != nil {
 			return nil, err
 		}
-		svc := c.fromContainer(ct, ref, repoDigests[ct.ImageID])
+		svc := c.fromContainer(ct, ref, repoDigests[ct.ImageID], self)
 		if svc.Kind == protocol.KindCompose {
 			if _, ok := stacks[svc.ComposeProject]; !ok {
 				stacks[svc.ComposeProject] = stackLabels{svc.WorkingDir, svc.ConfigFiles, ct.Labels[labelEnvFile]}
@@ -159,6 +196,7 @@ func (c *Collector) Snapshot(ctx context.Context) ([]protocol.Service, error) {
 				continue
 			}
 			svc := fromSwarmService(s)
+			svc.Self = self != nil && self.swarmService != "" && svc.SwarmService == self.swarmService
 			byKey[svc.Key()] = svc
 		}
 	}
@@ -189,12 +227,12 @@ func (c *Collector) imageRef(ctx context.Context, ct container.Summary) (string,
 	return insp.Config.Image, nil
 }
 
-func (c *Collector) fromContainer(ct container.Summary, ref string, repoDigests []string) protocol.Service {
+func (c *Collector) fromContainer(ct container.Summary, ref string, repoDigests []string, self *selfInfo) protocol.Service {
 	svc := protocol.Service{
 		Image:  ref,
 		Digest: digestFor(ref, repoDigests),
 		Status: containerStatus(ct.State),
-		Self:   c.selfID != "" && strings.HasPrefix(ct.ID, c.selfID),
+		Self:   c.isSelf(ct.ID, self),
 	}
 	project, service := ct.Labels[labelProject], ct.Labels[labelService]
 	if project != "" && service != "" {
@@ -213,6 +251,15 @@ func (c *Collector) fromContainer(ct container.Summary, ref string, repoDigests
 	return svc
 }
 
+// isSelf reports whether id is the agent's own container: by the inspected
+// ID when there is one, else by the selfID prefix.
+func (c *Collector) isSelf(id string, self *selfInfo) bool {
+	if self != nil {
+		return id == self.id
+	}
+	return c.selfID != "" && strings.HasPrefix(id, c.selfID)
+}
+
 func fromSwarmService(s swarm.Service) protocol.Service {
 	ref := s.Spec.TaskTemplate.ContainerSpec.Image
 	return protocol.Service{

internal/agent/inventory/collector_test.go

@@ -135,6 +135,19 @@ func TestSnapshot_PlainContainerAndSwarmTaskSkipped(t *testing.T) {
 	}}, got)
 }
 
+func TestSnapshot_SelfUpdateHelperSkipped(t *testing.T) {
+	cli := &fakeDocker{containers: []container.Summary{
+		{ID: "h1", Names: []string{"/quirky_helper"}, Image: "sha256:agentimage", State: "running",
+			Labels: map[string]string{labelSelfUpdateJob: "7"}},
+		{ID: "p1", Names: []string{"/adhoc"}, Image: "redis:alpine", State: "exited"},
+	}}
+	got, err := New(cli, "").Snapshot(context.Background())
+	require.NoError(t, err)
+	assert.Equal(t, []protocol.Service{{
+		Kind: protocol.KindContainer, ContainerName: "adhoc", Image: "redis:alpine", Status: "stopped",
+	}}, got)
+}
+
 func TestSnapshot_SelfFlag(t *testing.T) {
 	cli := &fakeDocker{containers: []container.Summary{
 		composeContainer("0123456789abcdef", "updater-agent", "agent", "pu:1", "sha256:i", "running"),
@@ -145,6 +158,63 @@ func TestSnapshot_SelfFlag(t *testing.T) {
 	assert.True(t, got[0].Self)
 }
 
+func TestSnapshot_SelfFlagByContainerName(t *testing.T) {
+	cli := &fakeDocker{
+		containers: []container.Summary{
+			composeContainer("0123456789abcdef", "updater-agent", "agent", "pu:1", "sha256:i", "running"),
+			composeContainer("fedcba9876543210", "slusk", "web", "slusk:1", "sha256:j", "running"),
+		},
+		inspect: map[string]container.InspectResponse{
+			"updater-agent-agent-1": {ContainerJSONBase: &container.ContainerJSONBase{ID: "0123456789abcdef"}},
+		},
+	}
+	got, err := New(cli, "updater-agent-agent-1").Snapshot(context.Background())
+	require.NoError(t, err)
+	require.Len(t, got, 2)
+	assert.False(t, got[0].Self, "slusk")
+	assert.True(t, got[1].Self, "updater-agent")
+}
+
+func TestSnapshot_SelfFlagFallsBackToIDPrefixWhenInspectFails(t *testing.T) {
+	cli := &fakeDocker{
+		containers: []container.Summary{composeContainer("0123456789abcdef", "updater-agent", "agent", "pu:1", "sha256:i", "running")},
+		inspectErr: errors.New("boom"),
+	}
+	got, err := New(cli, "0123456789ab").Snapshot(context.Background())
+	require.NoError(t, err)
+	require.Len(t, got, 1)
+	assert.True(t, got[0].Self)
+}
+
+func TestSnapshot_AgentsOwnSwarmServiceIsSelf(t *testing.T) {
+	swarmSvc := func(name string) swarm.Service {
+		return swarm.Service{
+			Spec: swarm.ServiceSpec{
+				Annotations:  swarm.Annotations{Name: name},
+				TaskTemplate: swarm.TaskSpec{ContainerSpec: &swarm.ContainerSpec{Image: "pu:1"}},
+			},
+			ServiceStatus: &swarm.ServiceStatus{RunningTasks: 1, DesiredTasks: 1},
+		}
+	}
+	cli := &fakeDocker{
+		info:     managerInfo(),
+		services: []swarm.Service{swarmSvc("pu_agent"), swarmSvc("pu_master")},
+		inspect: map[string]container.InspectResponse{
+			"abc123": {
+				ContainerJSONBase: &container.ContainerJSONBase{ID: "abc123full"},
+				Config:            &container.Config{Labels: map[string]string{labelSwarmName: "pu_agent"}},
+			},
+		},
+	}
+	got, err := New(cli, "abc123").Snapshot(context.Background())
+	require.NoError(t, err)
+	require.Len(t, got, 2)
+	assert.Equal(t, "pu_agent", got[0].SwarmService)
+	assert.True(t, got[0].Self)
+	assert.Equal(t, "pu_master", got[1].SwarmService)
+	assert.False(t, got[1].Self)
+}
+
 func TestSnapshot_SwarmServicesOnlyOnManager(t *testing.T) {
 	svc := swarm.Service{
 		Spec: swarm.ServiceSpec{

internal/agent/updater/compose.go

@@ -6,13 +6,12 @@ import (
 	"io"
 	"os"
 	"os/exec"
+	"slices"
 	"strings"
 	"time"
 
 	"github.com/docker/docker/api/types/container"
 	"github.com/docker/docker/api/types/filters"
-
-	"github.com/shcizo/package-updater/internal/protocol"
 )
 
 const (
@@ -71,6 +70,7 @@ func dockerEnv() []string {
 // ComposeDocker is the subset of the Docker SDK ComposeExecutor needs.
 type ComposeDocker interface {
 	SelfInspector
+	HelperDocker
 	ContainerList(ctx context.Context, opts container.ListOptions) ([]container.Summary, error)
 }
 
@@ -95,7 +95,10 @@ func NewComposeExecutor(cli ComposeDocker, resolve *Resolver, selfID string, run
 // is), writes it to the stack's override file and recreates the service.
 // Working directory and config files come from the service's container
 // labels on this machine, never from the job. It returns the pinned ref.
-func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) {
+//
+// When the service is the agent's own container, the recreate runs in a
+// helper container instead (see selfUpdate), because it stops this process.
+func (e *ComposeExecutor) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
 	project, service := job.ComposeProject, job.ComposeService
 	cts, err := e.cli.ContainerList(ctx, container.ListOptions{
 		All: true,
@@ -110,14 +113,24 @@ func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, ou
 	if len(cts) == 0 {
 		return "", fmt.Errorf("no container for compose service %s/%s on this machine", project, service)
 	}
+	self, err := inspectSelf(ctx, e.cli, e.selfID)
+	if err != nil {
+		return "", err
+	}
+	isSelf := self.ContainerJSONBase != nil && self.ID != "" &&
+		slices.ContainsFunc(cts, func(ct container.Summary) bool { return ct.ID == self.ID })
+	// What an update of the agent's own service has printed so far goes into
+	// the helper's labels.
+	var soFar *tailBuffer
+	if isSelf {
+		soFar = &tailBuffer{max: selfUpdateOutputMax}
+		out = io.MultiWriter(out, tailWriter{soFar})
+	}
 	workingDir := cts[0].Labels[labelWorkingDir]
 	configFiles := cts[0].Labels[labelConfigFiles]
 	envFileLabel := cts[0].Labels[labelEnvFile]
 
-	roots, err := MountRoots(ctx, e.cli, e.selfID)
+	roots := rootsOf(self)
-	if err != nil {
-		return "", err
-	}
 	files, dir, err := composeFiles(roots, configFiles)
 	if err != nil {
 		return "", err
@@ -163,13 +176,29 @@ func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, ou
 	}
 
 	base := composeBase(project, workingDir, files, envs)
-	for _, step := range [][]string{{"pull", service}, {"up", "-d", service}} {
+	run := func(args []string) error {
-		args := append(append([]string(nil), base...), step...)
 		fmt.Fprintf(out, "$ docker %s\n", strings.Join(args, " "))
 		tail := &tailBuffer{max: errorOutputTail}
 		if err := e.run(ctx, workingDir, io.MultiWriter(out, tailWriter{tail}), args...); err != nil {
-			return "", fmt.Errorf("docker compose %s: %w (output: %s)", step[0], err, strings.TrimSpace(tail.String()))
+			return composeError(args[len(base)], err, tail)
 		}
+		return nil
+	}
+	if err := run(append(append([]string(nil), base...), "pull", service)); err != nil {
+		return "", err
+	}
+	up := append(append([]string(nil), base...), "up", "-d", service)
+	if isSelf {
+		return e.selfUpdate(ctx, job, self, workingDir, up, pinned, out, soFar)
+	}
+	if err := run(up); err != nil {
+		return "", err
 	}
 	return pinned, nil
 }
+
+// composeError describes a failed `docker compose <step>` call with the tail
+// of its output.
+func composeError(step string, err error, tail *tailBuffer) error {
+	return fmt.Errorf("docker compose %s: %w (output: %s)", step, err, strings.TrimSpace(tail.String()))
+}

internal/agent/updater/compose_test.go

@@ -1,6 +1,7 @@
 package updater
 
 import (
+	"bytes"
 	"context"
 	"errors"
 	"io"
@@ -8,9 +9,14 @@ import (
 	"path/filepath"
 	"strings"
 	"testing"
+	"time"
+	"unicode/utf8"
 
 	"github.com/docker/docker/api/types/container"
 	"github.com/docker/docker/api/types/mount"
+	"github.com/docker/docker/api/types/network"
+	"github.com/docker/docker/pkg/stdcopy"
+	ocispec "github.com/opencontainers/image-spec/specs-go/v1"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
@@ -26,6 +32,123 @@ func (f fakeCompose) ContainerList(context.Context, container.ListOptions) ([]co
 	return f.containers, nil
 }
 
+// fakeCompose is not meant to start helper containers; these fail the test
+// run if an executor tries.
+func (fakeCompose) ContainerCreate(context.Context, *container.Config, *container.HostConfig, *network.NetworkingConfig, *ocispec.Platform, string) (container.CreateResponse, error) {
+	panic("unexpected ContainerCreate")
+}
+func (fakeCompose) ContainerStart(context.Context, string, container.StartOptions) error {
+	panic("unexpected ContainerStart")
+}
+func (fakeCompose) ContainerWait(context.Context, string, container.WaitCondition) (<-chan container.WaitResponse, <-chan error) {
+	panic("unexpected ContainerWait")
+}
+func (fakeCompose) ContainerLogs(context.Context, string, container.LogsOptions) (io.ReadCloser, error) {
+	panic("unexpected ContainerLogs")
+}
+func (fakeCompose) ContainerRemove(context.Context, string, container.RemoveOptions) error {
+	panic("unexpected ContainerRemove")
+}
+
+// createdHelper is what a fakeHelper was asked to create.
+type createdHelper struct {
+	cfg  *container.Config
+	host *container.HostConfig
+}
+
+// fakeHelper is a helper container the test controls. Its logs are framed
+// like Docker's multiplexed stream.
+type fakeHelper struct {
+	createErr error
+	exit      int64
+	hold      bool   // ContainerWait never reports an exit
+	logs      string // what the helper printed
+	onStart   func()
+	state     container.ContainerState // what an inspect reports; exited when empty
+
+	created  []createdHelper
+	started  []string
+	removed  []string
+	waitCond container.WaitCondition
+}
+
+func (h *fakeHelper) ContainerCreate(_ context.Context, cfg *container.Config, host *container.HostConfig, _ *network.NetworkingConfig, _ *ocispec.Platform, _ string) (container.CreateResponse, error) {
+	if h.createErr != nil {
+		return container.CreateResponse{}, h.createErr
+	}
+	h.created = append(h.created, createdHelper{cfg, host})
+	return container.CreateResponse{ID: "helper1"}, nil
+}
+
+func (h *fakeHelper) ContainerInspect(context.Context, string) (container.InspectResponse, error) {
+	state := h.state
+	if state == "" {
+		state = container.StateExited
+	}
+	return container.InspectResponse{ContainerJSONBase: &container.ContainerJSONBase{State: &container.State{Status: state}}}, nil
+}
+
+func (h *fakeHelper) ContainerStart(_ context.Context, id string, _ container.StartOptions) error {
+	h.started = append(h.started, id)
+	if h.onStart != nil {
+		h.onStart()
+	}
+	return nil
+}
+
+func (h *fakeHelper) ContainerWait(_ context.Context, _ string, cond container.WaitCondition) (<-chan container.WaitResponse, <-chan error) {
+	h.waitCond = cond
+	res, errs := make(chan container.WaitResponse, 1), make(chan error, 1)
+	if !h.hold {
+		res <- container.WaitResponse{StatusCode: h.exit}
+	}
+	return res, errs
+}
+
+func (h *fakeHelper) ContainerLogs(context.Context, string, container.LogsOptions) (io.ReadCloser, error) {
+	var buf bytes.Buffer
+	stdcopy.NewStdWriter(&buf, stdcopy.Stdout).Write([]byte(h.logs))
+	return io.NopCloser(&buf), nil
+}
+
+func (h *fakeHelper) ContainerRemove(_ context.Context, id string, opts container.RemoveOptions) error {
+	if !opts.Force {
+		return errors.New("remove without force")
+	}
+	h.removed = append(h.removed, id)
+	return nil
+}
+
+// fakeHelperCompose is a fakeCompose whose helper containers are h.
+type fakeHelperCompose struct {
+	fakeCompose
+	h *fakeHelper
+}
+
+// ContainerInspect answers for the agent's own container ("me") like fakeCompose
+// and for any other ID like the helper.
+func (f fakeHelperCompose) ContainerInspect(ctx context.Context, id string) (container.InspectResponse, error) {
+	if id == "me" {
+		return f.fakeCompose.ContainerInspect(ctx, id)
+	}
+	return f.h.ContainerInspect(ctx, id)
+}
+func (f fakeHelperCompose) ContainerCreate(ctx context.Context, cfg *container.Config, host *container.HostConfig, n *network.NetworkingConfig, p *ocispec.Platform, name string) (container.CreateResponse, error) {
+	return f.h.ContainerCreate(ctx, cfg, host, n, p, name)
+}
+func (f fakeHelperCompose) ContainerStart(ctx context.Context, id string, o container.StartOptions) error {
+	return f.h.ContainerStart(ctx, id, o)
+}
+func (f fakeHelperCompose) ContainerWait(ctx context.Context, id string, c container.WaitCondition) (<-chan container.WaitResponse, <-chan error) {
+	return f.h.ContainerWait(ctx, id, c)
+}
+func (f fakeHelperCompose) ContainerLogs(ctx context.Context, id string, o container.LogsOptions) (io.ReadCloser, error) {
+	return f.h.ContainerLogs(ctx, id, o)
+}
+func (f fakeHelperCompose) ContainerRemove(ctx context.Context, id string, o container.RemoveOptions) error {
+	return f.h.ContainerRemove(ctx, id, o)
+}
+
 // outRunner is a fake OutputRunner for `docker compose config`.
 type outRunner struct {
 	json string
@@ -70,7 +193,7 @@ func TestComposeExecutor(t *testing.T) {
 	stack := filepath.Join(root, "slusk")
 	require.NoError(t, os.MkdirAll(stack, 0o755))
 	composeFile := stackFile(t, stack)
-	job := protocol.JobStart{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", Ref: "gitea.shcizo.se/shcizo/slusk:1.4"}
+	job := Job{ID: "7", Started: time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC), JobStart: protocol.JobStart{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", Ref: "gitea.shcizo.se/shcizo/slusk:1.4"}}
 	pinned := job.Ref + "@" + d
 
 	ct := container.Summary{Labels: map[string]string{
@@ -319,3 +442,139 @@ func TestExecDocker_UsesMinimalEnv(t *testing.T) {
 	assert.NotContains(t, string(raw), "AGENT_TOKEN")
 	assert.Contains(t, string(raw), "DOCKER_CONFIG=/docker-config")
 }
+
+func TestComposeSelfUpdate(t *testing.T) {
+	const d = "sha256:d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1"
+	root, _ := filepath.EvalSymlinks(t.TempDir())
+	stack := filepath.Join(root, "updater")
+	require.NoError(t, os.MkdirAll(stack, 0o755))
+	stackFile(t, stack)
+	job := Job{ID: "7", Started: time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC), JobStart: protocol.JobStart{
+		Kind: protocol.KindCompose, ComposeProject: "updater", ComposeService: "agent", Ref: "gitea.shcizo.se/shcizo/package-updater:1.4", Self: true}}
+	pinned := job.Ref + "@" + d
+	overridePath := filepath.Join(stack, OverrideName)
+	t.Cleanup(func() { os.Remove(overridePath) })
+
+	selfCt := container.Summary{ID: "selfid", Labels: map[string]string{
+		labelWorkingDir:  stack,
+		labelConfigFiles: filepath.Join(stack, "compose.yaml"),
+	}}
+	mounts := []container.MountPoint{
+		{Type: mount.TypeBind, Source: root, Destination: root, RW: true},
+		{Type: mount.TypeBind, Source: "/var/run/docker.sock", Destination: "/var/run/docker.sock", RW: true},
+		{Type: mount.TypeBind, Source: "/root/.docker/config.json", Destination: "/docker-config/config.json", RW: false},
+		{Type: mount.TypeVolume, Name: "agent-data", Source: "/var/lib/docker/volumes/agent-data/_data", Destination: "/data", RW: true},
+		{Type: mount.TypeTmpfs, Destination: "/tmp", RW: true},
+	}
+	self := fakeSelf{id: "selfid", image: "sha256:agentimage", mounts: mounts}
+	wantMounts := []mount.Mount{
+		{Type: mount.TypeBind, Source: root, Target: root},
+		{Type: mount.TypeBind, Source: "/var/run/docker.sock", Target: "/var/run/docker.sock"},
+		{Type: mount.TypeBind, Source: "/root/.docker/config.json", Target: "/docker-config/config.json", ReadOnly: true},
+		{Type: mount.TypeVolume, Source: "agent-data", Target: "/data"},
+	}
+	base := []string{"compose", "-p", "updater", "--project-directory", stack,
+		"-f", filepath.Join(stack, "compose.yaml"), "-f", overridePath}
+	withStep := func(step ...string) []string { return append(append([]string(nil), base...), step...) }
+
+	t.Setenv("PATH", "/usr/bin")
+	t.Setenv("DOCKER_CONFIG", "/docker-config")
+	t.Setenv("AGENT_TOKEN", "secret")
+
+	cases := []struct {
+		name        string
+		helper      *fakeHelper
+		cancelled   bool // ctx is cancelled when the helper starts
+		wantErr     string
+		wantCancel  bool
+		wantRemoved []string
+		wantOut     []string
+	}{
+		{name: "helper succeeds", helper: &fakeHelper{logs: "Container updater-agent-1  Recreated\n"},
+			wantRemoved: []string{"helper1"}, wantOut: []string{"Container updater-agent-1  Recreated"}},
+		{name: "helper fails", helper: &fakeHelper{exit: 1, logs: "network not found\n"},
+			wantErr: "docker compose up: exit status 1 (output: network not found)", wantRemoved: []string{"helper1"},
+			wantOut: []string{"network not found"}},
+		{name: "ctx ends while waiting", helper: &fakeHelper{hold: true}, cancelled: true, wantCancel: true},
+		{name: "create fails", helper: &fakeHelper{createErr: errors.New("no such image")}, wantErr: "create update helper: no such image"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			t.Cleanup(func() { os.Remove(overridePath) })
+			ctx, cancel := context.WithCancel(context.Background())
+			defer cancel()
+			if c.cancelled {
+				c.helper.onStart = cancel
+			}
+			rec := &recorder{}
+			cli := fakeHelperCompose{fakeCompose{self, []container.Summary{selfCt}}, c.helper}
+			e := NewComposeExecutor(cli, NewResolver(&fakeDist{digest: d}, nil), "me", rec.run, (&outRunner{json: `{"services":{"agent":{"image":"x"}}}`}).run)
+
+			var out strings.Builder
+			got, err := e.Execute(ctx, job, &out)
+
+			require.Len(t, rec.cmds, 1, "pull ran, up did not")
+			assert.Equal(t, ranCmd{stack, withStep("pull", "agent")}, rec.cmds[0])
+			assert.Equal(t, c.wantRemoved, c.helper.removed)
+			for _, s := range c.wantOut {
+				assert.Contains(t, out.String(), s)
+			}
+			switch {
+			case c.wantCancel:
+				assert.ErrorIs(t, err, context.Canceled)
+			case c.wantErr != "":
+				assert.EqualError(t, err, c.wantErr)
+				assert.Empty(t, got)
+			default:
+				require.NoError(t, err)
+				assert.Equal(t, pinned, got)
+			}
+			if len(c.helper.created) == 0 {
+				return
+			}
+			cfg, host := c.helper.created[0].cfg, c.helper.created[0].host
+			assert.Equal(t, "sha256:agentimage", cfg.Image)
+			assert.Equal(t, []string{"docker"}, []string(cfg.Entrypoint))
+			assert.Equal(t, withStep("up", "-d", "agent"), []string(cfg.Cmd))
+			assert.Equal(t, stack, cfg.WorkingDir)
+			assert.Equal(t, dockerEnv(), cfg.Env)
+			assert.Equal(t, wantMounts, host.Mounts)
+			assert.False(t, host.AutoRemove)
+			assert.Equal(t, container.RestartPolicy{}, host.RestartPolicy)
+			assert.Equal(t, []string{"helper1"}, c.helper.started)
+			assert.Equal(t, container.WaitConditionNextExit, c.helper.waitCond)
+			labels := cfg.Labels
+			assert.Len(t, labels, 6, "no compose labels")
+			assert.Equal(t, "7", labels[labelSelfUpdateJob])
+			assert.Equal(t, pinned, labels[labelSelfUpdateRef])
+			assert.Equal(t, "2026-10-13T09:00:00Z", labels[labelSelfUpdateStarted])
+			assert.Equal(t, "updater", labels[labelSelfUpdateProject])
+			assert.Equal(t, "agent", labels[labelSelfUpdateService])
+			assert.Contains(t, labels[labelSelfUpdateOutput], "resolved "+pinned)
+			assert.Contains(t, labels[labelSelfUpdateOutput], "$ docker "+strings.Join(withStep("pull", "agent"), " "))
+		})
+	}
+
+	t.Run("a service that is not the agent's own never uses a helper", func(t *testing.T) {
+		t.Cleanup(func() { os.Remove(overridePath) })
+		other := self
+		other.id = "otherid" // fakeCompose panics on ContainerCreate
+		rec := &recorder{}
+		e := NewComposeExecutor(fakeCompose{other, []container.Summary{selfCt}}, NewResolver(&fakeDist{digest: d}, nil), "me", rec.run,
+			(&outRunner{json: `{"services":{"agent":{"image":"x"}}}`}).run)
+		got, err := e.Execute(context.Background(), job, io.Discard)
+		require.NoError(t, err)
+		assert.Equal(t, pinned, got)
+		require.Len(t, rec.cmds, 2)
+		assert.Equal(t, withStep("up", "-d", "agent"), rec.cmds[1].args)
+	})
+
+	t.Run("output label keeps the tail and says it was cut", func(t *testing.T) {
+		b := &tailBuffer{max: 4}
+		b.Write([]byte("abcdef"))
+		assert.Equal(t, truncatedMarker+"cdef", outputLabel(b))
+		b2 := &tailBuffer{max: 3}
+		b2.Write([]byte("aéé")) // the cut falls inside a character
+		assert.True(t, utf8.ValidString(outputLabel(b2)))
+	})
+}

internal/agent/updater/dispatch.go

@@ -16,7 +16,7 @@ func NewDispatch(compose, swarm Executor) Executor {
 	return dispatch{compose: compose, swarm: swarm}
 }
 
-func (d dispatch) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) {
+func (d dispatch) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
 	switch job.Kind {
 	case protocol.KindCompose:
 		return d.compose.Execute(ctx, job, out)

internal/agent/updater/dispatch_test.go

@@ -12,20 +12,20 @@ import (
 )
 
 func TestDispatch(t *testing.T) {
-	compose := execFunc(func(context.Context, protocol.JobStart, io.Writer) (string, error) { return "compose", nil })
+	compose := execFunc(func(context.Context, Job, io.Writer) (string, error) { return "compose", nil })
-	swarm := execFunc(func(context.Context, protocol.JobStart, io.Writer) (string, error) { return "swarm", nil })
+	swarm := execFunc(func(context.Context, Job, io.Writer) (string, error) { return "swarm", nil })
 	d := NewDispatch(compose, swarm)
 
-	got, err := d.Execute(context.Background(), protocol.JobStart{Kind: protocol.KindCompose}, io.Discard)
+	got, err := d.Execute(context.Background(), Job{JobStart: protocol.JobStart{Kind: protocol.KindCompose}}, io.Discard)
 	require.NoError(t, err)
 	assert.Equal(t, "compose", got)
 
-	got, err = d.Execute(context.Background(), protocol.JobStart{Kind: protocol.KindSwarm}, io.Discard)
+	got, err = d.Execute(context.Background(), Job{JobStart: protocol.JobStart{Kind: protocol.KindSwarm}}, io.Discard)
 	require.NoError(t, err)
 	assert.Equal(t, "swarm", got)
 
-	_, err = d.Execute(context.Background(), protocol.JobStart{Kind: protocol.KindContainer}, io.Discard)
+	_, err = d.Execute(context.Background(), Job{JobStart: protocol.JobStart{Kind: protocol.KindContainer}}, io.Discard)
 	assert.ErrorContains(t, err, "cannot update")
-	_, err = d.Execute(context.Background(), protocol.JobStart{Kind: "weird"}, io.Discard)
+	_, err = d.Execute(context.Background(), Job{JobStart: protocol.JobStart{Kind: "weird"}}, io.Discard)
 	assert.ErrorContains(t, err, "cannot update")
 }

internal/agent/updater/override.go

@@ -177,20 +177,35 @@ var errNoRoots = errors.New("refused: no stack directory is mounted into the age
 // path inside the container. Compose labels hold host paths, so only those
 // mounts make a label path usable as is.
 func MountRoots(ctx context.Context, cli SelfInspector, selfID string) ([]string, error) {
+	insp, err := inspectSelf(ctx, cli, selfID)
+	if err != nil {
+		return nil, err
+	}
+	return rootsOf(insp), nil
+}
+
+// inspectSelf inspects the agent's own container.
+func inspectSelf(ctx context.Context, cli SelfInspector, selfID string) (container.InspectResponse, error) {
 	if selfID == "" {
-		return nil, errNoRoots
+		return container.InspectResponse{}, errNoRoots
 	}
 	insp, err := cli.ContainerInspect(ctx, selfID)
 	if err != nil {
-		return nil, fmt.Errorf("inspect own container %s: %w", selfID, err)
+		return container.InspectResponse{}, fmt.Errorf("inspect own container %s: %w", selfID, err)
 	}
+	return insp, nil
+}
+
+// rootsOf returns the directories MountRoots describes, from an inspected
+// container.
+func rootsOf(insp container.InspectResponse) []string {
 	var roots []string
 	for _, m := range insp.Mounts {
 		if m.Type == mount.TypeBind && m.RW && filepath.Clean(m.Source) == filepath.Clean(m.Destination) {
 			roots = append(roots, filepath.Clean(m.Destination))
 		}
 	}
-	return roots, nil
+	return roots
 }
 
 // composeFiles returns the -f arguments for a job: the stack's config files

internal/agent/updater/override_test.go

@@ -175,10 +175,20 @@ func TestWriteOverride(t *testing.T) {
 type fakeSelf struct {
 	mounts []container.MountPoint
 	err    error
+	id     string            // the container's ID, when set
+	image  string            // the container's image ID, when set
+	labels map[string]string // the container's labels, when set
 }
 
 func (f fakeSelf) ContainerInspect(context.Context, string) (container.InspectResponse, error) {
-	return container.InspectResponse{Mounts: f.mounts}, f.err
+	insp := container.InspectResponse{Mounts: f.mounts}
+	if f.id != "" || f.image != "" {
+		insp.ContainerJSONBase = &container.ContainerJSONBase{ID: f.id, Image: f.image}
+	}
+	if f.labels != nil {
+		insp.Config = &container.Config{Labels: f.labels}
+	}
+	return insp, f.err
 }
 
 func TestMountRoots(t *testing.T) {

internal/agent/updater/runner.go

@@ -16,12 +16,25 @@ import (
 // chatty job cannot grow a single job.progress message without bound.
 const maxProgress = 32 << 10
 
+// Job is a job as an Executor sees it: what the master asked for, plus the
+// ID and start time the runner gave it. A self-update records both in the
+// helper container, so the new agent can finish the job under the same ID.
+type Job struct {
+	protocol.JobStart
+	ID      string
+	Started time.Time
+}
+
 // Executor deploys one job and returns the "repo:tag@digest" it deployed.
 // Everything it writes to out is streamed to the master as job output.
 type Executor interface {
-	Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error)
+	Execute(ctx context.Context, job Job, out io.Writer) (string, error)
 }
 
+// ResumeFunc finishes a job that an earlier agent process began. It returns
+// the "repo:tag@digest" that was deployed, like Executor.Execute.
+type ResumeFunc func(ctx context.Context, out io.Writer) (string, error)
+
 // Progress is output a running job produced since the previous TakeProgress.
 type Progress struct{ ID, Output string }
 
@@ -52,6 +65,8 @@ func (b *tailBuffer) Reset()          { b.buf, b.truncated = nil, false }
 
 type jobState struct {
 	start    protocol.JobStart
+	resume   ResumeFunc // set for a job resumed after a restart; start is then empty
+	started  time.Time  // set with resume
 	running  bool
 	result   *protocol.JobResult
 	output   tailBuffer // everything, capped at protocol.MaxJobOutput
@@ -117,6 +132,28 @@ func (r *Runner) Submit(id string, start protocol.JobStart) {
 	signal(r.wake)
 }
 
+// Resume registers a job that an earlier agent process began and runs fn for
+// it, before every job that is already queued. The job is in Known at once,
+// so the agent's next hello does not report it lost. The result carries
+// started as its start time. A job whose ID the runner already holds is
+// ignored.
+func (r *Runner) Resume(id string, started time.Time, fn ResumeFunc) {
+	r.mu.Lock()
+	if _, ok := r.jobs[id]; ok {
+		r.mu.Unlock()
+		return
+	}
+	r.jobs[id] = &jobState{
+		resume:  fn,
+		started: started,
+		output:  tailBuffer{max: protocol.MaxJobOutput},
+		unsent:  tailBuffer{max: maxProgress},
+	}
+	r.queue = append([]string{id}, r.queue...)
+	r.mu.Unlock()
+	signal(r.wake)
+}
+
 // Ack forgets a finished job. Acks for unknown, queued or running jobs are
 // ignored.
 func (r *Runner) Ack(id string) {
@@ -196,10 +233,12 @@ func (r *Runner) Pending() []Finished {
 func (r *Runner) Changed() <-chan struct{} { return r.changed }
 
 // Run works through the queue until ctx ends. Jobs still queued then are
-// dropped; the master marks them lost on the agent's next hello.
+// dropped; the master marks them lost on the agent's next hello. A job that
+// ctx interrupts is dropped too, without a result: a self-updating agent is
+// stopped that way, and its successor reports the outcome.
 func (r *Runner) Run(ctx context.Context) {
 	for {
-		id, start, ok := r.next()
+		id, ok := r.next()
 		if !ok {
 			select {
 			case <-ctx.Done():
@@ -211,32 +250,62 @@ func (r *Runner) Run(ctx context.Context) {
 		if ctx.Err() != nil {
 			return
 		}
-		r.runOne(ctx, id, start)
+		r.runOne(ctx, id)
 	}
 }
 
-// next pops the next queued job and marks it running.
+// next pops the next job to run and marks it running. Jobs for the agent's
-func (r *Runner) next() (string, protocol.JobStart, bool) {
+// own service run last: a master and an agent on the same host share an
+// image, and the master's job must finish before the agent replaces itself.
+func (r *Runner) next() (string, bool) {
 	r.mu.Lock()
 	defer r.mu.Unlock()
 	if len(r.queue) == 0 {
-		return "", protocol.JobStart{}, false
+		return "", false
 	}
-	id := r.queue[0]
+	pick := 0
-	r.queue = r.queue[1:]
+	for i, id := range r.queue {
-	j := r.jobs[id]
+		if !r.jobs[id].start.Self {
-	j.running = true
+			pick = i
+			break
+		}
+	}
+	id := r.queue[pick]
+	r.queue = append(r.queue[:pick:pick], r.queue[pick+1:]...)
+	r.jobs[id].running = true
 	signal(r.changed)
-	return id, j.start, true
+	return id, true
 }
 
-func (r *Runner) runOne(ctx context.Context, id string, start protocol.JobStart) {
+func (r *Runner) runOne(ctx context.Context, id string) {
-	started := r.now()
+	r.mu.Lock()
+	j := r.jobs[id]
+	start, resume, started := j.start, j.resume, j.started
+	r.mu.Unlock()
+	if resume == nil {
+		started = r.now()
+	}
+
 	jobCtx, cancel := context.WithTimeout(ctx, r.timeout)
-	ref, err := r.exec.Execute(jobCtx, start, &jobWriter{r: r, id: id})
+	var ref string
+	var err error
+	if resume != nil {
+		ref, err = resume(jobCtx, &jobWriter{r: r, id: id})
+	} else {
+		ref, err = r.exec.Execute(jobCtx, Job{JobStart: start, ID: id, Started: started}, &jobWriter{r: r, id: id})
+	}
 	timedOut := errors.Is(jobCtx.Err(), context.DeadlineExceeded)
 	cancel()
 
+	if err != nil && ctx.Err() != nil && !timedOut {
+		// The agent is stopping, not the job failing. Nothing is recorded, so
+		// no result can pre-empt the one a successor agent may report.
+		r.mu.Lock()
+		delete(r.jobs, id)
+		r.mu.Unlock()
+		return
+	}
+
 	res := protocol.JobResult{Status: protocol.ResultSucceeded, Ref: ref, StartedAt: started, FinishedAt: r.now()}
 	switch {
 	case timedOut:
@@ -246,7 +315,6 @@ func (r *Runner) runOne(ctx context.Context, id string, start protocol.JobStart)
 	}
 
 	r.mu.Lock()
-	j := r.jobs[id]
 	res.Output, res.OutputTruncated = j.output.String(), j.output.Truncated()
 	j.running, j.result, j.finished = false, &res, res.FinishedAt
 	j.unsent.Reset()

internal/agent/updater/runner_test.go

@@ -29,7 +29,7 @@ func newFakeExec() *fakeExec {
 	return &fakeExec{gate: make(chan struct{}), started: make(chan string, 10)}
 }
 
-func (f *fakeExec) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) {
+func (f *fakeExec) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
 	n := f.cur.Add(1)
 	defer f.cur.Add(-1)
 	for {
@@ -148,7 +148,7 @@ func TestRunner_TakeProgress(t *testing.T) {
 func TestRunner_TakeProgressReportsStartedJobOnce(t *testing.T) {
 	gate := make(chan struct{})
 	started := make(chan struct{})
-	r := startRunner(t, execFunc(func(ctx context.Context, _ protocol.JobStart, _ io.Writer) (string, error) {
+	r := startRunner(t, execFunc(func(ctx context.Context, _ Job, _ io.Writer) (string, error) {
 		close(started)
 		<-gate
 		return "x", nil
@@ -160,14 +160,14 @@ func TestRunner_TakeProgressReportsStartedJobOnce(t *testing.T) {
 	close(gate)
 }
 
-type execFunc func(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error)
+type execFunc func(ctx context.Context, job Job, out io.Writer) (string, error)
 
-func (f execFunc) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) {
+func (f execFunc) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
 	return f(ctx, job, out)
 }
 
 func TestRunner_Timeout(t *testing.T) {
-	r := startRunner(t, execFunc(func(ctx context.Context, _ protocol.JobStart, _ io.Writer) (string, error) {
+	r := startRunner(t, execFunc(func(ctx context.Context, _ Job, _ io.Writer) (string, error) {
 		<-ctx.Done()
 		return "", ctx.Err()
 	}), 20*time.Millisecond)
@@ -178,7 +178,7 @@ func TestRunner_Timeout(t *testing.T) {
 }
 
 func TestRunner_OutputCapped(t *testing.T) {
-	r := startRunner(t, execFunc(func(_ context.Context, _ protocol.JobStart, out io.Writer) (string, error) {
+	r := startRunner(t, execFunc(func(_ context.Context, _ Job, out io.Writer) (string, error) {
 		io.WriteString(out, strings.Repeat("a", 10))
 		io.WriteString(out, strings.Repeat("b", protocol.MaxJobOutput))
 		return "x", nil
@@ -190,7 +190,7 @@ func TestRunner_OutputCapped(t *testing.T) {
 }
 
 func TestRunner_ChangedSignals(t *testing.T) {
-	r := startRunner(t, execFunc(func(context.Context, protocol.JobStart, io.Writer) (string, error) { return "x", nil }), time.Minute)
+	r := startRunner(t, execFunc(func(context.Context, Job, io.Writer) (string, error) { return "x", nil }), time.Minute)
 	r.Submit("1", protocol.JobStart{Ref: "a:1"})
 	select {
 	case <-r.Changed():
@@ -198,3 +198,105 @@ func TestRunner_ChangedSignals(t *testing.T) {
 		t.Fatal("no change signal")
 	}
 }
+
+func TestRunner_ResumeJumpsTheQueue(t *testing.T) {
+	fe := newFakeExec()
+	r := NewRunner(fe, time.Minute)
+	r.Submit("1", protocol.JobStart{Ref: "a:1"})
+	started := time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC)
+	r.Resume("9", started, func(_ context.Context, out io.Writer) (string, error) {
+		io.WriteString(out, "from the helper\n")
+		return "resumed@sha256:bb", nil
+	})
+	assert.Equal(t, []string{"1", "9"}, r.Known(), "known before the worker starts")
+
+	ctx, cancel := context.WithCancel(context.Background())
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() { defer wg.Done(); r.Run(ctx) }()
+	t.Cleanup(func() { cancel(); wg.Wait() })
+
+	got := waitPending(t, r, 1)
+	require.Len(t, got, 1, "the resumed job runs first; job 1 waits on the gate")
+	assert.Equal(t, "9", got[0].ID)
+	assert.Equal(t, protocol.ResultSucceeded, got[0].Result.Status)
+	assert.Equal(t, "resumed@sha256:bb", got[0].Result.Ref)
+	assert.Equal(t, "from the helper\n", got[0].Result.Output)
+	assert.True(t, started.Equal(got[0].Result.StartedAt))
+
+	r.Resume("9", started, nil) // already held: ignored
+	assert.Equal(t, "a:1", <-fe.started)
+	fe.gate <- struct{}{}
+	waitPending(t, r, 2)
+}
+
+func TestRunner_ResumedJobFailure(t *testing.T) {
+	r := startRunner(t, newFakeExec(), time.Minute)
+	r.Resume("9", time.Now(), func(context.Context, io.Writer) (string, error) { return "", errors.New("helper failed") })
+	got := waitPending(t, r, 1)
+	assert.Equal(t, protocol.ResultFailed, got[0].Result.Status)
+	assert.Equal(t, "helper failed", got[0].Result.Error)
+}
+
+func TestRunner_SelfJobsRunLast(t *testing.T) {
+	fe := newFakeExec()
+	r := NewRunner(fe, time.Minute)
+	r.Submit("1", protocol.JobStart{Ref: "self:1", Self: true})
+	r.Submit("2", protocol.JobStart{Ref: "a:2"})
+	r.Submit("3", protocol.JobStart{Ref: "self:3", Self: true})
+	r.Submit("4", protocol.JobStart{Ref: "a:4"})
+
+	ctx, cancel := context.WithCancel(context.Background())
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() { defer wg.Done(); r.Run(ctx) }()
+	t.Cleanup(func() { cancel(); wg.Wait() })
+
+	var order []string
+	for i := 0; i < 4; i++ {
+		order = append(order, <-fe.started)
+		fe.gate <- struct{}{}
+	}
+	assert.Equal(t, []string{"a:2", "a:4", "self:1", "self:3"}, order)
+}
+
+func TestRunner_SelfJobRunsWhenAlone(t *testing.T) {
+	fe := newFakeExec()
+	r := startRunner(t, fe, time.Minute)
+	r.Submit("1", protocol.JobStart{Ref: "self:1", Self: true})
+	assert.Equal(t, "self:1", <-fe.started)
+	fe.gate <- struct{}{}
+	waitPending(t, r, 1)
+}
+
+func TestRunner_StoppingTheAgentRecordsNoResult(t *testing.T) {
+	r := NewRunner(execFunc(func(ctx context.Context, _ Job, _ io.Writer) (string, error) {
+		<-ctx.Done()
+		return "", ctx.Err()
+	}), time.Minute)
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() { r.Run(ctx); close(done) }()
+	r.Submit("1", protocol.JobStart{Ref: "a:1"})
+	require.Eventually(t, func() bool { return len(r.TakeProgress()) > 0 }, time.Second, 5*time.Millisecond)
+
+	cancel()
+	<-done
+	assert.Empty(t, r.Pending(), "no \"context canceled\" result")
+	assert.Empty(t, r.Known(), "the job is dropped, so the master marks it lost unless a successor resumes it")
+}
+
+func TestRunner_ExecutorGetsIDAndStart(t *testing.T) {
+	var got Job
+	r := startRunner(t, execFunc(func(_ context.Context, job Job, _ io.Writer) (string, error) {
+		got = job
+		return "x", nil
+	}), time.Minute)
+	now := time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC)
+	r.now = func() time.Time { return now }
+	r.Submit("5", protocol.JobStart{Ref: "a:1"})
+	waitPending(t, r, 1)
+	assert.Equal(t, "5", got.ID)
+	assert.Equal(t, "a:1", got.Ref)
+	assert.True(t, now.Equal(got.Started))
+}

internal/agent/updater/selfupdate.go

@@ -0,0 +1,279 @@
+package updater
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"log/slog"
+	"strings"
+	"time"
+
+	"github.com/docker/docker/api/types/container"
+	"github.com/docker/docker/api/types/filters"
+	"github.com/docker/docker/api/types/mount"
+	"github.com/docker/docker/api/types/network"
+	"github.com/docker/docker/pkg/stdcopy"
+	ocispec "github.com/opencontainers/image-spec/specs-go/v1"
+)
+
+// Labels on a self-update helper container. The new agent finds the helper
+// by them, and the inventory skips containers that carry the job label.
+const (
+	labelSelfUpdateJob     = "se.shcizo.package-updater.self-update.job"
+	labelSelfUpdateRef     = "se.shcizo.package-updater.self-update.ref"
+	labelSelfUpdateStarted = "se.shcizo.package-updater.self-update.started"
+	labelSelfUpdateProject = "se.shcizo.package-updater.self-update.project"
+	labelSelfUpdateService = "se.shcizo.package-updater.self-update.service"
+	labelSelfUpdateOutput  = "se.shcizo.package-updater.self-update.output"
+)
+
+// selfUpdateOutputMax caps the job output stored in the helper container's
+// output label. The old agent's output so far is all the new agent has of it.
+const selfUpdateOutputMax = 32 << 10
+
+// truncatedMarker starts the stored output when its head was cut off.
+const truncatedMarker = "…truncated\n"
+
+// resumeSeparator follows the old agent's output in a resumed job's output.
+const resumeSeparator = "--- agent restarted; output of the update helper ---\n"
+
+// HelperDocker is the subset of the Docker SDK the self-update helper
+// container needs. *client.Client satisfies it.
+type HelperDocker interface {
+	ContainerInspect(ctx context.Context, containerID string) (container.InspectResponse, error)
+	ContainerCreate(ctx context.Context, config *container.Config, hostConfig *container.HostConfig, networkingConfig *network.NetworkingConfig, platform *ocispec.Platform, containerName string) (container.CreateResponse, error)
+	ContainerStart(ctx context.Context, containerID string, options container.StartOptions) error
+	ContainerWait(ctx context.Context, containerID string, condition container.WaitCondition) (<-chan container.WaitResponse, <-chan error)
+	ContainerLogs(ctx context.Context, containerID string, options container.LogsOptions) (io.ReadCloser, error)
+	ContainerRemove(ctx context.Context, containerID string, options container.RemoveOptions) error
+}
+
+// selfUpdate runs `up -d` for the agent's own service in a short-lived helper
+// container. Recreating the service stops this process, so the helper, made
+// from the agent's own image with the agent's own mounts, does it and
+// outlives the agent. args are the complete `docker` arguments of the call.
+//
+// If the helper exits while this agent still runs, selfUpdate copies its logs
+// into the job output, removes it and returns the outcome. If ctx ends first
+// (the agent is being replaced, or the job timed out) it returns ctx's error
+// and leaves the helper to the new agent, see ResumeSelfUpdates.
+func (e *ComposeExecutor) selfUpdate(ctx context.Context, job Job, self container.InspectResponse, workingDir string, args []string, pinned string, out io.Writer, soFar *tailBuffer) (string, error) {
+	fmt.Fprintf(out, "$ docker %s (in an update helper container)\n", strings.Join(args, " "))
+	cfg := &container.Config{
+		Image:      self.Image,
+		Entrypoint: []string{"docker"},
+		Cmd:        args,
+		WorkingDir: workingDir,
+		Env:        dockerEnv(),
+		Labels: map[string]string{
+			labelSelfUpdateJob:     job.ID,
+			labelSelfUpdateRef:     pinned,
+			labelSelfUpdateStarted: job.Started.UTC().Format(time.RFC3339Nano),
+			labelSelfUpdateProject: job.ComposeProject,
+			labelSelfUpdateService: job.ComposeService,
+			labelSelfUpdateOutput:  outputLabel(soFar),
+		},
+	}
+	created, err := e.cli.ContainerCreate(ctx, cfg, &container.HostConfig{Mounts: helperMounts(self)}, nil, nil, "")
+	if err != nil {
+		return "", fmt.Errorf("create update helper: %w", err)
+	}
+	// Wait is registered before the start so a fast exit is not missed.
+	waitCh, errCh := e.cli.ContainerWait(ctx, created.ID, container.WaitConditionNextExit)
+	if err := e.cli.ContainerStart(ctx, created.ID, container.StartOptions{}); err != nil {
+		if rmErr := e.cli.ContainerRemove(context.WithoutCancel(ctx), created.ID, container.RemoveOptions{Force: true}); rmErr != nil {
+			fmt.Fprintf(out, "warning: remove the update helper: %v\n", rmErr)
+		}
+		return "", fmt.Errorf("start update helper: %w", err)
+	}
+	code, err := awaitExit(ctx, waitCh, errCh)
+	if err != nil {
+		return "", err
+	}
+
+	tail := &tailBuffer{max: errorOutputTail}
+	if err := finishHelper(ctx, e.cli, created.ID, io.MultiWriter(out, tailWriter{tail})); err != nil {
+		fmt.Fprintf(out, "warning: %v\n", err)
+	}
+	if code != 0 {
+		return "", exitError(code, tail)
+	}
+	return pinned, nil
+}
+
+// outputLabel returns the job output so far as a label value.
+func outputLabel(b *tailBuffer) string {
+	s := strings.ToValidUTF8(b.String(), "")
+	if b.Truncated() {
+		s = truncatedMarker + s
+	}
+	return s
+}
+
+// helperMounts copies the agent's own bind and volume mounts: the Docker
+// socket, the stack directories and the registry credentials the compose call
+// needs are all among them.
+func helperMounts(self container.InspectResponse) []mount.Mount {
+	var mounts []mount.Mount
+	for _, m := range self.Mounts {
+		switch m.Type {
+		case mount.TypeBind:
+			mounts = append(mounts, mount.Mount{Type: mount.TypeBind, Source: m.Source, Target: m.Destination, ReadOnly: !m.RW})
+		case mount.TypeVolume:
+			mounts = append(mounts, mount.Mount{Type: mount.TypeVolume, Source: m.Name, Target: m.Destination, ReadOnly: !m.RW})
+		}
+	}
+	return mounts
+}
+
+// awaitExit waits for a ContainerWait result and returns the exit code. When
+// ctx ends it returns ctx's error.
+func awaitExit(ctx context.Context, waitCh <-chan container.WaitResponse, errCh <-chan error) (int64, error) {
+	select {
+	case r := <-waitCh:
+		if r.Error != nil {
+			return 0, fmt.Errorf("wait for update helper: %s", r.Error.Message)
+		}
+		return r.StatusCode, nil
+	case err := <-errCh:
+		if ctx.Err() != nil {
+			return 0, ctx.Err()
+		}
+		return 0, fmt.Errorf("wait for update helper: %w", err)
+	case <-ctx.Done():
+		return 0, ctx.Err()
+	}
+}
+
+// finishHelper copies the exited helper's logs to w and removes the helper.
+// Both are attempted; the first failure is returned.
+func finishHelper(ctx context.Context, cli HelperDocker, id string, w io.Writer) error {
+	var first error
+	rc, err := cli.ContainerLogs(ctx, id, container.LogsOptions{ShowStdout: true, ShowStderr: true})
+	if err == nil {
+		_, err = stdcopy.StdCopy(w, w, rc)
+		rc.Close()
+	}
+	if err != nil {
+		first = fmt.Errorf("read the update helper's logs: %w", err)
+	}
+	if err := cli.ContainerRemove(ctx, id, container.RemoveOptions{Force: true}); err != nil && first == nil {
+		first = fmt.Errorf("remove the update helper: %w", err)
+	}
+	return first
+}
+
+// exitError describes a failed helper like a failed `docker compose` call.
+func exitError(code int64, tail *tailBuffer) error {
+	return composeError("up", fmt.Errorf("exit status %d", code), tail)
+}
+
+// ResumeDocker is the subset of the Docker SDK ResumeSelfUpdates needs.
+// *client.Client satisfies it.
+type ResumeDocker interface {
+	SelfInspector
+	HelperDocker
+	ContainerList(ctx context.Context, opts container.ListOptions) ([]container.Summary, error)
+}
+
+// ResumeSelfUpdates finishes the jobs a previous agent process started with
+// selfUpdate. It finds the helper containers made for this agent's own
+// Compose service and registers each with runner.Resume, which reports the
+// job under its old ID once the helper has exited. Call it before the runner
+// and the connection start, so the first hello already lists these jobs. A
+// helper with unusable labels is resumed too; its job fails with an error
+// that says so.
+func ResumeSelfUpdates(ctx context.Context, cli ResumeDocker, selfID string, runner *Runner, log *slog.Logger) error {
+	self, err := inspectSelf(ctx, cli, selfID)
+	if err != nil {
+		return err
+	}
+	if self.Config == nil {
+		return nil
+	}
+	project, service := self.Config.Labels[labelComposeProject], self.Config.Labels[labelComposeService]
+	if project == "" || service == "" {
+		return nil
+	}
+	cts, err := cli.ContainerList(ctx, container.ListOptions{
+		All:     true,
+		Filters: filters.NewArgs(filters.Arg("label", labelSelfUpdateJob)),
+	})
+	if err != nil {
+		return fmt.Errorf("list update helpers: %w", err)
+	}
+	for _, ct := range cts {
+		if ct.Labels[labelSelfUpdateProject] != project || ct.Labels[labelSelfUpdateService] != service {
+			continue
+		}
+		jobID := ct.Labels[labelSelfUpdateJob]
+		if jobID == "" {
+			log.Warn("self_update_helper_without_job", "helper", ct.ID)
+			continue
+		}
+		ref, started, labelErr := parseHelperLabels(ct.Labels)
+		if started.IsZero() {
+			started = time.Now()
+		}
+		log.Info("self_update_resumed", "job_id", jobID, "helper", ct.ID)
+		runner.Resume(jobID, started, resumeFunc(cli, ct.ID, ct.Labels[labelSelfUpdateOutput], ref, labelErr))
+	}
+	return nil
+}
+
+// parseHelperLabels reads the pinned ref and start time from a helper's
+// labels. started is zero when it is missing or unreadable, and err then says
+// what is wrong.
+func parseHelperLabels(labels map[string]string) (ref string, started time.Time, err error) {
+	ref = labels[labelSelfUpdateRef]
+	if ref == "" {
+		err = errors.New("update helper has no ref label")
+	}
+	raw := labels[labelSelfUpdateStarted]
+	if t, perr := time.Parse(time.RFC3339Nano, raw); perr == nil {
+		started = t
+	} else if err == nil {
+		err = fmt.Errorf("update helper has no usable started label %q", raw)
+	}
+	return ref, started, err
+}
+
+// resumeFunc waits for the helper to stop, then reports the old agent's
+// output, a separator line and the helper's own logs as the job output,
+// removes the helper and returns the pinned ref, or an error when the helper
+// failed or its labels were unusable.
+func resumeFunc(cli HelperDocker, helperID, earlier, ref string, labelErr error) ResumeFunc {
+	return func(ctx context.Context, out io.Writer) (string, error) {
+		// A helper that was created but never started would read as exited
+		// with code 0, so it must not be mistaken for a finished update.
+		if insp, err := cli.ContainerInspect(ctx, helperID); err == nil && insp.ContainerJSONBase != nil &&
+			insp.State != nil && insp.State.Status == container.StateCreated {
+			if err := cli.ContainerRemove(context.WithoutCancel(ctx), helperID, container.RemoveOptions{Force: true}); err != nil {
+				fmt.Fprintf(out, "warning: remove the update helper: %v\n", err)
+			}
+			return "", errors.New("update helper was never started")
+		}
+		waitCh, errCh := cli.ContainerWait(ctx, helperID, container.WaitConditionNotRunning)
+		code, err := awaitExit(ctx, waitCh, errCh)
+		if err != nil {
+			return "", err
+		}
+		io.WriteString(out, earlier)
+		if earlier != "" && !strings.HasSuffix(earlier, "\n") {
+			io.WriteString(out, "\n")
+		}
+		io.WriteString(out, resumeSeparator)
+		tail := &tailBuffer{max: errorOutputTail}
+		if err := finishHelper(ctx, cli, helperID, io.MultiWriter(out, tailWriter{tail})); err != nil {
+			fmt.Fprintf(out, "warning: %v\n", err)
+		}
+		switch {
+		case labelErr != nil:
+			return "", labelErr
+		case code != 0:
+			return "", exitError(code, tail)
+		}
+		return ref, nil
+	}
+}

internal/agent/updater/selfupdate_test.go

@@ -0,0 +1,116 @@
+package updater
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/docker/docker/api/types/container"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/shcizo/package-updater/internal/protocol"
+)
+
+func TestResumeSelfUpdates(t *testing.T) {
+	const pinned = "gitea.shcizo.se/shcizo/package-updater:1.4@sha256:aa"
+	started := time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC)
+	self := fakeSelf{id: "selfid", labels: map[string]string{labelComposeProject: "updater", labelComposeService: "agent"}}
+
+	helper := func(id, project, service string, extra map[string]string) container.Summary {
+		labels := map[string]string{
+			labelSelfUpdateJob:     "j-" + id,
+			labelSelfUpdateRef:     pinned,
+			labelSelfUpdateStarted: started.Format(time.RFC3339Nano),
+			labelSelfUpdateProject: project,
+			labelSelfUpdateService: service,
+			labelSelfUpdateOutput:  "resolved " + pinned + "\n$ docker compose pull agent\n",
+		}
+		for k, v := range extra {
+			if v == "" {
+				delete(labels, k)
+			} else {
+				labels[k] = v
+			}
+		}
+		return container.Summary{ID: id, Labels: labels}
+	}
+	separator := "--- agent restarted; output of the update helper ---\n"
+
+	cases := []struct {
+		name       string
+		helper     container.Summary
+		exit       int64
+		state      container.ContainerState
+		noStarted  bool // the old agent's start time is lost with the label
+		wantStatus string
+		wantErr    string
+		wantOutput string
+	}{
+		{name: "helper succeeded", helper: helper("h1", "updater", "agent", nil), wantStatus: protocol.ResultSucceeded,
+			wantOutput: "resolved " + pinned + "\n$ docker compose pull agent\n" + separator + "Recreated\n"},
+		{name: "helper failed", helper: helper("h1", "updater", "agent", nil), exit: 1, wantStatus: protocol.ResultFailed,
+			wantErr: "docker compose up: exit status 1 (output: Recreated)"},
+		{name: "no ref label", helper: helper("h1", "updater", "agent", map[string]string{labelSelfUpdateRef: ""}),
+			wantStatus: protocol.ResultFailed, wantErr: "update helper has no ref label"},
+		{name: "no started label", helper: helper("h1", "updater", "agent", map[string]string{labelSelfUpdateStarted: ""}),
+			noStarted:  true,
+			wantStatus: protocol.ResultFailed, wantErr: `update helper has no usable started label ""`},
+		{name: "helper never started", helper: helper("h1", "updater", "agent", nil), state: container.StateCreated,
+			wantStatus: protocol.ResultFailed, wantErr: "update helper was never started"},
+		{name: "no output label", helper: helper("h1", "updater", "agent", map[string]string{labelSelfUpdateOutput: ""}),
+			wantStatus: protocol.ResultSucceeded, wantOutput: separator + "Recreated\n"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			h := &fakeHelper{exit: c.exit, state: c.state, logs: "Recreated\n"}
+			unrelated := []container.Summary{
+				helper("other-project", "other", "agent", nil),
+				helper("other-service", "updater", "web", nil),
+			}
+			cli := fakeHelperCompose{fakeCompose{self, append(unrelated, c.helper)}, h}
+			r := NewRunner(execFunc(func(context.Context, Job, io.Writer) (string, error) {
+				t.Error("a resumed job must not reach the executor")
+				return "", nil
+			}), time.Minute)
+
+			require.NoError(t, ResumeSelfUpdates(context.Background(), cli, "me", r, slog.New(slog.NewTextHandler(io.Discard, nil))))
+			assert.Equal(t, []string{"j-h1"}, r.Known(), "known before the runner starts, so the first hello lists it")
+
+			ctx, cancel := context.WithCancel(context.Background())
+			var wg sync.WaitGroup
+			wg.Add(1)
+			go func() { defer wg.Done(); r.Run(ctx) }()
+			t.Cleanup(func() { cancel(); wg.Wait() })
+
+			got := waitPending(t, r, 1)
+			require.Len(t, got, 1)
+			assert.Equal(t, "j-h1", got[0].ID)
+			res := got[0].Result
+			assert.Equal(t, c.wantStatus, res.Status)
+			if c.state != container.StateCreated {
+				assert.Equal(t, container.WaitConditionNotRunning, h.waitCond)
+			}
+			assert.Equal(t, []string{"h1"}, h.removed, "only the matching helper is touched")
+			if c.wantStatus == protocol.ResultSucceeded {
+				assert.Equal(t, pinned, res.Ref)
+				assert.Equal(t, c.wantOutput, res.Output)
+			} else {
+				assert.Equal(t, c.wantErr, res.Error)
+			}
+			if !c.noStarted {
+				assert.True(t, started.Equal(res.StartedAt), "the old agent's start time is kept")
+			}
+		})
+	}
+
+	t.Run("an agent outside Compose resumes nothing", func(t *testing.T) {
+		cli := fakeHelperCompose{fakeCompose{fakeSelf{id: "selfid"}, []container.Summary{helper("h1", "updater", "agent", nil)}}, &fakeHelper{}}
+		r := NewRunner(nil, time.Minute)
+		require.NoError(t, ResumeSelfUpdates(context.Background(), cli, "me", r, slog.New(slog.NewTextHandler(io.Discard, nil))))
+		assert.Empty(t, r.Known())
+	})
+}

internal/agent/updater/swarm.go

@@ -9,7 +9,6 @@ import (
 	"github.com/docker/docker/api/types/system"
 
 	"github.com/shcizo/package-updater/internal/imageref"
-	"github.com/shcizo/package-updater/internal/protocol"
 )
 
 // SwarmClient is the subset of the Docker SDK SwarmExecutor depends on.
@@ -40,7 +39,7 @@ func NewSwarmExecutor(cli SwarmClient, resolve *Resolver, auth *RegistryAuth) *S
 // the service that was inspected, so it changes exactly that object. Success
 // means the daemon accepted the new spec, not that the tasks have converged.
 // It returns the pinned ref.
-func (e *SwarmExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) {
+func (e *SwarmExecutor) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
 	info, err := e.cli.Info(ctx)
 	if err != nil {
 		return "", fmt.Errorf("docker info: %w", err)

internal/agent/updater/swarm_test.go

@@ -79,7 +79,7 @@ func swarmSvc(name string) swarm.Service {
 
 func TestSwarmUsesPinnedRef(t *testing.T) {
 	const ref = "gitea/gitea:1.26@sha256:abababababababababababababababababababababababababababababababab"
-	job := protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: ref}
+	job := Job{JobStart: protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: ref}}
 	fake := &fakeSwarm{info: managerOf("c1"), svc: swarmSvc("gitea_gitea")}
 	dist := &fakeDist{digest: "sha256:other"}
 	var out bytes.Buffer
@@ -94,7 +94,7 @@ func TestSwarmUsesPinnedRef(t *testing.T) {
 }
 
 func TestSwarmExecutor(t *testing.T) {
-	job := protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: "gitea/gitea:1.27"}
+	job := Job{JobStart: protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: "gitea/gitea:1.27"}}
 	workerInfo := managerOf("c1")
 	workerInfo.Swarm.ControlAvailable = false
 	noContainer := swarmSvc("gitea_gitea")

internal/e2e/agent_master_test.go

@@ -11,7 +11,6 @@ import (
 	"net"
 	"net/http"
 	"net/http/httptest"
-	"regexp"
 	"sort"
 	"strings"
 	"sync"
@@ -25,6 +24,7 @@ import (
 	"github.com/shcizo/package-updater/internal/agent/conn"
 	"github.com/shcizo/package-updater/internal/agent/updater"
 	"github.com/shcizo/package-updater/internal/agenttoken"
+	"github.com/shcizo/package-updater/internal/imageref"
 	"github.com/shcizo/package-updater/internal/master/events"
 	"github.com/shcizo/package-updater/internal/master/hub"
 	"github.com/shcizo/package-updater/internal/master/model"
@@ -32,9 +32,6 @@ import (
 	"github.com/shcizo/package-updater/internal/protocol"
 )
 
-// digestPattern is the digest format the real store records.
-var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
-
 type memStore struct {
 	mu        sync.Mutex
 	agents    map[string]model.Agent
@@ -107,6 +104,19 @@ func (m *memStore) SetLastTagDigest(_ context.Context, id int64, digest string)
 	return nil
 }
 
+func (m *memStore) SetNewer(_ context.Context, id int64, ref string, at time.Time) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	svc, ok := m.svcRows[id]
+	if !ok {
+		return model.ErrNotFound
+	}
+	svc.NewerRef = ref
+	svc.NewerAt = &at
+	m.svcRows[id] = svc
+	return nil
+}
+
 // seedService adds a service row that jobs can be triggered for.
 func (m *memStore) seedService(svc model.Service) {
 	m.mu.Lock()
@@ -245,7 +255,8 @@ func (m *memStore) FinishJob(_ context.Context, agentID, jobID int64, f model.Jo
 	if f.Status == model.JobSucceeded {
 		svc := m.svcRows[j.ServiceID]
 		svc.DesiredRef = f.Ref
-		if _, d, _ := strings.Cut(f.Ref, "@"); !strings.Contains(requestedRef, "@") && digestPattern.MatchString(d) {
+		svc.AwaitingInventory = true
+		if _, d, _ := strings.Cut(f.Ref, "@"); !strings.Contains(requestedRef, "@") && imageref.IsDigest(d) {
 			svc.LastTagDigest = d
 		}
 		// Same rule as the SQL: the job was created for the suggested ref after it was reported.

internal/e2e/drift_test.go

@@ -0,0 +1,113 @@
+package e2e
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/master/poller"
+	"github.com/shcizo/package-updater/internal/protocol"
+)
+
+const (
+	driftDesired = "sha256:" + "d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1"
+	driftOther   = "sha256:" + "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a"
+)
+
+// seedDrifted replaces the environment's service with an auto service that was
+// updated to slusk:1.4.2 at driftDesired but runs driftOther now.
+func (e *jobEnv) seedDrifted() {
+	e.store.seedService(model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",
+		AgentID: ptr(agentID), ImageRepo: slusk, UpdateMode: model.ModeAuto,
+		DesiredRef: slusk + ":1.4.2@" + driftDesired,
+		ActualRef:  slusk + ":1.4.2", ActualDigest: driftOther})
+}
+
+// inventoryWrite is what the store does for an agent's upsert of the service.
+func (e *jobEnv) inventoryWrite(id int64, actualRef, actualDigest string) {
+	e.store.mu.Lock()
+	defer e.store.mu.Unlock()
+	svc := e.store.svcRows[id]
+	svc.ActualRef, svc.ActualDigest, svc.AwaitingInventory = actualRef, actualDigest, false
+	e.store.svcRows[id] = svc
+}
+
+func TestDriftedAutoServiceIsNotUpdatedByPollOrWebhook(t *testing.T) {
+	e := newJobEnv(t)
+	e.seedDrifted()
+	reg := &fixedRegistry{digests: map[string]string{slusk + ":1.4.2": "sha256:new"}}
+	p := poller.New(e.store, e.jobs, reg, e.m.broker, poller.Config{Workers: 1}, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	p.Cycle(context.Background())
+
+	assert.Equal(t, 1, reg.calls)
+	assert.Zero(t, e.store.jobCount(), "polling starts no job for a drifted service")
+	svc, err := e.store.GetService(context.Background(), 1)
+	require.NoError(t, err)
+	assert.Equal(t, slusk+":1.4.2", svc.AvailableRef, "the suggestion is still recorded")
+
+	code, body := callWebhook(t, e.webhookServer(t), `{"image":"`+slusk+`:ignored","tag":"2.0.0"}`)
+
+	assert.Equal(t, 200, code)
+	require.Len(t, body.Results, 1)
+	assert.Equal(t, "refused", body.Results[0].Status)
+	assert.Contains(t, body.Results[0].Error, "drifted")
+	assert.Zero(t, e.store.jobCount(), "the webhook starts no job either")
+}
+
+func TestRestoreDeploysThePinnedRefOnTheAgent(t *testing.T) {
+	e := newJobEnv(t)
+	e.seedDrifted()
+	close(e.exec.release)
+
+	job, err := e.jobs.RestoreDesired(context.Background(), 1, "alice")
+
+	require.NoError(t, err)
+	assert.Equal(t, model.TriggerManual, job.Trigger)
+	select {
+	case ref := <-e.exec.started:
+		assert.Equal(t, slusk+":1.4.2@"+driftDesired, ref, "the agent receives exactly the desired ref")
+	case <-time.After(waitFor):
+		t.Fatal("the agent never received the job")
+	}
+	done := e.waitStatus(t, job.ID, model.JobSucceeded)
+	assert.Equal(t, slusk+":1.4.2@"+driftDesired, done.ToRef)
+	svc, err := e.store.GetService(context.Background(), 1)
+	require.NoError(t, err)
+	assert.Equal(t, driftOther, svc.LastTagDigest, "the drifted digest is remembered so polling does not redeploy it")
+	assert.False(t, svc.Drifted(), "the inventory has not caught up yet")
+
+	e.inventoryWrite(1, slusk+":1.4.2", driftDesired)
+	svc, err = e.store.GetService(context.Background(), 1)
+	require.NoError(t, err)
+	assert.False(t, svc.Drifted())
+}
+
+func TestNoDriftAfterASuccessfulJob(t *testing.T) {
+	e := newJobEnv(t)
+	close(e.exec.release)
+
+	job, err := e.jobs.Trigger(context.Background(), 1, "alice")
+
+	require.NoError(t, err)
+	e.waitStatus(t, job.ID, model.JobSucceeded)
+	svc, err := e.store.GetService(context.Background(), 1)
+	require.NoError(t, err)
+	assert.Equal(t, slusk+":1.4.2@"+e2eDigest, svc.DesiredRef)
+	assert.Equal(t, slusk+":1.4.2@sha256:old", svc.ActualRef, "the agent's inventory write has not arrived")
+	assert.False(t, svc.Drifted(), "between job.result and the inventory write the row is not drifted")
+
+	e.inventoryWrite(1, slusk+":1.4.2", e2eDigest)
+	svc, _ = e.store.GetService(context.Background(), 1)
+	assert.False(t, svc.Drifted())
+
+	e.inventoryWrite(1, slusk+":1.4.2", driftOther)
+	svc, _ = e.store.GetService(context.Background(), 1)
+	assert.True(t, svc.Drifted(), "a later inventory write of another digest is drift")
+}

internal/e2e/jobs_test.go

@@ -7,6 +7,7 @@ import (
 	"io"
 	"log/slog"
 	"net/http"
+	"strconv"
 	"strings"
 	"testing"
 	"time"
@@ -46,7 +47,7 @@ func newGateExecutor() *gateExecutor {
 	return &gateExecutor{started: make(chan string, 8), release: make(chan struct{})}
 }
 
-func (g *gateExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) {
+func (g *gateExecutor) Execute(ctx context.Context, job updater.Job, out io.Writer) (string, error) {
 	_, _ = io.WriteString(out, "pulling "+job.Ref+"\n")
 	g.started <- job.Ref
 	select {
@@ -69,11 +70,17 @@ type jobAgent struct {
 	stopRun  func()
 }
 
-func startJobAgent(t *testing.T, m testMaster, token string, exec updater.Executor) *jobAgent {
+// startJobAgent starts an agent. before, if given, runs on the new runner
+// before the worker and the connection start, like ResumeSelfUpdates in the
+// real agent.
+func startJobAgent(t *testing.T, m testMaster, token string, exec updater.Executor, before ...func(*updater.Runner)) *jobAgent {
 	t.Helper()
 	src := &staticSource{}
 	src.set(web)
 	runner := updater.NewRunner(exec, time.Minute)
+	for _, f := range before {
+		f(runner)
+	}
 	runCtx, stopRun := context.WithCancel(context.Background())
 	runDone := make(chan struct{})
 	go func() { runner.Run(runCtx); close(runDone) }()
@@ -221,6 +228,54 @@ func TestAgentRestartMarksJobLost(t *testing.T) {
 	assert.Equal(t, lostMessage, lost.Error)
 }
 
+// TestAgentSelfUpdateResumedByNewAgent follows a self-update: the old agent
+// dies while its job runs, and the new agent finishes the job under the same
+// ID before it says hello, so the master neither loses nor re-runs it.
+func TestAgentSelfUpdateResumedByNewAgent(t *testing.T) {
+	pinned := slusk + ":1.4.2@" + e2eDigest
+	tests := []struct {
+		name      string
+		resume    updater.ResumeFunc
+		wantError string
+	}{
+		{"helper succeeded", func(_ context.Context, out io.Writer) (string, error) {
+			_, _ = io.WriteString(out, "helper output\n")
+			return pinned, nil
+		}, ""},
+		{"helper failed", func(context.Context, io.Writer) (string, error) {
+			return "", errors.New("docker compose up: exit status 1 (output: boom)")
+		}, "docker compose up: exit status 1 (output: boom)"},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			e := newJobEnv(t)
+			j := e.trigger(t)
+			e.waitStarted(t)
+			started := time.Now().Add(-time.Second)
+
+			e.agent.stopConn()
+			e.agent.stopRun()
+			startJobAgent(t, e.m, "pua_ok", newGateExecutor(), func(r *updater.Runner) {
+				r.Resume(strconv.FormatInt(j.ID, 10), started, tt.resume)
+			})
+
+			if tt.wantError != "" {
+				done := e.waitStatus(t, j.ID, model.JobFailed)
+				assert.Equal(t, tt.wantError, done.Error)
+				assert.NotEqual(t, lostMessage, done.Error)
+				return
+			}
+			done := e.waitStatus(t, j.ID, model.JobSucceeded)
+			assert.Empty(t, done.Error)
+			assert.Equal(t, pinned, done.ToRef)
+			assert.Contains(t, done.Output, "helper output")
+			svc, err := e.store.GetService(context.Background(), 1)
+			require.NoError(t, err)
+			assert.Equal(t, pinned, svc.DesiredRef)
+		})
+	}
+}
+
 func TestPhaseOneAgentGetsNoJobs(t *testing.T) {
 	store := &memStore{agents: map[string]model.Agent{agenttoken.Hash("pua_old"): {ID: agentID, Name: "node-2"}}}
 	svcRow := model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",

internal/e2e/versions_test.go

@@ -34,6 +34,9 @@ func (f *fixedRegistry) Digest(_ context.Context, ref string) (string, error) {
 	return f.digests[ref], nil
 }
 
+// Tags lists no tags, so no newer tag is ever suggested.
+func (f *fixedRegistry) Tags(context.Context, string) ([]string, error) { return nil, nil }
+
 func TestPickPinnedVersion(t *testing.T) {
 	env := newJobEnv(t)
 	env.store.seedVersion(model.ImageVersion{ID: 5, Repo: slusk, Tag: "1.3.0", Digest: "sha256:old130", Source: model.SourceRunning})

internal/imageref/imageref.go

@@ -2,7 +2,10 @@
 // and digest.
 package imageref
 
-import "strings"
+import (
+	"regexp"
+	"strings"
+)
 
 // Ref is an image reference split into its parts. Tag and Digest are empty
 // when the reference has none.
@@ -30,6 +33,12 @@ func Parse(ref string) Ref {
 	return r
 }
 
+// IsRegistryHost reports whether the first path segment of a repository names
+// a registry: it contains a "." or ":" or is "localhost".
+func IsRegistryHost(segment string) bool {
+	return strings.ContainsAny(segment, ".:") || segment == "localhost"
+}
+
 // String reassembles the reference as repo[:tag][@digest].
 func (r Ref) String() string {
 	s := r.Repo
@@ -41,3 +50,8 @@ func (r Ref) String() string {
 	}
 	return s
 }
+
+var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
+
+// IsDigest reports whether d is a well-formed sha256 digest.
+func IsDigest(d string) bool { return digestPattern.MatchString(d) }

internal/imageref/imageref_test.go

@@ -1,6 +1,7 @@
 package imageref
 
 import (
+	"strings"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -27,3 +28,39 @@ func TestParse(t *testing.T) {
 		})
 	}
 }
+
+func TestIsRegistryHost(t *testing.T) {
+	cases := []struct {
+		in   string
+		want bool
+	}{
+		{"lscr.io", true},
+		{"localhost:5000", true},
+		{"localhost", true},
+		{"registry:5000", true},
+		{"gitea", false},
+		{"library", false},
+		{"", false},
+	}
+	for _, c := range cases {
+		t.Run(c.in, func(t *testing.T) {
+			assert.Equal(t, c.want, IsRegistryHost(c.in))
+		})
+	}
+}
+
+func TestIsDigest(t *testing.T) {
+	cases := []struct {
+		in   string
+		want bool
+	}{
+		{"sha256:" + strings.Repeat("a", 64), true},
+		{"sha256:" + strings.Repeat("a", 63), false},
+		{"sha256:" + strings.Repeat("A", 64), false},
+		{"sha512:" + strings.Repeat("a", 64), false},
+		{"", false},
+	}
+	for _, c := range cases {
+		assert.Equal(t, c.want, IsDigest(c.in), c.in)
+	}
+}

internal/master/hub/route.go

@@ -16,8 +16,9 @@ type candidate struct {
 
 // Route picks the connection a job for svc goes to, among agents that speak
 // at least protocol version minProtocol. It returns model.ErrAgentOffline or
-// model.ErrAgentTooOld (model.ErrAgentTooOldForPinned when minProtocol asks
+// model.ErrAgentTooOld (model.ErrAgentTooOldForSelfUpdate when minProtocol
-// for a pinned ref) when there is none.
+// asks for self-update, model.ErrAgentTooOldForPinned when it asks for a
+// pinned ref) when there is none.
 func (h *Hub) Route(svc model.Service, minProtocol int) (model.Route, error) {
 	h.mu.Lock()
 	conns := make([]candidate, 0, len(h.conns))
@@ -65,6 +66,9 @@ func route(svc model.Service, conns []candidate, minProtocol int) (model.Route,
 		}
 	}
 	if tooOld {
+		if minProtocol >= protocol.MinSelfUpdateVersion {
+			return model.Route{}, model.ErrAgentTooOldForSelfUpdate
+		}
 		if minProtocol >= protocol.MinPinnedRefVersion {
 			return model.Route{}, model.ErrAgentTooOldForPinned
 		}

internal/master/hub/route_test.go

@@ -14,9 +14,10 @@ func TestRoute(t *testing.T) {
 	swarmSvc := model.Service{Kind: protocol.KindSwarm, AgentID: ptr(int64(1)), SwarmClusterID: "c1"}
 	v2 := protocol.Hello{ProtocolVersion: 2}
 	v3 := protocol.Hello{ProtocolVersion: 3}
+	v4 := protocol.Hello{ProtocolVersion: 4}
 	mgrV2 := protocol.Hello{ProtocolVersion: 2, SwarmManager: true, SwarmClusterID: "c1"}
 	mgrV3 := protocol.Hello{ProtocolVersion: 3, SwarmManager: true, SwarmClusterID: "c1"}
-	jobs, pinned := protocol.MinJobVersion, protocol.MinPinnedRefVersion
+	jobs, pinned, selfUpdate := protocol.MinJobVersion, protocol.MinPinnedRefVersion, protocol.MinSelfUpdateVersion
 
 	tests := []struct {
 		name  string
@@ -47,6 +48,8 @@ func TestRoute(t *testing.T) {
 			[]candidate{{1, 10, protocol.Hello{ProtocolVersion: 2, SwarmManager: true}}}, jobs, model.Route{}, model.ErrAgentOffline},
 		{"swarm, pinned ref falls back to a v3 manager when the reporting one is v2", swarmSvc,
 			[]candidate{{1, 10, mgrV2}, {3, 30, mgrV3}}, pinned, model.Route{AgentID: 3, ConnID: 30}, nil},
+		{"compose, own agent is v3 and the job updates the agent itself", composeSvc, []candidate{{2, 20, v3}}, selfUpdate, model.Route{}, model.ErrAgentTooOldForSelfUpdate},
+		{"compose, v4 agent and a self-update", composeSvc, []candidate{{2, 20, v4}}, selfUpdate, model.Route{AgentID: 2, ConnID: 20}, nil},
 		{"swarm, pinned ref and only v2 managers", swarmSvc, []candidate{{1, 10, mgrV2}}, pinned, model.Route{}, model.ErrAgentTooOldForPinned},
 	}
 	for _, tt := range tests {
@@ -57,3 +60,11 @@ func TestRoute(t *testing.T) {
 		})
 	}
 }
+
+func TestRoute_TooOldForSelfUpdateIsAlsoTooOld(t *testing.T) {
+	svc := model.Service{Kind: protocol.KindCompose, AgentID: ptr(int64(2))}
+	_, err := route(svc, []candidate{{2, 20, protocol.Hello{ProtocolVersion: 3}}}, protocol.MinSelfUpdateVersion)
+	assert.ErrorIs(t, err, model.ErrAgentTooOldForSelfUpdate)
+	assert.ErrorIs(t, err, model.ErrAgentTooOld)
+	assert.EqualError(t, err, "agent too old to update itself")
+}

internal/master/jobs/drift_test.go

@@ -0,0 +1,238 @@
+package jobs
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/protocol"
+)
+
+const (
+	driftRepo = "gitea.shcizo.se/shcizo/slusk"
+	driftD1   = "sha256:1111111111111111111111111111111111111111111111111111111111111111"
+	driftD2   = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
+	driftD3   = "sha256:3333333333333333333333333333333333333333333333333333333333333333"
+)
+
+// driftedSvc was updated to 1.4.2 at D1 but now runs 1.4.2 at D2.
+func driftedSvc() model.Service {
+	svc := composeSvc
+	svc.ImageRepo = driftRepo
+	svc.DesiredRef = driftRepo + ":1.4.2@" + driftD1
+	svc.ActualRef = driftRepo + ":1.4.2"
+	svc.ActualDigest = driftD2
+	return svc
+}
+
+func notUpdatableReason(t *testing.T, err error) string {
+	t.Helper()
+	var nu *model.NotUpdatableError
+	require.True(t, errors.As(err, &nu), "got %v", err)
+	return nu.Reason
+}
+
+func TestRestorable(t *testing.T) {
+	tests := []struct {
+		name   string
+		mutate func(*model.Service)
+		reason string // "" means restorable
+	}{
+		{"drifted", func(*model.Service) {}, ""},
+		{"drifted service that follows no tag", func(s *model.Service) {
+			s.ActualRef = driftRepo + "@" + driftD2
+		}, ""},
+		{"not drifted", func(s *model.Service) { s.ActualDigest = driftD1 }, model.ReasonNotDrifted},
+		{"never updated", func(s *model.Service) { s.DesiredRef = "" }, model.ReasonNotDrifted},
+		{"awaiting inventory", func(s *model.Service) { s.AwaitingInventory = true }, model.ReasonNotDrifted},
+		{"removed", func(s *model.Service) { s.RemovedAt = ptr(time.Now()) }, model.ReasonRemoved},
+		{"agent's own Compose service", func(s *model.Service) { s.Self = true }, ""},
+		{"agent's own Swarm service", func(s *model.Service) { s.Self = true; s.Kind = protocol.KindSwarm }, model.ReasonSelfSwarm},
+		{"built", func(s *model.Service) { s.Built = true }, model.ReasonBuilt},
+		{"container", func(s *model.Service) { s.Kind = protocol.KindContainer }, model.ReasonContainer},
+		{"desired ref is the agent image", func(s *model.Service) {
+			s.DesiredRef = "gitea.shcizo.se/shcizo/package-updater:latest@" + driftD1
+		}, ""},
+	}
+	s, _ := newService(&fakeStore{}, &fakeHub{})
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			svc := driftedSvc()
+			tt.mutate(&svc)
+			err := s.Restorable(svc)
+			if tt.reason == "" {
+				assert.NoError(t, err)
+				return
+			}
+			assert.Equal(t, tt.reason, notUpdatableReason(t, err))
+		})
+	}
+}
+
+func TestRestoreDesired_StartsAManualJobToTheDesiredRef(t *testing.T) {
+	svc := driftedSvc()
+	st := &fakeStore{services: map[int64]model.Service{1: svc}, jobs: map[int64]model.Job{}}
+	h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}}
+	s, _ := newService(st, h)
+
+	job, err := s.RestoreDesired(context.Background(), 1, "sub-1")
+
+	require.NoError(t, err)
+	assert.Equal(t, model.JobQueued, job.Status)
+	require.Len(t, st.created, 1)
+	assert.Equal(t, svc.DesiredRef, st.created[0].ToRef)
+	assert.Equal(t, svc.ActualRef, st.created[0].FromRef)
+	assert.Equal(t, "sub-1", st.created[0].RequestedBy)
+	assert.Equal(t, model.TriggerManual, st.created[0].Trigger)
+	assert.Equal(t, []int{protocol.MinPinnedRefVersion}, h.mins)
+	require.Len(t, h.starts, 1)
+	assert.Equal(t, svc.DesiredRef, h.starts[0].start.Ref)
+}
+
+func TestRestoreDesired_Refusals(t *testing.T) {
+	tests := []struct {
+		name   string
+		mutate func(*model.Service)
+		reason string
+	}{
+		{"not drifted", func(s *model.Service) { s.ActualDigest = driftD1 }, model.ReasonNotDrifted},
+		{"own Swarm service", func(s *model.Service) { s.Self = true; s.Kind = protocol.KindSwarm }, model.ReasonSelfSwarm},
+		{"built", func(s *model.Service) { s.Built = true }, model.ReasonBuilt},
+		{"container", func(s *model.Service) { s.Kind = protocol.KindContainer }, model.ReasonContainer},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			svc := driftedSvc()
+			tt.mutate(&svc)
+			st := &fakeStore{services: map[int64]model.Service{1: svc}, jobs: map[int64]model.Job{}}
+			h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}}
+			s, pub := newService(st, h)
+
+			_, err := s.RestoreDesired(context.Background(), 1, "sub-1")
+
+			assert.Equal(t, tt.reason, notUpdatableReason(t, err))
+			assert.Empty(t, st.created)
+			assert.Empty(t, h.starts)
+			assert.Empty(t, pub.topics)
+			assert.Empty(t, st.lastTag, "last_tag_digest untouched")
+		})
+	}
+}
+
+func TestRestoreDesired_UnknownService(t *testing.T) {
+	s, _ := newService(&fakeStore{services: map[int64]model.Service{}}, &fakeHub{})
+
+	_, err := s.RestoreDesired(context.Background(), 9, "sub-1")
+
+	assert.ErrorIs(t, err, model.ErrNotFound)
+}
+
+func TestRestoreDesired_ActiveJobReturnsItAndMarksNothing(t *testing.T) {
+	activeJob := model.Job{ID: 99, Status: model.JobRunning}
+	st := &fakeStore{services: map[int64]model.Service{1: driftedSvc()}, jobs: map[int64]model.Job{99: activeJob}, active: &activeJob}
+	s, _ := newService(st, &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}})
+
+	job, err := s.RestoreDesired(context.Background(), 1, "sub-1")
+
+	assert.ErrorIs(t, err, model.ErrJobActive)
+	assert.Equal(t, int64(99), job.ID)
+	assert.Empty(t, st.lastTag)
+}
+
+func TestRestoreDesired_OldAgentGivesAFailedJob(t *testing.T) {
+	st := &fakeStore{services: map[int64]model.Service{1: driftedSvc()}, jobs: map[int64]model.Job{}}
+	h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}, agentProtocol: 2}
+	s, _ := newService(st, h)
+
+	job, err := s.RestoreDesired(context.Background(), 1, "sub-1")
+
+	require.NoError(t, err)
+	assert.Equal(t, model.JobFailed, job.Status)
+	assert.Equal(t, model.ErrAgentTooOldForPinned.Error(), job.Error)
+	assert.Empty(t, h.starts)
+}
+
+func TestRestoreDesired_LastTagDigest(t *testing.T) {
+	tests := []struct {
+		name   string
+		mutate func(*model.Service)
+		want   string // "" means SetLastTagDigest is not called
+	}{
+		{"drifted to another digest of the same tag", func(*model.Service) {}, driftD2},
+		{"digest in the actual ref", func(s *model.Service) {
+			s.ActualRef, s.ActualDigest = driftRepo+":1.4.2@"+driftD3, ""
+		}, driftD3},
+		{"last tag digest holds back a newer one", func(s *model.Service) {
+			s.DesiredRef = driftRepo + ":1.4.2@" + driftD1
+			s.LastTagDigest = driftD3
+		}, ""},
+		{"last tag digest equals the desired digest", func(s *model.Service) { s.LastTagDigest = driftD1 }, driftD2},
+		{"already remembered", func(s *model.Service) { s.LastTagDigest = driftD2 }, ""},
+		{"different tag", func(s *model.Service) { s.ActualRef = driftRepo + ":1.5.0" }, ""},
+		{"actual digest malformed", func(s *model.Service) { s.ActualDigest = "sha256:abc" }, ""},
+		{"desired follows no tag", func(s *model.Service) { s.DesiredRef = driftRepo + "@" + driftD1 }, ""},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			svc := driftedSvc()
+			tt.mutate(&svc)
+			st := &fakeStore{services: map[int64]model.Service{1: svc}, jobs: map[int64]model.Job{}}
+			s, _ := newService(st, &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}})
+
+			_, err := s.RestoreDesired(context.Background(), 1, "sub-1")
+
+			require.NoError(t, err)
+			if tt.want == "" {
+				assert.Empty(t, st.lastTag)
+				return
+			}
+			assert.Equal(t, map[int64]string{1: tt.want}, st.lastTag)
+		})
+	}
+}
+
+func TestStart_RefusesPollAndWebhookWhileDrifted(t *testing.T) {
+	tests := []struct {
+		trigger model.JobTrigger
+		refused bool
+	}{
+		{model.TriggerPoll, true},
+		{model.TriggerWebhook, true},
+		{model.TriggerManual, false},
+	}
+	for _, tt := range tests {
+		t.Run(string(tt.trigger), func(t *testing.T) {
+			svc := driftedSvc()
+			st := &fakeStore{services: map[int64]model.Service{1: svc}, jobs: map[int64]model.Job{}}
+			h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}}
+			s, _ := newService(st, h)
+
+			_, err := s.Start(context.Background(), svc, StartRequest{Trigger: tt.trigger, Ref: driftRepo + ":2.0.0"})
+
+			if tt.refused {
+				assert.Equal(t, model.ReasonDrifted, notUpdatableReason(t, err))
+				assert.Empty(t, st.created)
+				return
+			}
+			require.NoError(t, err)
+			assert.Len(t, st.created, 1)
+		})
+	}
+}
+
+func TestStart_PollAllowedWhenNotDrifted(t *testing.T) {
+	svc := driftedSvc()
+	svc.AwaitingInventory = true
+	st := &fakeStore{services: map[int64]model.Service{1: svc}, jobs: map[int64]model.Job{}}
+	s, _ := newService(st, &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}})
+
+	_, err := s.Start(context.Background(), svc, StartRequest{Trigger: model.TriggerPoll})
+
+	require.NoError(t, err)
+	assert.Len(t, st.created, 1)
+}

internal/master/jobs/jobs.go

@@ -21,6 +21,7 @@ type Store interface {
 	FailJob(ctx context.Context, id int64, msg string, at time.Time) (bool, error)
 	GetJob(ctx context.Context, id int64) (model.Job, error)
 	GetImageVersion(ctx context.Context, id int64) (model.ImageVersion, error)
+	SetLastTagDigest(ctx context.Context, id int64, digest string) error
 }
 
 // Hub routes jobs to agent connections. *hub.Hub satisfies it.
@@ -47,8 +48,11 @@ type Service struct {
 	now      func() time.Time
 }
 
-// New returns a Service. agentImage is the master's AGENT_IMAGE; services
+// New returns a Service. agentImage is the master's AGENT_IMAGE. A service
-// whose image repo equals its repo are the updater itself and are refused.
+// whose image repo equals its repo may be an agent that reports itself as
+// another service (an agent older than protocol version 4 does not tell), so
+// such a job is only sent to an agent that speaks protocol version
+// protocol.MinSelfUpdateVersion.
 func New(store Store, hub Hub, pub Publisher, agentImage string, log *slog.Logger) *Service {
 	return &Service{
 		store: store, hub: hub, pub: pub, log: log, now: time.Now,
@@ -101,8 +105,8 @@ func (s *Service) updatable(svc model.Service, explicitRef bool) error {
 	switch {
 	case svc.RemovedAt != nil:
 		return &model.NotUpdatableError{Reason: model.ReasonRemoved}
-	case svc.Self || imageref.Parse(svc.ActualRef).Repo == s.selfRepo:
+	case svc.Self && svc.Kind == protocol.KindSwarm:
-		return &model.NotUpdatableError{Reason: model.ReasonSelf}
+		return &model.NotUpdatableError{Reason: model.ReasonSelfSwarm}
 	case svc.Built:
 		return &model.NotUpdatableError{Reason: model.ReasonBuilt}
 	case svc.Kind == protocol.KindContainer:
@@ -114,6 +118,75 @@ func (s *Service) updatable(svc model.Service, explicitRef bool) error {
 	return nil
 }
 
+// Restorable returns nil or a *model.NotUpdatableError saying why the service
+// cannot be restored to its desired version: it must have drifted, and the
+// desired version is deployed like an explicit ref, so a service that follows
+// no tag is fine. The service page calls it for every render.
+func (s *Service) Restorable(svc model.Service) error {
+	if svc.RemovedAt != nil {
+		return &model.NotUpdatableError{Reason: model.ReasonRemoved}
+	}
+	if !svc.Drifted() {
+		return &model.NotUpdatableError{Reason: model.ReasonNotDrifted}
+	}
+	return s.updatable(svc, true)
+}
+
+// RestoreDesired starts a manual job that deploys the drifted service's
+// desired ref again. The ref is read from the row, never from the caller. It
+// returns model.ErrNotFound for an unknown service and otherwise what Start
+// returns, with a *model.NotUpdatableError from Restorable if it cannot be
+// restored.
+//
+// When the service had drifted to another digest of the same tag, that digest
+// becomes last_tag_digest, so the poller does not deploy it again before the
+// registry shows a newer one (the same rule as for any rollback). It is not
+// written when last_tag_digest holds back a newer digest, that is when it is
+// set and differs from the digest the desired ref names.
+func (s *Service) RestoreDesired(ctx context.Context, serviceID int64, requestedBy string) (model.Job, error) {
+	svc, err := s.store.GetService(ctx, serviceID)
+	if err != nil {
+		return model.Job{}, err
+	}
+	if err := s.Restorable(svc); err != nil {
+		return model.Job{}, err
+	}
+	job, err := s.Start(ctx, svc, StartRequest{Trigger: model.TriggerManual, RequestedBy: requestedBy, Ref: svc.DesiredRef})
+	if err != nil {
+		return job, err
+	}
+	if d := driftedTagDigest(svc); d != "" && d != svc.LastTagDigest && holdsNothingNewer(svc) {
+		if err := s.store.SetLastTagDigest(ctx, svc.ID, d); err != nil {
+			s.log.Warn("last_tag_digest_not_set", "service_id", svc.ID, "error", err)
+		}
+	}
+	return job, nil
+}
+
+// holdsNothingNewer reports whether last_tag_digest may be overwritten by a
+// restore: it is empty or equals the digest the desired ref names. Anything
+// else is a digest the poller must keep holding back.
+func holdsNothingNewer(svc model.Service) bool {
+	return svc.LastTagDigest == "" || svc.LastTagDigest == imageref.Parse(svc.DesiredRef).Digest
+}
+
+// driftedTagDigest returns the digest a drifted service runs when it runs the
+// same tag as desired at another digest, else "".
+func driftedTagDigest(svc model.Service) string {
+	tag := FollowedRef(svc.DesiredRef)
+	if tag == "" || tag != FollowedRef(svc.ActualRef) {
+		return ""
+	}
+	d := imageref.Parse(svc.ActualRef).Digest
+	if d == "" {
+		d = svc.ActualDigest
+	}
+	if !imageref.IsDigest(d) || d == imageref.Parse(svc.DesiredRef).Digest {
+		return ""
+	}
+	return d
+}
+
 // StartRequest says who asks for an update and what to deploy.
 type StartRequest struct {
 	Trigger     model.JobTrigger
@@ -156,13 +229,18 @@ func (s *Service) TriggerVersion(ctx context.Context, serviceID, versionID int64
 }
 
 // Start starts an update of svc. It returns the job, which may already be
-// failed ("agent offline", "agent too old for updates", or "agent too old to
+// failed ("agent offline", "agent too old for updates", "agent too old to
 // deploy a pinned version" when the ref has a digest or no tag and the agent
-// speaks protocol version 2); a
+// speaks protocol version 2, or "agent too old to update itself" for an
+// agent's own service before protocol version 4); a
 // *model.NotUpdatableError when the service cannot be updated (no job
 // created); the active job and model.ErrJobActive when one is queued or
-// running.
+// running. Poll and webhook jobs are refused for a drifted service; only a
+// person starts one then.
 func (s *Service) Start(ctx context.Context, svc model.Service, req StartRequest) (model.Job, error) {
+	if req.Trigger != model.TriggerManual && svc.Drifted() {
+		return model.Job{}, &model.NotUpdatableError{Reason: model.ReasonDrifted}
+	}
 	if err := s.updatable(svc, req.Ref != ""); err != nil {
 		return model.Job{}, err
 	}
@@ -175,6 +253,9 @@ func (s *Service) Start(ctx context.Context, svc model.Service, req StartRequest
 	if p := imageref.Parse(ref); p.Digest != "" || p.Tag == "" {
 		minProtocol = protocol.MinPinnedRefVersion
 	}
+	if svc.Self || imageref.Parse(svc.ActualRef).Repo == s.selfRepo {
+		minProtocol = protocol.MinSelfUpdateVersion
+	}
 	r, routeErr := s.hub.Route(svc, minProtocol)
 	nj := model.NewJob{
 		ServiceID: svc.ID, FromRef: svc.ActualRef, ToRef: ref,
@@ -196,7 +277,7 @@ func (s *Service) Start(ctx context.Context, svc model.Service, req StartRequest
 
 	failure := routeErr
 	if failure == nil {
-		start := protocol.JobStart{Kind: svc.Kind, Ref: ref}
+		start := protocol.JobStart{Kind: svc.Kind, Ref: ref, Self: svc.Self}
 		if svc.Kind == protocol.KindSwarm {
 			start.SwarmClusterID, start.SwarmService = svc.SwarmClusterID, svc.SwarmService
 		} else {

internal/master/jobs/jobs_test.go

@@ -16,6 +16,8 @@ import (
 	"github.com/shcizo/package-updater/internal/protocol"
 )
 
+const selfImage = "gitea.shcizo.se/shcizo/package-updater:latest"
+
 type fakeStore struct {
 	services map[int64]model.Service
 	jobs     map[int64]model.Job
@@ -23,7 +25,8 @@ type fakeStore struct {
 	active   *model.Job // returned with ErrJobActive when set
 	versions map[int64]model.ImageVersion
 
-	revokedAgents map[int64]bool // CreateJob returns ErrAgentOffline for these
+	revokedAgents map[int64]bool   // CreateJob returns ErrAgentOffline for these
+	lastTag       map[int64]string // SetLastTagDigest calls: service ID to digest
 }
 
 func (f *fakeStore) GetService(_ context.Context, id int64) (model.Service, error) {
@@ -63,6 +66,14 @@ func (f *fakeStore) GetImageVersion(_ context.Context, id int64) (model.ImageVer
 	return v, nil
 }
 
+func (f *fakeStore) SetLastTagDigest(_ context.Context, id int64, digest string) error {
+	if f.lastTag == nil {
+		f.lastTag = map[int64]string{}
+	}
+	f.lastTag[id] = digest
+	return nil
+}
+
 func (f *fakeStore) GetJob(_ context.Context, id int64) (model.Job, error) { return f.jobs[id], nil }
 
 type startCall struct {
@@ -83,6 +94,9 @@ type fakeHub struct {
 func (h *fakeHub) Route(_ model.Service, minProtocol int) (model.Route, error) {
 	h.mins = append(h.mins, minProtocol)
 	if h.agentProtocol != 0 && minProtocol > h.agentProtocol {
+		if minProtocol >= protocol.MinSelfUpdateVersion {
+			return model.Route{}, model.ErrAgentTooOldForSelfUpdate
+		}
 		if minProtocol >= protocol.MinPinnedRefVersion {
 			return model.Route{}, model.ErrAgentTooOldForPinned
 		}
@@ -100,8 +114,6 @@ type fakePub struct{ topics []events.Topic }
 
 func (p *fakePub) Publish(t events.Topic) { p.topics = append(p.topics, t) }
 
-const selfImage = "gitea.shcizo.se/shcizo/package-updater:latest"
-
 var (
 	composeSvc = model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",
 		AgentID: ptr(int64(7)), ActualRef: "gitea.shcizo.se/shcizo/slusk:1.4.2@sha256:d1"}
@@ -125,9 +137,10 @@ func TestUpdatable(t *testing.T) {
 		reason string
 	}{
 		{"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved},
-		{"flagged self", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, model.ReasonSelf},
+		{"agent's own Compose service", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, ""},
+		{"agent's own Swarm service", model.Service{Kind: protocol.KindSwarm, ActualRef: "a:1", Self: true}, model.ReasonSelfSwarm},
 		{"same repo as the agent image", model.Service{Kind: protocol.KindCompose,
-			ActualRef: "gitea.shcizo.se/shcizo/package-updater:1.2.0@sha256:abc"}, model.ReasonSelf},
+			ActualRef: "gitea.shcizo.se/shcizo/package-updater:1.2.0@sha256:abc"}, ""},
 		{"built from source", model.Service{Kind: protocol.KindCompose, ActualRef: "slusk:1", Built: true}, model.ReasonBuilt},
 		{"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer},
 		{"digest without tag", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, model.ReasonNoTag},
@@ -289,7 +302,8 @@ func TestStart_ExplicitRefBypassesOnlyTheNoTagRefusal(t *testing.T) {
 	}{
 		{"digest only", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, ""},
 		{"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved},
-		{"self", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, model.ReasonSelf},
+		{"own Swarm service", model.Service{Kind: protocol.KindSwarm, ActualRef: "a:1", Self: true}, model.ReasonSelfSwarm},
+		{"own Compose service", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, ""},
 		{"built", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Built: true}, model.ReasonBuilt},
 		{"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer},
 	}
@@ -333,8 +347,9 @@ func TestAutoAllowed(t *testing.T) {
 		reason string
 	}{
 		{"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved},
-		{"self", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, model.ReasonSelf},
+		{"own Swarm service", model.Service{Kind: protocol.KindSwarm, ActualRef: "a:1", Self: true}, model.ReasonSelfSwarm},
-		{"agent image repo", model.Service{Kind: protocol.KindCompose, ActualRef: selfImage}, model.ReasonSelf},
+		{"own Compose service", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, ""},
+		{"agent image repo", model.Service{Kind: protocol.KindCompose, ActualRef: "gitea.shcizo.se/shcizo/package-updater:latest"}, ""},
 		{"built", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Built: true}, model.ReasonBuilt},
 		{"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer},
 		{"digest without tag is allowed", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, ""},
@@ -387,6 +402,61 @@ func TestStart_RoutesWithTheMinimumProtocolOfTheRef(t *testing.T) {
 	}
 }
 
+func TestStart_SelfServiceNeedsASelfUpdatingAgent(t *testing.T) {
+	self := composeSvc
+	self.Self = true
+	ok := model.Route{AgentID: 7, ConnID: 70}
+
+	t.Run("routes with the self-update protocol and marks the job", func(t *testing.T) {
+		st := &fakeStore{services: map[int64]model.Service{1: self}, jobs: map[int64]model.Job{}}
+		h := &fakeHub{route: ok}
+		s, _ := newService(st, h)
+
+		_, err := s.Start(context.Background(), self, StartRequest{Trigger: model.TriggerManual, Ref: "x/y:2@sha256:a"})
+
+		require.NoError(t, err)
+		assert.Equal(t, []int{protocol.MinSelfUpdateVersion}, h.mins, "also above the pinned-ref minimum")
+		require.Len(t, h.starts, 1)
+		assert.True(t, h.starts[0].start.Self)
+	})
+	t.Run("a service with the agent image repo needs a self-updating agent but is not marked", func(t *testing.T) {
+		svc := composeSvc
+		svc.ActualRef = selfImage
+		st := &fakeStore{services: map[int64]model.Service{1: svc}, jobs: map[int64]model.Job{}}
+		h := &fakeHub{route: ok}
+		s, _ := newService(st, h)
+
+		_, err := s.Start(context.Background(), svc, StartRequest{Trigger: model.TriggerManual})
+
+		require.NoError(t, err)
+		assert.Equal(t, []int{protocol.MinSelfUpdateVersion}, h.mins)
+		require.Len(t, h.starts, 1)
+		assert.False(t, h.starts[0].start.Self)
+	})
+	t.Run("a service that is not the agent's own is not marked", func(t *testing.T) {
+		st := &fakeStore{services: map[int64]model.Service{1: composeSvc}, jobs: map[int64]model.Job{}}
+		h := &fakeHub{route: ok}
+		s, _ := newService(st, h)
+
+		_, err := s.Start(context.Background(), composeSvc, StartRequest{Trigger: model.TriggerManual})
+
+		require.NoError(t, err)
+		assert.False(t, h.starts[0].start.Self)
+	})
+	t.Run("an older agent fails the job with a clear error", func(t *testing.T) {
+		st := &fakeStore{services: map[int64]model.Service{1: self}, jobs: map[int64]model.Job{}}
+		h := &fakeHub{route: ok, agentProtocol: 3}
+		s, _ := newService(st, h)
+
+		job, err := s.Start(context.Background(), self, StartRequest{Trigger: model.TriggerManual})
+
+		require.NoError(t, err)
+		assert.Equal(t, model.JobFailed, job.Status)
+		assert.Equal(t, "agent too old to update itself", job.Error)
+		assert.Empty(t, h.starts)
+	})
+}
+
 func TestTriggerVersion_Ref(t *testing.T) {
 	const repo = "gitea.shcizo.se/shcizo/slusk"
 	noTag := model.Service{ID: 4, Kind: protocol.KindCompose, ComposeProject: "n", ComposeService: "n",
@@ -402,6 +472,7 @@ func TestTriggerVersion_Ref(t *testing.T) {
 	}{
 		{"tag and digest", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"}, repo + ":1.3.0@sha256:old", protocol.MinPinnedRefVersion},
 		{"tag only", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "main"}, repo + ":main", protocol.MinJobVersion},
+		{"digest-less version from the tag list", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.28.0", Source: model.SourceTags}, repo + ":1.28.0", protocol.MinJobVersion},
 		{"digest only", svc, model.ImageVersion{ID: 10, Repo: repo, Digest: "sha256:old"}, repo + "@sha256:old", protocol.MinPinnedRefVersion},
 		{"service that follows no tag", noTag, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"}, repo + ":1.3.0@sha256:old", protocol.MinPinnedRefVersion},
 	}

internal/master/model/job.go

@@ -19,6 +19,10 @@ var (
 	// speaks protocol version 2, which cannot deploy a digest-pinned or
 	// tagless ref. errors.Is(err, ErrAgentTooOld) is true for it as well.
 	ErrAgentTooOldForPinned error = agentTooOldForPinned{}
+	// ErrAgentTooOldForSelfUpdate means the agent that runs the service speaks
+	// a protocol version before 4 and cannot update itself.
+	// errors.Is(err, ErrAgentTooOld) is true for it as well.
+	ErrAgentTooOldForSelfUpdate error = agentTooOldForSelfUpdate{}
 )
 
 type agentTooOldForPinned struct{}
@@ -29,6 +33,12 @@ func (agentTooOldForPinned) Error() string {
 
 func (agentTooOldForPinned) Is(target error) bool { return target == ErrAgentTooOld }
 
+type agentTooOldForSelfUpdate struct{}
+
+func (agentTooOldForSelfUpdate) Error() string { return "agent too old to update itself" }
+
+func (agentTooOldForSelfUpdate) Is(target error) bool { return target == ErrAgentTooOld }
+
 // NotUpdatableError says why a service cannot be updated. Reason is shown
 // in the UI as is.
 type NotUpdatableError struct {
@@ -39,11 +49,34 @@ func (e *NotUpdatableError) Error() string { return "not updatable: " + e.Reason
 
 // Reasons a service cannot be updated.
 const (
-	ReasonSelf      = "updated manually outside the system"
 	ReasonContainer = "plain containers cannot be updated"
 	ReasonNoTag     = "follows no tag"
 	ReasonRemoved   = "service no longer exists"
 	ReasonBuilt     = "built from source (build:)"
+	// ReasonSelfSwarm is for an agent's own Swarm service, which cannot be
+	// updated by the agent that runs in it.
+	ReasonSelfSwarm = "an agent's own Swarm service cannot update itself"
+
+	ReasonDrifted    = "drifted: restore or accept the running version first"
+	ReasonNotDrifted = "runs its desired version"
+)
+
+// MergeRefusedError says why two services cannot be merged. Reason is shown
+// in the UI as is.
+type MergeRefusedError struct {
+	Reason string
+}
+
+func (e *MergeRefusedError) Error() string { return "cannot merge: " + e.Reason }
+
+// Reasons two services cannot be merged.
+const (
+	ReasonMergeSame      = "a service cannot be merged into itself"
+	ReasonMergeLiveOld   = "the old service still exists"
+	ReasonMergeGoneNew   = "the new service no longer exists"
+	ReasonMergeKind      = "only services of the same kind can be merged"
+	ReasonMergeContainer = "plain containers cannot be merged"
+	ReasonMergeSelf      = "an agent's own service cannot be merged"
 )
 
 // JobStatus is the state of an update job.

internal/master/model/model.go

@@ -15,6 +15,8 @@ import (
 // Sentinel errors returned by storage.
 var (
 	ErrNotFound = errors.New("not found")
+	// ErrConflict means the row already exists, or that it changed meanwhile
+	// (AcceptRunning, MergeService).
 	ErrConflict = errors.New("already exists")
 	ErrRevoked  = errors.New("agent revoked")
 )
@@ -27,9 +29,12 @@ type Agent struct {
 	Hostname     string
 	AgentVersion string
 	SwarmManager bool
-	LastSeenAt   *time.Time
+	// SwarmClusterID is the Swarm cluster the agent's daemon was in when it last
-	RevokedAt    *time.Time
+	// connected; "" outside a swarm.
-	CreatedAt    time.Time
+	SwarmClusterID string
+	LastSeenAt     *time.Time
+	RevokedAt      *time.Time
+	CreatedAt      time.Time
 }
 
 // Service is one service as last reported by an agent.
@@ -61,6 +66,11 @@ type Service struct {
 	AvailableSource string // SourceWebhook or SourcePoll; "" when AvailableRef is empty
 	AvailableAt     *time.Time
 	LastPollError   string // last registry error for the followed ref; "" when fine
+	// NewerRef is the newest repo:tag from the registry's tag list that
+	// tagver.Newer finds for the followed tag; "" if none. A suggestion only:
+	// it is never deployed automatically, not even for auto services.
+	NewerRef string
+	NewerAt  *time.Time // when NewerRef was first suggested; nil when NewerRef is empty
 	// LastTagDigest is the newest digest of the followed tag that was deployed
 	// or started: set when the poller starts an auto job, when it sees the service
 	// already running that digest, and when any plain tag job succeeds (the
@@ -68,9 +78,13 @@ type Service struct {
 	// not start another job for the same digest, so a rollback sticks until the
 	// registry shows a different one.
 	LastTagDigest string
-	FirstSeenAt   time.Time
+	// AwaitingInventory is true from a successful job until the next inventory
-	UpdatedAt     time.Time
+	// write for the row. Until then DesiredRef is newer than ActualRef without
-	RemovedAt     *time.Time
+	// the service having drifted.
+	AwaitingInventory bool
+	FirstSeenAt       time.Time
+	UpdatedAt         time.Time
+	RemovedAt         *time.Time
 }
 
 // Update modes of a service.
@@ -86,6 +100,9 @@ const (
 	// SourceRunning is used by image_versions only: an agent reported the
 	// version as running.
 	SourceRunning = "running"
+	// SourceTags is used by image_versions only: the tag appeared in the
+	// registry's tag list as a newer tag.
+	SourceTags = "tags"
 )
 
 // Available is a newer ref reported for a service.
@@ -104,7 +121,7 @@ type ImageVersion struct {
 	Repo        string
 	Tag         string
 	Digest      string
-	Source      string // SourceWebhook, SourcePoll or SourceRunning: the first source that saw it
+	Source      string // SourceWebhook, SourcePoll, SourceRunning or SourceTags: the first source that saw it
 	FirstSeenAt time.Time
 }
 
@@ -113,6 +130,68 @@ func (v ImageVersion) Ref() string {
 	return imageref.Ref{Repo: v.Repo, Tag: v.Tag, Digest: v.Digest}.String()
 }
 
+// Drifted reports whether the service runs something other than the version
+// the last job deployed. A service the updater never touched, a removed one
+// and one whose inventory has not caught up with the last job do not drift.
+func (s Service) Drifted() bool {
+	return s.DesiredRef != "" && s.RemovedAt == nil && !s.AwaitingInventory &&
+		!RunsRef(s.DesiredRef, s.ActualRef, s.ActualDigest)
+}
+
+// RunsRef reports whether a service reporting actualRef and actualDigest runs
+// the desired ref. The digest decides when both sides know one, so
+// "repo:tag@D" is run by "repo:tag" reporting digest D, and by "repo@D".
+// Otherwise the repo (compared exactly, without registry normalisation) and
+// the followed tag must be equal.
+func RunsRef(desired, actualRef, actualDigest string) bool {
+	d, a := imageref.Parse(desired), imageref.Parse(actualRef)
+	if d.Repo != a.Repo {
+		return false
+	}
+	ad := a.Digest
+	if ad == "" {
+		ad = actualDigest
+	}
+	if d.Digest != "" && ad != "" {
+		return d.Digest == ad
+	}
+	return followedTag(d) == followedTag(a)
+}
+
+// followedTag is the tag a ref follows: a bare image follows "latest" and a
+// ref with a digest but no tag follows nothing.
+func followedTag(r imageref.Ref) string {
+	if r.Tag == "" && r.Digest == "" {
+		return "latest"
+	}
+	return r.Tag
+}
+
+// RunningRef is the ref to record as the desired one when the running version
+// is accepted: actualRef itself if it carries a digest, "repo:tag@digest" if
+// only actualDigest is known (a bare image is "latest"), else actualRef.
+func RunningRef(actualRef, actualDigest string) string {
+	r := imageref.Parse(actualRef)
+	if r.Digest != "" || !imageref.IsDigest(actualDigest) {
+		return actualRef
+	}
+	r.Tag, r.Digest = followedTag(r), actualDigest
+	return r.String()
+}
+
+// RunningRef returns the ref accepting the running version records as the
+// desired one; see the function RunningRef. Templates post it as the guard of
+// the accept form.
+func (s Service) RunningRef() string { return RunningRef(s.ActualRef, s.ActualDigest) }
+
+// Move says that the removed service FromID has probably moved to the live
+// service ToID on another agent. The names are the agents' own; a revoked
+// agent keeps its name.
+type Move struct {
+	FromID, ToID       int64
+	FromAgent, ToAgent string
+}
+
 // DisplayName is the human-facing name of the service.
 func (s Service) DisplayName() string {
 	switch s.Kind {

internal/master/model/model_test.go

@@ -2,6 +2,7 @@ package model
 
 import (
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 
@@ -58,6 +59,84 @@ func TestImageVersionRef(t *testing.T) {
 	}
 }
 
+const (
+	dA = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+	dB = "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
+)
+
+func TestRunsRef(t *testing.T) {
+	cases := []struct {
+		name                          string
+		desired, actualRef, actualDig string
+		want                          bool
+	}{
+		{"same string", "repo:v1@" + dA, "repo:v1@" + dA, "", true},
+		{"tag and digest, actual reports it separately", "repo:tag@" + dA, "repo:tag", dA, true},
+		{"bare image reports latest digest", "postgres:latest@" + dA, "postgres", dA, true},
+		{"tagless actual with the same digest", "repo:tag@" + dA, "repo@" + dA, "", true},
+		{"same digest under another tag", "repo:v1@" + dA, "repo:v2@" + dA, "", true},
+		{"same digest, actual tag differs, digest reported", "repo:v1@" + dA, "repo:v2", dA, true},
+		{"same tag, actual digest unknown", "repo:tag@" + dA, "repo:tag", "", true},
+		{"desired without digest, same tag", "repo:tag", "repo:tag@" + dA, "", true},
+		{"bare desired, latest actual", "repo", "repo:latest", "", true},
+		{"registry with port", "localhost:5000/foo:v1@" + dA, "localhost:5000/foo:v1", dA, true},
+		{"registry with port, other digest", "localhost:5000/foo:v1@" + dA, "localhost:5000/foo:v1", dB, false},
+		{"same tag, other digest", "repo:tag@" + dA, "repo:tag", dB, false},
+		{"other digest in the ref", "repo:tag@" + dA, "repo:tag@" + dB, "", false},
+		{"other tag, digest unknown", "repo:v1@" + dA, "repo:v2", "", false},
+		{"tagless desired, tagged actual, digest unknown", "repo@" + dA, "repo:v1", "", false},
+		{"other repo", "repo:tag@" + dA, "other:tag", dA, false},
+		{"no docker.io normalisation", "nginx:1@" + dA, "docker.io/library/nginx:1", dA, false},
+		{"case-sensitive repo", "Repo:1@" + dA, "repo:1", dA, false},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			assert.Equal(t, c.want, RunsRef(c.desired, c.actualRef, c.actualDig))
+		})
+	}
+}
+
+func TestDrifted(t *testing.T) {
+	now := time.Now()
+	drifted := Service{DesiredRef: "repo:tag@" + dA, ActualRef: "repo:tag", ActualDigest: dB}
+	cases := []struct {
+		name   string
+		mutate func(*Service)
+		want   bool
+	}{
+		{"runs another digest", func(*Service) {}, true},
+		{"never updated", func(s *Service) { s.DesiredRef = "" }, false},
+		{"removed", func(s *Service) { s.RemovedAt = &now }, false},
+		{"awaiting inventory", func(s *Service) { s.AwaitingInventory = true }, false},
+		{"runs the desired digest", func(s *Service) { s.ActualDigest = dA }, false},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			svc := drifted
+			c.mutate(&svc)
+			assert.Equal(t, c.want, svc.Drifted())
+		})
+	}
+}
+
+func TestRunningRef(t *testing.T) {
+	cases := []struct {
+		name, actualRef, actualDigest, want string
+	}{
+		{"ref with digest is kept", "repo:v1@" + dB, dA, "repo:v1@" + dB},
+		{"tag plus reported digest", "repo:v1", dA, "repo:v1@" + dA},
+		{"bare image becomes latest", "postgres", dA, "postgres:latest@" + dA},
+		{"registry with port", "localhost:5000/foo:v1", dA, "localhost:5000/foo:v1@" + dA},
+		{"no digest known", "repo:v1", "", "repo:v1"},
+		{"malformed digest", "repo:v1", "sha256:abc", "repo:v1"},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			assert.Equal(t, c.want, RunningRef(c.actualRef, c.actualDigest))
+		})
+	}
+}
+
 func TestJob_Active(t *testing.T) {
 	cases := []struct {
 		status JobStatus

internal/master/poller/poller.go

@@ -1,24 +1,32 @@
 // Package poller checks the registry for new digests of the tags the
 // services follow. A new digest becomes the service's available version, and
-// a service in auto mode gets an update job.
+// a service in auto mode gets an update job. It also lists the tags of the
+// repos whose followed tag looks like a version and suggests a newer one;
+// that suggestion never starts a job.
 package poller
 
 import (
 	"context"
 	"errors"
 	"log/slog"
+	"slices"
 	"strings"
 	"sync"
 	"time"
 
+	"github.com/shcizo/package-updater/internal/imageref"
 	"github.com/shcizo/package-updater/internal/master/events"
 	"github.com/shcizo/package-updater/internal/master/jobs"
 	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/master/tagver"
 )
 
 // lookupTimeout bounds one registry lookup.
 const lookupTimeout = 30 * time.Second
 
+// listTimeout bounds one tag listing, which may take several requests.
+const listTimeout = 60 * time.Second
+
 // maxErrorBytes caps the registry error stored in services.last_poll_error.
 const maxErrorBytes = 500
 
@@ -28,6 +36,7 @@ type Store interface {
 	GetService(ctx context.Context, id int64) (model.Service, error)
 	SetAvailable(ctx context.Context, id int64, a model.Available) error
 	SetLastTagDigest(ctx context.Context, id int64, digest string) error
+	SetNewer(ctx context.Context, id int64, ref string, at time.Time) error
 	SetPollError(ctx context.Context, ids []int64, msg string) error
 }
 
@@ -36,10 +45,11 @@ type Jobs interface {
 	Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error)
 }
 
-// Registry looks up the digest a "repo:tag" ref points at. *registry.Client
+// Registry looks up the digest a "repo:tag" ref points at and lists the tags
-// satisfies it.
+// of a repo. *registry.Client satisfies it.
 type Registry interface {
 	Digest(ctx context.Context, ref string) (string, error)
+	Tags(ctx context.Context, repo string) ([]string, error)
 }
 
 // Publisher receives change signals. *events.Broker satisfies it.
@@ -49,7 +59,7 @@ type Publisher interface{ Publish(t events.Topic) }
 type Config struct {
 	Interval   time.Duration // time between the end of a cycle and the start of the next (POLL_INTERVAL)
 	FirstDelay time.Duration // wait before the first cycle; 1 minute in production
-	Workers    int           // registry lookups in flight at once; 4 in production
+	Workers    int           // registry requests in flight at once; 4 in production
 }
 
 // Poller runs polling cycles.
@@ -112,13 +122,16 @@ type lookup struct {
 // cycleStats is what one cycle did, for the poll_cycle log line.
 type cycleStats struct {
 	refs, ok, failed, suggested, jobs, skipped int
+	reposListed, tagsFailed, newer             int
 	wrote                                      bool
 }
 
 // Cycle runs one poll over all services. It looks up every distinct followed
 // ref once, with at most Config.Workers lookups at a time, then records the
-// results. Writing results and starting jobs is not cut short by ctx ending
+// results. Then it lists the tags of each repo that has a service following a
-// during the lookups, so a job that was marked is always started.
+// version-like tag, once per repo, and records newer tags. Writing results and
+// starting jobs is not cut short by ctx ending during the lookups, so a job
+// that was marked is always started.
 func (p *Poller) Cycle(ctx context.Context) {
 	began := p.now()
 	svcs, err := p.store.ListServices(ctx)
@@ -126,6 +139,17 @@ func (p *Poller) Cycle(ctx context.Context) {
 		p.log.Error("poll_list_failed", "error", err)
 		return
 	}
+	stats := p.pollServices(ctx, svcs)
+	p.log.Info("poll_cycle", "refs", stats.refs, "ok", stats.ok, "failed", stats.failed,
+		"suggested", stats.suggested, "jobs", stats.jobs, "skipped", stats.skipped,
+		"repos_listed", stats.reposListed, "tags_failed", stats.tagsFailed, "newer", stats.newer,
+		"duration_ms", p.now().Sub(began).Milliseconds())
+}
+
+// pollServices does the work of a cycle for svcs: it looks up every distinct
+// followed ref once, records the results and starts auto jobs, then syncs the
+// newer-tag suggestions of svcs.
+func (p *Poller) pollServices(ctx context.Context, svcs []model.Service) cycleStats {
 	var stats cycleStats
 	byRef := map[string][]model.Service{}
 	var refs []string
@@ -152,19 +176,61 @@ func (p *Poller) Cycle(ctx context.Context) {
 		}
 		p.record(write, l, byRef[l.ref], &stats)
 	}
-	if stats.wrote {
+	// Publish after the digest phase too: listing tags can take a while and
-		p.pub.Publish(events.Services)
+	// an "Update available" should not wait for it.
-	}
+	p.publishIfWrote(&stats)
-	p.log.Info("poll_cycle", "refs", stats.refs, "ok", stats.ok, "failed", stats.failed,
+	p.syncNewer(ctx, write, svcs, &stats)
-		"suggested", stats.suggested, "jobs", stats.jobs, "skipped", stats.skipped,
+	p.publishIfWrote(&stats)
-		"duration_ms", p.now().Sub(began).Milliseconds())
+	return stats
 }
 
-// lookupAll asks the registry for every ref, Config.Workers at a time. It
+// NotPolledError says a service cannot be checked against the registry.
-// returns the answers in ref order and stops starting new lookups once ctx
+type NotPolledError struct{ Reason string }
-// is done; refs never looked up are missing from the result.
+
-func (p *Poller) lookupAll(ctx context.Context, refs []string) []lookup {
+func (e *NotPolledError) Error() string { return "not polled: " + e.Reason }
-	results := make([]*lookup, len(refs))
+
+// PollService does what Cycle does, for the service id only: it looks up the
+// followed ref, records the digest, starts an auto job if one is wanted and
+// syncs the newer-tag suggestion. Other services that follow the same ref are
+// updated at the next cycle. It is safe to run while Run is running: the
+// unique index update_jobs_one_active prevents duplicate jobs. A registry
+// error is not returned; it is stored as last_poll_error, like in a cycle. It
+// returns the store's error when the service cannot be read, model.ErrNotFound
+// for a removed service, *NotPolledError when the service is not polled, and
+// ctx.Err() when ctx ended.
+func (p *Poller) PollService(ctx context.Context, id int64) error {
+	began := p.now()
+	svc, err := p.store.GetService(ctx, id)
+	if err != nil {
+		return err
+	}
+	if svc.RemovedAt != nil {
+		return model.ErrNotFound
+	}
+	if r := SkipReason(svc); r != "" {
+		return &NotPolledError{Reason: r}
+	}
+	stats := p.pollServices(ctx, []model.Service{svc})
+	p.log.Info("poll_service", "service_id", id, "ref", jobs.FollowedRef(svc.ActualRef),
+		"ok", stats.ok, "failed", stats.failed, "suggested", stats.suggested, "jobs", stats.jobs,
+		"tags_failed", stats.tagsFailed, "newer", stats.newer,
+		"duration_ms", p.now().Sub(began).Milliseconds())
+	return ctx.Err()
+}
+
+// publishIfWrote tells the browsers to refresh if a phase of the cycle wrote
+// anything, and starts the next phase with a clean slate.
+func (p *Poller) publishIfWrote(stats *cycleStats) {
+	if stats.wrote {
+		p.pub.Publish(events.Services)
+		stats.wrote = false
+	}
+}
+
+// forEach calls fn(i) for i in [0, n), Config.Workers at a time, and stops
+// starting new calls once ctx is done. It returns when all started calls
+// have returned.
+func (p *Poller) forEach(ctx context.Context, n int, fn func(i int)) {
 	next := make(chan int)
 	var wg sync.WaitGroup
 	for w := 0; w < p.cfg.Workers; w++ {
@@ -172,15 +238,12 @@ func (p *Poller) lookupAll(ctx context.Context, refs []string) []lookup {
 		go func() {
 			defer wg.Done()
 			for i := range next {
-				lctx, cancel := context.WithTimeout(ctx, lookupTimeout)
+				fn(i)
-				d, err := p.reg.Digest(lctx, refs[i])
-				cancel()
-				results[i] = &lookup{ref: refs[i], digest: d, err: err}
 			}
 		}()
 	}
 feed:
-	for i := range refs {
+	for i := 0; i < n; i++ {
 		select {
 		case next <- i:
 		case <-ctx.Done():
@@ -189,6 +252,19 @@ feed:
 	}
 	close(next)
 	wg.Wait()
+}
+
+// lookupAll asks the registry for every ref, Config.Workers at a time. It
+// returns the answers in ref order and stops starting new lookups once ctx
+// is done; refs never looked up are missing from the result.
+func (p *Poller) lookupAll(ctx context.Context, refs []string) []lookup {
+	results := make([]*lookup, len(refs))
+	p.forEach(ctx, len(refs), func(i int) {
+		lctx, cancel := context.WithTimeout(ctx, lookupTimeout)
+		d, err := p.reg.Digest(lctx, refs[i])
+		cancel()
+		results[i] = &lookup{ref: refs[i], digest: d, err: err}
+	})
 	var out []lookup
 	for _, r := range results {
 		if r != nil {
@@ -198,6 +274,118 @@ feed:
 	return out
 }
 
+// tagList is the registry's answer for the tags of one repo.
+type tagList struct {
+	repo string
+	tags []string
+	err  error
+}
+
+// listAll lists the tags of every repo like lookupAll looks up refs.
+func (p *Poller) listAll(ctx context.Context, repos []string) []tagList {
+	results := make([]*tagList, len(repos))
+	p.forEach(ctx, len(repos), func(i int) {
+		lctx, cancel := context.WithTimeout(ctx, listTimeout)
+		tags, err := p.reg.Tags(lctx, repos[i])
+		cancel()
+		results[i] = &tagList{repo: repos[i], tags: tags, err: err}
+	})
+	var out []tagList
+	for _, r := range results {
+		if r != nil {
+			out = append(out, *r)
+		}
+	}
+	return out
+}
+
+// Followed returns the repo and tag svc follows: a bare image follows
+// "latest", and a digest-only reference follows no tag.
+func Followed(svc model.Service) (repo, tag string) {
+	ref := imageref.Parse(jobs.FollowedRef(svc.ActualRef))
+	return ref.Repo, ref.Tag
+}
+
+// followedComparable returns the repo and tag svc follows. ok is false unless
+// svc is polled and the tag can have a newer tag.
+func followedComparable(svc model.Service) (repo, tag string, ok bool) {
+	if SkipReason(svc) != "" {
+		return "", "", false
+	}
+	repo, tag = Followed(svc)
+	return repo, tag, tagver.Comparable(tag)
+}
+
+// syncNewer lists the tags of every repo that has a service following a
+// version-like tag, once per repo, and stores the newer tag of each service.
+// A service whose tag cannot have a newer tag any more gets its suggestion
+// cleared; one whose repo could not be listed keeps it, unless it names
+// another repo than the service follows now. Nothing is written
+// when the value is unchanged. It never starts a job, not even for an auto
+// service: a newer tag may be a major step the user has not chosen.
+func (p *Poller) syncNewer(ctx, write context.Context, svcs []model.Service, stats *cycleStats) {
+	seen := map[string]bool{}
+	var repos []string
+	for _, svc := range svcs {
+		if repo, _, ok := followedComparable(svc); ok && svc.RemovedAt == nil && !seen[repo] {
+			seen[repo] = true
+			repos = append(repos, repo)
+		}
+	}
+	lists := map[string][]string{}
+	for _, l := range p.listAll(ctx, repos) {
+		if l.err != nil {
+			p.log.Warn("poll_tags_failed", "repo", l.repo, "error", truncate(l.err.Error()))
+			if ctx.Err() == nil { // cancelled mid-listing: not the registry's fault
+				stats.tagsFailed++
+			}
+			continue
+		}
+		stats.reposListed++
+		lists[l.repo] = l.tags
+	}
+
+	type repoTag struct{ repo, tag string }
+	newest := map[repoTag]string{}
+	for _, svc := range svcs {
+		if svc.RemovedAt != nil {
+			continue
+		}
+		want := ""
+		if repo, tag, ok := followedComparable(svc); ok {
+			tags, listed := lists[repo]
+			if !listed && imageref.Parse(svc.NewerRef).Repo == repo {
+				if svc.NewerRef != "" {
+					stats.newer++
+				}
+				continue // not listed this cycle: keep what is stored
+			}
+			if listed {
+				key := repoTag{repo, tag}
+				n, done := newest[key]
+				if !done {
+					n, _ = tagver.Newer(tag, tags)
+					newest[key] = n
+				}
+				if n != "" {
+					want = repo + ":" + n
+				}
+			}
+		}
+		if want != "" {
+			stats.newer++
+		}
+		if want == svc.NewerRef {
+			continue
+		}
+		if err := p.store.SetNewer(write, svc.ID, want, p.now()); err != nil {
+			p.log.Error("poll_set_newer_failed", "service_id", svc.ID, "ref", want, "error", err)
+			continue
+		}
+		stats.wrote = true
+	}
+}
+
 // record stores one lookup result for the services that follow l.ref.
 func (p *Poller) record(ctx context.Context, l lookup, svcs []model.Service, stats *cycleStats) {
 	if l.err != nil {
@@ -209,14 +397,29 @@ func (p *Poller) record(ctx context.Context, l lookup, svcs []model.Service, sta
 	}
 	stats.ok++
 	p.setPollError(ctx, svcs, "", stats, func(s model.Service) bool { return s.LastPollError != "" })
+	var auto []model.Service
 	for _, svc := range svcs {
 		if !p.suggest(ctx, svc, l, stats) {
 			continue
 		}
 		if autoWanted(svc, l.ref, l.digest) {
-			p.startAuto(ctx, svc.ID, l.ref, l.digest, stats)
+			auto = append(auto, svc)
 		}
 	}
+	// An agent's own service goes last: jobs that reach an agent while it
+	// replaces itself are lost, and messages to one agent arrive in order.
+	slices.SortStableFunc(auto, func(a, b model.Service) int { return selfRank(a) - selfRank(b) })
+	for _, svc := range auto {
+		p.startAuto(ctx, svc.ID, l.ref, l.digest, stats)
+	}
+}
+
+// selfRank sorts an agent's own service after all others.
+func selfRank(svc model.Service) int {
+	if svc.Self {
+		return 1
+	}
+	return 0
 }
 
 // suggest marks l.digest as the available version of svc unless the service

internal/master/poller/poller_test.go

@@ -30,6 +30,12 @@ type pollErrorCall struct {
 	msg string
 }
 
+type newerCall struct {
+	id  int64
+	ref string
+	at  time.Time
+}
+
 type lastTagCall struct {
 	id     int64
 	digest string
@@ -40,6 +46,7 @@ type fakeStore struct {
 	services   []model.Service
 	fresh      map[int64]model.Service // what GetService returns instead of the listed row
 	lastTag    []lastTagCall
+	newer      []newerCall
 	listErr    error
 	available  []availableCall
 	pollErrors []pollErrorCall
@@ -86,6 +93,21 @@ func (f *fakeStore) SetAvailable(ctx context.Context, id int64, a model.Availabl
 	return nil
 }
 
+// SetNewer records the call and, like the database, changes the row the
+// next cycle lists.
+func (f *fakeStore) SetNewer(ctx context.Context, id int64, ref string, at time.Time) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.ctxErrs = append(f.ctxErrs, ctx.Err())
+	f.newer = append(f.newer, newerCall{id, ref, at})
+	for i := range f.services {
+		if f.services[i].ID == id {
+			f.services[i].NewerRef = ref
+		}
+	}
+	return nil
+}
+
 func (f *fakeStore) SetPollError(ctx context.Context, ids []int64, msg string) error {
 	f.mu.Lock()
 	defer f.mu.Unlock()
@@ -125,6 +147,36 @@ type fakeRegistry struct {
 	started  chan struct{} // receives a value whenever a lookup starts, if set
 	release  chan struct{} // lookups wait for it to be closed, if set
 	onResult func()        // called after a lookup computed its answer
+
+	tags       map[string][]string // repo -> tags
+	tagErrs    map[string]error    // repo -> error
+	tagCalls   []string
+	tagStarted chan struct{} // receives a value whenever a tag listing starts, if set
+	tagRelease chan struct{} // tag listings wait for it to be closed, if set
+	onTags     func()        // called after a tag listing computed its answer
+}
+
+func (f *fakeRegistry) Tags(ctx context.Context, repo string) ([]string, error) {
+	f.mu.Lock()
+	f.tagCalls = append(f.tagCalls, repo)
+	f.mu.Unlock()
+	if f.tagStarted != nil {
+		f.tagStarted <- struct{}{}
+	}
+	if f.tagRelease != nil {
+		select {
+		case <-f.tagRelease:
+		case <-ctx.Done():
+			return nil, ctx.Err()
+		}
+	}
+	if f.onTags != nil {
+		f.onTags()
+	}
+	if err := f.tagErrs[repo]; err != nil {
+		return nil, err
+	}
+	return f.tags[repo], nil
 }
 
 func (f *fakeRegistry) Digest(ctx context.Context, ref string) (string, error) {
@@ -297,6 +349,24 @@ func TestNewDigest(t *testing.T) {
 	}
 }
 
+func TestAgentsOwnServiceIsStartedLast(t *testing.T) {
+	ref := repoA + ":1.4"
+	own := svc(1, ref+"@sha256:old")
+	own.UpdateMode, own.Self = model.ModeAuto, true
+	other := svc(2, ref+"@sha256:old")
+	other.UpdateMode = model.ModeAuto
+	st := &fakeStore{services: []model.Service{own, other}}
+	reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
+	j := &fakeJobs{}
+	p, _ := newPoller(st, j, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	require.Len(t, j.starts, 2)
+	assert.Equal(t, int64(2), j.starts[0].svcID)
+	assert.Equal(t, int64(1), j.starts[1].svcID)
+}
+
 func TestSameSuggestionIsIdempotent(t *testing.T) {
 	ref := repoA + ":1.4"
 	running := svc(1, ref+"@sha256:old")
@@ -332,7 +402,7 @@ func TestStartErrorsDoNotAbortCycle(t *testing.T) {
 	reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
 	j := &fakeJobs{errFor: map[int64]error{
 		1: model.ErrJobActive,
-		2: &model.NotUpdatableError{Reason: model.ReasonSelf},
+		2: &model.NotUpdatableError{Reason: model.ReasonContainer},
 		3: errors.New("database down"),
 	}}
 	p, _ := newPoller(st, j, reg, Config{})
@@ -649,3 +719,379 @@ func TestSeedsTheTagDigestOfARunningService(t *testing.T) {
 	assert.Empty(t, j.starts, "the rollback stays")
 	assert.Len(t, st.lastTag, 1, "nothing more to seed")
 }
+
+// newTagRegistry returns a fake registry that lists tags by repo.
+func newTagRegistry(tags map[string][]string) *fakeRegistry {
+	return &fakeRegistry{digests: map[string]string{}, tags: tags}
+}
+
+// newerSvc is a polled service running tag of repo that already runs the
+// registry's digest, so the digest phase writes nothing.
+func newerSvc(id int64, repo, tag string) model.Service {
+	s := svc(id, repo+":"+tag)
+	s.ActualDigest = "sha256:new"
+	return s
+}
+
+// newerRegistry is newTagRegistry that also answers sha256:new for refs.
+func newerRegistry(tags map[string][]string, refs ...string) *fakeRegistry {
+	reg := newTagRegistry(tags)
+	for _, r := range refs {
+		reg.digests[r] = "sha256:new"
+	}
+	return reg
+}
+
+func TestNewerTagSuggested(t *testing.T) {
+	ref := repoB + ":1.27.3"
+	st := &fakeStore{services: []model.Service{newerSvc(1, repoB, "1.27.3")}}
+	reg := newerRegistry(map[string][]string{repoB: {"1.27.4", "1.28.0", "2.0.0", "latest", "1.28.1-rootless"}}, ref)
+	p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	assert.Equal(t, []newerCall{{1, repoB + ":1.28.0", time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}}, st.newer)
+	assert.Equal(t, []events.Topic{events.Services}, pub.topics)
+}
+
+func TestOneTagListPerRepo(t *testing.T) {
+	st := &fakeStore{services: []model.Service{
+		newerSvc(1, repoB, "1.27.3"),
+		newerSvc(2, repoB, "2.1.0"),
+		newerSvc(3, repoB, "1.27.3"),
+	}}
+	reg := newerRegistry(map[string][]string{repoB: {"2.1.1", "1.27.4"}},
+		repoB+":1.27.3", repoB+":2.1.0")
+	p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	assert.Equal(t, []string{repoB}, reg.tagCalls)
+	got := map[int64]string{}
+	for _, c := range st.newer {
+		got[c.id] = c.ref
+	}
+	assert.Equal(t, map[int64]string{1: repoB + ":1.27.4", 2: repoB + ":2.1.1", 3: repoB + ":1.27.4"}, got)
+}
+
+func TestNoTagListWithoutComparableTag(t *testing.T) {
+	built := newerSvc(5, repoB, "1.2.3")
+	built.Built = true
+	noDigest := newerSvc(6, repoB, "1.2.3")
+	noDigest.ActualDigest = ""
+	st := &fakeStore{services: []model.Service{
+		newerSvc(1, repoB, "latest"),
+		newerSvc(2, repoB, "7"),
+		newerSvc(3, repoB, "pg16"),
+		newerSvc(4, repoB+"@sha256:abc", ""),
+		built,
+		noDigest,
+	}}
+	st.services[3].ActualRef = repoB + "@sha256:abc"
+	reg := newTagRegistry(map[string][]string{repoB: {"1.2.4"}})
+	p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	assert.Empty(t, reg.tagCalls)
+	assert.Empty(t, st.newer)
+}
+
+func TestNewerClearedWhenNoCandidate(t *testing.T) {
+	s := newerSvc(1, repoB, "1.27.3")
+	s.NewerRef = repoB + ":1.27.4"
+	st := &fakeStore{services: []model.Service{s}}
+	reg := newerRegistry(map[string][]string{repoB: {"1.27.3", "2.0.0"}}, repoB+":1.27.3")
+	p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	assert.Equal(t, []newerCall{{1, "", time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}}, st.newer)
+}
+
+func TestNewerClearedWhenTagNoLongerComparable(t *testing.T) {
+	moved := newerSvc(1, repoB, "latest")
+	moved.NewerRef = repoB + ":1.27.4"
+	built := newerSvc(2, repoB, "1.27.3")
+	built.Built = true
+	built.NewerRef = repoB + ":1.27.4"
+	st := &fakeStore{services: []model.Service{moved, built}}
+	reg := newerRegistry(map[string][]string{repoB: {"1.27.4"}}, repoB+":latest")
+	p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	require.Len(t, st.newer, 2)
+	assert.Equal(t, "", st.newer[0].ref)
+	assert.Equal(t, "", st.newer[1].ref)
+	assert.Empty(t, reg.tagCalls)
+	assert.Equal(t, []events.Topic{events.Services}, pub.topics)
+}
+
+func TestTagListErrorKeepsNewerAndPollError(t *testing.T) {
+	s := newerSvc(1, repoB, "1.27.3")
+	s.NewerRef = repoB + ":1.27.4"
+	st := &fakeStore{services: []model.Service{s}}
+	reg := newerRegistry(nil, repoB+":1.27.3")
+	reg.tagErrs = map[string]error{repoB: errors.New("list x: registry answered 500 Internal Server Error")}
+	p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	assert.Empty(t, st.newer, "the stored suggestion stays")
+	assert.Empty(t, st.pollErrors, "a failed tag list is not a poll error")
+	assert.Empty(t, pub.topics)
+}
+
+func TestNewerIdempotent(t *testing.T) {
+	s := newerSvc(1, repoB, "1.27.3")
+	s.NewerRef = repoB + ":1.28.0"
+	st := &fakeStore{services: []model.Service{s}}
+	reg := newerRegistry(map[string][]string{repoB: {"1.28.0"}}, repoB+":1.27.3")
+	p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	assert.Empty(t, st.newer)
+	assert.Empty(t, pub.topics)
+}
+
+func TestNewerNeverStartsJobs(t *testing.T) {
+	s := newerSvc(1, repoB, "1.27.3")
+	s.UpdateMode = model.ModeAuto
+	st := &fakeStore{services: []model.Service{s}}
+	reg := newerRegistry(map[string][]string{repoB: {"1.27.4"}}, repoB+":1.27.3")
+	j := &fakeJobs{}
+	p, _ := newPoller(st, j, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	require.Len(t, st.newer, 1)
+	assert.Empty(t, j.starts)
+}
+
+func TestNewerWritesSurviveCancel(t *testing.T) {
+	st := &fakeStore{services: []model.Service{newerSvc(1, repoB, "1.27.3")}}
+	ctx, cancel := context.WithCancel(context.Background())
+	reg := newerRegistry(map[string][]string{repoB: {"1.27.4"}}, repoB+":1.27.3")
+	reg.onTags = cancel
+	p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(ctx)
+
+	require.Len(t, st.newer, 1)
+	assert.NoError(t, st.ctxErrs[0], "the write does not see the cancelled context")
+}
+
+func TestCancelledListKeepsNewer(t *testing.T) {
+	s := newerSvc(1, repoB, "1.27.3")
+	s.NewerRef = repoB + ":1.27.4"
+	st := &fakeStore{services: []model.Service{s}}
+	reg := newerRegistry(map[string][]string{repoB: {"1.28.0"}}, repoB+":1.27.3")
+	reg.tagRelease = make(chan struct{})
+	reg.tagStarted = make(chan struct{}, 1)
+	p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() { p.Cycle(ctx); close(done) }()
+	<-reg.tagStarted
+
+	cancel()
+
+	select {
+	case <-done:
+	case <-time.After(2 * time.Second):
+		t.Fatal("Cycle did not return after cancel")
+	}
+	assert.Empty(t, st.newer)
+	assert.Empty(t, pub.topics)
+}
+
+func TestFailedListClearsNewerFromAnotherRepo(t *testing.T) {
+	s := newerSvc(1, repoB, "1.27.3")
+	s.NewerRef = repoA + ":1.27.4"
+	st := &fakeStore{services: []model.Service{s}}
+	reg := newerRegistry(nil, repoB+":1.27.3")
+	reg.tagErrs = map[string]error{repoB: errors.New("list x: registry answered 500 Internal Server Error")}
+	p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	p.Cycle(context.Background())
+
+	assert.Equal(t, []newerCall{{1, "", time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}}, st.newer)
+}
+
+func TestDigestWritesArePublishedBeforeTagsAreListed(t *testing.T) {
+	st := &fakeStore{services: []model.Service{svc(1, repoB+":1.27.3")}}
+	reg := newerRegistry(map[string][]string{repoB: {"1.27.3"}}, repoB+":1.27.3")
+	reg.tagRelease = make(chan struct{})
+	reg.tagStarted = make(chan struct{}, 1)
+	p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
+	done := make(chan struct{})
+	go func() { p.Cycle(context.Background()); close(done) }()
+	<-reg.tagStarted
+
+	pub.mu.Lock()
+	published := append([]events.Topic(nil), pub.topics...)
+	pub.mu.Unlock()
+	assert.Equal(t, []events.Topic{events.Services}, published, "published while the listing is still blocked")
+
+	close(reg.tagRelease)
+	<-done
+}
+
+func TestPollService(t *testing.T) {
+	ref := repoA + ":latest" // not version-like: no tag listing
+	newDigest := map[string]string{ref: "sha256:new"}
+	auto := func(edit func(*model.Service)) model.Service {
+		s := svc(1, ref)
+		s.UpdateMode = model.ModeAuto
+		if edit != nil {
+			edit(&s)
+		}
+		return s
+	}
+	tests := []struct {
+		name        string
+		svc         model.Service
+		reg         *fakeRegistry
+		startErr    error
+		wantErr     error // matched with ErrorIs
+		wantNotPoll string
+		wantHeads   []string
+		wantTags    []string
+		check       func(t *testing.T, st *fakeStore, j *fakeJobs)
+	}{
+		{name: "manual with a new digest is only marked",
+			svc: svc(1, ref), reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
+			check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
+				require.Len(t, st.available, 1)
+				assert.Equal(t, model.SourcePoll, st.available[0].a.Source)
+				assert.Empty(t, j.starts)
+			}},
+		{name: "auto with a new digest starts a poll job",
+			svc: auto(nil), reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
+			check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
+				assert.Equal(t, []startCall{{1, jobs.StartRequest{Trigger: model.TriggerPoll, Ref: ref}}}, j.starts)
+				assert.Equal(t, []lastTagCall{{1, "sha256:new"}}, st.lastTag)
+			}},
+		{name: "auto that cannot be updated is marked but last_tag_digest stays",
+			svc: auto(nil), reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
+			startErr: &model.NotUpdatableError{Reason: model.ReasonDrifted},
+			check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
+				assert.Len(t, st.available, 1)
+				assert.Len(t, j.starts, 1)
+				assert.Empty(t, st.lastTag)
+			}},
+		{name: "auto that already got the digest is not started",
+			svc: auto(func(s *model.Service) { s.LastTagDigest = "sha256:new" }),
+			reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
+			check: func(t *testing.T, st *fakeStore, j *fakeJobs) { assert.Empty(t, j.starts) }},
+		{name: "auto whose desired_ref names the digest is not started",
+			svc: auto(func(s *model.Service) { s.DesiredRef = ref + "@sha256:new" }),
+			reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
+			check: func(t *testing.T, st *fakeStore, j *fakeJobs) { assert.Empty(t, j.starts) }},
+		{name: "registry error is stored, not returned",
+			svc: svc(1, ref), reg: &fakeRegistry{errs: map[string]error{ref: errors.New("head: boom")}}, wantHeads: []string{ref},
+			check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
+				assert.Equal(t, []pollErrorCall{{[]int64{1}, "head: boom"}}, st.pollErrors)
+			}},
+		{name: "a previous poll error is cleared on success",
+			svc: auto(func(s *model.Service) {
+				s.UpdateMode, s.ActualDigest, s.LastPollError = model.ModeManual, "sha256:new", "head: boom"
+			}),
+			reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
+			check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
+				assert.Equal(t, []pollErrorCall{{[]int64{1}, ""}}, st.pollErrors)
+			}},
+		{name: "no tag is not polled",
+			svc: svc(1, repoA+"@sha256:old"), reg: &fakeRegistry{}, wantNotPoll: model.ReasonNoTag},
+		{name: "built is not polled",
+			svc: auto(func(s *model.Service) { s.Built = true }), reg: &fakeRegistry{}, wantNotPoll: "built from source"},
+		{name: "no digest is not polled",
+			svc: auto(func(s *model.Service) { s.ActualDigest = "" }), reg: &fakeRegistry{}, wantNotPoll: "no registry digest"},
+		{name: "removed is not found",
+			svc: auto(func(s *model.Service) { s.RemovedAt = ptr(time.Now()) }), reg: &fakeRegistry{}, wantErr: model.ErrNotFound},
+		{name: "a comparable tag lists the repo once and suggests the newer tag",
+			svc:       newerSvc(1, repoB, "1.27.3"),
+			reg:       newerRegistry(map[string][]string{repoB: {"1.27.4", "1.28.0"}}, repoB+":1.27.3"),
+			wantHeads: []string{repoB + ":1.27.3"}, wantTags: []string{repoB},
+			check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
+				require.Len(t, st.newer, 1)
+				assert.Equal(t, newerCall{1, repoB + ":1.28.0", time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}, st.newer[0])
+			}},
+		{name: "a tag that cannot have a newer tag lists nothing",
+			svc: newerSvc(1, repoB, "latest"), reg: newerRegistry(nil, repoB+":latest"), wantHeads: []string{repoB + ":latest"}},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			st := &fakeStore{services: []model.Service{tt.svc}}
+			j := &fakeJobs{errFor: map[int64]error{1: tt.startErr}}
+			p, _ := newPoller(st, j, tt.reg, Config{})
+
+			err := p.PollService(context.Background(), 1)
+
+			var np *NotPolledError
+			switch {
+			case tt.wantNotPoll != "":
+				require.ErrorAs(t, err, &np)
+				assert.Equal(t, tt.wantNotPoll, np.Reason)
+			case tt.wantErr != nil:
+				assert.ErrorIs(t, err, tt.wantErr)
+			default:
+				require.NoError(t, err)
+			}
+			assert.Equal(t, tt.wantHeads, tt.reg.sortedCalls())
+			assert.Equal(t, tt.wantTags, tt.reg.tagCalls)
+			if tt.check != nil {
+				tt.check(t, st, j)
+			}
+		})
+	}
+}
+
+func TestPollServiceUnknownID(t *testing.T) {
+	reg := &fakeRegistry{}
+	p, _ := newPoller(&fakeStore{}, &fakeJobs{}, reg, Config{})
+
+	err := p.PollService(context.Background(), 9)
+
+	assert.ErrorIs(t, err, model.ErrNotFound)
+	assert.Empty(t, reg.calls)
+}
+
+func TestPollServiceOnlyThisService(t *testing.T) {
+	ref := repoA + ":1.4"
+	st := &fakeStore{services: []model.Service{svc(1, ref), svc(2, ref)}}
+	reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
+	p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
+
+	require.NoError(t, p.PollService(context.Background(), 1))
+
+	require.Len(t, st.available, 1)
+	assert.Equal(t, int64(1), st.available[0].id)
+	assert.Equal(t, []string{ref}, reg.sortedCalls())
+	for _, c := range st.pollErrors {
+		assert.NotContains(t, c.ids, int64(2))
+	}
+	for _, c := range st.newer {
+		assert.Equal(t, int64(1), c.id)
+	}
+	for _, c := range st.lastTag {
+		assert.Equal(t, int64(1), c.id)
+	}
+}
+
+func TestPollServiceCancelled(t *testing.T) {
+	ref := repoA + ":1.4"
+	st := &fakeStore{services: []model.Service{svc(1, ref)}}
+	reg := &fakeRegistry{release: make(chan struct{})} // lookups only end by cancel
+	p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+
+	err := p.PollService(ctx, 1)
+
+	assert.ErrorIs(t, err, context.Canceled)
+	assert.Empty(t, st.pollErrors)
+	assert.Empty(t, st.available)
+}

internal/master/registry/registry.go

@@ -1,5 +1,6 @@
 // Package registry looks up the digest a tag points at in a container
-// registry, using HEAD requests only.
+// registry and lists the tags of a repository. Manifests are only ever
+// requested with HEAD.
 package registry
 
 import (
@@ -21,9 +22,10 @@ import (
 	"github.com/shcizo/package-updater/internal/imageref"
 )
 
-// Client looks up tag digests with HEAD requests only. Docker Hub counts a
+// Client looks up tag digests and lists tags. Docker Hub counts a GET of a
-// GET of a manifest as a pull against the rate limit and a HEAD as nothing,
+// manifest as a pull against the rate limit and a HEAD as nothing, so Client
-// so Client never issues a manifest GET.
+// never issues a manifest GET. The only GETs it sends are the /v2/ ping and
+// tags/list, which Docker Hub does not count as a pull.
 type Client struct {
 	keychain  authn.Keychain
 	userAgent string
@@ -65,16 +67,46 @@ func (c *Client) Digest(ctx context.Context, ref string) (string, error) {
 		remote.WithAuth(auth),
 		remote.WithUserAgent(c.userAgent))
 	if err != nil {
-		var terr *transport.Error
+		return "", c.lookupError("lookup", r.String(), ref, err)
-		if errors.As(err, &terr) {
-			return "", fmt.Errorf("lookup %s: %s", r, describe(terr))
-		}
-		c.log.Warn("registry_lookup_failed", "ref", ref, "error", err)
-		return "", fmt.Errorf("lookup %s: %s", r, classify(err))
 	}
 	return desc.Digest.String(), nil
 }
 
+// Tags returns every tag of repo ("host/path", or a Docker Hub name), in the
+// order the registry lists them. It sends GET /v2/<repo>/tags/list, following
+// the registry's pagination links. Errors are sanitised like Digest's.
+func (c *Client) Tags(ctx context.Context, repo string) ([]string, error) {
+	r, err := name.NewRepository(repo)
+	if err != nil {
+		return nil, fmt.Errorf("parse %s: %w", repo, err)
+	}
+	auth, err := c.keychain.Resolve(r)
+	if err != nil {
+		return nil, fmt.Errorf("list %s: %w", r, err)
+	}
+	tags, err := remote.List(r,
+		remote.WithContext(ctx),
+		remote.WithAuth(auth),
+		remote.WithUserAgent(c.userAgent))
+	if err != nil {
+		return nil, c.lookupError("list", r.String(), repo, err)
+	}
+	return tags, nil
+}
+
+// lookupError turns a failed registry request into an error that is safe to
+// show: a registry answer is described by its status and codes, anything
+// else by a fixed text. The full error goes to the log. verb and target
+// start the message; ref is what the log line names.
+func (c *Client) lookupError(verb, target, ref string, err error) error {
+	var terr *transport.Error
+	if errors.As(err, &terr) {
+		return fmt.Errorf("%s %s: %s", verb, target, describe(terr))
+	}
+	c.log.Warn("registry_request_failed", "ref", ref, "error", err)
+	return fmt.Errorf("%s %s: %s", verb, target, classify(err))
+}
+
 // classify returns a fixed text for an error that is not a registry answer.
 // ggcr copies registry-controlled text (token responses, authentication
 // challenges, redirect URLs, certificate names) into such errors, so none of

internal/master/registry/registry_test.go

@@ -134,6 +134,8 @@ func TestDigestIgnoresDigestSuffix(t *testing.T) {
 	assert.Equal(t, want, got)
 }
 
+// Digest asks the manifest endpoint with HEAD only; the one other request is
+// the GET /v2/ ping that every client sends first.
 func TestDigestUsesHeadOnly(t *testing.T) {
 	dockerConfig(t, "")
 	reg := newTestRegistry(t, "", "")
@@ -236,7 +238,7 @@ func TestCredsStoreWithoutHelperFailsTheLookup(t *testing.T) {
 }
 
 // A registry that leaves out Docker-Content-Digest must not make the client
-// retry with a manifest download, which Docker Hub counts as a pull.
+// retry with a manifest GET, which Docker Hub counts as a pull.
 func TestMissingDigestHeaderDoesNotFallBackToDownload(t *testing.T) {
 	dockerConfig(t, "")
 	var mu sync.Mutex
@@ -334,3 +336,101 @@ func TestClassify(t *testing.T) {
 		})
 	}
 }
+
+// pushTags stores a random index under each tag of repo.
+func (tr *testRegistry) pushTags(t *testing.T, repo string, tags ...string) {
+	t.Helper()
+	for _, tag := range tags {
+		tr.pushIndex(t, repo, tag)
+	}
+}
+
+// Digest and Tags together never GET a manifest: every manifest request is a
+// HEAD, and the only other requests are the /v2/ ping and tags/list.
+func TestNoManifestGet(t *testing.T) {
+	dockerConfig(t, "")
+	reg := newTestRegistry(t, "", "")
+	reg.pushTags(t, "team/app", "1.2", "1.3")
+	reg.reset()
+
+	_, err := newClient().Digest(context.Background(), reg.host+"/team/app:1.2")
+	require.NoError(t, err)
+	_, err = newClient().Tags(context.Background(), reg.host+"/team/app")
+	require.NoError(t, err)
+
+	reg.mu.Lock()
+	defer reg.mu.Unlock()
+	for _, r := range reg.requests {
+		switch {
+		case strings.Contains(r.path, "/manifests/"):
+			assert.Equal(t, http.MethodHead, r.method, r.path)
+		case r.path == "/v2/", r.path == "/v2/team/app/tags/list":
+			assert.Equal(t, http.MethodGet, r.method, r.path)
+		default:
+			assert.Failf(t, "unexpected request", "%s %s", r.method, r.path)
+		}
+	}
+}
+
+func TestTagsListsAllTags(t *testing.T) {
+	dockerConfig(t, "")
+	reg := newTestRegistry(t, "", "")
+	reg.pushTags(t, "team/app", "1.2", "1.3", "latest")
+
+	got, err := newClient().Tags(context.Background(), reg.host+"/team/app")
+
+	require.NoError(t, err)
+	assert.ElementsMatch(t, []string{"1.2", "1.3", "latest"}, got)
+}
+
+// ggcr's in-memory registry sends no Link header, so pagination is tested
+// against a handler that does.
+func TestTagsFollowsLinkPagination(t *testing.T) {
+	dockerConfig(t, "")
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case r.URL.Path == "/v2/":
+		case r.URL.Path == "/v2/a/b/tags/list" && r.URL.Query().Get("last") == "":
+			w.Header().Set("Link", `</v2/a/b/tags/list?last=1.3&n=2>; rel="next"`)
+			_, _ = w.Write([]byte(`{"name":"a/b","tags":["1.2","1.3"]}`))
+		case r.URL.Path == "/v2/a/b/tags/list":
+			_, _ = w.Write([]byte(`{"name":"a/b","tags":["1.4"]}`))
+		default:
+			w.WriteHeader(http.StatusNotFound)
+		}
+	}))
+	t.Cleanup(srv.Close)
+
+	got, err := newClient().Tags(context.Background(), strings.TrimPrefix(srv.URL, "http://")+"/a/b")
+
+	require.NoError(t, err)
+	assert.Equal(t, []string{"1.2", "1.3", "1.4"}, got)
+}
+
+func TestTagsUsesDockerConfigCredentials(t *testing.T) {
+	reg := newTestRegistry(t, "bot", "s3cret")
+	reg.pushIndex(t, "team/app", "1.2", remote.WithAuth(&authn.Basic{Username: "bot", Password: "s3cret"}))
+	auth := base64.StdEncoding.EncodeToString([]byte("bot:s3cret"))
+	dockerConfig(t, `{"auths":{"`+reg.host+`":{"auth":"`+auth+`"}}}`)
+
+	got, err := newClient().Tags(context.Background(), reg.host+"/team/app")
+
+	require.NoError(t, err)
+	assert.Equal(t, []string{"1.2"}, got)
+}
+
+func TestTagsErrorBodyDoesNotLeak(t *testing.T) {
+	dockerConfig(t, "")
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusBadRequest)
+		_, _ = w.Write([]byte(`{"errors":[{"code":"NAME_UNKNOWN","message":"internal-secret-body"}]}`))
+	}))
+	t.Cleanup(srv.Close)
+	host := strings.TrimPrefix(srv.URL, "http://")
+
+	_, err := newClient().Tags(context.Background(), host+"/a/b")
+
+	require.Error(t, err)
+	assert.NotContains(t, err.Error(), "internal-secret-body")
+	assert.ErrorContains(t, err, "list "+host+"/a/b: registry answered 400 Bad Request")
+}

internal/master/store/agents.go

@@ -13,11 +13,11 @@ import (
 	"github.com/shcizo/package-updater/internal/protocol"
 )
 
-const agentColumns = `id, name, hostname, agent_version, swarm_manager, last_seen_at, revoked_at, created_at`
+const agentColumns = `id, name, hostname, agent_version, swarm_manager, swarm_cluster_id, last_seen_at, revoked_at, created_at`
 
 func scanAgent(row pgx.Row) (model.Agent, error) {
 	var a model.Agent
-	err := row.Scan(&a.ID, &a.Name, &a.Hostname, &a.AgentVersion, &a.SwarmManager, &a.LastSeenAt, &a.RevokedAt, &a.CreatedAt)
+	err := row.Scan(&a.ID, &a.Name, &a.Hostname, &a.AgentVersion, &a.SwarmManager, &a.SwarmClusterID, &a.LastSeenAt, &a.RevokedAt, &a.CreatedAt)
 	if errors.Is(err, pgx.ErrNoRows) {
 		return model.Agent{}, model.ErrNotFound
 	}
@@ -101,8 +101,8 @@ func (s *Store) RevokeAgent(ctx context.Context, id int64, at time.Time) ([]int6
 // RecordHello stores what the agent reported about itself on connect.
 func (s *Store) RecordHello(ctx context.Context, id int64, h protocol.Hello, at time.Time) error {
 	if _, err := s.pool.Exec(ctx,
-		`UPDATE agents SET hostname = $2, agent_version = $3, swarm_manager = $4, last_seen_at = $5 WHERE id = $1`,
+		`UPDATE agents SET hostname = $2, agent_version = $3, swarm_manager = $4, swarm_cluster_id = $6, last_seen_at = $5 WHERE id = $1`,
-		id, h.Hostname, h.AgentVersion, h.SwarmManager, at); err != nil {
+		id, h.Hostname, h.AgentVersion, h.SwarmManager, at, h.SwarmClusterID); err != nil {
 		return fmt.Errorf("record hello: %w", err)
 	}
 	return nil

internal/master/store/agents_integration_test.go

@@ -80,6 +80,7 @@ func TestAgents_RecordHelloAndMarkSeen(t *testing.T) {
 	assert.Equal(t, "swarm-mgr", agents[0].Hostname)
 	assert.Equal(t, "v1", agents[0].AgentVersion)
 	assert.True(t, agents[0].SwarmManager)
+	assert.Equal(t, "c1", agents[0].SwarmClusterID)
 	require.NotNil(t, agents[0].LastSeenAt)
 	assert.True(t, agents[0].LastSeenAt.Equal(at.Add(time.Minute)))
 }

internal/master/store/drift_merge_integration_test.go

@@ -0,0 +1,554 @@
+//go:build integration
+
+package store
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/protocol"
+)
+
+var (
+	digNew     = "sha256:" + strings.Repeat("e", 64)
+	digOld     = "sha256:" + strings.Repeat("f", 64)
+	sluskDrift = "gitea.shcizo.se/shcizo/slusk"
+)
+
+// serviceID returns the ID of the service the agent reports under svc's key.
+func serviceID(t *testing.T, s *Store, agentID int64, svc protocol.Service) int64 {
+	t.Helper()
+	var id int64
+	require.NoError(t, s.pool.QueryRow(context.Background(),
+		`SELECT id FROM services WHERE location_key = $1`, model.LocationKey(agentID, "", svc.Key())).Scan(&id))
+	return id
+}
+
+// finishJob runs a job for the service to success with the deployed ref.
+func finishJob(t *testing.T, s *Store, agentID, svcID int64, toRef, deployed string) {
+	t.Helper()
+	ctx := context.Background()
+	nj := newJob(svcID, agentID)
+	nj.ToRef = toRef
+	j, err := s.CreateJob(ctx, nj)
+	require.NoError(t, err)
+	ok, err := s.FinishJob(ctx, agentID, j.ID, model.JobFinish{Status: model.JobSucceeded, Ref: deployed, StartedAt: t0, FinishedAt: t0})
+	require.NoError(t, err)
+	require.True(t, ok)
+}
+
+func mustExec(t *testing.T, s *Store, sql string, args ...any) {
+	t.Helper()
+	_, err := s.pool.Exec(context.Background(), sql, args...)
+	require.NoError(t, err)
+}
+
+func TestFinishJob_SetsAwaitingInventoryUntilTheNextUpsert(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	agent, svc := jobFixture(t, s, "a")
+
+	got, err := s.GetService(ctx, svc)
+	require.NoError(t, err)
+	assert.False(t, got.AwaitingInventory)
+
+	finishJob(t, s, agent, svc, "r:1", "r:1@"+digNew)
+	got, err = s.GetService(ctx, svc)
+	require.NoError(t, err)
+	assert.True(t, got.AwaitingInventory, "set by the successful job")
+	assert.False(t, got.Drifted(), "desired is ahead of actual only until the inventory arrives")
+
+	require.NoError(t, s.UpsertService(ctx, agent, "", slusk, t0.Add(time.Minute)))
+	got, err = s.GetService(ctx, svc)
+	require.NoError(t, err)
+	assert.False(t, got.AwaitingInventory, "cleared by the agent's upsert")
+	assert.True(t, got.Drifted(), "the agent still reports the old version")
+}
+
+func TestFinishJob_FailedJobDoesNotSetAwaitingInventory(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	agent, svc := jobFixture(t, s, "a")
+	j, err := s.CreateJob(ctx, newJob(svc, agent))
+	require.NoError(t, err)
+
+	ok, err := s.FinishJob(ctx, agent, j.ID, model.JobFinish{Status: model.JobFailed, Error: "boom", StartedAt: t0, FinishedAt: t0})
+	require.NoError(t, err)
+	require.True(t, ok)
+
+	got, err := s.GetService(ctx, svc)
+	require.NoError(t, err)
+	assert.False(t, got.AwaitingInventory)
+}
+
+func TestCreateJob_UnknownServiceIsNotFound(t *testing.T) {
+	s := openTestStore(t)
+	agent := newAgent(t, s, "a")
+
+	_, err := s.CreateJob(context.Background(), newJob(9999, agent))
+
+	assert.ErrorIs(t, err, model.ErrNotFound)
+}
+
+// driftedFixture is a service updated to digNew whose agent now reports digD1.
+func driftedFixture(t *testing.T, s *Store) (agent, svc int64) {
+	t.Helper()
+	agent, svc = jobFixture(t, s, "a")
+	finishJob(t, s, agent, svc, "r:1", sluskDrift+":1.4.2@"+digNew)
+	require.NoError(t, s.UpsertService(context.Background(), agent, "", slusk, t0.Add(time.Minute)))
+	return agent, svc
+}
+
+func TestAcceptRunning_WritesTheDesiredRef(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	_, svc := driftedFixture(t, s)
+	require.NoError(t, s.SetLastTagDigest(ctx, svc, "sha256:ltd"))
+	require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskDrift + ":1.5.0", Source: model.SourcePoll, At: t0}))
+	require.NoError(t, s.SetUpdateMode(ctx, svc, model.ModeAuto))
+	before, err := s.GetService(ctx, svc)
+	require.NoError(t, err)
+	require.True(t, before.Drifted())
+
+	running := model.RunningRef(before.ActualRef, before.ActualDigest)
+	require.NoError(t, s.AcceptRunning(ctx, svc, before.ActualRef, before.ActualDigest, running))
+
+	got, err := s.GetService(ctx, svc)
+	require.NoError(t, err)
+	assert.Equal(t, sluskDrift+":1.4.2@"+digD1, got.DesiredRef)
+	assert.False(t, got.Drifted())
+	assert.False(t, got.AwaitingInventory)
+	assert.Equal(t, "sha256:ltd", got.LastTagDigest, "last_tag_digest untouched")
+	assert.Equal(t, sluskDrift+":1.5.0", got.AvailableRef, "available suggestion untouched")
+	assert.Equal(t, model.ModeAuto, got.UpdateMode)
+}
+
+func TestAcceptRunning_Refusals(t *testing.T) {
+	tests := []struct {
+		name      string
+		prepare   func(t *testing.T, s *Store, agent, svc int64) (id int64, actualRef, actualDigest string)
+		wantError error
+	}{
+		{"running version changed", func(t *testing.T, s *Store, agent, svc int64) (int64, string, string) {
+			return svc, sluskDrift + ":1.4.2", digOld
+		}, model.ErrConflict},
+		{"actual ref changed", func(t *testing.T, s *Store, agent, svc int64) (int64, string, string) {
+			return svc, sluskDrift + ":1.3.0", digD1
+		}, model.ErrConflict},
+		{"never updated", func(t *testing.T, s *Store, agent, svc int64) (int64, string, string) {
+			mustExec(t, s, `UPDATE services SET desired_ref = '' WHERE id = $1`, svc)
+			return svc, sluskDrift + ":1.4.2", digD1
+		}, model.ErrConflict},
+		{"removed", func(t *testing.T, s *Store, agent, svc int64) (int64, string, string) {
+			require.NoError(t, s.RemoveService(context.Background(), agent, "", slusk.Key(), t0.Add(time.Hour)))
+			return svc, sluskDrift + ":1.4.2", digD1
+		}, model.ErrNotFound},
+		{"unknown", func(t *testing.T, s *Store, agent, svc int64) (int64, string, string) {
+			return 9999, sluskDrift + ":1.4.2", digD1
+		}, model.ErrNotFound},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			s := openTestStore(t)
+			agent, svc := driftedFixture(t, s)
+			id, ref, digest := tt.prepare(t, s, agent, svc)
+
+			err := s.AcceptRunning(context.Background(), id, ref, digest, "x:1@"+digOld)
+
+			assert.ErrorIs(t, err, tt.wantError)
+			got, gerr := s.GetService(context.Background(), svc)
+			require.NoError(t, gerr)
+			assert.NotEqual(t, "x:1@"+digOld, got.DesiredRef, "nothing written")
+		})
+	}
+}
+
+// moveFixture has agent "a" with slusk removed after it was updated (history,
+// auto mode) and agent "b" reporting a new slusk row.
+type moveFixture struct {
+	a, b     int64
+	from, to int64
+}
+
+func newMoveFixture(t *testing.T, s *Store) moveFixture {
+	t.Helper()
+	ctx := context.Background()
+	f := moveFixture{a: newAgent(t, s, "a"), b: newAgent(t, s, "b")}
+	require.NoError(t, s.ApplySnapshot(ctx, f.a, worker, []protocol.Service{slusk}, t0))
+	f.from = serviceID(t, s, f.a, slusk)
+	finishJob(t, s, f.a, f.from, "r:1", sluskDrift+":1.4.2@"+digNew)
+	require.NoError(t, s.SetUpdateMode(ctx, f.from, model.ModeAuto))
+	require.NoError(t, s.ApplySnapshot(ctx, f.a, worker, nil, t0.Add(time.Hour)))
+	require.NoError(t, s.ApplySnapshot(ctx, f.b, worker, []protocol.Service{slusk}, t0.Add(2*time.Hour)))
+	f.to = serviceID(t, s, f.b, slusk)
+	return f
+}
+
+func TestMoveCandidates(t *testing.T) {
+	tests := []struct {
+		name   string
+		mutate func(t *testing.T, s *Store, f moveFixture)
+		want   bool
+		from   string // agent name of the removed row
+	}{
+		{"one removed and one live", func(*testing.T, *Store, moveFixture) {}, true, "a"},
+		{"removed row's agent was revoked", func(t *testing.T, s *Store, f moveFixture) {
+			_, err := s.RevokeAgent(context.Background(), f.a, t0.Add(3*time.Hour))
+			require.NoError(t, err)
+		}, true, "a"},
+		{"two live targets", func(t *testing.T, s *Store, f moveFixture) {
+			c := newAgent(t, s, "c")
+			require.NoError(t, s.ApplySnapshot(context.Background(), c, worker, []protocol.Service{slusk}, t0.Add(3*time.Hour)))
+		}, false, ""},
+		{"two removed sources", func(t *testing.T, s *Store, f moveFixture) {
+			ctx := context.Background()
+			d := newAgent(t, s, "d")
+			require.NoError(t, s.ApplySnapshot(ctx, d, worker, []protocol.Service{slusk}, t0))
+			require.NoError(t, s.ApplySnapshot(ctx, d, worker, nil, t0.Add(time.Hour)))
+		}, false, ""},
+		{"target has history", func(t *testing.T, s *Store, f moveFixture) {
+			finishJob(t, s, f.b, f.to, "r:1", sluskDrift+":1.4.2@"+digNew)
+		}, false, ""},
+		{"target has a desired ref", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `UPDATE services SET desired_ref = 'r:1@x' WHERE id = $1`, f.to)
+		}, false, ""},
+		{"source has no history", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `DELETE FROM update_jobs WHERE service_id = $1`, f.from)
+			mustExec(t, s, `UPDATE services SET desired_ref = '', update_mode = 'manual' WHERE id = $1`, f.from)
+		}, false, ""},
+		{"source only has auto mode", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `DELETE FROM update_jobs WHERE service_id = $1`, f.from)
+			mustExec(t, s, `UPDATE services SET desired_ref = '' WHERE id = $1`, f.from)
+		}, true, "a"},
+		{"source only has jobs", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `UPDATE services SET desired_ref = '', update_mode = 'manual' WHERE id = $1`, f.from)
+		}, true, "a"},
+		{"target is older than the source", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `UPDATE services SET first_seen_at = $2 WHERE id = $1`, f.to, t0.Add(-time.Hour))
+		}, false, ""},
+		{"plain containers", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `UPDATE services SET kind = 'container' WHERE id IN ($1, $2)`, f.from, f.to)
+		}, false, ""},
+		{"swarm services", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `UPDATE services SET kind = 'swarm' WHERE id IN ($1, $2)`, f.from, f.to)
+		}, false, ""},
+		{"the updater itself (target)", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `UPDATE services SET self = true WHERE id = $1`, f.to)
+		}, false, ""},
+		{"the updater itself (source)", func(t *testing.T, s *Store, f moveFixture) {
+			mustExec(t, s, `UPDATE services SET self = true WHERE id = $1`, f.from)
+		}, false, ""},
+		{"live row also removed", func(t *testing.T, s *Store, f moveFixture) {
+			require.NoError(t, s.RemoveService(context.Background(), f.b, "", slusk.Key(), t0.Add(3*time.Hour)))
+		}, false, ""},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			s := openTestStore(t)
+			f := newMoveFixture(t, s)
+			tt.mutate(t, s, f)
+
+			got, err := s.MoveCandidates(context.Background())
+
+			require.NoError(t, err)
+			if !tt.want {
+				assert.Empty(t, got)
+				return
+			}
+			assert.Equal(t, []model.Move{{FromID: f.from, ToID: f.to, FromAgent: tt.from, ToAgent: "b"}}, got)
+		})
+	}
+}
+
+func TestMergeService_MovesHistoryModeAndDesiredToTheNewLocation(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	f := newMoveFixture(t, s)
+	finishJob(t, s, f.b, f.to, "r:2", sluskDrift+":2.0.0@"+digOld)
+	require.NoError(t, s.SetLastTagDigest(ctx, f.to, "sha256:fromto"))
+	require.NoError(t, s.SetAvailable(ctx, f.to, model.Available{Ref: sluskDrift + ":3", Digest: digNew, Source: model.SourcePoll, At: t0}))
+	require.NoError(t, s.SetPollError(ctx, []int64{f.to}, "registry down"))
+	require.NoError(t, s.SetNewer(ctx, f.to, sluskDrift+":3.1", t0))
+	// A job on the old row is finished, and awaiting_inventory is set on both.
+	before, err := s.GetService(ctx, f.from)
+	require.NoError(t, err)
+
+	require.NoError(t, s.MergeService(ctx, f.from, f.to))
+
+	got, err := s.GetService(ctx, f.from)
+	require.NoError(t, err)
+	assert.Equal(t, f.from, got.ID, "the old id survives")
+	assert.Nil(t, got.RemovedAt)
+	assert.False(t, got.AwaitingInventory)
+	assert.Equal(t, "b", got.AgentName)
+	assert.Equal(t, model.LocationKey(f.b, "", slusk.Key()), got.LocationKey)
+	assert.Equal(t, before.FirstSeenAt, got.FirstSeenAt, "first_seen_at kept")
+	assert.Equal(t, model.ModeAuto, got.UpdateMode, "mode kept")
+	assert.Equal(t, before.DesiredRef, got.DesiredRef, "desired ref kept")
+	assert.Equal(t, slusk.Image, got.ActualRef, "inventory from the duplicate")
+	assert.Equal(t, digD1, got.ActualDigest)
+	assert.Equal(t, "running", got.Status)
+	assert.Equal(t, sluskDrift+":3", got.AvailableRef)
+	assert.Equal(t, digNew, got.AvailableDigest)
+	assert.Equal(t, "registry down", got.LastPollError)
+	assert.Equal(t, sluskDrift+":3.1", got.NewerRef, "newer tag from the duplicate")
+	require.NotNil(t, got.NewerAt)
+	assert.True(t, t0.Equal(*got.NewerAt))
+	assert.Equal(t, before.LastTagDigest, got.LastTagDigest, "an existing last_tag_digest wins")
+
+	_, err = s.GetService(ctx, f.to)
+	assert.ErrorIs(t, err, model.ErrNotFound, "the duplicate is deleted")
+	jobs, err := s.ListJobs(ctx, model.JobFilter{ServiceID: f.from})
+	require.NoError(t, err)
+	assert.Len(t, jobs, 2, "the duplicate's job moved to the survivor")
+	live, err := s.ListServices(ctx)
+	require.NoError(t, err)
+	assert.Equal(t, []string{"slusk/web"}, names(live))
+}
+
+func TestMergeService_TakesLastTagDigestOfTheDuplicateWhenTheOldRowHasNone(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	f := newMoveFixture(t, s)
+	mustExec(t, s, `UPDATE services SET last_tag_digest = '' WHERE id = $1`, f.from)
+	require.NoError(t, s.SetLastTagDigest(ctx, f.to, "sha256:fromto"))
+
+	require.NoError(t, s.MergeService(ctx, f.from, f.to))
+
+	got, err := s.GetService(ctx, f.from)
+	require.NoError(t, err)
+	assert.Equal(t, "sha256:fromto", got.LastTagDigest)
+}
+
+func TestMergeService_ManualMergeOfAnyRemovedServiceOfTheSameKind(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	f := newMoveFixture(t, s)
+	other := protocol.Service{Kind: protocol.KindCompose, ComposeProject: "renamed", ComposeService: "app", Image: "x/y:1", Status: "running"}
+	require.NoError(t, s.UpsertService(ctx, f.b, "", other, t0.Add(3*time.Hour)))
+	target := serviceID(t, s, f.b, other)
+
+	require.NoError(t, s.MergeService(ctx, f.from, target))
+
+	got, err := s.GetService(ctx, f.from)
+	require.NoError(t, err)
+	assert.Equal(t, "renamed/app", got.DisplayName(), "names may differ")
+}
+
+func TestMergeService_SameAgentIsAllowed(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	f := newMoveFixture(t, s)
+	other := protocol.Service{Kind: protocol.KindCompose, ComposeProject: "renamed", ComposeService: "app", Image: "x/y:1", Status: "running"}
+	require.NoError(t, s.UpsertService(ctx, f.a, "", other, t0.Add(3*time.Hour)))
+
+	require.NoError(t, s.MergeService(ctx, f.from, serviceID(t, s, f.a, other)))
+
+	got, err := s.GetService(ctx, f.from)
+	require.NoError(t, err)
+	assert.Equal(t, "a", got.AgentName)
+}
+
+func TestMergeService_Swarm(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	m1 := newAgent(t, s, "m1")
+	require.NoError(t, s.ApplySnapshot(ctx, m1, manager, []protocol.Service{gitea}, t0))
+	oldID := serviceIDByName(t, s, "gitea_gitea")
+	require.NoError(t, s.ApplySnapshot(ctx, m1, manager, nil, t0.Add(time.Hour)))
+	renamed := gitea
+	renamed.SwarmService = "gitea2_gitea"
+	require.NoError(t, s.ApplySnapshot(ctx, m1, manager, []protocol.Service{renamed}, t0.Add(2*time.Hour)))
+	newID := serviceIDByName(t, s, "gitea2_gitea")
+
+	require.NoError(t, s.MergeService(ctx, oldID, newID))
+
+	got, err := s.GetService(ctx, oldID)
+	require.NoError(t, err)
+	assert.Equal(t, "gitea2_gitea", got.SwarmService)
+	assert.Nil(t, got.RemovedAt)
+}
+
+func serviceIDByName(t *testing.T, s *Store, swarmService string) int64 {
+	t.Helper()
+	var id int64
+	require.NoError(t, s.pool.QueryRow(context.Background(),
+		`SELECT id FROM services WHERE swarm_service = $1`, swarmService).Scan(&id))
+	return id
+}
+
+func TestMergeService_Refusals(t *testing.T) {
+	refused := func(reason string) func(*testing.T, error) {
+		return func(t *testing.T, err error) {
+			var mr *model.MergeRefusedError
+			require.True(t, errors.As(err, &mr), "got %v", err)
+			assert.Equal(t, reason, mr.Reason)
+		}
+	}
+	is := func(want error) func(*testing.T, error) {
+		return func(t *testing.T, err error) { assert.ErrorIs(t, err, want) }
+	}
+	tests := []struct {
+		name   string
+		run    func(t *testing.T, s *Store, f moveFixture) error
+		assert func(*testing.T, error)
+	}{
+		{"into itself", func(t *testing.T, s *Store, f moveFixture) error {
+			return s.MergeService(context.Background(), f.to, f.to)
+		}, refused(model.ReasonMergeSame)},
+		{"the old service still exists", func(t *testing.T, s *Store, f moveFixture) error {
+			mustExec(t, s, `UPDATE services SET removed_at = NULL WHERE id = $1`, f.from)
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, refused(model.ReasonMergeLiveOld)},
+		{"the new service was removed", func(t *testing.T, s *Store, f moveFixture) error {
+			require.NoError(t, s.RemoveService(context.Background(), f.b, "", slusk.Key(), t0.Add(3*time.Hour)))
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, refused(model.ReasonMergeGoneNew)},
+		{"different kinds", func(t *testing.T, s *Store, f moveFixture) error {
+			mustExec(t, s, `UPDATE services SET kind = 'swarm' WHERE id = $1`, f.to)
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, refused(model.ReasonMergeKind)},
+		{"plain containers", func(t *testing.T, s *Store, f moveFixture) error {
+			mustExec(t, s, `UPDATE services SET kind = 'container' WHERE id IN ($1, $2)`, f.from, f.to)
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, refused(model.ReasonMergeContainer)},
+		{"the updater itself (new)", func(t *testing.T, s *Store, f moveFixture) error {
+			mustExec(t, s, `UPDATE services SET self = true WHERE id = $1`, f.to)
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, refused(model.ReasonMergeSelf)},
+		{"the updater itself (old)", func(t *testing.T, s *Store, f moveFixture) error {
+			mustExec(t, s, `UPDATE services SET self = true WHERE id = $1`, f.from)
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, refused(model.ReasonMergeSelf)},
+		{"active job on the old service", func(t *testing.T, s *Store, f moveFixture) error {
+			_, err := s.CreateJob(context.Background(), newJob(f.from, f.a))
+			require.NoError(t, err)
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, is(model.ErrJobActive)},
+		{"active job on the new service", func(t *testing.T, s *Store, f moveFixture) error {
+			_, err := s.CreateJob(context.Background(), newJob(f.to, f.b))
+			require.NoError(t, err)
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, is(model.ErrJobActive)},
+		{"unknown old service", func(t *testing.T, s *Store, f moveFixture) error {
+			return s.MergeService(context.Background(), 9999, f.to)
+		}, is(model.ErrNotFound)},
+		{"unknown new service", func(t *testing.T, s *Store, f moveFixture) error {
+			return s.MergeService(context.Background(), f.from, 9999)
+		}, is(model.ErrNotFound)},
+		{"the new service's agent is revoked", func(t *testing.T, s *Store, f moveFixture) error {
+			_, err := s.RevokeAgent(context.Background(), f.b, t0.Add(3*time.Hour))
+			require.NoError(t, err)
+			return s.MergeService(context.Background(), f.from, f.to)
+		}, is(model.ErrRevoked)},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			s := openTestStore(t)
+			f := newMoveFixture(t, s)
+			var rowsBefore int
+			require.NoError(t, s.pool.QueryRow(context.Background(), `SELECT count(*) FROM services`).Scan(&rowsBefore))
+
+			err := tt.run(t, s, f)
+
+			tt.assert(t, err)
+			var rowsAfter int
+			require.NoError(t, s.pool.QueryRow(context.Background(), `SELECT count(*) FROM services`).Scan(&rowsAfter))
+			assert.Equal(t, rowsBefore, rowsAfter, "nothing is deleted when the merge is refused")
+		})
+	}
+}
+
+func TestMergeService_InventoryAfterTheMerge(t *testing.T) {
+	setup := func(t *testing.T) (*Store, moveFixture) {
+		s := openTestStore(t)
+		f := newMoveFixture(t, s)
+		require.NoError(t, s.MergeService(context.Background(), f.from, f.to))
+		return s, f
+	}
+
+	t.Run("an upsert for the new location updates the survivor", func(t *testing.T) {
+		s, f := setup(t)
+		ctx := context.Background()
+		moved := slusk
+		moved.Digest = digOld
+		require.NoError(t, s.UpsertService(ctx, f.b, "", moved, t0.Add(4*time.Hour)))
+
+		live, err := s.ListServices(ctx)
+		require.NoError(t, err)
+		require.Len(t, live, 1)
+		assert.Equal(t, f.from, live[0].ID)
+		assert.Equal(t, digOld, live[0].ActualDigest)
+	})
+
+	t.Run("a snapshot keeps the survivor", func(t *testing.T) {
+		s, f := setup(t)
+		ctx := context.Background()
+		require.NoError(t, s.ApplySnapshot(ctx, f.b, worker, []protocol.Service{slusk}, t0.Add(4*time.Hour)))
+
+		got, err := s.GetService(ctx, f.from)
+		require.NoError(t, err)
+		assert.Nil(t, got.RemovedAt)
+		live, err := s.ListServices(ctx)
+		require.NoError(t, err)
+		assert.Len(t, live, 1)
+	})
+
+	t.Run("the old agent returning with the stack creates a new row", func(t *testing.T) {
+		s, f := setup(t)
+		ctx := context.Background()
+		require.NoError(t, s.ApplySnapshot(ctx, f.a, worker, []protocol.Service{slusk}, t0.Add(4*time.Hour)))
+
+		returned := serviceID(t, s, f.a, slusk)
+		assert.NotEqual(t, f.from, returned)
+		got, err := s.GetService(ctx, returned)
+		require.NoError(t, err)
+		assert.Equal(t, model.ModeManual, got.UpdateMode)
+		assert.Empty(t, got.DesiredRef)
+		live, err := s.ListServices(ctx)
+		require.NoError(t, err)
+		assert.Len(t, live, 2)
+	})
+}
+
+func TestMergeSources(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	f := newMoveFixture(t, s)
+	// A removed service with another name, removed later than the matching one.
+	other := protocol.Service{Kind: protocol.KindCompose, ComposeProject: "renamed", ComposeService: "app", Image: "x/y:1", Status: "running"}
+	require.NoError(t, s.UpsertService(ctx, f.a, "", other, t0))
+	require.NoError(t, s.RemoveService(ctx, f.a, "", other.Key(), t0.Add(5*time.Hour)))
+	otherID := serviceID(t, s, f.a, other)
+	// Other kinds, the updater itself and a live service never show up.
+	require.NoError(t, s.ApplySnapshot(ctx, f.a, worker, []protocol.Service{adhoc}, t0.Add(6*time.Hour)))
+	require.NoError(t, s.ApplySnapshot(ctx, f.a, worker, nil, t0.Add(7*time.Hour)))
+	selfSvc := protocol.Service{Kind: protocol.KindCompose, ComposeProject: "pu", ComposeService: "agent", Image: "pu:1", Status: "running", Self: true}
+	require.NoError(t, s.UpsertService(ctx, f.a, "", selfSvc, t0))
+	require.NoError(t, s.RemoveService(ctx, f.a, "", selfSvc.Key(), t0.Add(8*time.Hour)))
+
+	got, err := s.MergeSources(ctx, f.to)
+
+	require.NoError(t, err)
+	var ids []int64
+	for _, m := range got {
+		ids = append(ids, m.ID)
+	}
+	assert.Equal(t, []int64{f.from, otherID}, ids, "same name first, then the most recently removed")
+	assert.Equal(t, "a", got[0].AgentName)
+
+	_, err = s.MergeSources(ctx, 9999)
+	assert.ErrorIs(t, err, model.ErrNotFound)
+
+	containerID := serviceID(t, s, f.a, adhoc)
+	got, err = s.MergeSources(ctx, containerID)
+	require.NoError(t, err)
+	assert.Empty(t, got, "plain containers cannot be merged")
+}

internal/master/store/jobs.go

@@ -10,6 +10,7 @@ import (
 	"github.com/jackc/pgx/v5"
 	"github.com/jackc/pgx/v5/pgconn"
 
+	"github.com/shcizo/package-updater/internal/imageref"
 	"github.com/shcizo/package-updater/internal/master/model"
 	"github.com/shcizo/package-updater/internal/protocol"
 )
@@ -90,6 +91,9 @@ func (s *Store) CreateJob(ctx context.Context, j model.NewJob) (model.Job, error
 			}
 			return active, model.ErrJobActive
 		}
+		if errors.As(err, &pgErr) && pgErr.Code == "23503" && pgErr.ConstraintName == "update_jobs_service_id_fkey" {
+			return model.Job{}, model.ErrNotFound
+		}
 		if err != nil {
 			return model.Job{}, fmt.Errorf("create job: %w", err)
 		}
@@ -232,11 +236,11 @@ func (s *Store) FinishJob(ctx context.Context, agentID, jobID int64, f model.Job
 		// A plain tag job, whoever started it, deployed the tag at the digest
 		// the agent pinned; remember it. Pinned and tagless picks do not.
 		tagDigest := ""
-		if _, d, _ := strings.Cut(f.Ref, "@"); requestedDigest == "" && digestPattern.MatchString(d) {
+		if _, d, _ := strings.Cut(f.Ref, "@"); requestedDigest == "" && imageref.IsDigest(d) {
 			tagDigest = d
 		}
 		if _, err := tx.Exec(ctx, `
-			UPDATE services SET desired_ref = $2,
+			UPDATE services SET desired_ref = $2, awaiting_inventory = true,
 				last_tag_digest  = CASE WHEN $6 <> '' THEN $6 ELSE last_tag_digest END,
 				available_ref    = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_ref END,
 				available_digest = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_digest END,

internal/master/store/merge.go

@@ -0,0 +1,250 @@
+package store
+
+import (
+	"context"
+	"errors"
+	"fmt"
+
+	"github.com/jackc/pgx/v5"
+
+	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/protocol"
+)
+
+// moveCandidatesSQL pairs a removed Compose service with a live one of the
+// same project and service name on another agent. A pair counts only when it
+// is the single one for both rows, and the live row is new (newer than the
+// removed one, never updated, no jobs) while the removed one has something
+// worth moving (a desired ref, auto mode or jobs).
+const moveCandidatesSQL = `
+WITH pairs AS (
+	SELECT f.id AS from_id, t.id AS to_id
+	FROM services f
+	JOIN services t ON t.kind = 'compose' AND t.removed_at IS NULL AND NOT t.self
+	 AND t.compose_project = f.compose_project AND t.compose_service = f.compose_service
+	 AND t.agent_id IS DISTINCT FROM f.agent_id
+	WHERE f.kind = 'compose' AND f.removed_at IS NOT NULL AND NOT f.self
+), uniq AS (
+	SELECT from_id, to_id FROM pairs p
+	WHERE (SELECT count(*) FROM pairs q WHERE q.from_id = p.from_id) = 1
+	  AND (SELECT count(*) FROM pairs q WHERE q.to_id   = p.to_id)   = 1
+)
+SELECT u.from_id, u.to_id, COALESCE(fa.name, ''), COALESCE(ta.name, '')
+FROM uniq u
+JOIN services f ON f.id = u.from_id
+JOIN services t ON t.id = u.to_id
+LEFT JOIN agents fa ON fa.id = f.agent_id
+LEFT JOIN agents ta ON ta.id = t.agent_id
+WHERE t.first_seen_at > f.first_seen_at
+  AND t.desired_ref = ''
+  AND NOT EXISTS (SELECT 1 FROM update_jobs j WHERE j.service_id = t.id)
+  AND (f.desired_ref <> '' OR f.update_mode = 'auto'
+       OR EXISTS (SELECT 1 FROM update_jobs j WHERE j.service_id = f.id))
+ORDER BY u.to_id`
+
+// MoveCandidates returns the removed Compose services that look like they
+// moved to another agent; see model.Move. It only suggests: nothing is merged.
+func (s *Store) MoveCandidates(ctx context.Context) ([]model.Move, error) {
+	rows, err := s.pool.Query(ctx, moveCandidatesSQL)
+	if err != nil {
+		return nil, fmt.Errorf("move candidates: %w", err)
+	}
+	defer rows.Close()
+	var out []model.Move
+	for rows.Next() {
+		var m model.Move
+		if err := rows.Scan(&m.FromID, &m.ToID, &m.FromAgent, &m.ToAgent); err != nil {
+			return nil, fmt.Errorf("scan move: %w", err)
+		}
+		out = append(out, m)
+	}
+	return out, rows.Err()
+}
+
+// MergeSources returns up to 50 removed services that could be merged into
+// the service toID: the same kind (Compose or Swarm), not the updater itself,
+// those with the same name first, then the most recently removed. Returns
+// model.ErrNotFound if toID does not exist.
+func (s *Store) MergeSources(ctx context.Context, toID int64) ([]model.Service, error) {
+	to, err := s.GetService(ctx, toID)
+	if err != nil {
+		return nil, err
+	}
+	if to.Kind == protocol.KindContainer {
+		return nil, nil
+	}
+	rows, err := s.pool.Query(ctx, `
+		SELECT `+serviceColumns+`
+		FROM services s LEFT JOIN agents a ON a.id = s.agent_id
+		WHERE s.removed_at IS NOT NULL AND s.kind = $1 AND NOT s.self AND s.id <> $2
+		ORDER BY (s.compose_project, s.compose_service, s.swarm_service) = ($3, $4, $5) DESC,
+		         s.removed_at DESC, s.id DESC
+		LIMIT 50`,
+		string(to.Kind), toID, to.ComposeProject, to.ComposeService, to.SwarmService)
+	if err != nil {
+		return nil, fmt.Errorf("merge sources: %w", err)
+	}
+	defer rows.Close()
+	var out []model.Service
+	for rows.Next() {
+		m, err := scanService(rows)
+		if err != nil {
+			return nil, fmt.Errorf("scan service: %w", err)
+		}
+		out = append(out, m)
+	}
+	return out, rows.Err()
+}
+
+// mergeRow is what MergeService needs to know about one of the two rows.
+type mergeRow struct {
+	kind    string
+	agentID *int64
+	removed bool
+	self    bool
+}
+
+// MergeService moves the live service toID onto the removed service fromID:
+// the old row keeps its id, first_seen_at, update mode and desired ref, takes
+// the location and inventory of the live row (and its available and newer-tag
+// suggestions), becomes live, and the live row is deleted. The jobs of the live
+// row move to the old one. This is the only place a services row is deleted.
+//
+// It returns *model.MergeRefusedError when the pair cannot be merged,
+// model.ErrJobActive when either row has a queued or running job,
+// model.ErrRevoked when the live row's agent is revoked, model.ErrNotFound
+// when a row does not exist, and model.ErrConflict when the live row moved to
+// another agent while the merge started.
+//
+// Locks are taken in the order inventory writes use: the agent row first,
+// then the service rows by id. The live row is deleted before the old one is
+// updated because location_key is unique.
+func (s *Store) MergeService(ctx context.Context, fromID, toID int64) error {
+	if fromID == toID {
+		return &model.MergeRefusedError{Reason: model.ReasonMergeSame}
+	}
+	tx, err := s.pool.Begin(ctx)
+	if err != nil {
+		return fmt.Errorf("begin merge: %w", err)
+	}
+	defer func() { _ = tx.Rollback(ctx) }()
+
+	var toAgent *int64
+	err = tx.QueryRow(ctx, `SELECT agent_id FROM services WHERE id = $1`, toID).Scan(&toAgent)
+	if errors.Is(err, pgx.ErrNoRows) {
+		return model.ErrNotFound
+	}
+	if err != nil {
+		return fmt.Errorf("merge: find new service: %w", err)
+	}
+	if toAgent != nil {
+		if err := lockActiveAgent(ctx, tx, *toAgent); err != nil {
+			return err
+		}
+	}
+
+	rows, err := tx.Query(ctx, `
+		SELECT id, kind, agent_id, removed_at IS NOT NULL, self FROM services
+		WHERE id IN ($1, $2) ORDER BY id FOR UPDATE`, fromID, toID)
+	if err != nil {
+		return fmt.Errorf("merge: lock services: %w", err)
+	}
+	locked := map[int64]mergeRow{}
+	for rows.Next() {
+		var id int64
+		var r mergeRow
+		if err := rows.Scan(&id, &r.kind, &r.agentID, &r.removed, &r.self); err != nil {
+			rows.Close()
+			return fmt.Errorf("merge: scan service: %w", err)
+		}
+		locked[id] = r
+	}
+	rows.Close()
+	if err := rows.Err(); err != nil {
+		return fmt.Errorf("merge: lock services: %w", err)
+	}
+	from, okFrom := locked[fromID]
+	to, okTo := locked[toID]
+	if !okFrom || !okTo {
+		return model.ErrNotFound
+	}
+	if (to.agentID == nil) != (toAgent == nil) || (toAgent != nil && *to.agentID != *toAgent) {
+		return model.ErrConflict
+	}
+	if reason := mergeRefusal(from, to); reason != "" {
+		return &model.MergeRefusedError{Reason: reason}
+	}
+
+	var active bool
+	if err := tx.QueryRow(ctx, `
+		SELECT EXISTS (SELECT 1 FROM update_jobs
+		               WHERE service_id IN ($1, $2) AND status IN ('queued', 'running'))`,
+		fromID, toID).Scan(&active); err != nil {
+		return fmt.Errorf("merge: check active jobs: %w", err)
+	}
+	if active {
+		return model.ErrJobActive
+	}
+
+	if _, err := tx.Exec(ctx, `UPDATE update_jobs SET service_id = $1 WHERE service_id = $2`, fromID, toID); err != nil {
+		return fmt.Errorf("merge: move jobs: %w", err)
+	}
+	var t model.Service
+	var lastTagDigest string
+	err = tx.QueryRow(ctx, `
+		SELECT agent_id, location_key, compose_project, compose_service, working_dir, config_files, built,
+		       swarm_cluster_id, swarm_service, container_name, image_repo, actual_ref, actual_digest, status, self,
+		       updated_at, available_ref, available_digest, available_source, available_at, last_poll_error,
+		       last_tag_digest, newer_ref, newer_at
+		FROM services WHERE id = $1`, toID).Scan(
+		&t.AgentID, &t.LocationKey, &t.ComposeProject, &t.ComposeService, &t.WorkingDir, &t.ConfigFiles, &t.Built,
+		&t.SwarmClusterID, &t.SwarmService, &t.ContainerName, &t.ImageRepo, &t.ActualRef, &t.ActualDigest, &t.Status, &t.Self,
+		&t.UpdatedAt, &t.AvailableRef, &t.AvailableDigest, &t.AvailableSource, &t.AvailableAt, &t.LastPollError,
+		&lastTagDigest, &t.NewerRef, &t.NewerAt)
+	if err != nil {
+		return fmt.Errorf("merge: read new service: %w", err)
+	}
+	if _, err := tx.Exec(ctx, `DELETE FROM services WHERE id = $1`, toID); err != nil {
+		return fmt.Errorf("merge: delete new service: %w", err)
+	}
+	if _, err := tx.Exec(ctx, `
+		UPDATE services SET
+			agent_id = $2, location_key = $3, compose_project = $4, compose_service = $5, working_dir = $6,
+			config_files = $7, built = $8, swarm_cluster_id = $9, swarm_service = $10, container_name = $11,
+			image_repo = $12, actual_ref = $13, actual_digest = $14, status = $15, self = $16, updated_at = $17,
+			available_ref = $18, available_digest = $19, available_source = $20, available_at = $21,
+			last_poll_error = $22,
+			last_tag_digest = CASE WHEN last_tag_digest <> '' THEN last_tag_digest ELSE $23 END,
+			newer_ref = $24, newer_at = $25,
+			removed_at = NULL, awaiting_inventory = false
+		WHERE id = $1`,
+		fromID, t.AgentID, t.LocationKey, t.ComposeProject, t.ComposeService, t.WorkingDir,
+		t.ConfigFiles, t.Built, t.SwarmClusterID, t.SwarmService, t.ContainerName,
+		t.ImageRepo, t.ActualRef, t.ActualDigest, t.Status, t.Self, t.UpdatedAt,
+		t.AvailableRef, t.AvailableDigest, t.AvailableSource, t.AvailableAt,
+		t.LastPollError, lastTagDigest, t.NewerRef, t.NewerAt); err != nil {
+		return fmt.Errorf("merge: update old service: %w", err)
+	}
+	if err := tx.Commit(ctx); err != nil {
+		return fmt.Errorf("commit merge: %w", err)
+	}
+	return nil
+}
+
+// mergeRefusal returns why from (the old row) cannot be merged with to (the
+// live one), or "".
+func mergeRefusal(from, to mergeRow) string {
+	switch {
+	case !from.removed:
+		return model.ReasonMergeLiveOld
+	case to.removed:
+		return model.ReasonMergeGoneNew
+	case from.kind != to.kind:
+		return model.ReasonMergeKind
+	case from.kind == string(protocol.KindContainer):
+		return model.ReasonMergeContainer
+	case from.self || to.self:
+		return model.ReasonMergeSelf
+	}
+	return ""
+}

internal/master/store/migrations/0005_drift_merge.sql

@@ -0,0 +1,7 @@
+-- True from a successful job until the next inventory write for the row. The
+-- agent's upsert arrives a debounce interval after job.result, and until then
+-- desired_ref is newer than actual_ref without the service having drifted.
+ALTER TABLE services ADD COLUMN awaiting_inventory boolean NOT NULL DEFAULT false;
+
+-- Move suggestions look up Compose rows by project and service across agents.
+CREATE INDEX services_by_compose_name ON services (compose_project, compose_service) WHERE kind = 'compose';

internal/master/store/migrations/0006_agent_cluster.sql

@@ -0,0 +1 @@
+ALTER TABLE agents ADD COLUMN swarm_cluster_id text NOT NULL DEFAULT '';

internal/master/store/migrations/0007_newer_tag.sql

@@ -0,0 +1,9 @@
+-- Newer version tag of the followed tag, found in the registry's tag list.
+-- A suggestion only: nothing deploys it automatically.
+ALTER TABLE services ADD COLUMN newer_ref text NOT NULL DEFAULT '';
+ALTER TABLE services ADD COLUMN newer_at  timestamptz;
+
+-- A suggested tag is recorded in image_versions with source 'tags'.
+ALTER TABLE image_versions DROP CONSTRAINT image_versions_source_check;
+ALTER TABLE image_versions ADD CONSTRAINT image_versions_source_check
+    CHECK (source IN ('webhook', 'poll', 'running', 'tags'));

internal/master/store/services.go

@@ -4,7 +4,6 @@ import (
 	"context"
 	"errors"
 	"fmt"
-	"regexp"
 	"time"
 
 	"github.com/jackc/pgx/v5"
@@ -55,10 +54,8 @@ ON CONFLICT (location_key) DO UPDATE SET
 	status        = EXCLUDED.status,
 	self          = EXCLUDED.self,
 	updated_at    = EXCLUDED.updated_at,
-	removed_at    = NULL`
+	removed_at    = NULL,
-
+	awaiting_inventory = false`
-// digestPattern is the form of a digest worth recording as a version.
-var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
 
 func upsertService(ctx context.Context, db execer, agentID int64, clusterID string, svc protocol.Service, at time.Time) error {
 	image := imageref.Parse(svc.Image)
@@ -75,7 +72,7 @@ func upsertService(ctx context.Context, db execer, agentID int64, clusterID stri
 	if err != nil {
 		return fmt.Errorf("upsert service %s: %w", svc.Key(), err)
 	}
-	if digestPattern.MatchString(svc.Digest) {
+	if imageref.IsDigest(svc.Digest) {
 		// A bare image runs "latest", as jobs.FollowedRef sees it, so it
 		// shares its row with the polled one. repo@digest has no tag.
 		if image.Tag == "" && image.Digest == "" {
@@ -178,7 +175,7 @@ const serviceColumns = `s.id, s.kind, s.location_key, s.agent_id, COALESCE(a.nam
 	s.compose_project, s.compose_service, s.working_dir, s.config_files, s.built, s.swarm_cluster_id,
 	s.swarm_service, s.container_name, s.image_repo, s.desired_ref, s.actual_ref, s.actual_digest,
 	s.status, s.self, s.update_mode, s.available_ref, s.available_digest, s.available_source, s.available_at,
-	s.last_poll_error, s.last_tag_digest, s.first_seen_at, s.updated_at, s.removed_at`
+	s.last_poll_error, s.newer_ref, s.newer_at, s.last_tag_digest, s.awaiting_inventory, s.first_seen_at, s.updated_at, s.removed_at`
 
 func scanService(row pgx.Row) (model.Service, error) {
 	var m model.Service
@@ -187,7 +184,7 @@ func scanService(row pgx.Row) (model.Service, error) {
 		&m.ComposeProject, &m.ComposeService, &m.WorkingDir, &m.ConfigFiles, &m.Built, &m.SwarmClusterID,
 		&m.SwarmService, &m.ContainerName, &m.ImageRepo, &m.DesiredRef, &m.ActualRef, &m.ActualDigest,
 		&m.Status, &m.Self, &m.UpdateMode, &m.AvailableRef, &m.AvailableDigest, &m.AvailableSource, &m.AvailableAt,
-		&m.LastPollError, &m.LastTagDigest, &m.FirstSeenAt, &m.UpdatedAt, &m.RemovedAt)
+		&m.LastPollError, &m.NewerRef, &m.NewerAt, &m.LastTagDigest, &m.AwaitingInventory, &m.FirstSeenAt, &m.UpdatedAt, &m.RemovedAt)
 	m.Kind = protocol.Kind(kind)
 	return m, err
 }
@@ -249,6 +246,33 @@ func (s *Store) SetAvailable(ctx context.Context, id int64, a model.Available) e
 	return nil
 }
 
+// SetNewer sets the service's newer-tag suggestion (model.Service.NewerRef);
+// ref "" clears it. A non-empty ref is also recorded in image_versions with
+// source "tags", in the same transaction, unless that (repo, tag) is already
+// there. The digest is unknown, so the row has none.
+func (s *Store) SetNewer(ctx context.Context, id int64, ref string, at time.Time) error {
+	tx, err := s.pool.Begin(ctx)
+	if err != nil {
+		return fmt.Errorf("set newer: begin: %w", err)
+	}
+	defer func() { _ = tx.Rollback(ctx) }()
+	if _, err := tx.Exec(ctx, `
+		UPDATE services SET newer_ref = $2, newer_at = CASE WHEN $2 = '' THEN NULL ELSE $3::timestamptz END
+		WHERE id = $1`, id, ref, at); err != nil {
+		return fmt.Errorf("set newer: %w", err)
+	}
+	if ref != "" {
+		p := imageref.Parse(ref)
+		if err := insertVersion(ctx, tx, p.Repo, p.Tag, "", model.SourceTags, at); err != nil {
+			return err
+		}
+	}
+	if err := tx.Commit(ctx); err != nil {
+		return fmt.Errorf("set newer: commit: %w", err)
+	}
+	return nil
+}
+
 // SetPollError sets last_poll_error on the given services; msg "" clears it.
 func (s *Store) SetPollError(ctx context.Context, ids []int64, msg string) error {
 	if len(ids) == 0 {
@@ -262,8 +286,8 @@ func (s *Store) SetPollError(ctx context.Context, ids []int64, msg string) error
 }
 
 // SetLastTagDigest sets last_tag_digest; see model.Service.LastTagDigest. The
-// poller calls it, and FinishJob sets the same column after a tag job.
+// poller and jobs.RestoreDesired call it, and FinishJob sets the same column
-// Inventory writes never change it.
+// after a tag job. Inventory writes never change it.
 func (s *Store) SetLastTagDigest(ctx context.Context, id int64, digest string) error {
 	if _, err := s.pool.Exec(ctx,
 		`UPDATE services SET last_tag_digest = $2 WHERE id = $1`, id, digest); err != nil {
@@ -316,6 +340,23 @@ func (s *Store) GetImageVersion(ctx context.Context, id int64) (model.ImageVersi
 	return v, nil
 }
 
+// GetTagVersion returns the version row of repo and tag with an empty digest, which
+// SetNewer and the webhook create. Returns model.ErrNotFound if there is none.
+func (s *Store) GetTagVersion(ctx context.Context, repo, tag string) (model.ImageVersion, error) {
+	var v model.ImageVersion
+	err := s.pool.QueryRow(ctx, `
+		SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions
+		WHERE repo = $1 AND tag = $2 AND digest = ''`, repo, tag,
+	).Scan(&v.ID, &v.Repo, &v.Tag, &v.Digest, &v.Source, &v.FirstSeenAt)
+	if errors.Is(err, pgx.ErrNoRows) {
+		return model.ImageVersion{}, model.ErrNotFound
+	}
+	if err != nil {
+		return model.ImageVersion{}, fmt.Errorf("get tag version: %w", err)
+	}
+	return v, nil
+}
+
 // SetUpdateMode sets the service's update mode (model.ModeManual or
 // model.ModeAuto). Returns model.ErrNotFound if the service does not exist
 // or is marked removed.
@@ -331,6 +372,34 @@ func (s *Store) SetUpdateMode(ctx context.Context, id int64, mode string) error
 	return nil
 }
 
+// AcceptRunning makes the running version the desired one: desired_ref =
+// desired, provided the service is live, has been updated before and still
+// reports actualRef and actualDigest (what the caller saw when it decided).
+// It starts no job and leaves last_tag_digest, the available suggestion and
+// the update mode alone. Returns model.ErrNotFound for an unknown or removed
+// service and model.ErrConflict when the service was never updated or the
+// running version changed meanwhile.
+func (s *Store) AcceptRunning(ctx context.Context, id int64, actualRef, actualDigest, desired string) error {
+	tag, err := s.pool.Exec(ctx, `
+		UPDATE services SET desired_ref = $4, awaiting_inventory = false
+		WHERE id = $1 AND removed_at IS NULL AND desired_ref <> ''
+		  AND actual_ref = $2 AND actual_digest = $3`, id, actualRef, actualDigest, desired)
+	if err != nil {
+		return fmt.Errorf("accept running: %w", err)
+	}
+	if tag.RowsAffected() == 1 {
+		return nil
+	}
+	svc, err := s.GetService(ctx, id)
+	if err != nil {
+		return err
+	}
+	if svc.RemovedAt != nil {
+		return model.ErrNotFound
+	}
+	return model.ErrConflict
+}
+
 // GetService returns one service, including a removed one (RemovedAt set).
 // Returns model.ErrNotFound if the ID does not exist.
 func (s *Store) GetService(ctx context.Context, id int64) (model.Service, error) {

internal/master/store/versions_integration_test.go

@@ -214,3 +214,79 @@ func TestSetLastTagDigest(t *testing.T) {
 	require.Len(t, list, 1)
 	assert.Equal(t, "sha256:n1", list[0].LastTagDigest)
 }
+
+func TestSetNewer_SetsAndClears(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	_, svc := jobFixture(t, s, "a")
+
+	require.NoError(t, s.SetNewer(ctx, svc, sluskRepo+":1.5.0", t0))
+
+	got, err := s.GetService(ctx, svc)
+	require.NoError(t, err)
+	assert.Equal(t, sluskRepo+":1.5.0", got.NewerRef)
+	require.NotNil(t, got.NewerAt)
+	assert.True(t, t0.Equal(*got.NewerAt))
+	v, err := s.GetTagVersion(ctx, sluskRepo, "1.5.0")
+	require.NoError(t, err)
+	assert.Equal(t, model.SourceTags, v.Source)
+	assert.Empty(t, v.Digest)
+	assert.True(t, t0.Equal(v.FirstSeenAt))
+
+	require.NoError(t, s.SetNewer(ctx, svc, "", t0.Add(time.Hour)))
+
+	got, err = s.GetService(ctx, svc)
+	require.NoError(t, err)
+	assert.Empty(t, got.NewerRef)
+	assert.Nil(t, got.NewerAt)
+	_, err = s.GetTagVersion(ctx, sluskRepo, "1.5.0")
+	assert.NoError(t, err, "clearing the suggestion keeps the version")
+}
+
+func TestSetNewer_KeepsEarlierVersionSource(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	_, svc := jobFixture(t, s, "a")
+	require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":1.5.0", Source: model.SourceWebhook, At: t0}))
+
+	require.NoError(t, s.SetNewer(ctx, svc, sluskRepo+":1.5.0", t0.Add(time.Hour)))
+
+	v, err := s.GetTagVersion(ctx, sluskRepo, "1.5.0")
+	require.NoError(t, err)
+	assert.Equal(t, model.SourceWebhook, v.Source)
+	assert.True(t, t0.Equal(v.FirstSeenAt))
+}
+
+func TestRecentVersions_TagsRowHiddenOnceDigestKnown(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	_, svc := jobFixture(t, s, "a")
+	require.NoError(t, s.SetNewer(ctx, svc, sluskRepo+":1.5.0", t0.Add(time.Minute)))
+	require.Len(t, versionsOf(t, s, sluskRepo), 2, "running 1.4.2 and the tags row")
+
+	require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":1.5.0", Digest: "sha256:n1", Source: model.SourcePoll, At: t0.Add(2 * time.Minute)}))
+
+	var refs []string
+	for _, v := range versionsOf(t, s, sluskRepo) {
+		refs = append(refs, v.Ref())
+	}
+	assert.Equal(t, []string{sluskRepo + ":1.5.0@sha256:n1", sluskRepo + ":1.4.2@" + digD1}, refs)
+}
+
+func TestGetTagVersion(t *testing.T) {
+	s := openTestStore(t)
+	ctx := context.Background()
+	_, svc := jobFixture(t, s, "a")
+	require.NoError(t, s.SetNewer(ctx, svc, sluskRepo+":1.5.0", t0))
+
+	got, err := s.GetTagVersion(ctx, sluskRepo, "1.5.0")
+	require.NoError(t, err)
+	assert.Equal(t, sluskRepo+":1.5.0", got.Ref())
+
+	_, err = s.GetTagVersion(ctx, sluskRepo, "9.9.9")
+	assert.ErrorIs(t, err, model.ErrNotFound)
+	_, err = s.GetTagVersion(ctx, "other/repo", "1.5.0")
+	assert.ErrorIs(t, err, model.ErrNotFound)
+	_, err = s.GetTagVersion(ctx, sluskRepo, "1.4.2")
+	assert.ErrorIs(t, err, model.ErrNotFound, "a row with a digest is not a tag version")
+}

internal/master/tagver/tagver.go

@@ -0,0 +1,101 @@
+// Package tagver decides which tag of a repository is a newer version of the
+// tag a service follows, using only the shape of the tag names.
+package tagver
+
+import (
+	"regexp"
+	"strconv"
+	"strings"
+)
+
+// tagPattern splits a tag into a prefix without digits, a dotted numeric
+// part and whatever follows it. "v1.2.3-alpine" gives "v", "1.2.3", "-alpine".
+var tagPattern = regexp.MustCompile(`^([^0-9]*)([0-9]+(?:\.[0-9]+)*)(.*)$`)
+
+// cosignPattern matches the tags cosign pushes next to an image to store its
+// signature, attestation or SBOM. They look like versions to a naive check.
+// The pattern is taken from Diun (crazy-max/diun, MIT, pkg/registry/tags.go).
+var cosignPattern = regexp.MustCompile(`^sha256-[0-9a-f]{64}(\.att|\.sbom|\.sig)?$`)
+
+// parsed is a tag split into its parts.
+type parsed struct {
+	prefix, suffix string
+	nums           []int
+}
+
+// parse splits tag. It returns false for a cosign tag, a tag without digits
+// and a numeric component that does not fit an int.
+func parse(tag string) (parsed, bool) {
+	if cosignPattern.MatchString(tag) {
+		return parsed{}, false
+	}
+	m := tagPattern.FindStringSubmatch(tag)
+	if m == nil {
+		return parsed{}, false
+	}
+	var nums []int
+	for _, part := range strings.Split(m[2], ".") {
+		n, err := strconv.Atoi(part)
+		if err != nil {
+			return parsed{}, false
+		}
+		nums = append(nums, n)
+	}
+	return parsed{prefix: m[1], suffix: m[3], nums: nums}, true
+}
+
+// Comparable reports whether tag can ever get a newer tag: it splits into
+// prefix, numeric part and suffix, has at least two numeric components, and
+// is not a cosign artifact tag. A tag with one component ("7", "pg16",
+// "8-bookworm") is a moving tag, not a version.
+func Comparable(tag string) bool {
+	p, ok := parse(tag)
+	return ok && len(p.nums) >= 2
+}
+
+// Newer returns the highest candidate that is newer than current, and false
+// when there is none. A candidate qualifies when it has the same prefix and
+// suffix and the same number of components as current, the same first
+// component, and a greater numeric part. Of two candidates with the same
+// numeric value the lexically smaller string wins, so the result does not
+// depend on the order of candidates. A tag such as "name2-v1.2.3", where a
+// digit comes before the version, is not comparable.
+func Newer(current string, candidates []string) (string, bool) {
+	cur, ok := parse(current)
+	if !ok || len(cur.nums) < 2 {
+		return "", false
+	}
+	best, found := "", false
+	var bestNums []int
+	for _, c := range candidates {
+		p, ok := parse(c)
+		if !ok || p.prefix != cur.prefix || p.suffix != cur.suffix ||
+			len(p.nums) != len(cur.nums) || p.nums[0] != cur.nums[0] {
+			continue
+		}
+		if compare(p.nums, cur.nums) <= 0 {
+			continue
+		}
+		if !found {
+			best, bestNums, found = c, p.nums, true
+			continue
+		}
+		if cmp := compare(p.nums, bestNums); cmp > 0 || (cmp == 0 && c < best) {
+			best, bestNums = c, p.nums
+		}
+	}
+	return best, found
+}
+
+// compare compares two numeric parts of equal length.
+func compare(a, b []int) int {
+	for i := range a {
+		switch {
+		case a[i] < b[i]:
+			return -1
+		case a[i] > b[i]:
+			return 1
+		}
+	}
+	return 0
+}

internal/master/tagver/tagver_test.go

@@ -0,0 +1,71 @@
+package tagver
+
+import (
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestComparable(t *testing.T) {
+	sig := "sha256-" + strings.Repeat("a", 64) + ".sig"
+	tests := []struct {
+		tag  string
+		want bool
+	}{
+		{"latest", false},
+		{"release", false},
+		{"stable", false},
+		{"edge", false},
+		{"alpine", false},
+		{"7", false},
+		{"7-alpine", false},
+		{"8-bookworm", false},
+		{"pg16", false},
+		{"14-vectorchord0.4.3-pgvectors0.2.0", false},
+		{"name2-v1.2.3", false},
+		{sig, false},
+		{"99999999999999999999.1", false},
+		{"1.2", true},
+		{"1.2.3", true},
+		{"v1.2.3", true},
+		{"11.0-latest", true},
+	}
+	for _, tt := range tests {
+		t.Run(tt.tag, func(t *testing.T) {
+			assert.Equal(t, tt.want, Comparable(tt.tag))
+		})
+	}
+}
+
+func TestNewer(t *testing.T) {
+	tests := []struct {
+		name       string
+		current    string
+		candidates []string
+		want       string
+		ok         bool
+	}{
+		{"highest within major", "1.2.3", []string{"1.2.4", "1.3.0", "2.0.0", "1.2.4-rc1", "1.3", "v1.2.4"}, "1.3.0", true},
+		{"v prefix kept", "v1.2.3", []string{"1.2.4", "v1.2.4"}, "v1.2.4", true},
+		{"name prefix kept", "name-v1.2.3", []string{"v1.2.4", "name-v1.2.4"}, "name-v1.2.4", true},
+		{"suffix kept", "11.0-latest", []string{"11.1", "11.1-latest"}, "11.1-latest", true},
+		{"two components", "7.0", []string{"8.0", "7.0.1", "7.2", "7.1"}, "7.2", true},
+		{"equal is not newer", "1.2.3", []string{"1.2.3"}, "", false},
+		{"older is not newer", "1.2.3", []string{"1.2.2", "1.1.9"}, "", false},
+		{"numeric not lexical", "1.9.0", []string{"1.10.0"}, "1.10.0", true},
+		{"empty candidates", "1.2.3", nil, "", false},
+		{"moving tag", "latest", []string{"1.2.3"}, "", false},
+		{"single component", "7", []string{"8"}, "", false},
+		{"digit before version", "name2-v1.2.3", []string{"name2-v1.2.4"}, "", false},
+		{"cosign tag ignored", "1.2.3", []string{"sha256-" + strings.Repeat("0", 64) + ".sig"}, "", false},
+		{"equal value picks smaller string", "1.2.3", []string{"1.02.4", "1.2.4"}, "1.02.4", true},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, ok := Newer(tt.current, tt.candidates)
+			assert.Equal(t, tt.ok, ok)
+			assert.Equal(t, tt.want, got)
+		})
+	}
+}

internal/master/web/drift_test.go

@@ -0,0 +1,297 @@
+package web
+
+import (
+	"context"
+	"io"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strconv"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/shcizo/package-updater/internal/master/events"
+	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/protocol"
+)
+
+const (
+	driftDesired = "sha256:1111111111111111111111111111111111111111111111111111111111111111"
+	driftActual  = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
+)
+
+type acceptCall struct {
+	id                                  int64
+	actualRef, actualDigest, desiredRef string
+}
+
+func (f *fakeStore) AcceptRunning(_ context.Context, id int64, actualRef, actualDigest, desired string) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.acceptCalls = append(f.acceptCalls, acceptCall{id, actualRef, actualDigest, desired})
+	return f.acceptErr
+}
+
+func (f *fakeStore) MoveCandidates(context.Context) ([]model.Move, error) { return f.moves, nil }
+
+func (f *fakeStore) MergeSources(context.Context, int64) ([]model.Service, error) {
+	return f.sources, nil
+}
+
+func (f *fakeStore) MergeService(_ context.Context, from, to int64) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.merged = append(f.merged, [2]int64{from, to})
+	return f.mergeErr
+}
+
+func (f *fakeJobs) Restorable(model.Service) error { return f.restorable }
+
+func (f *fakeJobs) RestoreDesired(_ context.Context, serviceID int64, by string) (model.Job, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.requestedBy = by
+	f.restored = append(f.restored, serviceID)
+	return f.job, f.err
+}
+
+// driftService makes the first fixture service a drifted one: updated to
+// 1.4.2 at driftDesired, now running 1.4.2 at driftActual.
+func driftService(store *fakeStore) {
+	store.services[0].DesiredRef = "gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftDesired
+	store.services[0].ActualDigest = driftActual
+}
+
+func postTo(t *testing.T, srv *httptest.Server, path string, form url.Values) (*http.Response, string) {
+	t.Helper()
+	form.Set("csrf_token", "csrf")
+	resp, err := noRedirectClient().PostForm(srv.URL+path, form)
+	require.NoError(t, err)
+	defer resp.Body.Close()
+	body, _ := io.ReadAll(resp.Body)
+	return resp, string(body)
+}
+
+func TestRestoreDesired(t *testing.T) {
+	tests := []struct {
+		name         string
+		job          model.Job
+		err          error
+		path         string
+		wantCode     int
+		wantLocation string
+		wantBody     string
+	}{
+		{name: "started", job: model.Job{ID: 42}, path: "/services/1/restore",
+			wantCode: http.StatusSeeOther, wantLocation: "/jobs/42"},
+		{name: "already running", job: model.Job{ID: 41}, err: model.ErrJobActive, path: "/services/1/restore",
+			wantCode: http.StatusSeeOther, wantLocation: "/jobs/41"},
+		{name: "refused", err: &model.NotUpdatableError{Reason: model.ReasonNotDrifted}, path: "/services/1/restore",
+			wantCode: http.StatusConflict, wantBody: "Cannot restore: runs its desired version."},
+		{name: "unknown service", err: model.ErrNotFound, path: "/services/1/restore", wantCode: http.StatusNotFound},
+		{name: "invalid id", path: "/services/x/restore", wantCode: http.StatusBadRequest},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			srv, _, _, _, jobs := newTestEnv(t)
+			jobs.job, jobs.err = tt.job, tt.err
+
+			resp, body := postTo(t, srv, tt.path, url.Values{})
+
+			assert.Equal(t, tt.wantCode, resp.StatusCode)
+			assert.Equal(t, tt.wantLocation, resp.Header.Get("Location"))
+			assert.Contains(t, body, tt.wantBody)
+			if tt.wantCode != http.StatusBadRequest {
+				assert.Equal(t, "alice", jobs.triggeredBy())
+			}
+		})
+	}
+}
+
+func TestAcceptRunning_WritesRunningRefAndPublishes(t *testing.T) {
+	srv, store, _, broker, _ := newTestEnv(t)
+	driftService(store)
+	ch, cancel := broker.Subscribe()
+	defer cancel()
+
+	resp, _ := postTo(t, srv, "/services/1/accept", url.Values{"actual": {"gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftActual}})
+
+	assert.Equal(t, http.StatusSeeOther, resp.StatusCode)
+	assert.Equal(t, "/services/1", resp.Header.Get("Location"))
+	assert.Equal(t, []acceptCall{{
+		id: 1, actualRef: "gitea.shcizo.se/shcizo/slusk:1.4.2", actualDigest: driftActual,
+		desiredRef: "gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftActual,
+	}}, store.acceptCalls, "the value written comes from the row, not the form")
+	select {
+	case topic := <-ch:
+		assert.Equal(t, events.Services, topic)
+	case <-time.After(time.Second):
+		t.Fatal("no services signal")
+	}
+}
+
+func TestAcceptRunning_ReturnToOverview(t *testing.T) {
+	srv, store, _, _, _ := newTestEnv(t)
+	driftService(store)
+
+	resp, _ := postTo(t, srv, "/services/1/accept",
+		url.Values{"actual": {"gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftActual}, "return": {"overview"}})
+
+	assert.Equal(t, http.StatusSeeOther, resp.StatusCode)
+	assert.Equal(t, "/", resp.Header.Get("Location"))
+}
+
+func TestAcceptRunning_Refusals(t *testing.T) {
+	tests := []struct {
+		name       string
+		path       string
+		form       url.Values
+		notDrifted bool
+		removed    bool
+		acceptErr  error
+		wantCode   int
+		wantBody   string
+	}{
+		{name: "missing actual", path: "/services/1/accept", form: url.Values{}, wantCode: http.StatusBadRequest},
+		{name: "stale actual", path: "/services/1/accept", form: url.Values{"actual": {"gitea.shcizo.se/shcizo/slusk:1.3.0"}},
+			wantCode: http.StatusConflict, wantBody: "The running version changed; check it again."},
+		{name: "same ref, other digest", path: "/services/1/accept",
+			form:     url.Values{"actual": {"gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftDesired}},
+			wantCode: http.StatusConflict, wantBody: "The running version changed; check it again."},
+		{name: "not drifted", path: "/services/1/accept", form: url.Values{"actual": {"gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftActual}},
+			notDrifted: true, wantCode: http.StatusConflict, wantBody: "Cannot accept: the service runs its desired version."},
+		{name: "changed under our feet", path: "/services/1/accept", form: url.Values{"actual": {"gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftActual}},
+			acceptErr: model.ErrConflict, wantCode: http.StatusConflict, wantBody: "The running version changed; check it again."},
+		{name: "unknown service", path: "/services/9/accept", form: url.Values{"actual": {"x"}}, wantCode: http.StatusNotFound},
+		{name: "removed", path: "/services/1/accept", form: url.Values{"actual": {"gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftActual}},
+			removed: true, wantCode: http.StatusNotFound},
+		{name: "removed meanwhile", path: "/services/1/accept", form: url.Values{"actual": {"gitea.shcizo.se/shcizo/slusk:1.4.2@" + driftActual}},
+			acceptErr: model.ErrNotFound, wantCode: http.StatusNotFound},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			srv, store, _, _, _ := newTestEnv(t)
+			if !tt.notDrifted {
+				driftService(store)
+			}
+			if tt.removed {
+				removedAt := time.Now()
+				store.services[0].RemovedAt = &removedAt
+			}
+			store.acceptErr = tt.acceptErr
+
+			resp, body := postTo(t, srv, tt.path, tt.form)
+
+			assert.Equal(t, tt.wantCode, resp.StatusCode)
+			assert.Contains(t, body, tt.wantBody)
+			if tt.acceptErr == nil {
+				assert.Empty(t, store.acceptCalls)
+			}
+		})
+	}
+}
+
+func TestMergeService_RedirectsToTheSurvivorAndPublishes(t *testing.T) {
+	srv, store, _, broker, _ := newTestEnv(t)
+	ch, cancel := broker.Subscribe()
+	defer cancel()
+
+	resp, _ := postTo(t, srv, "/services/1/merge", url.Values{"from": {"17"}})
+
+	assert.Equal(t, http.StatusSeeOther, resp.StatusCode)
+	assert.Equal(t, "/services/17", resp.Header.Get("Location"))
+	assert.Equal(t, [][2]int64{{17, 1}}, store.merged)
+	var got []events.Topic
+	for len(got) < 2 {
+		select {
+		case topic := <-ch:
+			got = append(got, topic)
+		case <-time.After(time.Second):
+			t.Fatalf("signals so far: %v", got)
+		}
+	}
+	assert.ElementsMatch(t, []events.Topic{events.Services, events.Jobs}, got)
+}
+
+func TestMergeService_Refusals(t *testing.T) {
+	tests := []struct {
+		name     string
+		id, from string
+		err      error
+		wantCode int
+		wantBody string
+	}{
+		{name: "invalid from", id: "1", from: "x", wantCode: http.StatusBadRequest},
+		{name: "empty from", id: "1", from: "", wantCode: http.StatusBadRequest},
+		{name: "refused", id: "1", from: "17", err: &model.MergeRefusedError{Reason: model.ReasonMergeLiveOld},
+			wantCode: http.StatusConflict, wantBody: "Cannot merge: the old service still exists."},
+		{name: "job active", id: "1", from: "17", err: model.ErrJobActive,
+			wantCode: http.StatusConflict, wantBody: "Cannot merge: an update is already in progress for this service."},
+		{name: "agent revoked", id: "1", from: "17", err: model.ErrRevoked, wantCode: http.StatusConflict, wantBody: "Cannot merge:"},
+		{name: "not found", id: "1", from: "17", err: model.ErrNotFound, wantCode: http.StatusNotFound},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			srv, store, _, _, _ := newTestEnv(t)
+			store.mergeErr = tt.err
+
+			resp, body := postTo(t, srv, "/services/"+tt.id+"/merge", url.Values{"from": {tt.from}})
+
+			assert.Equal(t, tt.wantCode, resp.StatusCode)
+			assert.Contains(t, body, tt.wantBody)
+			assert.Empty(t, resp.Header.Get("Location"))
+		})
+	}
+}
+
+// serviceDetail's data is what the service template reads.
+func TestServiceDetail_DriftAndMoveData(t *testing.T) {
+	removed := time.Now()
+	live := model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", ActualRef: "slusk:1"}
+	gone := model.Service{ID: 2, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", ActualRef: "slusk:1", RemovedAt: &removed}
+	swarm := model.Service{ID: 3, Kind: protocol.KindSwarm, SwarmService: "gitea", ActualRef: "gitea:1"}
+	container := model.Service{ID: 4, Kind: protocol.KindContainer, ContainerName: "adhoc", ActualRef: "redis:7"}
+	self := model.Service{ID: 5, Kind: protocol.KindCompose, ComposeProject: "pu", ComposeService: "agent", ActualRef: "pu:1", Self: true}
+	move := model.Move{FromID: 2, ToID: 1, FromAgent: "old", ToAgent: "new"}
+	sources := []model.Service{gone}
+
+	tests := []struct {
+		name        string
+		id          int64
+		restorable  error
+		wantBlocked string
+		wantFrom    *model.Move
+		wantTo      *model.Move
+		wantSources []model.Service
+	}{
+		{name: "live row gets the suggestion and merge choices", id: 1, wantFrom: &move, wantSources: sources},
+		{name: "removed row gets where it went", id: 2, wantTo: &move},
+		{name: "swarm service can merge, no suggestion", id: 3, wantSources: sources},
+		{name: "container cannot merge", id: 4},
+		{name: "self cannot merge", id: 5},
+		{name: "restore refusal is passed on", id: 1, restorable: &model.NotUpdatableError{Reason: model.ReasonContainer},
+			wantBlocked: model.ReasonContainer, wantFrom: &move, wantSources: sources},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			store := &fakeStore{services: []model.Service{live, gone, swarm, container, self},
+				moves: []model.Move{move}, sources: sources}
+			jobs := &fakeJobs{restorable: tt.restorable}
+			s, err := New(store, &fakeHub{}, jobs, events.NewBroker(), &fakePoller{}, Config{}, slog.New(slog.NewTextHandler(io.Discard, nil)))
+			require.NoError(t, err)
+			req := httptest.NewRequest(http.MethodGet, "/services/"+strconv.FormatInt(tt.id, 10), nil)
+
+			data, err := s.serviceDetail(req, tt.id)
+
+			require.NoError(t, err)
+			assert.Equal(t, tt.wantBlocked, data.RestoreBlocked)
+			assert.Equal(t, tt.wantFrom, data.MoveFrom)
+			assert.Equal(t, tt.wantTo, data.MoveTo)
+			assert.Equal(t, tt.wantSources, data.MergeSources)
+		})
+	}
+}

internal/master/web/icons/LICENSE-simple-icons.md

@@ -0,0 +1,30 @@
+# CC0 1.0 Universal
+
+## Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer exclusive Copyright and Related Rights (defined below) upon the creator and subsequent owner(s) (each and all, an “owner”) of an original work of authorship and/or a database (each, a “Work”).
+
+Certain owners wish to permanently relinquish those rights to a Work for the purpose of contributing to a commons of creative, cultural and scientific works (“Commons”) that the public can reliably and without fear of later claims of infringement build upon, modify, incorporate in other works, reuse and redistribute as freely as possible in any form whatsoever and for any purposes, including without limitation commercial purposes. These owners may contribute to the Commons to promote the ideal of a free culture and the further production of creative, cultural and scientific works, or to gain reputation or greater distribution for their Work in part through the use and efforts of others.
+
+For these and/or other purposes and motivations, and without any expectation of additional consideration or compensation, the person associating CC0 with a Work (the “Affirmer”), to the extent that he or she is an owner of Copyright and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and publicly distribute the Work under its terms, with knowledge of his or her Copyright and Related Rights in the Work and the meaning and intended legal effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be protected by copyright and related or neighboring rights (“Copyright and Related Rights”). Copyright and Related Rights include, but are not limited to, the following:
+    1. the right to reproduce, adapt, distribute, perform, display, communicate, and translate a Work;
+    2. moral rights retained by the original author(s) and/or performer(s);
+    3. publicity and privacy rights pertaining to a person’s image or likeness depicted in a Work;
+    4. rights protecting against unfair competition in regards to a Work, subject to the limitations in paragraph 4(i), below;
+    5. rights protecting the extraction, dissemination, use and reuse of data in a Work;
+    6. database rights (such as those arising under Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, and under any national implementation thereof, including any amended or successor version of such directive); and
+    7. other similar, equivalent or corresponding rights throughout the world based on applicable law or treaty, and any national implementations thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention of, applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and unconditionally waives, abandons, and surrenders all of Affirmer’s Copyright and Related Rights and associated claims and causes of action, whether now known or unknown (including existing as well as future claims and causes of action), in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the “Waiver”). Affirmer makes the Waiver for the benefit of each member of the public at large and to the detriment of Affirmer’s heirs and successors, fully intending that such Waiver shall not be subject to revocation, rescission, cancellation, termination, or any other legal or equitable action to disrupt the quiet enjoyment of the Work by the public as contemplated by Affirmer’s express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer’s express Statement of Purpose. In addition, to the extent the Waiver is so judged Affirmer hereby grants to each affected person a royalty-free, non transferable, non sublicensable, non exclusive, irrevocable and unconditional license to exercise Affirmer’s Copyright and Related Rights in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the “License”). The License shall be deemed effective as of the date CC0 was applied by Affirmer to the Work. Should any part of the License for any reason be judged legally invalid or ineffective under applicable law, such partial invalidity or ineffectiveness shall not invalidate the remainder of the License, and in such case Affirmer hereby affirms that he or she will not (i) exercise any of his or her remaining Copyright and Related Rights in the Work or (ii) assert any associated claims and causes of action with respect to the Work, in either case contrary to Affirmer’s express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+    1. No trademark or patent rights held by Affirmer are waived, abandoned, surrendered, licensed or otherwise affected by this document.
+    2. Affirmer offers the Work as-is and makes no representations or warranties of any kind concerning the Work, express, implied, statutory or otherwise, including without limitation warranties of title, merchantability, fitness for a particular purpose, non infringement, or the absence of latent or other defects, accuracy, or the present or absence of errors, whether or not discoverable, all to the greatest extent permissible under applicable law.
+    3. Affirmer disclaims responsibility for clearing rights of other persons that may apply to the Work or any use thereof, including without limitation any person’s Copyright and Related Rights in the Work. Further, Affirmer disclaims responsibility for obtaining any necessary consents, permissions or other rights required for any use of the Work.
+    4. Affirmer understands and acknowledges that Creative Commons is not a party to this document and has no duty or obligation with respect to this CC0 or use of the Work.
+
+For more information, please see <https://creativecommons.org/publicdomain/zero/1.0>.

internal/master/web/icons/gen/main.go

@@ -0,0 +1,128 @@
+// Command gen builds simple-icons.json.gz, the icon data the icons package
+// embeds. It downloads the pinned simple-icons npm tarball, checks its
+// sha256 and writes a gzipped JSON object {slug: svg path data}. The output
+// is deterministic. It writes into the current directory; run it with
+//
+//	go generate ./internal/master/web/icons
+package main
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path"
+	"regexp"
+	"strings"
+)
+
+const (
+	tarballURL    = "https://registry.npmjs.org/simple-icons/-/simple-icons-16.35.0.tgz"
+	tarballSHA256 = "20ebead4f6bc93464812a5e260d19fc4d84772e5eeff69a6fbea263ca2441c5f"
+)
+
+var pathData = regexp.MustCompile(`<path d="([^"]+)"`)
+
+func main() {
+	if err := run(); err != nil {
+		fmt.Fprintln(os.Stderr, "gen:", err)
+		os.Exit(1)
+	}
+}
+
+func run() error {
+	tgz, err := download(tarballURL)
+	if err != nil {
+		return err
+	}
+	if sum := sha256.Sum256(tgz); hex.EncodeToString(sum[:]) != tarballSHA256 {
+		return fmt.Errorf("sha256 mismatch for %s: got %x, want %s", tarballURL, sum, tarballSHA256)
+	}
+	icons, license, err := extract(tgz)
+	if err != nil {
+		return err
+	}
+	var js bytes.Buffer
+	enc := json.NewEncoder(&js)
+	enc.SetEscapeHTML(false)
+	if err := enc.Encode(icons); err != nil { // map keys are written sorted
+		return err
+	}
+	var gz bytes.Buffer
+	zw, err := gzip.NewWriterLevel(&gz, gzip.BestCompression)
+	if err != nil {
+		return err
+	}
+	if _, err := zw.Write(js.Bytes()); err != nil { // the header has no name and a zero mtime
+		return err
+	}
+	if err := zw.Close(); err != nil {
+		return err
+	}
+	if err := os.WriteFile("simple-icons.json.gz", gz.Bytes(), 0o644); err != nil {
+		return err
+	}
+	if err := os.WriteFile("LICENSE-simple-icons.md", license, 0o644); err != nil {
+		return err
+	}
+	fmt.Printf("wrote %d icons\n", len(icons))
+	return nil
+}
+
+func download(url string) ([]byte, error) {
+	resp, err := http.Get(url)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("GET %s: %s", url, resp.Status)
+	}
+	return io.ReadAll(resp.Body)
+}
+
+// extract returns the path data of every package/icons/*.svg by slug, and
+// package/LICENSE.md.
+func extract(tgz []byte) (icons map[string]string, license []byte, err error) {
+	zr, err := gzip.NewReader(bytes.NewReader(tgz))
+	if err != nil {
+		return nil, nil, err
+	}
+	tr := tar.NewReader(zr)
+	icons = map[string]string{}
+	for {
+		h, err := tr.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			return nil, nil, err
+		}
+		switch {
+		case h.Name == "package/LICENSE.md":
+			if license, err = io.ReadAll(tr); err != nil {
+				return nil, nil, err
+			}
+		case path.Dir(h.Name) == "package/icons" && strings.HasSuffix(h.Name, ".svg"):
+			svg, err := io.ReadAll(tr)
+			if err != nil {
+				return nil, nil, err
+			}
+			m := pathData.FindAllSubmatch(svg, -1)
+			if len(m) != 1 {
+				return nil, nil, fmt.Errorf("%s: want exactly one <path>, found %d", h.Name, len(m))
+			}
+			icons[strings.TrimSuffix(path.Base(h.Name), ".svg")] = string(m[0][1])
+		}
+	}
+	if license == nil || len(icons) == 0 {
+		return nil, nil, fmt.Errorf("tarball has no LICENSE.md or no icons")
+	}
+	return icons, license, nil
+}

internal/master/web/icons/icons.go

@@ -0,0 +1,109 @@
+// Package icons maps container image references to brand icons from the
+// Simple Icons set (CC0, see LICENSE-simple-icons.md). Only the master's UI
+// imports it, so the agent binary does not carry the data.
+package icons
+
+import (
+	"bytes"
+	"compress/gzip"
+	_ "embed"
+	"encoding/json"
+	"strings"
+	"sync"
+
+	"github.com/shcizo/package-updater/internal/imageref"
+)
+
+//go:generate go run ./gen
+
+//go:embed simple-icons.json.gz
+var data []byte
+
+var (
+	once  sync.Once
+	paths map[string]string
+)
+
+// aliases maps a normalised image name to the Simple Icons slug that differs
+// from it.
+var aliases = map[string]string{
+	"postgres":    "postgresql",
+	"goauthentik": "authentik",
+	"traefik":     "traefikproxy",
+	"node":        "nodedotjs",
+	"mongo":       "mongodb",
+}
+
+// skipNamespaces are registry namespaces that name a publisher, not a product.
+var skipNamespaces = map[string]bool{"library": true, "linuxserver": true, "bitnami": true, "hotio": true}
+
+// suffixes are stripped from the last path segment to find the product name
+// ("authentik-server" is authentik).
+var suffixes = []string{"-server", "-web", "-api", "-worker", "-agent"}
+
+func load() {
+	once.Do(func() {
+		paths = map[string]string{}
+		zr, err := gzip.NewReader(bytes.NewReader(data))
+		if err != nil {
+			return
+		}
+		_ = json.NewDecoder(zr).Decode(&paths)
+	})
+}
+
+// Path returns the SVG path data (24x24 viewBox) of the icon with this Simple
+// Icons slug.
+func Path(slug string) (string, bool) {
+	load()
+	p, ok := paths[slug]
+	return p, ok
+}
+
+// Count returns how many icons are embedded.
+func Count() int {
+	load()
+	return len(paths)
+}
+
+// ForImage returns the slug of the icon that best matches the image
+// reference, or "" when there is none. It tries the last path segment, that
+// segment without a role suffix, and the namespace, in that order.
+func ForImage(ref string) string {
+	segs := strings.Split(imageref.Parse(ref).Repo, "/")
+	if len(segs) > 1 && imageref.IsRegistryHost(segs[0]) {
+		segs = segs[1:]
+	}
+	last := segs[len(segs)-1]
+	candidates := []string{last}
+	for _, s := range suffixes {
+		if base, ok := strings.CutSuffix(last, s); ok && base != "" {
+			candidates = append(candidates, base)
+			break
+		}
+	}
+	if len(segs) > 1 && !skipNamespaces[segs[len(segs)-2]] {
+		candidates = append(candidates, segs[len(segs)-2])
+	}
+	for _, c := range candidates {
+		slug := normalise(c)
+		if a, ok := aliases[slug]; ok {
+			slug = a
+		}
+		if _, ok := Path(slug); ok {
+			return slug
+		}
+	}
+	return ""
+}
+
+// normalise lowercases s and keeps only a-z and 0-9, the form of Simple Icons slugs.
+func normalise(s string) string {
+	var b strings.Builder
+	for _, r := range strings.ToLower(s) {
+		if r >= 'a' && r <= 'z' || r >= '0' && r <= '9' {
+			b.WriteRune(r)
+		}
+	}
+	return b.String()
+}

internal/master/web/icons/icons_test.go

@@ -0,0 +1,46 @@
+package icons
+
+import (
+	"regexp"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestForImage(t *testing.T) {
+	cases := []struct {
+		ref, want string
+	}{
+		{"gitea/gitea:1.27.3", "gitea"},
+		{"ghcr.io/goauthentik/server:2026.8.2", "authentik"},
+		{"postgres:17.10", "postgresql"},
+		{"grafana/loki:3.4.2", "grafana"},
+		{"docker.n8n.io/n8nio/n8n:1.112.4", "n8n"},
+		{"lscr.io/linuxserver/prowlarr:latest", ""},
+		{"localhost:5000/foo:v1", ""},
+		{"gitea.shcizo.se/shcizo/slusk:v0.8.1", ""},
+	}
+	for _, c := range cases {
+		t.Run(c.ref, func(t *testing.T) {
+			assert.Equal(t, c.want, ForImage(c.ref))
+		})
+	}
+}
+
+// safePath is a strict allowlist for what an embedded path may contain, since
+// the paths are written into the page's HTML.
+var safePath = regexp.MustCompile(`^[MmLlHhVvCcSsQqTtAaZz0-9.,\- e]+$`)
+
+func TestEmbeddedSet(t *testing.T) {
+	assert.Greater(t, Count(), 3000)
+	for slug, p := range paths {
+		if !safePath.MatchString(p) {
+			t.Errorf("icon %q has unexpected characters in its path", slug)
+		}
+	}
+	p, ok := Path("gitea")
+	assert.True(t, ok)
+	assert.NotEmpty(t, p)
+	_, ok = Path("no-such-icon")
+	assert.False(t, ok)
+}

internal/master/web/ledger_test.go

@@ -0,0 +1,127 @@
+package web
+
+import (
+	"bufio"
+	"context"
+	"io"
+	"net/http"
+	"net/url"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestEvents_SendsPingWhenIdle(t *testing.T) {
+	old := pingInterval
+	pingInterval = 20 * time.Millisecond
+	t.Cleanup(func() { pingInterval = old })
+	srv, _, _, _ := newTestServer(t)
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, srv.URL+"/events", nil)
+	require.NoError(t, err)
+	resp, err := http.DefaultClient.Do(req)
+	require.NoError(t, err)
+	defer resp.Body.Close()
+
+	reader := bufio.NewReader(resp.Body)
+	for {
+		line, err := reader.ReadString('\n')
+		require.NoError(t, err)
+		if strings.HasPrefix(line, "event:") {
+			assert.Equal(t, "event: ping\n", line, "the Live box counts a named event; a comment would not reach the browser's listeners")
+			return
+		}
+	}
+}
+
+func TestCreateAgent_ValidationShowsInsideTheAddPanel(t *testing.T) {
+	srv, _, _, _ := newTestServer(t)
+	resp, err := http.PostForm(srv.URL+"/agents", url.Values{"name": {"node-2"}})
+	require.NoError(t, err)
+	defer resp.Body.Close()
+	body, _ := io.ReadAll(resp.Body)
+	assert.Equal(t, http.StatusConflict, resp.StatusCode)
+	assert.Contains(t, string(body), `class="drawer"`)
+	assert.Contains(t, string(body), `value="node-2"`, "the typed name is kept")
+	assert.NotContains(t, string(body), "pua_", "no token is shown")
+}
+
+func TestNavFragment(t *testing.T) {
+	srv, _, _, _ := newTestServer(t)
+	cases := []struct {
+		query, want, notWant string
+	}{
+		{"?section=agents", `<a href="/agents" class="cur" aria-current="page">Agents <span class="n">1/3</span>`, `<a href="/" class="cur"`},
+		{"?section=services", `<a href="/" class="cur" aria-current="page">Services <span class="n">4</span>`, `<a href="/agents" class="cur"`},
+		{"?section=bogus", `<a href="/jobs">Jobs</a>`, `class="cur"`},
+	}
+	for _, c := range cases {
+		t.Run(c.query, func(t *testing.T) {
+			code, body := get(t, srv, "/fragments/nav"+c.query)
+			assert.Equal(t, http.StatusOK, code)
+			assert.NotContains(t, body, "<html")
+			assert.Contains(t, body, c.want)
+			assert.NotContains(t, body, c.notWant)
+		})
+	}
+	_, body := get(t, srv, "/agents")
+	assert.Contains(t, body, `hx-get="/fragments/nav?section=agents"`, "every page refreshes the sidebar on services and agents events")
+	assert.Contains(t, body, `hx-trigger="sse:services, sse:agents"`)
+}
+
+func TestAgentConnectFragment(t *testing.T) {
+	cases := []struct {
+		name     string
+		path     string
+		online   bool
+		wantCode int
+		want     []string
+		notWant  []string
+	}{
+		{"never connected waits", "/fragments/agents/5/connect", false, http.StatusOK, []string{"Waiting for fresh", `action="/agents/5/revoke"`}, []string{"Connected", "add-steps"}},
+		{"online is connected", "/fragments/agents/5/connect", true, http.StatusOK, []string{"Connected · ", `id="add-steps" hx-swap-oob="innerHTML"`}, []string{"Waiting", "revoke"}},
+		{"seen before is connected", "/fragments/agents/3/connect", false, http.StatusOK, []string{"Connected · node-3.lan · v1.3.2"}, []string{"Waiting"}},
+		{"unknown", "/fragments/agents/99/connect", false, http.StatusNotFound, nil, nil},
+		{"revoked", "/fragments/agents/4/connect", false, http.StatusNotFound, nil, nil},
+		{"bad id", "/fragments/agents/abc/connect", false, http.StatusBadRequest, nil, nil},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			srv, _, hub, _, _ := newTestEnv(t)
+			hub.mu.Lock()
+			hub.online[5] = c.online
+			hub.mu.Unlock()
+			code, body := get(t, srv, c.path)
+			assert.Equal(t, c.wantCode, code)
+			for _, w := range c.want {
+				assert.Contains(t, body, w)
+			}
+			for _, w := range c.notWant {
+				assert.NotContains(t, body, w)
+			}
+			assert.NotContains(t, body, "pua_", "the token is never sent again")
+		})
+	}
+}
+
+func TestStatic_FontsAndAppJS(t *testing.T) {
+	srv, _, _, _ := newTestServer(t)
+	for _, path := range []string{
+		"/static/app.js", "/static/app.css",
+		"/static/schibsted-grotesk-latin-wght-normal.woff2", "/static/jetbrains-mono-latin-wght-normal.woff2",
+		"/static/OFL-schibsted-grotesk.txt", "/static/OFL-jetbrains-mono.txt",
+	} {
+		code, _ := get(t, srv, path)
+		assert.Equal(t, http.StatusOK, code, path)
+	}
+	_, css := get(t, srv, "/static/app.css")
+	assert.NotContains(t, css, "fonts.googleapis.com", "fonts are served by the master")
+	assert.Contains(t, css, "format('woff2')")
+	_, page := get(t, srv, "/")
+	assert.NotContains(t, page, "fonts.googleapis.com")
+	assert.Contains(t, page, `src="/static/app.js"`)
+}

internal/master/web/poll_test.go

@@ -0,0 +1,86 @@
+package web
+
+import (
+	"context"
+	"io"
+	"net/http"
+	"net/url"
+	"sync"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/master/poller"
+)
+
+// fakePoller records the services it was asked to check and answers err.
+type fakePoller struct {
+	mu    sync.Mutex
+	err   error
+	calls []int64
+}
+
+func (f *fakePoller) PollService(_ context.Context, id int64) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.calls = append(f.calls, id)
+	return f.err
+}
+
+func (f *fakePoller) called() []int64 {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	return append([]int64(nil), f.calls...)
+}
+
+func TestPollService(t *testing.T) {
+	tests := []struct {
+		name         string
+		id           string
+		err          error
+		wantStatus   int
+		wantLocation string
+		wantBody     string
+		wantCalls    []int64
+	}{
+		{name: "redirects to the service page", id: "1", wantStatus: http.StatusSeeOther, wantLocation: "/services/1", wantCalls: []int64{1}},
+		{name: "unknown service is 404", id: "1", err: model.ErrNotFound, wantStatus: http.StatusNotFound, wantCalls: []int64{1}},
+		{name: "invalid id is 400", id: "abc", wantStatus: http.StatusBadRequest},
+		{name: "not polled is 409 with the reason", id: "1", err: &poller.NotPolledError{Reason: model.ReasonNoTag},
+			wantStatus: http.StatusConflict, wantBody: "Cannot check: follows no tag.", wantCalls: []int64{1}},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			fp := &fakePoller{err: tt.err}
+			srv, _, _, _, _ := newTestEnvFull(t, func(*Config) {}, fp)
+
+			resp, err := noRedirectClient().PostForm(srv.URL+"/services/"+tt.id+"/poll", url.Values{"csrf_token": {"csrf"}})
+			require.NoError(t, err)
+			defer resp.Body.Close()
+			body, _ := io.ReadAll(resp.Body)
+
+			assert.Equal(t, tt.wantStatus, resp.StatusCode)
+			assert.Equal(t, tt.wantLocation, resp.Header.Get("Location"))
+			assert.Contains(t, string(body), tt.wantBody)
+			assert.Equal(t, tt.wantCalls, fp.called())
+		})
+	}
+}
+
+func TestServicePage_CheckNowForm(t *testing.T) {
+	srv, store, _, _, _ := newTestEnv(t)
+	store.services[0].ActualDigest = "sha256:" + "a"
+	_, body := get(t, srv, "/services/1")
+	assert.Contains(t, body, `action="/services/1/poll"`)
+	assert.Contains(t, body, `<button type="submit">Check now</button>`)
+	assert.NotContains(t, body, "Check unavailable")
+}
+
+func TestServicePage_CheckNowDisabledWhenNotPolled(t *testing.T) {
+	srv, _, _, _, _ := newTestEnv(t)
+	_, body := get(t, srv, "/services/2") // follows no tag
+	assert.Contains(t, body, `<button type="submit" disabled>Check now</button>`)
+	assert.Contains(t, body, "Check unavailable: follows no tag.")
+}

internal/master/web/server.go

@@ -10,6 +10,7 @@ import (
 	"log/slog"
 	"net/http"
 	"regexp"
+	"slices"
 	"strconv"
 	"strings"
 	"time"
@@ -17,7 +18,10 @@ import (
 	"github.com/shcizo/package-updater/internal/agenttoken"
 	"github.com/shcizo/package-updater/internal/master/events"
 	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/master/poller"
+	"github.com/shcizo/package-updater/internal/master/web/icons"
 	"github.com/shcizo/package-updater/internal/master/webauth"
+	"github.com/shcizo/package-updater/internal/protocol"
 )
 
 //go:embed templates/*.html static/*
@@ -35,11 +39,24 @@ var agentNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`)
 type Store interface {
 	ListServices(ctx context.Context) ([]model.Service, error)
 	GetService(ctx context.Context, id int64) (model.Service, error)
+	// GetTagVersion returns the version of repo with this tag and no digest;
+	// model.ErrNotFound if there is none.
+	GetTagVersion(ctx context.Context, repo, tag string) (model.ImageVersion, error)
 	// RecentVersions returns up to limit versions of repo the master has
 	// seen, most recently first seen first.
 	RecentVersions(ctx context.Context, repo string, limit int) ([]model.ImageVersion, error)
 	// SetUpdateMode returns model.ErrNotFound for an unknown or removed service.
 	SetUpdateMode(ctx context.Context, id int64, mode string) error
+	// AcceptRunning sets the desired ref of a drifted service; model.ErrNotFound
+	// for an unknown or removed service, model.ErrConflict when the running
+	// version is no longer actualRef@actualDigest.
+	AcceptRunning(ctx context.Context, id int64, actualRef, actualDigest, desired string) error
+	// MoveCandidates returns the suggested moves of removed services.
+	MoveCandidates(ctx context.Context) ([]model.Move, error)
+	// MergeSources returns the removed services that could be merged into toID.
+	MergeSources(ctx context.Context, toID int64) ([]model.Service, error)
+	// MergeService moves the live service toID onto the removed fromID.
+	MergeService(ctx context.Context, fromID, toID int64) error
 	GetJob(ctx context.Context, id int64) (model.Job, error)
 	ListJobs(ctx context.Context, f model.JobFilter) ([]model.Job, error)
 	ListAgents(ctx context.Context) ([]model.Agent, error)
@@ -63,6 +80,17 @@ type Jobs interface {
 	// TriggerVersion updates to exactly the image version versionID of the
 	// service's repo; model.ErrNotFound for an unknown service or version.
 	TriggerVersion(ctx context.Context, serviceID, versionID int64, requestedBy string) (model.Job, error)
+	// Restorable explains why a drifted service cannot be restored.
+	Restorable(svc model.Service) error
+	// RestoreDesired deploys the service's desired ref again.
+	RestoreDesired(ctx context.Context, serviceID int64, requestedBy string) (model.Job, error)
+}
+
+// Poller checks the registry for one service. *poller.Poller satisfies it.
+type Poller interface {
+	// PollService returns *poller.NotPolledError when the service is not
+	// polled and model.ErrNotFound for an unknown or removed one.
+	PollService(ctx context.Context, id int64) error
 }
 
 // Broker delivers change signals to open streams and accepts signals from
@@ -72,12 +100,14 @@ type Broker interface {
 	Publish(t events.Topic)
 }
 
-// Config holds values shown in the "add agent" snippet and on the service page.
+// Config holds values shown in the "add agent" snippet, on the agents page and
+// on the service page.
 type Config struct {
 	AgentURL     string // what agents dial, e.g. wss://10.0.0.5:8090/agent
 	MasterKey    string // the master's fingerprint
 	AgentImage   string
 	PollInterval time.Duration // shown as "every <interval>" on the service page
+	Version      string        // the master's build version; agents on another version are marked old
 }
 
 // Server renders the UI.
@@ -86,6 +116,7 @@ type Server struct {
 	hub    Hub
 	jobs   Jobs
 	broker Broker
+	poller Poller
 	cfg    Config
 	log    *slog.Logger
 	now    func() time.Time
@@ -94,17 +125,17 @@ type Server struct {
 
 // New parses the templates. Each page is its own template set so every page
 // can define its own "content" block.
-func New(store Store, hub Hub, jobs Jobs, broker Broker, cfg Config, log *slog.Logger) (*Server, error) {
+func New(store Store, hub Hub, jobs Jobs, broker Broker, poller Poller, cfg Config, log *slog.Logger) (*Server, error) {
-	funcs := template.FuncMap{"shortRef": shortRef, "duration": jobDuration, "followsNoTag": followsNoTag}
+	funcs := template.FuncMap{"shortRef": shortRef, "duration": jobDuration, "followsNoTag": followsNoTag, "middle": middle}
 	pages := map[string]*template.Template{}
-	for _, page := range []string{"overview.html", "agents.html", "agent_created.html", "service.html", "jobs.html", "job.html"} {
+	for _, page := range []string{"overview.html", "agents.html", "service.html", "jobs.html", "job.html"} {
 		t, err := template.New(page).Funcs(funcs).ParseFS(assets, "templates/layout.html", "templates/"+page)
 		if err != nil {
 			return nil, fmt.Errorf("parse %s: %w", page, err)
 		}
 		pages[page] = t
 	}
-	return &Server{store: store, hub: hub, jobs: jobs, broker: broker, cfg: cfg, log: log, now: time.Now, pages: pages}, nil
+	return &Server{store: store, hub: hub, jobs: jobs, broker: broker, poller: poller, cfg: cfg, log: log, now: time.Now, pages: pages}, nil
 }
 
 // Register adds the UI routes to mux. protect wraps every route except
@@ -118,35 +149,84 @@ func (s *Server) Register(mux *http.ServeMux, protect func(http.Handler) http.Ha
 	route("GET /fragments/services/{id}/jobs", s.serviceJobsFragment)
 	route("POST /services/{id}/update", s.updateService)
 	route("POST /services/{id}/mode", s.setMode)
+	route("POST /services/{id}/poll", s.pollService)
+	route("POST /services/{id}/restore", s.restoreDesired)
+	route("POST /services/{id}/accept", s.acceptRunning)
+	route("POST /services/{id}/merge", s.mergeService)
 	route("GET /jobs", s.jobsPage)
 	route("GET /fragments/jobs", s.jobsFragment)
 	route("GET /jobs/{id}", s.jobPage)
 	route("GET /fragments/jobs/{id}", s.jobFragment)
 	route("GET /agents", s.agents)
 	route("GET /fragments/agents", s.agentsFragment)
+	route("GET /fragments/agents/{id}/connect", s.agentConnectFragment)
+	route("GET /fragments/nav", s.navFragment)
 	route("POST /agents", s.createAgent)
 	route("POST /agents/{id}/revoke", s.revokeAgent)
 	route("GET /events", s.events)
 }
 
+// navData is what the sidebar shows on every page.
+type navData struct {
+	Section      string // services, jobs or agents; "" for none
+	Services     int
+	AgentsOnline int
+	AgentsTotal  int // agents that are not revoked
+}
+
+// spriteIcon is one icon of the sprite the services fragment carries.
+type spriteIcon struct {
+	Slug string
+	Path string
+}
+
+// sections maps a page to the sidebar entry that is current on it.
+var sections = map[string]string{
+	"overview.html": "services", "service.html": "services",
+	"jobs.html": "jobs", "job.html": "jobs",
+	"agents.html": "agents",
+}
+
 type pageData struct {
-	Title   string
+	Title  string
-	CSRF    string
+	CSRF   string
-	Groups  []serviceGroup
+	Nav    navData
+	Groups []serviceGroup
+	// ServiceCount is how many services the groups hold.
+	ServiceCount int
+	// Sprite holds the icons the groups use, so the page carries only those.
+	Sprite  []spriteIcon
 	Agents  []agentRow
 	Error   string
 	Created model.Agent
 	Snippet string
 
+	// AddStep is the step of the "add agent" panel: 0 closed, 1 name,
+	// 2 deploy (Created, Token, MasterKey and Snippet are set).
+	AddStep       int
+	FormName      string // what was typed in step 1
+	Token         string
+	MasterKey     string
+	Connected     bool   // the agent Created has connected
+	MasterVersion string // the master's build version; "" if unknown
+
 	Service     model.Service
 	ConfigFiles []string
 	Reason      string // why Service cannot be updated; "" when it can
 	AutoBlocked string // why Service cannot be set to auto; "" when it can
+	PollBlocked string // why Service cannot be checked against the registry; "" when it can
 	Versions    []versionRow
+	NewerTag    string // Service's newer tag while it still holds; "" otherwise
 	Polling     string // what the service page says about registry polling
 	PollFailed  bool   // Polling is the last registry error
-	Jobs        []model.Job
+	// RestoreBlocked is why a drifted Service cannot be restored; "" when it
-	Job         model.Job
+	// can. Only meaningful when Service.Drifted().
+	RestoreBlocked string
+	MoveFrom       *model.Move     // a live Service: the removed service it probably replaces
+	MoveTo         *model.Move     // a removed Service: the live service it probably moved to
+	MergeSources   []model.Service // a live Compose or Swarm Service: removed ones to merge in
+	Jobs           []model.Job
+	Job            model.Job
 }
 
 func (s *Server) base(r *http.Request, title string) pageData {
@@ -154,6 +234,58 @@ func (s *Server) base(r *http.Request, title string) pageData {
 	return pageData{Title: title, CSRF: sess.CSRFToken}
 }
 
+// page renders a full page: the sidebar numbers are looked up and put in data.
+func (s *Server) page(w http.ResponseWriter, r *http.Request, status int, page string, data pageData) {
+	nav, err := s.navData(r.Context(), sections[page])
+	if err != nil {
+		s.fail(w, err)
+		return
+	}
+	data.Nav = nav
+	s.render(w, status, page, "layout", data)
+}
+
+// navData counts what the sidebar shows next to its entries.
+func (s *Server) navData(ctx context.Context, section string) (navData, error) {
+	nav := navData{Section: section}
+	services, err := s.store.ListServices(ctx)
+	if err != nil {
+		return nav, err
+	}
+	nav.Services = len(services)
+	agents, err := s.store.ListAgents(ctx)
+	if err != nil {
+		return nav, err
+	}
+	for _, a := range agents {
+		if a.RevokedAt != nil {
+			continue
+		}
+		nav.AgentsTotal++
+		if s.hub.Online(a.ID) {
+			nav.AgentsOnline++
+		}
+	}
+	return nav, nil
+}
+
+// navFragment renders the sidebar entries. The page asks for it with its own
+// section so the current entry stays marked.
+func (s *Server) navFragment(w http.ResponseWriter, r *http.Request) {
+	section := r.URL.Query().Get("section")
+	if section != "services" && section != "jobs" && section != "agents" {
+		section = ""
+	}
+	nav, err := s.navData(r.Context(), section)
+	if err != nil {
+		s.fail(w, err)
+		return
+	}
+	data := s.base(r, "")
+	data.Nav = nav
+	s.render(w, http.StatusOK, "overview.html", "nav", data)
+}
+
 func (s *Server) render(w http.ResponseWriter, status int, page, block string, data pageData) {
 	w.Header().Set("Content-Type", "text/html; charset=utf-8")
 	w.WriteHeader(status)
@@ -168,17 +300,41 @@ func (s *Server) serviceData(r *http.Request) (pageData, error) {
 	if err != nil {
 		return data, err
 	}
-	data.Groups = groupServices(services, s.autoBlocked)
+	agents, err := s.store.ListAgents(r.Context())
+	if err != nil {
+		return data, err
+	}
+	data.Groups = groupServices(services, agents, s.hub.Online, s.autoBlocked)
+	data.ServiceCount = len(services)
+	data.Sprite = spriteFor(data.Groups)
 	return data, nil
 }
 
+// spriteFor collects the icons the groups' rows use, each once.
+func spriteFor(groups []serviceGroup) []spriteIcon {
+	var sprite []spriteIcon
+	seen := map[string]bool{}
+	for _, g := range groups {
+		for _, row := range g.Services {
+			if row.Icon == "" || seen[row.Icon] {
+				continue
+			}
+			seen[row.Icon] = true
+			if path, ok := icons.Path(row.Icon); ok {
+				sprite = append(sprite, spriteIcon{Slug: row.Icon, Path: path})
+			}
+		}
+	}
+	return sprite
+}
+
 func (s *Server) overview(w http.ResponseWriter, r *http.Request) {
 	data, err := s.serviceData(r)
 	if err != nil {
 		s.fail(w, err)
 		return
 	}
-	s.render(w, http.StatusOK, "overview.html", "layout", data)
+	s.page(w, r, http.StatusOK, "overview.html", data)
 }
 
 func (s *Server) servicesFragment(w http.ResponseWriter, r *http.Request) {
@@ -187,7 +343,7 @@ func (s *Server) servicesFragment(w http.ResponseWriter, r *http.Request) {
 		s.fail(w, err)
 		return
 	}
-	s.render(w, http.StatusOK, "overview.html", "services", data)
+	s.render(w, http.StatusOK, "overview.html", "services-fragment", data)
 }
 
 // autoBlocked returns the reason svc cannot be set to auto mode, or "".
@@ -235,16 +391,57 @@ func (s *Server) serviceDetail(r *http.Request, id int64) (pageData, error) {
 		return data, err
 	}
 	data.AutoBlocked = s.autoBlocked(svc)
+	data.PollBlocked = pollBlocked(svc)
+	if err := s.jobs.Restorable(svc); errors.As(err, &nu) {
+		data.RestoreBlocked = nu.Reason
+	} else if err != nil {
+		return data, err
+	}
+	if err := s.moveData(r, &data); err != nil {
+		return data, err
+	}
 	versions, err := s.store.RecentVersions(r.Context(), svc.ImageRepo, versionsLimit)
 	if err != nil {
 		return data, err
 	}
+	data.NewerTag = newerTag(svc)
+	if data.NewerTag != "" && !slices.ContainsFunc(versions, func(v model.ImageVersion) bool { return v.Tag == data.NewerTag && v.Digest == "" }) {
+		// The suggested tag can be older than the listed versions; it must
+		// still be pickable.
+		v, err := s.store.GetTagVersion(r.Context(), svc.ImageRepo, data.NewerTag)
+		if err == nil {
+			versions = append(versions, v)
+		} else if !errors.Is(err, model.ErrNotFound) {
+			return data, err
+		}
+	}
 	data.Versions = versionRows(versions, svc.ActualDigest)
 	data.Polling, data.PollFailed = pollingStatus(svc, s.cfg.PollInterval)
 	data.Jobs, err = s.store.ListJobs(r.Context(), model.JobFilter{ServiceID: id, Limit: serviceJobsLimit})
 	return data, err
 }
 
+// moveData fills the move suggestion and the merge choices of the service page.
+func (s *Server) moveData(r *http.Request, data *pageData) error {
+	svc := data.Service
+	moves, err := s.store.MoveCandidates(r.Context())
+	if err != nil {
+		return err
+	}
+	for i, m := range moves {
+		switch svc.ID {
+		case m.ToID:
+			data.MoveFrom = &moves[i]
+		case m.FromID:
+			data.MoveTo = &moves[i]
+		}
+	}
+	if svc.RemovedAt == nil && !svc.Self && svc.Kind != protocol.KindContainer {
+		data.MergeSources, err = s.store.MergeSources(r.Context(), svc.ID)
+	}
+	return err
+}
+
 func (s *Server) servicePage(w http.ResponseWriter, r *http.Request) {
 	id, ok := pathID(w, r)
 	if !ok {
@@ -255,7 +452,7 @@ func (s *Server) servicePage(w http.ResponseWriter, r *http.Request) {
 		s.failLookup(w, err)
 		return
 	}
-	s.render(w, http.StatusOK, "service.html", "layout", data)
+	s.page(w, r, http.StatusOK, "service.html", data)
 }
 
 func (s *Server) serviceJobsFragment(w http.ResponseWriter, r *http.Request) {
@@ -296,18 +493,158 @@ func (s *Server) updateService(w http.ResponseWriter, r *http.Request) {
 	case err == nil || errors.Is(err, model.ErrJobActive):
 		http.Redirect(w, r, "/jobs/"+strconv.FormatInt(job.ID, 10), http.StatusSeeOther)
 	case errors.As(err, &nu):
-		data, derr := s.serviceDetail(r, id)
+		s.renderServiceError(w, r, id, http.StatusConflict, "Cannot update: "+nu.Reason+".")
-		if derr != nil {
-			s.failLookup(w, derr)
-			return
-		}
-		data.Error = "Cannot update: " + nu.Reason + "."
-		s.render(w, http.StatusConflict, "service.html", "layout", data)
 	default:
 		s.failLookup(w, err)
 	}
 }
 
+// restoreDesired deploys the desired ref of a drifted service again.
+func (s *Server) restoreDesired(w http.ResponseWriter, r *http.Request) {
+	id, ok := pathID(w, r)
+	if !ok {
+		return
+	}
+	sess, _ := webauth.SessionFrom(r.Context())
+	job, err := s.jobs.RestoreDesired(r.Context(), id, sess.Subject)
+	var nu *model.NotUpdatableError
+	switch {
+	case err == nil || errors.Is(err, model.ErrJobActive):
+		http.Redirect(w, r, "/jobs/"+strconv.FormatInt(job.ID, 10), http.StatusSeeOther)
+	case errors.As(err, &nu):
+		s.renderServiceError(w, r, id, http.StatusConflict, "Cannot restore: "+nu.Reason+".")
+	default:
+		s.failLookup(w, err)
+	}
+}
+
+// acceptRunning makes the running version of a drifted service the desired
+// one. The form's actual field is the running ref the user saw (Service.RunningRef);
+// it only guards against the service having changed since, the value written
+// comes from the row.
+func (s *Server) acceptRunning(w http.ResponseWriter, r *http.Request) {
+	id, ok := pathID(w, r)
+	if !ok {
+		return
+	}
+	actual := r.PostFormValue("actual")
+	if actual == "" {
+		http.Error(w, "missing actual", http.StatusBadRequest)
+		return
+	}
+	svc, err := s.store.GetService(r.Context(), id)
+	if err == nil && svc.RemovedAt != nil {
+		err = model.ErrNotFound
+	}
+	if err != nil {
+		s.failLookup(w, err)
+		return
+	}
+	if !svc.Drifted() {
+		s.renderServiceError(w, r, id, http.StatusConflict, "Cannot accept: the service runs its desired version.")
+		return
+	}
+	if actual != svc.RunningRef() {
+		s.renderServiceError(w, r, id, http.StatusConflict, "The running version changed; check it again.")
+		return
+	}
+	desired := svc.RunningRef()
+	err = s.store.AcceptRunning(r.Context(), id, svc.ActualRef, svc.ActualDigest, desired)
+	switch {
+	case errors.Is(err, model.ErrConflict):
+		s.renderServiceError(w, r, id, http.StatusConflict, "The running version changed; check it again.")
+		return
+	case err != nil:
+		s.failLookup(w, err)
+		return
+	}
+	sess, _ := webauth.SessionFrom(r.Context())
+	s.log.Info("drift_accepted", "service_id", id, "subject", sess.Subject, "desired", desired)
+	s.broker.Publish(events.Services)
+	if r.PostFormValue("return") == "overview" {
+		http.Redirect(w, r, "/", http.StatusSeeOther)
+		return
+	}
+	http.Redirect(w, r, "/services/"+strconv.FormatInt(id, 10), http.StatusSeeOther)
+}
+
+// mergeService moves the live service {id} onto the removed service in the
+// form field "from", which keeps its history, mode and desired ref.
+func (s *Server) mergeService(w http.ResponseWriter, r *http.Request) {
+	id, ok := pathID(w, r)
+	if !ok {
+		return
+	}
+	from, err := strconv.ParseInt(r.PostFormValue("from"), 10, 64)
+	if err != nil {
+		http.Error(w, "invalid service", http.StatusBadRequest)
+		return
+	}
+	err = s.store.MergeService(r.Context(), from, id)
+	var refused *model.MergeRefusedError
+	switch {
+	case errors.As(err, &refused):
+		s.renderServiceError(w, r, id, http.StatusConflict, "Cannot merge: "+refused.Reason+".")
+		return
+	case errors.Is(err, model.ErrJobActive):
+		s.renderServiceError(w, r, id, http.StatusConflict, "Cannot merge: "+model.ErrJobActive.Error()+".")
+		return
+	case errors.Is(err, model.ErrRevoked):
+		s.renderServiceError(w, r, id, http.StatusConflict, "Cannot merge: the agent is revoked.")
+		return
+	case errors.Is(err, model.ErrConflict):
+		s.renderServiceError(w, r, id, http.StatusConflict, "Cannot merge: the service changed; check it again.")
+		return
+	case err != nil:
+		s.failLookup(w, err)
+		return
+	}
+	sess, _ := webauth.SessionFrom(r.Context())
+	s.log.Info("service_merged", "from_id", from, "to_id", id, "subject", sess.Subject)
+	s.broker.Publish(events.Services)
+	s.broker.Publish(events.Jobs)
+	http.Redirect(w, r, "/services/"+strconv.FormatInt(from, 10), http.StatusSeeOther)
+}
+
+// renderServiceError renders the service page of id with msg as its error.
+func (s *Server) renderServiceError(w http.ResponseWriter, r *http.Request, id int64, status int, msg string) {
+	data, err := s.serviceDetail(r, id)
+	if err != nil {
+		s.failLookup(w, err)
+		return
+	}
+	data.Error = msg
+	s.page(w, r, status, "service.html", data)
+}
+
+// pollService checks the registry for one service now and waits for the
+// answer. A registry error is not a failure of the request: it shows on the
+// service page as the last check failing. The poller publishes the change
+// itself when it writes.
+func (s *Server) pollService(w http.ResponseWriter, r *http.Request) {
+	id, ok := pathID(w, r)
+	if !ok {
+		return
+	}
+	err := s.poller.PollService(r.Context(), id)
+	var np *poller.NotPolledError
+	switch {
+	case errors.As(err, &np):
+		s.renderServiceError(w, r, id, http.StatusConflict, "Cannot check: "+np.Reason+".")
+		return
+	case r.Context().Err() != nil:
+		// The client is gone or the master is shutting down. Redirect anyway:
+		// a browser still waiting gets the page, not an empty 200.
+	case err != nil:
+		s.failLookup(w, err)
+		return
+	default:
+		sess, _ := webauth.SessionFrom(r.Context())
+		s.log.Info("poll_requested", "service_id", id, "subject", sess.Subject)
+	}
+	http.Redirect(w, r, "/services/"+strconv.FormatInt(id, 10), http.StatusSeeOther)
+}
+
 // setMode switches a service between manual and auto. Manual is always
 // allowed; auto only when jobs.AutoAllowed says so.
 func (s *Server) setMode(w http.ResponseWriter, r *http.Request) {
@@ -331,13 +668,11 @@ func (s *Server) setMode(w http.ResponseWriter, r *http.Request) {
 		}
 		var nu *model.NotUpdatableError
 		if err := s.jobs.AutoAllowed(svc); errors.As(err, &nu) {
-			data, derr := s.serviceDetail(r, id)
+			if isHTMX(r) {
-			if derr != nil {
+				// htmx does not swap a 409; the service page shows the reason.
-				s.failLookup(w, derr)
+				w.Header().Set("HX-Redirect", "/services/"+strconv.FormatInt(id, 10))
-				return
 			}
-			data.Error = "Cannot enable auto: " + nu.Reason + "."
+			s.renderServiceError(w, r, id, http.StatusConflict, "Cannot enable auto: "+nu.Reason+".")
-			s.render(w, http.StatusConflict, "service.html", "layout", data)
 			return
 		} else if err != nil {
 			s.fail(w, err)
@@ -350,6 +685,11 @@ func (s *Server) setMode(w http.ResponseWriter, r *http.Request) {
 	}
 	s.broker.Publish(events.Services)
 	if r.PostFormValue("return") == "overview" {
+		if isHTMX(r) {
+			// Swap the list in place instead of redirecting, so the page keeps its scroll position.
+			s.servicesFragment(w, r)
+			return
+		}
 		http.Redirect(w, r, "/", http.StatusSeeOther)
 		return
 	}
@@ -369,7 +709,7 @@ func (s *Server) jobsPage(w http.ResponseWriter, r *http.Request) {
 		s.fail(w, err)
 		return
 	}
-	s.render(w, http.StatusOK, "jobs.html", "layout", data)
+	s.page(w, r, http.StatusOK, "jobs.html", data)
 }
 
 func (s *Server) jobsFragment(w http.ResponseWriter, r *http.Request) {
@@ -400,7 +740,7 @@ func (s *Server) jobData(w http.ResponseWriter, r *http.Request) (pageData, bool
 
 func (s *Server) jobPage(w http.ResponseWriter, r *http.Request) {
 	if data, ok := s.jobData(w, r); ok {
-		s.render(w, http.StatusOK, "job.html", "layout", data)
+		s.page(w, r, http.StatusOK, "job.html", data)
 	}
 }
 
@@ -412,23 +752,28 @@ func (s *Server) jobFragment(w http.ResponseWriter, r *http.Request) {
 
 func (s *Server) agentData(r *http.Request) (pageData, error) {
 	data := s.base(r, "Agents")
+	data.MasterVersion = s.cfg.Version
 	agents, err := s.store.ListAgents(r.Context())
 	if err != nil {
 		return data, err
 	}
 	for _, a := range agents {
-		data.Agents = append(data.Agents, agentRow{Agent: a, Online: s.hub.Online(a.ID)})
+		data.Agents = append(data.Agents, newAgentRow(a, s.hub.Online(a.ID), s.cfg.Version))
 	}
 	return data, nil
 }
 
+// agents shows the agent list; with ?add=1 the "add agent" panel is open.
 func (s *Server) agents(w http.ResponseWriter, r *http.Request) {
 	data, err := s.agentData(r)
 	if err != nil {
 		s.fail(w, err)
 		return
 	}
-	s.render(w, http.StatusOK, "agents.html", "layout", data)
+	if r.URL.Query().Get("add") == "1" {
+		data.AddStep = 1
+	}
+	s.page(w, r, http.StatusOK, "agents.html", data)
 }
 
 func (s *Server) agentsFragment(w http.ResponseWriter, r *http.Request) {
@@ -437,13 +782,41 @@ func (s *Server) agentsFragment(w http.ResponseWriter, r *http.Request) {
 		s.fail(w, err)
 		return
 	}
-	s.render(w, http.StatusOK, "agents.html", "agents", data)
+	if data.Nav, err = s.navData(r.Context(), "agents"); err != nil {
+		s.fail(w, err)
+		return
+	}
+	s.render(w, http.StatusOK, "agents.html", "agents-fragment", data)
+}
+
+// agentConnectFragment is the last step of the "add agent" panel: it says
+// whether the agent that was just added has connected. It never repeats the
+// token.
+func (s *Server) agentConnectFragment(w http.ResponseWriter, r *http.Request) {
+	id, ok := pathID(w, r)
+	if !ok {
+		return
+	}
+	agents, err := s.store.ListAgents(r.Context())
+	if err != nil {
+		s.fail(w, err)
+		return
+	}
+	i := slices.IndexFunc(agents, func(a model.Agent) bool { return a.ID == id && a.RevokedAt == nil })
+	if i < 0 {
+		http.Error(w, "not found", http.StatusNotFound)
+		return
+	}
+	data := s.base(r, "Agents")
+	data.Created = agents[i]
+	data.Connected = s.hub.Online(id) || agents[i].LastSeenAt != nil
+	s.render(w, http.StatusOK, "agents.html", "connect", data)
 }
 
 func (s *Server) createAgent(w http.ResponseWriter, r *http.Request) {
 	name := strings.TrimSpace(r.FormValue("name"))
 	if !agentNamePattern.MatchString(name) {
-		s.agentsWithError(w, r, http.StatusBadRequest, "Agent names use lowercase letters, digits and dashes.")
+		s.agentsWithError(w, r, http.StatusBadRequest, name, "Agent names use lowercase letters, digits and dashes.")
 		return
 	}
 	token, err := agenttoken.Generate()
@@ -453,28 +826,39 @@ func (s *Server) createAgent(w http.ResponseWriter, r *http.Request) {
 	}
 	agent, err := s.store.CreateAgent(r.Context(), name, agenttoken.Hash(token))
 	if errors.Is(err, model.ErrConflict) {
-		s.agentsWithError(w, r, http.StatusConflict, fmt.Sprintf("An agent named %q already exists.", name))
+		s.agentsWithError(w, r, http.StatusConflict, name, fmt.Sprintf("An agent named %q already exists.", name))
 		return
 	}
 	if err != nil {
 		s.fail(w, err)
 		return
 	}
-	data := s.base(r, "Agent created")
-	data.Created = agent
-	data.Snippet = agentSnippet(s.cfg, token)
-	w.Header().Set("Cache-Control", "no-store") // the page shows the one-time token
-	s.render(w, http.StatusOK, "agent_created.html", "layout", data)
-}
-
-func (s *Server) agentsWithError(w http.ResponseWriter, r *http.Request, status int, msg string) {
 	data, err := s.agentData(r)
 	if err != nil {
 		s.fail(w, err)
 		return
 	}
+	data.AddStep = 2
+	data.Created = agent
+	data.Token = token
+	data.MasterKey = s.cfg.MasterKey
+	data.Snippet = agentSnippet(s.cfg, token)
+	w.Header().Set("Cache-Control", "no-store") // the page shows the one-time token
+	s.page(w, r, http.StatusOK, "agents.html", data)
+}
+
+// agentsWithError shows the agent list with the "add agent" panel open at
+// its first step, name kept and msg under the field.
+func (s *Server) agentsWithError(w http.ResponseWriter, r *http.Request, status int, name, msg string) {
+	data, err := s.agentData(r)
+	if err != nil {
+		s.fail(w, err)
+		return
+	}
+	data.AddStep = 1
+	data.FormName = name
 	data.Error = msg
-	s.render(w, status, "agents.html", "layout", data)
+	s.page(w, r, status, "agents.html", data)
 }
 
 func (s *Server) revokeAgent(w http.ResponseWriter, r *http.Request) {
@@ -489,6 +873,7 @@ func (s *Server) revokeAgent(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	s.hub.Disconnect(id)
+	s.broker.Publish(events.Agents)   // the agent list and the sidebar change
 	s.broker.Publish(events.Services) // the agent's services were marked removed
 	s.broker.Publish(events.Jobs)     // the agent's active jobs were failed
 	for _, jobID := range failedJobs {
@@ -497,6 +882,9 @@ func (s *Server) revokeAgent(w http.ResponseWriter, r *http.Request) {
 	http.Redirect(w, r, "/agents", http.StatusSeeOther)
 }
 
+// pingInterval is how often an idle event stream sends "event: ping".
+var pingInterval = 25 * time.Second
+
 // events streams change signals as Server-Sent Events. htmx's SSE extension
 // turns "event: services" into a refetch of the services fragment. The event
 // carries no state, so a dropped signal is recovered by the next one.
@@ -516,7 +904,7 @@ func (s *Server) events(w http.ResponseWriter, r *http.Request) {
 		s.log.Error("sse_flush_unsupported", "error", err)
 		return
 	}
-	keepalive := time.NewTicker(25 * time.Second)
+	keepalive := time.NewTicker(pingInterval)
 	defer keepalive.Stop()
 	for {
 		var msg string
@@ -526,7 +914,7 @@ func (s *Server) events(w http.ResponseWriter, r *http.Request) {
 		case t := <-ch:
 			msg = fmt.Sprintf("event: %s\ndata: changed\n\n", t)
 		case <-keepalive.C:
-			msg = ": keepalive\n\n"
+			msg = "event: ping\ndata: ok\n\n" // keeps proxies open; the Live box in the sidebar counts it
 		}
 		if _, err := fmt.Fprint(w, msg); err != nil {
 			return
@@ -542,6 +930,11 @@ func (s *Server) fail(w http.ResponseWriter, err error) {
 	http.Error(w, "internal error", http.StatusInternalServerError)
 }
 
+// isHTMX reports whether r was sent by htmx rather than by a plain form submit.
+func isHTMX(r *http.Request) bool {
+	return r.Header.Get("HX-Request") == "true"
+}
+
 // agentSnippet is the compose file shown once when an agent is added.
 func agentSnippet(cfg Config, token string) string {
 	return fmt.Sprintf(`services:
@@ -559,6 +952,7 @@ func agentSnippet(cfg Config, token string) string {
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
       # - /path/to/stacks:/path/to/stacks  # mount each stacks directory at the same path, read-write
+      #   (also this agent's own stack directory, or it cannot update itself)
       # - /root/.docker/config.json:/docker-config/config.json:ro
 `, cfg.AgentImage, cfg.AgentURL, cfg.MasterKey, token)
 }

internal/master/web/server_test.go

@@ -35,6 +35,13 @@ type fakeStore struct {
 	revokedID    int64
 	revokeFailed []int64 // job IDs RevokeAgent reports as failed
 	versions     []model.ImageVersion
+
+	acceptCalls []acceptCall
+	acceptErr   error
+	moves       []model.Move
+	sources     []model.Service
+	merged      [][2]int64 // from, to
+	mergeErr    error
 }
 
 func (f *fakeStore) RecentVersions(_ context.Context, repo string, limit int) ([]model.ImageVersion, error) {
@@ -142,6 +149,9 @@ type fakeJobs struct {
 	triggered   []int64 // version IDs passed to TriggerVersion
 	triggerSvc  []int64 // service IDs passed to TriggerVersion
 	plainCalls  int     // Trigger calls
+
+	restorable error   // returned by Restorable
+	restored   []int64 // service IDs passed to RestoreDesired
 }
 
 func (f *fakeJobs) Updatable(model.Service) error { return f.updatable }
@@ -211,30 +221,52 @@ func newTestEnv(t *testing.T) (*httptest.Server, *fakeStore, *fakeHub, *events.B
 // the service page shows.
 func newTestEnvPolling(t *testing.T, pollInterval time.Duration) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) {
 	t.Helper()
-	agentID := int64(2)
+	return newTestEnvConfig(t, func(c *Config) { c.PollInterval = pollInterval })
+}
+
+// newTestEnvVersion is newTestEnv with version as the master's build version.
+func newTestEnvVersion(t *testing.T, version string) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) {
+	t.Helper()
+	return newTestEnvConfig(t, func(c *Config) { c.Version = version })
+}
+
+// newTestEnvConfig is newTestEnv with edit applied to the UI's Config.
+func newTestEnvConfig(t *testing.T, edit func(*Config)) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) {
+	t.Helper()
+	return newTestEnvFull(t, edit, &fakePoller{})
+}
+
+// newTestEnvFull is newTestEnvConfig with p as the poller.
+func newTestEnvFull(t *testing.T, edit func(*Config), p Poller) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) {
+	t.Helper()
+	agentID, offlineAgentID := int64(2), int64(3)
 	store := &fakeStore{
 		created: map[string]string{"node-2": "x"},
 		services: []model.Service{
 			{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", AgentID: &agentID, AgentName: "node-2",
 				ActualRef: "gitea.shcizo.se/shcizo/slusk:1.4.2", Status: "running", UpdateMode: "manual"},
-			{ID: 2, Kind: protocol.KindSwarm, SwarmService: "jellyfin_jellyfin",
+			{ID: 2, Kind: protocol.KindSwarm, SwarmService: "jellyfin_jellyfin", SwarmClusterID: "c1abcdefgh",
 				ActualRef: "lscr.io/linuxserver/jellyfin@sha256:0f42497a69fa0441bfd5f9d6bba8694f2a656ec984e571d0ac04c6dd91250039", Status: "degraded", UpdateMode: "manual"},
 			{ID: 3, Kind: protocol.KindContainer, ContainerName: "updater-agent", AgentID: &agentID, AgentName: "node-2",
 				ActualRef: "pu:1", Status: "running", UpdateMode: "manual", Self: true},
+			{ID: 4, Kind: protocol.KindCompose, ComposeProject: "media", ComposeService: "radarr", AgentID: &offlineAgentID, AgentName: "node-3",
+				ActualRef: "lscr.io/linuxserver/radarr:latest", Status: "running", UpdateMode: "manual"},
 		},
 		agents: []model.Agent{
-			{ID: 2, Name: "node-2", Hostname: "node-2"},
+			{ID: 2, Name: "node-2", Hostname: "node-2.lan", AgentVersion: "v1.4.0", SwarmManager: true, SwarmClusterID: "c1abcdefgh", LastSeenAt: ptr(time.Now().UTC())},
-			{ID: 3, Name: "node-3"},
+			{ID: 3, Name: "node-3", Hostname: "node-3.lan", AgentVersion: "v1.3.2", LastSeenAt: ptr(time.Date(2026, 10, 12, 13, 51, 0, 0, time.UTC))},
 			{ID: 4, Name: "old", RevokedAt: ptr(time.Now())},
+			{ID: 5, Name: "fresh"},
 		},
 	}
 	hub := &fakeHub{online: map[int64]bool{2: true}}
 	broker := events.NewBroker()
 	jobs := &fakeJobs{}
-	s, err := New(store, hub, jobs, broker, Config{
+	cfg := Config{
 		AgentURL: "wss://10.0.0.5:8090/agent", MasterKey: "sha256:abc", AgentImage: "gitea.shcizo.se/shcizo/package-updater:latest",
-		PollInterval: pollInterval,
+	}
-	}, slog.New(slog.NewTextHandler(io.Discard, nil)))
+	edit(&cfg)
+	s, err := New(store, hub, jobs, broker, p, cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
 	require.NoError(t, err)
 	mux := http.NewServeMux()
 	s.Register(mux, func(h http.Handler) http.Handler {
@@ -269,13 +301,45 @@ func TestOverview_GroupsServicesPerMachineAndSwarm(t *testing.T) {
 	code, body := get(t, srv, "/")
 	assert.Equal(t, http.StatusOK, code)
 	assert.Contains(t, body, "<html")
-	assert.Contains(t, body, "node-2")
+	assert.Contains(t, body, ">Swarm</h2>", "one cluster is just called Swarm")
-	assert.Contains(t, body, "Swarm")
+	assert.Contains(t, body, `title="c1abcdefgh"`, "the full cluster id is the hover text")
-	assert.Contains(t, body, "slusk/web")
+	assert.Contains(t, body, "managers node-2")
-	assert.Contains(t, body, "jellyfin_jellyfin")
+	assert.Contains(t, body, ">node-2</h2>")
+	assert.Contains(t, body, `<span class="nmt">web</span>`)
+	assert.Contains(t, body, `<span class="stk">slusk</span>`)
+	assert.Contains(t, body, "jellyfin_jellyfin", "Swarm service names are shown as they are")
 	assert.Contains(t, body, "sha256:0f42497a69fa…", "digests are shortened")
-	assert.Contains(t, body, "managed outside")
+	assert.Contains(t, body, `<span class="tag f-out">This agent</span>`)
-	assert.Less(t, strings.Index(body, "node-2"), strings.Index(body, "Swarm"), "machines before Swarm")
+	assert.NotContains(t, body, "Managed outside")
+	assert.Less(t, strings.Index(body, ">Swarm</h2>"), strings.Index(body, ">node-2</h2>"), "a cluster is followed by its managers' machines")
+	assert.Contains(t, body, `class="sub" data-machine="agent-2"`, "the machine filter lists the manager under the cluster")
+}
+
+func TestOverview_OfflineGroupShowsSince(t *testing.T) {
+	srv, store, hub, _, _ := newTestEnv(t)
+	_, body := get(t, srv, "/")
+	assert.Regexp(t, `Agent offline since <time datetime="2026-10-12T13:51:00Z">13:51</time> · showing last known state`, body)
+	assert.NotContains(t, body, "No manager online", "the cluster has a connected manager")
+
+	hub.mu.Lock()
+	hub.online[2] = false
+	hub.mu.Unlock()
+	_, body = get(t, srv, "/")
+	since := store.agents[0].LastSeenAt.Format("2006-01-02T15:04:05Z07:00")
+	assert.Contains(t, body, `No manager online since <time datetime="`+since+`">`)
+	assert.Contains(t, body, ` · showing last known state`)
+}
+
+func TestOverview_IconOrLetter(t *testing.T) {
+	srv, store, _, _, _ := newTestEnv(t)
+	store.services[0].ActualRef = "gitea/gitea:1.27.3"
+	for _, path := range []string{"/", "/fragments/services"} {
+		_, body := get(t, srv, path)
+		assert.Contains(t, body, `<symbol id="si-gitea"`, path)
+		assert.Equal(t, 1, strings.Count(body, `<symbol id="si-gitea"`), "the sprite holds each icon once: "+path)
+		assert.Contains(t, body, `href="#si-gitea"`, path)
+		assert.Contains(t, body, `class="ltr lc-`, "services without an icon get a letter: "+path)
+	}
 }
 
 func TestServicesFragment_IsNotAFullPage(t *testing.T) {
@@ -283,15 +347,64 @@ func TestServicesFragment_IsNotAFullPage(t *testing.T) {
 	code, body := get(t, srv, "/fragments/services")
 	assert.Equal(t, http.StatusOK, code)
 	assert.NotContains(t, body, "<html")
-	assert.Contains(t, body, "slusk/web")
+	assert.Contains(t, body, `title="slusk/web"`)
+	assert.Contains(t, body, `id="machines" hx-swap-oob`, "the machine filter in the sidebar is refreshed with the rows")
 }
 
 func TestAgentsPage_ShowsConnectionState(t *testing.T) {
 	srv, _, _, _ := newTestServer(t)
 	_, body := get(t, srv, "/agents")
-	assert.Regexp(t, regexp.MustCompile(`node-2[\s\S]*online`), body)
+	assert.Regexp(t, regexp.MustCompile(`node-2[\s\S]*Online`), body)
-	assert.Regexp(t, regexp.MustCompile(`node-3[\s\S]*offline`), body)
+	assert.Regexp(t, regexp.MustCompile(`node-3[\s\S]*Offline`), body)
-	assert.Regexp(t, regexp.MustCompile(`old[\s\S]*revoked`), body)
+	assert.Regexp(t, regexp.MustCompile(`old[\s\S]*Revoked`), body)
+	assert.Regexp(t, regexp.MustCompile(`fresh[\s\S]*Never connected`), body)
+	assert.Contains(t, body, "Manager")
+	assert.Contains(t, body, "3 machines", "revoked agents are not counted")
+	assert.NotContains(t, body, `class="drawer"`, "the add panel is closed by default")
+}
+
+func TestAgentsFragment_RefreshesHeaderCount(t *testing.T) {
+	srv, _, _, _ := newTestServer(t)
+	code, body := get(t, srv, "/fragments/agents")
+	assert.Equal(t, http.StatusOK, code)
+	assert.Contains(t, body, `id="agents-sub" hx-swap-oob="true">3 machines`)
+}
+
+func TestAgentsPage_RevokeUsesPopover(t *testing.T) {
+	srv, _, _, _ := newTestServer(t)
+	_, body := get(t, srv, "/agents")
+	assert.Contains(t, body, `<details class="rv" data-id="2">`)
+	assert.Contains(t, body, `action="/agents/2/revoke"`)
+	assert.NotContains(t, body, "confirm(")
+	assert.NotContains(t, body, `action="/agents/4/revoke"`, "a revoked agent cannot be revoked again")
+}
+
+func TestAgentsPage_MasterVersionAndOldAgent(t *testing.T) {
+	cases := []struct {
+		name, version string
+		wantOld       bool
+	}{
+		{"other version is old", "v1.4.0", true},
+		{"dev master marks nothing", "dev", false},
+		{"unknown master version marks nothing", "", false},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			srv, _, _, _, _ := newTestEnvVersion(t, c.version)
+			_, body := get(t, srv, "/agents")
+			assert.Equal(t, c.version != "", strings.Contains(body, "master "+c.version))
+			assert.Equal(t, c.wantOld, strings.Contains(body, `<div class="ver old">`+"<span class=\"k\">Agent</span>v1.3.2"), "node-3 runs v1.3.2")
+			assert.NotContains(t, body, `<div class="ver old"><span class="k">Agent</span>v1.4.0`, "node-2 is current")
+		})
+	}
+}
+
+func TestAgentsPage_AddPanelOpensWithQuery(t *testing.T) {
+	srv, _, _, _ := newTestServer(t)
+	_, body := get(t, srv, "/agents?add=1")
+	assert.Contains(t, body, `class="drawer"`)
+	assert.Contains(t, body, `<form method="post" action="/agents"`)
+	assert.Contains(t, body, `name="csrf_token" value="csrf"`)
 }
 
 func TestCreateAgent_ShowsSnippetOnceAndStoresHash(t *testing.T) {
@@ -309,6 +422,10 @@ func TestCreateAgent_ShowsSnippetOnceAndStoresHash(t *testing.T) {
 	assert.Contains(t, string(body), "MASTER_KEY: sha256:abc")
 	assert.Contains(t, string(body), "MASTER_URL: wss://10.0.0.5:8090/agent")
 	assert.Contains(t, string(body), "shown only once")
+	assert.Contains(t, string(body), `<pre id="snippet">`)
+	assert.Contains(t, string(body), `data-copy="`+token+`"`)
+	assert.Contains(t, string(body), `hx-get="/fragments/agents/9/connect"`, "step 3 waits for the agent")
+	assert.Contains(t, string(body), ">node-3</b>", "the agent list is still on the page")
 }
 
 func TestCreateAgent_Validation(t *testing.T) {
@@ -348,16 +465,16 @@ func TestRevokeAgent_RevokesAndDisconnects(t *testing.T) {
 	assert.Equal(t, int64(2), store.revoked())
 	assert.Equal(t, int64(2), hub.lastDisconnected(), "a revoked agent must be cut off immediately")
 	var got []events.Topic
-	for len(got) < 4 {
+	for len(got) < 5 {
 		select {
 		case topic := <-ch:
 			got = append(got, topic)
 		default:
-			t.Fatalf("revoke published %v, want services, jobs and one topic per failed job", got)
+			t.Fatalf("revoke published %v, want agents, services, jobs and one topic per failed job", got)
 		}
 	}
-	assert.ElementsMatch(t, []events.Topic{events.Services, events.Jobs, events.JobTopic(5), events.JobTopic(6)}, got,
+	assert.ElementsMatch(t, []events.Topic{events.Agents, events.Services, events.Jobs, events.JobTopic(5), events.JobTopic(6)}, got,
-		"the overview drops the agent's services; job lists and open job pages show the failures")
+		"the agent list and sidebar change; the overview drops the agent's services; job lists and open job pages show the failures")
 }
 
 func TestRevokeAgent_BadID(t *testing.T) {
@@ -403,6 +520,7 @@ func TestAgentSnippet(t *testing.T) {
 	assert.Contains(t, snippet, "UPDATE_TIMEOUT: 5m")
 	assert.Contains(t, snippet, "      # DOCKER_CONFIG: /docker-config\n")
 	assert.Contains(t, snippet, "      # - /path/to/stacks:/path/to/stacks  # mount each stacks directory at the same path, read-write\n")
+	assert.Contains(t, snippet, "also this agent's own stack directory")
 	assert.Contains(t, snippet, "AGENT_TOKEN: tok")
 }
 
@@ -427,7 +545,7 @@ func TestServicePage_ShowsUpdateButton(t *testing.T) {
 }
 
 func TestServicePage_DisabledReason(t *testing.T) {
-	for _, reason := range []string{model.ReasonSelf, model.ReasonContainer, model.ReasonNoTag, model.ReasonRemoved, model.ReasonBuilt} {
+	for _, reason := range []string{model.ReasonContainer, model.ReasonNoTag, model.ReasonRemoved, model.ReasonBuilt} {
 		t.Run(reason, func(t *testing.T) {
 			srv, _, _, _, jobs := newTestEnv(t)
 			jobs.updatable = &model.NotUpdatableError{Reason: reason}
@@ -595,6 +713,47 @@ func TestSetMode_ReturnToOverview(t *testing.T) {
 	assert.Equal(t, "auto", store.modeOf(1))
 }
 
+func postModeHTMX(t *testing.T, srv *httptest.Server, id string, form url.Values) (*http.Response, string) {
+	t.Helper()
+	form.Set("csrf_token", "csrf")
+	req, err := http.NewRequest(http.MethodPost, srv.URL+"/services/"+id+"/mode", strings.NewReader(form.Encode()))
+	require.NoError(t, err)
+	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req.Header.Set("HX-Request", "true")
+	resp, err := noRedirectClient().Do(req)
+	require.NoError(t, err)
+	defer resp.Body.Close()
+	body, _ := io.ReadAll(resp.Body)
+	return resp, string(body)
+}
+
+func TestSetMode_HTMXFromOverviewReturnsServicesFragment(t *testing.T) {
+	srv, store, _, broker, _ := newTestEnv(t)
+	sub, cancel := broker.Subscribe()
+	defer cancel()
+
+	resp, body := postModeHTMX(t, srv, "1", url.Values{"mode": {"auto"}, "return": {"overview"}})
+
+	assert.Equal(t, http.StatusOK, resp.StatusCode, "no redirect, so the page keeps its scroll position")
+	assert.Empty(t, resp.Header.Get("Location"))
+	assert.NotContains(t, body, "<html")
+	assert.Contains(t, body, `id="machines" hx-swap-oob`)
+	assert.Contains(t, body, `action="/services/1/mode"`)
+	assert.Equal(t, "auto", store.modeOf(1))
+	assert.Equal(t, events.Services, <-sub)
+}
+
+func TestSetMode_HTMXRefusedRedirectsToService(t *testing.T) {
+	srv, store, _, _, jobs := newTestEnv(t)
+	jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonBuilt}
+
+	resp, _ := postModeHTMX(t, srv, "1", url.Values{"mode": {"auto"}, "return": {"overview"}})
+
+	assert.Equal(t, http.StatusConflict, resp.StatusCode)
+	assert.Equal(t, "/services/1", resp.Header.Get("HX-Redirect"), "htmx does not swap a 409, so send the user to the page that shows why")
+	assert.Equal(t, "manual", store.modeOf(1))
+}
+
 func TestSetMode_AutoRefusedShows409(t *testing.T) {
 	srv, store, _, _, jobs := newTestEnv(t)
 	jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonBuilt}
@@ -640,16 +799,20 @@ func TestOverview_ShowsModeToggleAndAvailableBadge(t *testing.T) {
 	srv, store, _, _, jobs := newTestEnv(t)
 	store.services[0].AvailableRef = "gitea.shcizo.se/shcizo/slusk:1.5.0"
 	store.services[1].UpdateMode = "auto"
-	jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonSelf}
+	jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonBuilt}
 
 	for _, path := range []string{"/", "/fragments/services"} {
 		_, body := get(t, srv, path)
 		assert.Contains(t, body, `action="/services/1/mode"`, path)
 		assert.Contains(t, body, `name="csrf_token" value="csrf"`, path)
 		assert.Contains(t, body, `name="return" value="overview"`, path)
-		assert.Contains(t, body, `class="badge badge-update" title="gitea.shcizo.se/shcizo/slusk:1.5.0"`, path)
+		assert.Contains(t, body, `class="tag f-up" title="gitea.shcizo.se/shcizo/slusk:1.5.0">Update available<em>1.5.0</em>`, path)
-		assert.Contains(t, body, `<input type="hidden" name="mode" value="manual">`, "auto services toggle to manual: "+path)
+		assert.Contains(t, body, `<input type="hidden" name="mode" value="manual">`+"\n"+`        <button type="submit" class="sw" role="switch" aria-checked="true">`, "auto services toggle to manual: "+path)
-		assert.Contains(t, body, `disabled>Make auto</button> <span class="muted">`+model.ReasonSelf+`</span>`, "blocked manual services get a disabled button: "+path)
+		assert.Contains(t, body, `<input type="hidden" name="mode" value="auto">`, "manual services toggle to auto: "+path)
+		assert.Contains(t, body, `aria-checked="false" disabled title="`+model.ReasonBuilt+`">`, "blocked manual services get a disabled switch: "+path)
+		assert.Contains(t, body, `<span class="na">`+model.ReasonBuilt+`</span>`, "and say why: "+path)
+		assert.Contains(t, body, `action="/services/3/mode"`, "the agent's own service has a switch like any other: "+path)
+		assert.NotContains(t, body, `<span class="na">n/a</span>`, path)
 	}
 }
 
@@ -674,3 +837,34 @@ func TestServicePage_AutoBlockedDisablesToggle(t *testing.T) {
 	_, body := get(t, srv, "/services/1")
 	assert.Contains(t, body, "Auto unavailable: "+model.ReasonContainer+".")
 }
+
+// GetTagVersion returns the digest-less version of repo with this tag.
+func (f *fakeStore) GetTagVersion(_ context.Context, repo, tag string) (model.ImageVersion, error) {
+	for _, v := range f.versions {
+		if v.Repo == repo && v.Tag == tag && v.Digest == "" {
+			return v, nil
+		}
+	}
+	return model.ImageVersion{}, model.ErrNotFound
+}
+
+func TestOverview_ShowsNewerTagChipBesideUpdateChip(t *testing.T) {
+	srv, store, _, _, _ := newTestEnv(t)
+	store.services[0].AvailableRef = "gitea.shcizo.se/shcizo/slusk:1.4.2"
+	store.services[0].NewerRef = "gitea.shcizo.se/shcizo/slusk:1.5.0"
+
+	for _, path := range []string{"/", "/fragments/services"} {
+		_, body := get(t, srv, path)
+		assert.Contains(t, body, `class="tag f-newer" title="gitea.shcizo.se/shcizo/slusk:1.5.0">Newer tag<em>1.5.0</em>`, path)
+		assert.Contains(t, body, `class="tag f-up"`, path)
+	}
+}
+
+func TestOverview_HidesStaleNewerTag(t *testing.T) {
+	srv, store, _, _, _ := newTestEnv(t)
+	store.services[0].NewerRef = "gitea.shcizo.se/shcizo/slusk:1.4.1" // not newer than 1.4.2
+
+	_, body := get(t, srv, "/")
+
+	assert.NotContains(t, body, "f-newer")
+}

internal/master/web/static/OFL-jetbrains-mono.txt

@@ -0,0 +1,93 @@
+Copyright 2020 The JetBrains Mono Project Authors (https://github.com/JetBrains/JetBrainsMono) JetBrainsMono-Italic[wght].ttf: Copyright 2020 The JetBrains Mono Project Authors (https://github.com/JetBrains/JetBrainsMono)
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded,
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.

internal/master/web/static/OFL-schibsted-grotesk.txt

@@ -0,0 +1,93 @@
+Copyright 2023 The Schibsted-Grotesk Project Authors (https://github.com/schibsted/schibsted-grotesk) SchibstedGrotesk-Italic[wght].ttf: Copyright 2023 The Schibsted-Grotesk Project Authors (https://github.com/schibsted/schibsted-grotesk)
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded,
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.

internal/master/web/static/app.css

@@ -1,34 +1,224 @@
-:root { --bg: #111418; --fg: #e6e8eb; --muted: #8a929c; --line: #262b31; --ok: #3fb950; --warn: #d29922; --bad: #f85149; --accent: #58a6ff; }
+@font-face{font-family:'Schibsted Grotesk';font-style:normal;font-weight:400 900;font-display:swap;src:url(/static/schibsted-grotesk-latin-wght-normal.woff2) format('woff2')}
-@media (prefers-color-scheme: light) {
+@font-face{font-family:'JetBrains Mono';font-style:normal;font-weight:100 800;font-display:swap;src:url(/static/jetbrains-mono-latin-wght-normal.woff2) format('woff2')}
-  :root { --bg: #ffffff; --fg: #1f2328; --muted: #59636e; --line: #d1d9e0; --ok: #1a7f37; --warn: #9a6700; --bad: #cf222e; --accent: #0969da; }
+
+/* Ledger. Tokens live on <html class="app">; a script in <head> adds t-light before first paint. */
+.app{color-scheme:dark;--bg:#0f1218;--side:#0b0d12;--card:#151a23;--hover:#171c26;--line:#232a37;--line2:#323a4a;--fg:#e7eaf1;--mut:#a0a9bb;--dim:#828ba0;--blue:#74a8ff;--bluebg:rgba(116,168,255,.12);--orange:#ffa262;--orangebg:rgba(255,162,98,.12);--green:#62d4a0;--red:#ff7d88;--redbg:rgba(255,125,136,.10);--violet:#c5a9ff;--violetbg:rgba(197,169,255,.12);--l0:#74a8ff;--l1:#c5a9ff;--l2:#5fd0e0;--l3:#f29bd0;--onblue:#06142e;--term:#080a0e;--shadow:0 12px 32px rgba(0,0,0,.5);--sans:'Schibsted Grotesk',system-ui,-apple-system,sans-serif;--mono:'JetBrains Mono',ui-monospace,Menlo,monospace;min-height:100vh;background:var(--bg);color:var(--fg);font:14px/1.45 var(--sans)}
+.app.t-light{color-scheme:light;--bg:#f7f8fa;--side:#eceff4;--card:#ffffff;--hover:#f0f2f6;--line:#dde1e8;--line2:#c6ccd7;--fg:#121620;--mut:#4b5468;--dim:#5c6579;--blue:#2156c8;--bluebg:rgba(33,86,200,.08);--orange:#a84b0b;--orangebg:rgba(200,96,20,.09);--green:#177649;--red:#b32a3b;--redbg:rgba(179,42,59,.07);--violet:#6943c1;--violetbg:rgba(105,67,193,.08);--l0:#2156c8;--l1:#6943c1;--l2:#0b7285;--l3:#a8306f;--onblue:#ffffff;--term:#121620;--shadow:0 12px 32px rgba(20,30,50,.16)}
+*{box-sizing:border-box}
+body{margin:0}
+[hidden]{display:none !important}
+a{color:var(--blue)}a:hover{color:var(--fg)}
+button,input{font-family:inherit}
+:focus-visible{outline:2px solid var(--blue);outline-offset:2px}
+
+/* Shell */
+.shell{display:flex;flex-wrap:wrap;min-height:100vh}
+.side{flex:0 0 232px;background:var(--side);border-right:1px solid var(--line);display:flex;flex-direction:column;gap:28px;padding:20px 14px}
+.brand{display:flex;align-items:center;gap:10px;font:700 15px var(--sans);letter-spacing:-.01em;padding:0 8px;color:var(--fg);text-decoration:none}
+.brand svg{color:var(--blue)}
+.snav{display:flex;flex-direction:column;gap:2px}
+.snav a,.mlist a{display:flex;align-items:center;gap:10px;padding:7px 8px;border-radius:6px;color:var(--mut);text-decoration:none;font-weight:500}
+.snav a:hover,.mlist a:hover{background:var(--hover);color:var(--fg)}
+.snav a.cur,.mlist a.cur{background:var(--card);color:var(--fg);box-shadow:inset 0 0 0 1px var(--line)}
+.snav .n,.mlist .n{margin-left:auto;font:12px var(--mono);color:var(--dim)}
+.slabel{font:600 11px var(--sans);text-transform:uppercase;letter-spacing:.08em;color:var(--dim);padding:0 8px 6px}
+.mlist{display:flex;flex-direction:column;gap:2px}
+.mlist .sub{padding-left:22px;font-size:13px}
+.mdot{width:7px;height:7px;border-radius:50%;background:var(--green);flex:none}
+.mdot.off{background:transparent;box-shadow:inset 0 0 0 1.5px var(--red)}
+.sw-icon{width:7px;height:7px;flex:none;border-radius:2px;box-shadow:inset 0 0 0 1.5px var(--blue)}
+.livebox{margin-top:auto;padding:10px 8px;border-top:1px solid var(--line);font-size:12px;color:var(--mut);display:flex;flex-direction:column;gap:2px}
+.livebox b{display:flex;align-items:center;gap:8px;color:var(--fg);font-weight:600}
+.pulse{width:8px;height:8px;border-radius:50%;background:var(--green);animation:breathe 2.4s ease-in-out infinite}
+.livebox.down .pulse{background:var(--red);animation:none}
+@keyframes breathe{0%{box-shadow:0 0 0 0 rgba(98,212,160,.45)}50%{box-shadow:0 0 0 5px rgba(98,212,160,0)}}
+.sfoot{display:flex;flex-wrap:wrap;align-items:center;gap:4px 8px}
+.sfoot form{margin:0}
+.work{flex:999 1 640px;min-width:0;display:flex;flex-wrap:wrap}
+.main{flex:999 1 520px;min-width:0;padding:28px 36px 48px;container-type:inline-size}
+.sprite{position:absolute;width:0;height:0;overflow:hidden}
+
+/* Page header, buttons */
+.hdr{display:flex;flex-wrap:wrap;align-items:flex-end;gap:12px 24px;margin-bottom:18px}
+.hdr h1{margin:0;font:700 34px/1 var(--sans);letter-spacing:-.025em}
+.hdr .sub{font-size:14px;color:var(--mut)}
+.hdr .end{margin-left:auto}
+.btn,button:not([class]){height:32px;padding:0 14px;border-radius:6px;border:1px solid var(--line2);background:var(--card);color:var(--fg);font:600 13px var(--sans);cursor:pointer;display:inline-flex;align-items:center;gap:8px;white-space:nowrap;text-decoration:none}
+.btn:hover,button:not([class]):hover{border-color:var(--dim);color:var(--fg)}
+button:not([class]):disabled{opacity:.5;cursor:not-allowed}
+.btn.pri,.btn.pri:hover{background:var(--blue);border-color:var(--blue);color:var(--onblue)}
+.btn.sm{height:28px;padding:0 10px;font-size:12.5px}
+.btn.quiet{border-color:transparent;background:transparent;color:var(--mut)}
+.btn.quiet:hover{color:var(--fg)}
+.btn.red{color:var(--red);border-color:var(--line2);background:transparent}
+.btn.redfill,.btn.redfill:hover{background:var(--red);border-color:var(--red);color:#1d0408}
+summary.btn{list-style:none}
+summary.btn::-webkit-details-marker{display:none}
+
+/* Overview */
+.tools{display:flex;flex-wrap:wrap;align-items:center;gap:10px 18px;padding:12px 0;border-top:1px solid var(--line);border-bottom:1px solid var(--line);margin-bottom:8px}
+.search{flex:0 1 340px;min-width:220px;display:flex;align-items:center;gap:8px;height:34px;padding:0 12px;border-radius:6px;background:var(--card);border:1px solid var(--line2);color:var(--dim)}
+.search input{flex:1;min-width:0;border:0;outline:0;background:transparent;color:var(--fg);font-size:14px}
+.search input::placeholder{color:var(--dim)}
+.tools .end{margin-left:auto;display:flex;align-items:center;gap:10px}
+.tools .lbl{font-size:12.5px;color:var(--dim)}
+.seg{display:inline-flex;padding:2px;border-radius:7px;background:var(--card);box-shadow:inset 0 0 0 1px var(--line2)}
+.seg button{display:inline-flex;align-items:center;gap:6px;height:28px;padding:0 10px;border:0;border-radius:5px;background:transparent;color:var(--mut);font:500 12.5px var(--sans);cursor:pointer}
+.seg button:hover{color:var(--fg)}
+.seg button[aria-pressed=true]{background:var(--hover);color:var(--fg);box-shadow:inset 0 0 0 1px var(--line2)}
+/* htmx inserts the new list before removing the old one; scroll anchoring on the
+   focused switch would scroll by the list height and then land at the bottom. */
+#services{overflow-anchor:none}
+.grp{margin-top:28px}
+.gh{display:flex;flex-wrap:wrap;align-items:baseline;gap:4px 12px;padding-bottom:8px;border-bottom:2px solid var(--fg)}
+.gh h2{margin:0;font:700 20px var(--sans);letter-spacing:-.015em}
+.gh .kind{font:600 11px var(--sans);text-transform:uppercase;letter-spacing:.08em;color:var(--mut)}
+.gh .meta{font-size:13px;color:var(--dim)}
+.gh .warn{font-size:13px;color:var(--red);font-weight:500}
+.gh .count{margin-left:auto;font:12px var(--mono);color:var(--dim)}
+.grp.offline .gh{border-bottom-color:var(--line2)}
+.grp.offline .row{color:var(--dim)}
+.row{position:relative;display:grid;grid-template-columns:minmax(200px,1.4fr) minmax(200px,2fr) 130px minmax(160px,1.5fr) minmax(104px,150px);align-items:center;column-gap:18px;min-height:42px;padding:0 6px;border-bottom:1px solid var(--line)}
+.row:hover{background:var(--hover)}
+.nm{display:flex;align-items:center;gap:8px;min-width:0}
+.nm a{display:inline-flex;align-items:center;gap:10px;min-width:0;color:var(--fg);font-weight:600;text-decoration:none;white-space:nowrap}
+.nm a:hover .nmt{text-decoration:underline;text-underline-offset:3px}
+.nm .nmt{overflow:hidden;text-overflow:ellipsis}
+.nm .stk{font-size:12.5px;color:var(--dim);white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
+.ico{flex:none;display:grid;place-items:center;width:26px;height:26px;border-radius:7px;background:var(--card);box-shadow:inset 0 0 0 1px var(--line2);color:var(--fg)}
+.ico svg{fill:currentColor}
+.ltr{flex:none;display:grid;place-items:center;width:26px;height:26px;border-radius:50%;font:700 12.5px var(--sans)}
+.lc-0{color:var(--l0);background:color-mix(in srgb,var(--l0) 18%,transparent)}
+.lc-1{color:var(--l1);background:color-mix(in srgb,var(--l1) 18%,transparent)}
+.lc-2{color:var(--l2);background:color-mix(in srgb,var(--l2) 18%,transparent)}
+.lc-3{color:var(--l3);background:color-mix(in srgb,var(--l3) 18%,transparent)}
+.grp.offline .ico,.grp.offline .ltr{opacity:.65}
+.d-icon .nmt,.d-icon .stk{display:none}
+.d-text .ico,.d-text .ltr{display:none}
+.ver{display:flex;min-width:0;font:12.5px var(--mono);color:var(--dim);white-space:nowrap}
+.ver .r{overflow:hidden;text-overflow:ellipsis;min-width:0}
+.ver .t{flex:none}
+.ver b{color:var(--fg);font-weight:500}
+.st{display:inline-flex;align-items:center;gap:8px;font-size:13px;font-weight:500;white-space:nowrap}
+.st svg{flex:none}
+.st circle{fill:currentColor}
+.s-running{color:var(--green)}.s-degraded{color:var(--orange)}.s-stopped{color:var(--dim)}.s-unknown{color:var(--dim)}
+.s-stopped circle,.s-unknown circle{fill:none}
+.flags{display:flex;flex-wrap:wrap;gap:6px}
+.tag{font:500 12px/20px var(--sans);padding:0 8px;border-radius:999px;white-space:nowrap}
+.tag em{font:500 11.5px var(--mono);font-style:normal;margin-left:6px}
+.tag em:empty{display:none}
+.f-up{color:var(--orange);background:var(--orangebg)}
+.f-newer em{color:var(--orange)}
+.f-err{color:var(--red);background:var(--redbg)}
+.f-newer,.f-notag{color:var(--mut);box-shadow:inset 0 0 0 1px var(--line2)}
+.f-out{color:var(--dim);box-shadow:inset 0 0 0 1px var(--line)}
+.mode{display:flex;flex-direction:column;align-items:flex-start;gap:2px}
+.mode form{margin:0;display:flex;flex-direction:column;align-items:flex-start;gap:2px}
+.sw{display:inline-flex;align-items:center;gap:8px;padding:0;border:0;background:none;font:inherit;font-size:13px;color:var(--mut);cursor:pointer}
+.sw i{flex:none;width:30px;height:18px;border-radius:999px;background:var(--line2);position:relative;transition:background .15s}
+.sw i::after{content:"";position:absolute;top:2px;left:2px;width:14px;height:14px;border-radius:50%;background:var(--fg);transition:transform .15s}
+.sw[aria-checked=true] i{background:var(--blue)}
+.sw[aria-checked=true] i::after{transform:translateX(12px);background:var(--onblue)}
+.sw[aria-checked=true]{color:var(--blue);font-weight:600}
+.sw:disabled{opacity:.55;cursor:not-allowed}
+.na{font-size:12.5px;color:var(--dim)}
+.nomatch{padding:24px 6px;color:var(--dim)}
+
+/* Agents */
+.hdr.ruled{padding-bottom:16px;border-bottom:2px solid var(--fg);margin-bottom:0}
+.agent{position:relative;display:grid;grid-template-columns:minmax(150px,1fr) 130px 90px 110px 120px 84px;grid-template-areas:"id st ver mgr last act";align-items:center;column-gap:18px;padding:16px 6px;border-bottom:1px solid var(--line)}
+.agent .id{grid-area:id}
+.agent .id b{display:block;font:700 16px var(--sans)}
+.agent .id span{font:12.5px var(--mono);color:var(--dim)}
+.agent .stc{grid-area:st}.agent .ver{grid-area:ver;display:block;font:13px var(--mono);color:var(--mut)}.agent .mgr{grid-area:mgr;font-size:13px;color:var(--mut)}.agent .last{grid-area:last;font-size:13px;color:var(--mut)}.agent .act{grid-area:act;display:flex;justify-content:flex-end}
+.agent .k{display:block;font:600 10.5px var(--sans);text-transform:uppercase;letter-spacing:.08em;color:var(--dim);margin-bottom:2px}
+.agent .ver.old{color:var(--orange)}
+.agent .st{font-size:13.5px;font-weight:600}
+.agent .st i{width:8px;height:8px;border-radius:50%;background:currentColor}
+.a-on{color:var(--green)}.a-off{color:var(--red)}.a-never,.a-revoked{color:var(--dim)}
+.a-off i,.a-never i{background:transparent !important;box-shadow:inset 0 0 0 1.5px currentColor}
+/* After the .agent rules: a container query adds no specificity, so it must come later to win. */
+@container (max-width:760px){
+  .agent{grid-template-columns:1fr auto;grid-template-areas:"id st" "ver mgr" "last act";row-gap:10px}
+}
+.pop{position:absolute;right:6px;top:calc(100% - 8px);z-index:5;width:360px;padding:14px 16px;border-radius:8px;background:var(--card);border:1px solid var(--line2);box-shadow:var(--shadow);font-size:13px;color:var(--mut)}
+.pop::before{content:"";position:absolute;top:-6px;right:30px;width:10px;height:10px;background:var(--card);border-left:1px solid var(--line2);border-top:1px solid var(--line2);transform:rotate(45deg)}
+.pop h4{margin:0 0 4px;font:700 14px var(--sans);color:var(--fg)}
+.pop p{margin:0 0 12px}
+.pop .acts{display:flex;gap:8px;justify-content:flex-end;margin:0}
+.drawer{flex:1 1 440px;min-width:0;max-width:520px;border-left:1px solid var(--line2);background:var(--card);padding:28px 28px 40px;display:flex;flex-direction:column;gap:18px}
+.dh{display:flex;align-items:center;gap:12px}
+.dh h2{margin:0;font:700 20px var(--sans);letter-spacing:-.015em}
+.dh .x{margin-left:auto}
+.steps{display:flex;gap:8px;font-size:12.5px;color:var(--dim)}
+.steps span{display:flex;align-items:center;gap:6px}
+.steps b{display:inline-grid;place-items:center;width:20px;height:20px;border-radius:50%;font:600 11px var(--mono);box-shadow:inset 0 0 0 1px var(--line2)}
+.steps .done{color:var(--green)}.steps .done b{box-shadow:none;background:var(--green);color:var(--bg)}
+.steps .now{color:var(--fg)}.steps .now b{box-shadow:none;background:var(--blue);color:var(--onblue)}
+.steps .sep{width:16px;height:1px;background:var(--line2)}
+.dform{display:flex;flex-direction:column;gap:18px;margin:0}
+.field{display:flex;flex-direction:column;gap:6px;font-size:13.5px;color:var(--mut)}
+.field input{height:36px;padding:0 12px;border-radius:6px;border:1px solid var(--line2);background:var(--bg);color:var(--fg);font:500 14px var(--mono)}
+.field small{color:var(--dim);font-size:12.5px}
+.namebox{display:flex;align-items:center;justify-content:space-between;padding:10px 12px;border-radius:6px;background:var(--bg);font-size:13.5px}
+.namebox code{font:500 14px var(--mono);color:var(--fg)}
+.notice{display:flex;gap:10px;padding:12px 14px;border-radius:6px;background:var(--orangebg);color:var(--fg);font-size:13.5px}
+.notice svg{flex:none;color:var(--orange);margin-top:1px}
+.code{border-radius:6px;background:var(--term);overflow:hidden}
+.code-h{display:flex;align-items:center;justify-content:space-between;padding:6px 8px 6px 12px;border-bottom:1px solid #1d2230;font:12px var(--mono);color:#8d96a8}
+.code pre{margin:0;padding:12px;border:0;color:#cdd3df;font:12px/1.6 var(--mono);overflow-x:auto}
+.secret{display:grid;grid-template-columns:auto 1fr auto;align-items:center;gap:8px 12px;font-size:13px;color:var(--mut)}
+.secret code{font:12.5px var(--mono);color:var(--fg);overflow-wrap:anywhere}
+.df{display:flex;gap:8px;justify-content:flex-end;align-items:center;margin-top:auto}
+.df form{margin:0}
+.df > span{margin-right:auto;font-size:13px;color:var(--mut);display:flex;align-items:center;gap:8px}
+.df .ok{color:var(--green)}
+.spin{width:12px;height:12px;border-radius:50%;box-shadow:inset 0 0 0 2px var(--line2);border-top:2px solid var(--blue);box-sizing:border-box;animation:spin 1s linear infinite}
+@keyframes spin{to{transform:rotate(360deg)}}
+
+/* Pages that keep the plain markup: service, jobs, job */
+h1{font:700 34px/1.1 var(--sans);letter-spacing:-.025em;margin:0 0 18px}
+h2{font:700 20px var(--sans);letter-spacing:-.015em;margin:28px 0 8px}
+table{width:100%;border-collapse:collapse;margin-bottom:24px}
+th,td{text-align:left;padding:6px 8px;border-bottom:1px solid var(--line)}
+th{color:var(--mut);font-weight:500}
+code,pre{font:13px var(--mono)}
+pre{padding:12px;border:1px solid var(--line);border-radius:6px;overflow-x:auto}
+.badge{font-size:12px;color:var(--mut);border:1px solid var(--line);padding:0 6px;border-radius:8px}
+.status-running{color:var(--green)}
+.status-degraded,.status-unknown{color:var(--orange)}
+.status-stopped{color:var(--red)}
+.error{color:var(--red)}
+.warning{color:var(--orange)}
+.muted{color:var(--dim)}
+.status-job-succeeded{color:var(--green)}
+.status-job-queued,.status-job-running{color:var(--orange)}
+.status-job-failed{color:var(--red)}
+table.details{width:auto}
+table.details th{padding-right:24px}
+pre.log{white-space:pre-wrap}
+
+@media (max-width:760px){
+  .side{flex:1 1 100%;flex-direction:row;flex-wrap:wrap;align-items:center;gap:8px 16px;padding:12px 16px;border-right:0;border-bottom:1px solid var(--line)}
+  #snav{order:3;width:100%}
+  .snav{flex-direction:row;overflow-x:auto}
+  .snav a{padding:10px 12px}
+  .snav .n{margin-left:4px}
+  .mblock{display:none}
+  .livebox{margin:0 0 0 auto;border:0;padding:0}
+  .livebox #live-age{display:none}
+  .main{padding:20px 16px 40px}
+  .hdr h1{font-size:28px}
+  .hdr .end,.tools .end{margin-left:0}
+  .search{flex:1 1 100%}
+  .btn,button:not([class]){height:44px}
+  .btn.sm{height:36px}
+  .row{grid-template-columns:1fr auto;grid-template-areas:"nm st" "ver ver" "fl fl" "mode mode";row-gap:8px;padding:12px 4px}
+  .nm{grid-area:nm}.stc{grid-area:st}.row .ver{grid-area:ver}.flags{grid-area:fl}.mode{grid-area:mode}
+  .flags:empty{display:none}
+  .agent{grid-template-columns:1fr auto;grid-template-areas:"id st" "ver mgr" "last act";row-gap:10px}
+  .drawer{order:-1;max-width:none;border-left:0;border-bottom:1px solid var(--line2);padding:20px 16px}
+  .pop{left:0;right:0;width:auto}
+  table{display:block;overflow-x:auto}
 }
-* { box-sizing: border-box; }
-body { margin: 0; background: var(--bg); color: var(--fg); font: 14px/1.5 system-ui, sans-serif; }
-nav { display: flex; gap: 16px; align-items: center; padding: 12px 16px; border-bottom: 1px solid var(--line); }
-nav form { margin-left: auto; }
-main { padding: 16px; max-width: 1200px; margin: 0 auto; }
-a { color: var(--accent); }
-table { width: 100%; border-collapse: collapse; margin-bottom: 24px; }
-th, td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--line); }
-th { color: var(--muted); font-weight: 500; }
-code, pre { font: 13px ui-monospace, monospace; }
-pre { padding: 12px; border: 1px solid var(--line); overflow-x: auto; }
-.badge { font-size: 12px; color: var(--muted); border: 1px solid var(--line); padding: 0 6px; border-radius: 8px; }
-.status-running { color: var(--ok); }
-.status-degraded, .status-unknown { color: var(--warn); }
-.status-stopped { color: var(--bad); }
-.error { color: var(--bad); }
-.warning { color: var(--warn); }
-form.inline { display: flex; gap: 8px; margin-bottom: 16px; }
-@media (max-width: 600px) { table { display: block; overflow-x: auto; } }
-.muted { color: var(--muted); }
-.status-job-succeeded { color: var(--ok); }
-.status-job-queued, .status-job-running { color: var(--warn); }
-.status-job-failed { color: var(--bad); }
-table.details { width: auto; }
-table.details th { padding-right: 24px; }
-pre.log { white-space: pre-wrap; }
-.badge-update { color: var(--warn); border-color: var(--warn); }
-.badge-error { color: var(--bad); border-color: var(--bad); }
-form.mode { display: inline; margin-left: 8px; }
-form.mode button { font-size: 12px; padding: 0 6px; }

internal/master/web/static/app.js

@@ -0,0 +1,226 @@
+// Behaviour the server-rendered pages need in the browser: theme, display
+// mode, search and machine filter, the Live box, local times, copy buttons and
+// the revoke popovers. The pages work without it; htmx and the SSE extension
+// do the live updates. Everything stored in the browser is wrapped in
+// try/catch because storage can be blocked.
+(function () {
+  'use strict';
+  var root = document.documentElement;
+
+  function load(area, key) {
+    try { return window[area].getItem(key); } catch (e) { return null; }
+  }
+  function save(area, key, value) {
+    try { window[area].setItem(key, value); } catch (e) { /* storage is blocked */ }
+  }
+  function each(selector, fn) {
+    Array.prototype.forEach.call(document.querySelectorAll(selector), fn);
+  }
+
+  // Theme: the head script set the initial class; the toggle overrides it.
+  var colorScheme = window.matchMedia ? window.matchMedia('(prefers-color-scheme: light)') : null;
+  var themeButton = document.getElementById('theme-toggle');
+  if (themeButton) {
+    themeButton.addEventListener('click', function () {
+      var light = !root.classList.contains('t-light');
+      root.classList.toggle('t-light', light);
+      themeButton.setAttribute('aria-pressed', String(light));
+      save('localStorage', 'pu-theme', light ? 'light' : 'dark');
+    });
+    themeButton.setAttribute('aria-pressed', String(root.classList.contains('t-light')));
+  }
+  if (colorScheme && colorScheme.addEventListener) {
+    colorScheme.addEventListener('change', function (e) {
+      if (!load('localStorage', 'pu-theme')) root.classList.toggle('t-light', e.matches);
+    });
+  }
+
+  // Display mode: icon, icon + text, or text.
+  function setDisplay(mode) {
+    ['icon', 'both', 'text'].forEach(function (m) { root.classList.toggle('d-' + m, m === mode); });
+    each('[data-display]', function (b) { b.setAttribute('aria-pressed', String(b.dataset.display === mode)); });
+  }
+  setDisplay(root.classList.contains('d-icon') ? 'icon' : root.classList.contains('d-text') ? 'text' : 'both');
+
+  // Search and machine filter. The rows are all in the page; this hides them.
+  var filter = { q: '', machine: '' };
+  try {
+    var saved = JSON.parse(load('sessionStorage', 'pu-filter') || '{}');
+    if (typeof saved.q === 'string') filter.q = saved.q;
+    if (typeof saved.machine === 'string') filter.machine = saved.machine;
+  } catch (e) { /* ignore a broken saved filter */ }
+  var search = document.getElementById('q');
+  if (search) search.value = filter.q;
+
+  // The machine list is hidden on narrow screens; a filter that cannot be
+  // seen or cleared there is ignored. Search still applies.
+  var narrow = window.matchMedia ? window.matchMedia('(max-width:760px)') : null;
+
+  function applyFilter() {
+    var box = document.getElementById('services');
+    if (!box) return;
+    var groups = box.querySelectorAll('.grp');
+    var known = Array.prototype.some.call(groups, function (g) { return g.dataset.key === filter.machine; });
+    if (filter.machine && !known) filter.machine = '';
+    var machine = narrow && narrow.matches ? '' : filter.machine;
+    var q = filter.q.trim().toLowerCase();
+    var any = false;
+    Array.prototype.forEach.call(groups, function (g) {
+      var inMachine = !machine || g.dataset.key === machine;
+      var shown = 0;
+      Array.prototype.forEach.call(g.querySelectorAll('.row'), function (row) {
+        var match = inMachine && (!q || (row.dataset.search || '').indexOf(q) !== -1);
+        row.hidden = !match;
+        if (match) shown++;
+      });
+      g.hidden = shown === 0;
+      if (shown > 0) any = true;
+    });
+    var none = document.getElementById('nomatch');
+    if (none) none.hidden = any || groups.length === 0;
+    each('#machines a', function (a) {
+      var cur = a.dataset.machine === filter.machine;
+      a.classList.toggle('cur', cur);
+      if (cur) a.setAttribute('aria-current', 'true'); else a.removeAttribute('aria-current');
+    });
+  }
+  if (narrow && narrow.addEventListener) narrow.addEventListener('change', function () { applyFilter(); });
+  function saveFilter() { save('sessionStorage', 'pu-filter', JSON.stringify(filter)); }
+
+  if (search) {
+    search.addEventListener('input', function () { filter.q = search.value; saveFilter(); applyFilter(); });
+  }
+
+  // Live box: counts any event on the stream, including the server's ping.
+  var live = document.getElementById('live');
+  var liveAge = document.getElementById('live-age');
+  var lastEvent = null;
+  var down = false;
+  var wired = typeof WeakSet === 'function' ? new WeakSet() : null;
+  function touch() { lastEvent = Date.now(); }
+  document.addEventListener('htmx:sseOpen', function (e) {
+    var source = e.detail && e.detail.source;
+    touch();
+    down = false;
+    if (source && wired && !wired.has(source)) {
+      wired.add(source);
+      ['ping', 'services', 'agents', 'jobs'].forEach(function (name) { source.addEventListener(name, touch); });
+    }
+  });
+  document.addEventListener('htmx:sseError', function () { down = true; });
+  function renderLive() {
+    if (!live || !liveAge) return;
+    live.classList.toggle('down', down);
+    if (down) { liveAge.textContent = 'Reconnecting…'; return; }
+    if (lastEvent === null) { liveAge.textContent = 'connecting…'; return; }
+    var s = Math.max(0, Math.round((Date.now() - lastEvent) / 1000));
+    liveAge.textContent = 'last event ' + (s < 60 ? s + ' s ago' : Math.floor(s / 60) + ' min ago');
+  }
+
+  // Times: the server writes <time datetime="..."> with its own clock as
+  // fallback text; show them in the browser's time zone.
+  var months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
+  function pad(n) { return (n < 10 ? '0' : '') + n; }
+  function localiseTimes() {
+    each('time[datetime]', function (t) {
+      var d = new Date(t.getAttribute('datetime'));
+      if (isNaN(d.getTime())) return;
+      var text;
+      if (t.hasAttribute('data-rel')) {
+        var min = Math.floor((Date.now() - d.getTime()) / 60000);
+        text = min < 1 ? 'just now' : min < 60 ? min + ' min ago' : min < 1440 ? Math.floor(min / 60) + ' h ago' : Math.floor(min / 1440) + ' d ago';
+        t.title = d.toLocaleString();
+      } else {
+        var clock = pad(d.getHours()) + ':' + pad(d.getMinutes());
+        text = d.toDateString() === new Date().toDateString() ? clock : months[d.getMonth()] + ' ' + d.getDate() + ' ' + clock;
+      }
+      if (t.textContent !== text) t.textContent = text;
+    });
+  }
+
+  // Copy buttons: data-copy holds the text, data-copy-target the id of the element to copy.
+  function flash(button, text) {
+    if (!button.dataset.label) button.dataset.label = button.textContent;
+    button.textContent = text;
+    setTimeout(function () { button.textContent = button.dataset.label; }, 2000);
+  }
+  function copyFallback(button, text) {
+    var target = button.dataset.copyTarget ? document.getElementById(button.dataset.copyTarget) : null;
+    if (target) {
+      var range = document.createRange();
+      range.selectNodeContents(target);
+      var selection = window.getSelection();
+      selection.removeAllRanges();
+      selection.addRange(range);
+      flash(button, 'Press Ctrl+C');
+      return;
+    }
+    var area = document.createElement('textarea');
+    area.value = text;
+    area.setAttribute('readonly', '');
+    area.style.position = 'fixed';
+    area.style.opacity = '0';
+    document.body.appendChild(area);
+    area.select();
+    var ok = false;
+    try { ok = document.execCommand('copy'); } catch (e) { /* not allowed */ }
+    document.body.removeChild(area);
+    flash(button, ok ? 'Copied' : 'Copy failed');
+  }
+
+  // Revoke popovers are <details>. A swap of the agent list would close them, so reopen those that were open.
+  var openRevokes = [];
+
+  document.addEventListener('click', function (e) {
+    var display = e.target.closest('[data-display]');
+    if (display) {
+      setDisplay(display.dataset.display);
+      save('localStorage', 'pu-display', display.dataset.display);
+      return;
+    }
+    var machine = e.target.closest('#machines a');
+    if (machine) {
+      e.preventDefault();
+      filter.machine = filter.machine === machine.dataset.machine ? '' : machine.dataset.machine;
+      saveFilter();
+      applyFilter();
+      return;
+    }
+    var close = e.target.closest('[data-close]');
+    if (close) {
+      var details = close.closest('details');
+      if (details) details.open = false;
+      return;
+    }
+    var copy = e.target.closest('[data-copy], [data-copy-target]');
+    if (copy) {
+      var text = copy.dataset.copy;
+      if (text === undefined) {
+        var el = document.getElementById(copy.dataset.copyTarget);
+        text = el ? el.textContent : '';
+      }
+      if (navigator.clipboard && navigator.clipboard.writeText) {
+        navigator.clipboard.writeText(text).then(function () { flash(copy, 'Copied'); }, function () { copyFallback(copy, text); });
+      } else {
+        copyFallback(copy, text);
+      }
+    }
+  });
+  document.addEventListener('keydown', function (e) {
+    if (e.key === 'Escape') each('details.rv[open]', function (d) { d.open = false; });
+  });
+  document.addEventListener('htmx:beforeSwap', function () {
+    openRevokes = [];
+    each('details.rv[open]', function (d) { openRevokes.push(d.dataset.id); });
+  });
+  document.addEventListener('htmx:afterSwap', function () {
+    each('details.rv', function (d) { if (openRevokes.indexOf(d.dataset.id) !== -1) d.open = true; });
+  });
+  document.addEventListener('htmx:afterSettle', function () { applyFilter(); localiseTimes(); });
+
+  applyFilter();
+  localiseTimes();
+  renderLive();
+  setInterval(renderLive, 1000);
+  setInterval(localiseTimes, 15000);
+})();

internal/master/web/templates/agent_created.html

@@ -1,6 +0,0 @@
-{{define "content"}}
-<h1>Agent "{{.Created.Name}}" created</h1>
-<p class="warning">The token is shown only once. Save this as <code>docker-compose.yml</code> on the machine and run <code>docker compose up -d</code>.</p>
-<pre>{{.Snippet}}</pre>
-<p><a href="/agents">Back to agents</a></p>
-{{end}}

internal/master/web/templates/agents.html

@@ -1,38 +1,108 @@
 {{define "content"}}
-<h1>Agents</h1>
+<div class="hdr ruled">
-<form method="post" action="/agents" class="inline">
+  <h1>Agents</h1>
-  <input type="hidden" name="csrf_token" value="{{.CSRF}}">
+  <span class="sub" id="agents-sub">{{template "agents-sub" .}}</span>
-  <label>Name <input name="name" required placeholder="node-2"></label>
+  <div class="end"><a class="btn pri" href="/agents?add=1">Add agent</a></div>
-  <button type="submit">Add agent</button>
+</div>
-</form>
-{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
 <div id="agents" hx-get="/fragments/agents" hx-trigger="sse:agents" hx-swap="innerHTML">
   {{template "agents" .}}
 </div>
 {{end}}
 
-{{define "agents"}}
+{{define "agents-sub"}}{{.Nav.AgentsTotal}} {{if eq .Nav.AgentsTotal 1}}machine{{else}}machines{{end}}{{if .MasterVersion}} · master {{.MasterVersion}}{{end}}{{end}}
-<table>
+
-  <thead><tr><th>Name</th><th>Hostname</th><th>State</th><th>Version</th><th>Swarm manager</th><th>Last seen</th><th></th></tr></thead>
+{{/* What /fragments/agents returns: the list, and the header count swapped in out of band. */}}
-  <tbody>
+{{define "agents-fragment"}}
-  {{range .Agents}}
+{{template "agents" .}}
-    <tr>
+<span class="sub" id="agents-sub" hx-swap-oob="true">{{template "agents-sub" .}}</span>
-      <td>{{.Name}}</td>
-      <td>{{.Hostname}}</td>
-      <td>{{if .RevokedAt}}<span class="status status-stopped">revoked</span>{{else if .Online}}<span class="status status-running">online</span>{{else}}<span class="status status-degraded">offline</span>{{end}}</td>
-      <td>{{.AgentVersion}}</td>
-      <td>{{if .SwarmManager}}yes{{end}}</td>
-      <td>{{with .LastSeenAt}}{{.Format "2006-01-02 15:04"}}{{end}}</td>
-      <td>{{if not .RevokedAt}}
-        <form method="post" action="/agents/{{.ID}}/revoke" onsubmit="return confirm('Revoke {{.Name}}? It is disconnected immediately and must be added again to come back.')">
-          <input type="hidden" name="csrf_token" value="{{$.CSRF}}">
-          <button type="submit">Revoke</button>
-        </form>
-      {{end}}</td>
-    </tr>
-  {{else}}
-    <tr><td colspan="7">No agents yet.</td></tr>
-  {{end}}
-  </tbody>
-</table>
 {{end}}
+
+{{define "agents"}}
+{{range .Agents}}
+<div class="agent">
+  <div class="id"><b>{{.Name}}</b><span>{{.Hostname}}</span></div>
+  <div class="stc"><span class="st {{.StatusClass}}"><i></i>{{.Status}}</span></div>
+  <div class="ver{{if .VersionOld}} old{{end}}"><span class="k">Agent</span>{{.AgentVersion}}</div>
+  <div class="mgr"><span class="k">Swarm</span>{{.Role}}</div>
+  <div class="last"><span class="k">Last seen</span>{{if .Online}}now{{else}}{{with .LastSeenAt}}<time datetime="{{.Format "2006-01-02T15:04:05Z07:00"}}" data-rel>{{.Format "2006-01-02 15:04"}}</time>{{else}}—{{end}}{{end}}</div>
+  <div class="act">{{if not .RevokedAt}}
+    <details class="rv" data-id="{{.ID}}">
+      <summary class="btn sm red">Revoke</summary>
+      <div class="pop" role="dialog" aria-label="Confirm revoke">
+        <h4>Revoke {{.Name}}?</h4>
+        <p>Its token stops working immediately. Services on {{.Name}} keep running, but drop out of package-updater.</p>
+        <form method="post" action="/agents/{{.ID}}/revoke" class="acts">
+          <input type="hidden" name="csrf_token" value="{{$.CSRF}}">
+          <button type="button" class="btn sm quiet" data-close>Cancel</button>
+          <button type="submit" class="btn sm redfill">Revoke</button>
+        </form>
+      </div>
+    </details>
+  {{end}}</div>
+</div>
+{{else}}
+<p class="nomatch">No agents yet.</p>
+{{end}}
+{{end}}
+
+{{/* The "add agent" panel beside the list. Step 1 asks for a name, step 2 shows the one-time token, step 3 waits for the agent to connect. */}}
+{{define "drawer"}}
+{{if .AddStep}}
+<section class="drawer" aria-label="Add agent">
+  <div class="dh">
+    <h2>Add agent</h2>
+    <a class="btn sm quiet x" href="/agents" aria-label="Close">
+      <svg width="14" height="14" viewBox="0 0 14 14" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true"><path d="M3 3l8 8M11 3l-8 8"></path></svg>
+    </a>
+  </div>
+  {{if eq .AddStep 1}}
+  <div class="steps" id="add-steps">{{template "steps" 1}}</div>
+  <form method="post" action="/agents" class="dform">
+    <input type="hidden" name="csrf_token" value="{{.CSRF}}">
+    <label class="field">Name
+      <input name="name" required autofocus autocomplete="off" placeholder="node-2" value="{{.FormName}}">
+      <small>Lowercase letters, digits and dashes. Use the machine's name.</small>
+    </label>
+    {{if .Error}}<p class="error" role="alert">{{.Error}}</p>{{end}}
+    <div class="df">
+      <a class="btn quiet" href="/agents">Cancel</a>
+      <button type="submit" class="btn pri">Create agent</button>
+    </div>
+  </form>
+  {{else}}
+  <div class="steps" id="add-steps">{{template "steps" 2}}</div>
+  <div class="namebox"><span>Name</span><code>{{.Created.Name}}</code></div>
+  <div class="notice" role="note">
+    <svg width="16" height="16" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.5" aria-hidden="true"><path d="M8 1.8L15 14H1L8 1.8z"></path><path d="M8 6.5v3.5M8 12v.01"></path></svg>
+    <span><b>This token is shown only once.</b> Save the snippet as <code>docker-compose.yml</code> on the machine and run <code>docker compose up -d</code>.</span>
+  </div>
+  <div class="code">
+    <div class="code-h"><span>docker-compose.yml</span><button type="button" class="btn sm" data-copy-target="snippet">Copy</button></div>
+    <pre id="snippet">{{.Snippet}}</pre>
+  </div>
+  <div class="secret">
+    <span>Token</span><code>{{middle .Token}}</code><button type="button" class="btn sm" data-copy="{{.Token}}">Copy</button>
+    <span>Master key</span><code>{{middle .MasterKey}}</code><button type="button" class="btn sm" data-copy="{{.MasterKey}}">Copy</button>
+  </div>
+  <div class="df" id="connect" hx-get="/fragments/agents/{{.Created.ID}}/connect" hx-trigger="load, sse:agents" hx-swap="innerHTML">
+    {{template "connect" .}}
+  </div>
+  {{end}}
+</section>
+{{end}}
+{{end}}
+
+{{/* step is the one in progress (1 to 3); 4 marks all three as done. */}}
+{{define "steps"}}<span class="{{if gt . 1}}done{{else}}now{{end}}"><b>1</b>Name</span><span class="sep"></span><span class="{{if gt . 2}}done{{else if eq . 2}}now{{end}}"><b>2</b>Deploy</span><span class="sep"></span><span class="{{if gt . 3}}done{{else if eq . 3}}now{{end}}"><b>3</b>Connect</span>{{end}}
+
+{{/* Step 3. Until the agent has connected, Cancel revokes the token that was just created. */}}
+{{define "connect"}}{{if .Connected}}<span class="ok">Connected · {{.Created.Hostname}} · {{.Created.AgentVersion}}</span>
+<a class="btn pri" href="/agents">Done</a>
+<div id="add-steps" hx-swap-oob="innerHTML">{{template "steps" 4}}</div>
+{{else}}<span><span class="spin" aria-hidden="true"></span>Waiting for {{.Created.Name}}…</span>
+<form method="post" action="/agents/{{.Created.ID}}/revoke">
+  <input type="hidden" name="csrf_token" value="{{.CSRF}}">
+  <button type="submit" class="btn quiet">Cancel</button>
+</form>
+<a class="btn pri" href="/agents">Done</a>
+{{end}}{{end}}

internal/master/web/templates/layout.html

@@ -1,23 +1,62 @@
 {{define "layout"}}<!doctype html>
-<html lang="en">
+<html lang="en" class="app">
 <head>
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width, initial-scale=1">
+  <meta name="color-scheme" content="dark light">
   <title>{{.Title}} · package-updater</title>
+  <script>
+    // Sets the theme and the display mode before first paint. app.js changes them later.
+    (function () {
+      var root = document.documentElement, theme = null, display = null;
+      try { theme = localStorage.getItem('pu-theme'); display = localStorage.getItem('pu-display'); } catch (e) {}
+      if (theme !== 'light' && theme !== 'dark') {
+        theme = window.matchMedia && window.matchMedia('(prefers-color-scheme: light)').matches ? 'light' : 'dark';
+      }
+      if (theme === 'light') root.classList.add('t-light');
+      root.classList.add(display === 'icon' || display === 'text' ? 'd-' + display : 'd-both');
+    })();
+  </script>
+  <link rel="preload" href="/static/schibsted-grotesk-latin-wght-normal.woff2" as="font" type="font/woff2" crossorigin>
   <link rel="stylesheet" href="/static/app.css">
   <script src="/static/htmx.min.js"></script>
   <script src="/static/htmx-ext-sse.js"></script>
+  <script src="/static/app.js" defer></script>
 </head>
 <body hx-headers='{"X-CSRF-Token": "{{.CSRF}}"}' hx-ext="sse" sse-connect="/events">
-  <nav>
+  <div class="shell">
-    <a href="/">Services</a>
+    <aside class="side">
-    <a href="/jobs">Jobs</a>
+      <div class="brand">
-    <a href="/agents">Agents</a>
+        <svg width="20" height="20" viewBox="0 0 20 20" fill="none" stroke="currentColor" stroke-width="1.7" aria-hidden="true"><path d="M3 6h14M3 10h9M3 14h6"></path><path d="M15 16V10m-2.5 2.5L15 10l2.5 2.5"></path></svg>
-    <form method="post" action="/auth/logout">
+        package-updater
-      <input type="hidden" name="csrf_token" value="{{.CSRF}}">
+      </div>
-      <button type="submit">Log out</button>
+      <div id="snav" hx-get="/fragments/nav?section={{.Nav.Section}}" hx-trigger="sse:services, sse:agents" hx-swap="innerHTML">{{template "nav" .}}</div>
-    </form>
+      {{block "machines" .}}{{end}}
-  </nav>
+      <div class="livebox" id="live">
-  <main>{{template "content" .}}</main>
+        <b><span class="pulse" aria-hidden="true"></span>Live</b>
+        <span id="live-age">connecting…</span>
+      </div>
+      <div class="sfoot">
+        <button type="button" class="btn sm quiet" id="theme-toggle">
+          <svg width="14" height="14" viewBox="0 0 14 14" fill="none" stroke="currentColor" stroke-width="1.5" aria-hidden="true"><circle cx="7" cy="7" r="5"></circle><path d="M7 2v10" ></path></svg>
+          Theme
+        </button>
+        <form method="post" action="/auth/logout">
+          <input type="hidden" name="csrf_token" value="{{.CSRF}}">
+          <button type="submit" class="btn sm quiet">Log out</button>
+        </form>
+      </div>
+    </aside>
+    <div class="work">
+      <main class="main">{{template "content" .}}</main>
+      {{block "drawer" .}}{{end}}
+    </div>
+  </div>
 </body>
 </html>{{end}}
+
+{{define "nav"}}<nav class="snav" aria-label="Main">
+  <a href="/"{{if eq .Nav.Section "services"}} class="cur" aria-current="page"{{end}}>Services <span class="n">{{.Nav.Services}}</span></a>
+  <a href="/jobs"{{if eq .Nav.Section "jobs"}} class="cur" aria-current="page"{{end}}>Jobs</a>
+  <a href="/agents"{{if eq .Nav.Section "agents"}} class="cur" aria-current="page"{{end}}>Agents <span class="n">{{.Nav.AgentsOnline}}/{{.Nav.AgentsTotal}}</span></a>
+</nav>{{end}}

internal/master/web/templates/overview.html

@@ -1,38 +1,103 @@
 {{define "content"}}
-<h1>Services</h1>
+<div class="hdr">
-<div id="services" hx-get="/fragments/services" hx-trigger="sse:services" hx-swap="innerHTML">
+  <h1>Services</h1>
+</div>
+<div class="tools">
+  <label class="search">
+    <svg width="15" height="15" viewBox="0 0 15 15" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true"><circle cx="6.5" cy="6.5" r="5"></circle><path d="M10.3 10.3L14 14"></path></svg>
+    <input type="search" id="q" placeholder="Search services" aria-label="Search services" autocomplete="off">
+  </label>
+  <div class="end">
+    <span class="lbl" id="showlbl">Show</span>
+    <div class="seg" role="group" aria-labelledby="showlbl">
+      <button type="button" data-display="icon" aria-pressed="false">
+        <svg width="14" height="14" viewBox="0 0 14 14" fill="none" stroke="currentColor" stroke-width="1.5" aria-hidden="true"><rect x="2" y="2" width="10" height="10" rx="2.5"></rect></svg>
+        Icon
+      </button>
+      <button type="button" data-display="both" aria-pressed="true">
+        <svg width="18" height="14" viewBox="0 0 18 14" fill="none" stroke="currentColor" stroke-width="1.5" aria-hidden="true"><rect x="1" y="3" width="8" height="8" rx="2"></rect><path d="M12 7h5"></path></svg>
+        Icon + text
+      </button>
+      <button type="button" data-display="text" aria-pressed="false">
+        <svg width="14" height="14" viewBox="0 0 14 14" fill="none" stroke="currentColor" stroke-width="1.5" aria-hidden="true"><path d="M2 4.5h10M2 9.5h7"></path></svg>
+        Text
+      </button>
+    </div>
+  </div>
+</div>
+<div id="services" hx-get="/fragments/services" hx-trigger="sse:services, sse:agents" hx-swap="innerHTML">
   {{template "services" .}}
 </div>
 {{end}}
 
-{{define "services"}}
+{{/* The machine filter in the sidebar. The page defines it; the services fragment replaces its links out of band. */}}
+{{define "machines"}}
+<div class="mblock">
+  <div class="slabel">Machines</div>
+  <nav class="mlist" id="machines" aria-label="Filter by machine">{{template "machine-links" .}}</nav>
+</div>
+{{end}}
+
+{{define "machine-links"}}
+<a class="cur" href="/" aria-current="true" data-machine="">All <span class="n">{{.ServiceCount}}</span></a>
 {{range .Groups}}
-<section>
+<a href="/"{{if .Sub}} class="sub"{{end}} data-machine="{{.Key}}">{{if .Cluster}}<span class="sw-icon" aria-hidden="true"></span>{{else}}<span class="mdot{{if .Offline}} off{{end}}" aria-hidden="true"></span>{{end}}{{.Title}} <span class="n">{{len .Services}}</span></a>
-  <h2>{{.Title}}</h2>
+{{end}}
-  <table>
+{{end}}
-    <thead><tr><th>Service</th><th>Image</th><th>Status</th><th>Mode</th></tr></thead>
+
-    <tbody>
+{{/* What /fragments/services returns: the groups, and the machine filter swapped in out of band. */}}
-    {{range .Services}}
+{{define "services-fragment"}}
-      <tr>
+{{template "services" .}}
-        <td><a href="/services/{{.ID}}">{{.DisplayName}}</a>{{if .Self}} <span class="badge">managed outside</span>{{end}}{{if .AvailableRef}} <span class="badge badge-update" title="{{.AvailableRef}}">update available</span>{{end}}{{if followsNoTag .Service}} <span class="badge">follows no tag</span>{{end}}{{if .LastPollError}} <span class="badge badge-error" title="{{.LastPollError}}">poll failed</span>{{end}}</td>
+<div id="machines" hx-swap-oob="innerHTML">{{template "machine-links" .}}</div>
-        <td><code>{{shortRef .ActualRef}}</code></td>
+{{end}}
-        <td class="status status-{{.Status}}">{{.Status}}</td>
+
-        <td>{{.UpdateMode}}
+{{define "services"}}
-          <form method="post" action="/services/{{.ID}}/mode" class="mode">
+{{if .Sprite}}<svg class="sprite" aria-hidden="true">{{range .Sprite}}<symbol id="si-{{.Slug}}" viewBox="0 0 24 24"><path fill="currentColor" d="{{.Path}}"/></symbol>{{end}}</svg>{{end}}
-            <input type="hidden" name="csrf_token" value="{{$.CSRF}}">
+{{range .Groups}}
-            <input type="hidden" name="return" value="overview">
+<section class="grp{{if .Offline}} offline{{end}}" data-key="{{.Key}}" aria-label="{{.Title}}">
-            {{if eq .UpdateMode "auto"}}<input type="hidden" name="mode" value="manual">
+  <div class="gh">
-            <button type="submit">Make manual</button>
+    <h2{{if .TitleHint}} title="{{.TitleHint}}"{{end}}>{{.Title}}</h2>
-            {{else}}<input type="hidden" name="mode" value="auto">
+    <span class="kind">{{.Kind}}</span>
-            <button type="submit"{{if .AutoBlocked}} disabled{{end}}>Make auto</button>{{if .AutoBlocked}} <span class="muted">{{.AutoBlocked}}</span>{{end}}{{end}}
+    {{if .Meta}}<span class="meta">{{.Meta}}</span>{{end}}
-          </form>
+    {{if .Offline}}<span class="warn">{{.OfflineLabel}}{{with .OfflineSince}} since <time datetime="{{.Format "2006-01-02T15:04:05Z07:00"}}">{{.Format "15:04"}}</time>{{end}} · showing last known state</span>{{end}}
-        </td>
+    <span class="count">{{len .Services}}</span>
-      </tr>
+  </div>
-    {{end}}
+  {{range .Services}}
-    </tbody>
+  <div class="row" data-search="{{.Search}}">
-  </table>
+    <div class="nm">
+      <a href="/services/{{.ID}}" title="{{.DisplayName}}" aria-label="{{.DisplayName}}">
+        {{if .Icon}}<span class="ico" aria-hidden="true"><svg width="15" height="15" viewBox="0 0 24 24"><use href="#si-{{.Icon}}"></use></svg></span>{{else}}<span class="ltr {{.LetterClass}}" aria-hidden="true">{{.Letter}}</span>{{end}}
+        <span class="nmt">{{.Name}}</span>
+      </a>
+      {{if .Stack}}<span class="stk">{{.Stack}}</span>{{end}}
+    </div>
+    <div class="ver" title="{{.ActualRef}}"><span class="r">{{.Repo}}</span><span class="t"><b>{{.Tag}}</b>{{.Digest}}</span></div>
+    <div class="stc">
+      <span class="st {{.StatusClass}}"><svg width="10" height="10" viewBox="0 0 10 10" aria-hidden="true"><circle cx="5" cy="5" r="4" stroke="currentColor" stroke-width="1.5"></circle></svg>{{.StatusLabel}}</span>
+    </div>
+    <div class="flags">
+      {{if .AvailableRef}}<span class="tag f-up" title="{{.AvailableRef}}">Update available<em>{{.AvailableTag}}</em></span>{{end}}
+      {{if .NewerTag}}<span class="tag f-newer" title="{{.NewerRef}}">Newer tag<em>{{.NewerTag}}</em></span>{{end}}
+      {{if followsNoTag .Service}}<span class="tag f-notag">Follows no tag</span>{{end}}
+      {{if .LastPollError}}<span class="tag f-err" title="{{.LastPollError}}">Poll failed</span>{{end}}
+      {{if .Self}}<span class="tag f-out">This agent</span>{{end}}
+    </div>
+    <div class="mode">
+      <form method="post" action="/services/{{.ID}}/mode" hx-post="/services/{{.ID}}/mode" hx-target="#services" hx-swap="innerHTML">
+        <input type="hidden" name="csrf_token" value="{{$.CSRF}}">
+        <input type="hidden" name="return" value="overview">
+        {{if eq .UpdateMode "auto"}}<input type="hidden" name="mode" value="manual">
+        <button type="submit" class="sw" role="switch" aria-checked="true"><i></i>Auto</button>
+        {{else}}<input type="hidden" name="mode" value="auto">
+        <button type="submit" class="sw" role="switch" aria-checked="false"{{if .AutoBlocked}} disabled title="{{.AutoBlocked}}"{{end}}><i></i>Manual</button>
+        {{if .AutoBlocked}}<span class="na">{{.AutoBlocked}}</span>{{end}}{{end}}
+      </form>
+    </div>
+  </div>
+  {{end}}
 </section>
 {{else}}
-<p>No services reported yet. <a href="/agents">Add an agent</a> to get started.</p>
+<p>No services reported yet. <a href="/agents?add=1">Add an agent</a> to get started.</p>
 {{end}}
+<p class="nomatch" id="nomatch" hidden>No services match.</p>
 {{end}}

internal/master/web/templates/service.html

@@ -9,6 +9,7 @@
   <tr><th>Status</th><td class="status status-{{.Service.Status}}">{{.Service.Status}}</td></tr>
   {{if .Service.DesiredRef}}<tr><th>Desired image</th><td><code>{{shortRef .Service.DesiredRef}}</code></td></tr>{{end}}
   {{if .Service.AvailableRef}}<tr><th>Available</th><td><code>{{.Service.AvailableRef}}</code> <span class="muted">via {{.Service.AvailableSource}}{{with .Service.AvailableAt}}, {{.Format "2006-01-02 15:04"}}{{end}}</span></td></tr>{{end}}
+  {{if .NewerTag}}<tr><th>Newer tag</th><td><code>{{.Service.NewerRef}}</code> <span class="muted">from the registry's tag list{{with .Service.NewerAt}}, {{.Format "2006-01-02 15:04"}}{{end}}; deploy it from Versions</span></td></tr>{{end}}
   <tr><th>Mode</th><td>{{.Service.UpdateMode}}</td></tr>
   <tr><th>Polling</th><td{{if .PollFailed}} class="error"{{end}}>{{.Polling}}</td></tr>
   {{if .ConfigFiles}}<tr><th>Config files</th><td>{{range .ConfigFiles}}<code>{{.}}</code><br>{{end}}</td></tr>{{end}}
@@ -22,6 +23,11 @@
 </form>
 <p class="muted">Pulls the current tag again and redeploys the service.</p>
 {{end}}
+<form method="post" action="/services/{{.Service.ID}}/poll">
+  <input type="hidden" name="csrf_token" value="{{.CSRF}}">
+  <button type="submit"{{if .PollBlocked}} disabled{{end}}>Check now</button>
+  <p class="muted">{{if .PollBlocked}}Check unavailable: {{.PollBlocked}}.{{else}}Asks the registry now instead of waiting for the next poll.{{end}}</p>
+</form>
 <form method="post" action="/services/{{.Service.ID}}/mode">
   <input type="hidden" name="csrf_token" value="{{.CSRF}}">
   {{if eq .Service.UpdateMode "auto"}}<input type="hidden" name="mode" value="manual">

internal/master/web/versions_test.go

@@ -68,7 +68,7 @@ func TestServicePage_VersionPickerWarnsForAutoServices(t *testing.T) {
 }
 
 func TestServicePage_VersionPickerHiddenWhenServiceCannotBeUpdated(t *testing.T) {
-	for _, reason := range []string{model.ReasonSelf, model.ReasonContainer, model.ReasonRemoved, model.ReasonBuilt} {
+	for _, reason := range []string{model.ReasonContainer, model.ReasonRemoved, model.ReasonBuilt} {
 		t.Run(reason, func(t *testing.T) {
 			srv, store, _, _, jobs := newTestEnv(t)
 			store.services[0].ImageRepo = sluskRepo
@@ -214,8 +214,58 @@ func TestOverview_ShowsFollowsNoTagAndPollFailedBadges(t *testing.T) {
 
 	for _, path := range []string{"/", "/fragments/services"} {
 		_, body := get(t, srv, path)
-		assert.Equal(t, 1, strings.Count(body, "follows no tag"), "only the digest-only Swarm service: "+path)
+		assert.Equal(t, 1, strings.Count(body, "Follows no tag"), "only the digest-only Swarm service: "+path)
-		assert.Contains(t, body, `class="badge badge-error" title="head x: &#34;boom&#34; &lt;script&gt;">poll failed</span>`, path)
+		assert.Contains(t, body, `class="tag f-err" title="head x: &#34;boom&#34; &lt;script&gt;">Poll failed</span>`, path)
-		assert.Equal(t, 1, strings.Count(body, "poll failed"), path)
+		assert.Equal(t, 1, strings.Count(body, "Poll failed"), path)
 	}
 }
+
+func TestServicePage_VersionPickerAppendsNewerTagBeyondLimit(t *testing.T) {
+	srv, store, _, _, _ := newTestEnv(t)
+	store.services[0].ImageRepo = sluskRepo
+	store.services[0].NewerRef = sluskRepo + ":1.5.0"
+	for i := int64(1); i <= 12; i++ {
+		store.versions = append(store.versions, model.ImageVersion{ID: i, Repo: sluskRepo, Tag: "t", Digest: "sha256:d", Source: "poll"})
+	}
+	store.versions = append(store.versions, model.ImageVersion{ID: 99, Repo: sluskRepo, Tag: "1.5.0", Source: "tags"})
+
+	_, body := get(t, srv, "/services/1")
+
+	assert.Equal(t, 11, strings.Count(body, `name="version"`), "the ten recent versions and the newer tag")
+	assert.Contains(t, body, `<input type="hidden" name="version" value="99">`)
+	assert.Contains(t, body, "<th>Newer tag</th>")
+}
+
+func TestServicePage_NewerTagAlreadyListedIsNotAppendedTwice(t *testing.T) {
+	srv, store, _, _, _ := newTestEnv(t)
+	store.services[0].ImageRepo = sluskRepo
+	store.services[0].NewerRef = sluskRepo + ":1.5.0"
+	store.versions = []model.ImageVersion{{ID: 31, Repo: sluskRepo, Tag: "1.5.0", Source: "tags"}}
+
+	_, body := get(t, srv, "/services/1")
+
+	assert.Equal(t, 1, strings.Count(body, `name="version"`))
+}
+
+func TestServicePage_PinnedRowOfTheNewerTagDoesNotHideThePlainRow(t *testing.T) {
+	srv, store, _, _, _ := newTestEnv(t)
+	store.services[0].ImageRepo = sluskRepo
+	store.services[0].NewerRef = sluskRepo + ":1.5.0"
+	store.versions = []model.ImageVersion{
+		{ID: 31, Repo: sluskRepo, Tag: "1.5.0", Digest: "sha256:d", Source: "poll"},
+		{ID: 32, Repo: sluskRepo, Tag: "1.5.0", Source: "tags"},
+	}
+
+	_, body := get(t, srv, "/services/1")
+
+	assert.Equal(t, 2, strings.Count(body, `name="version"`))
+	assert.Contains(t, body, `<input type="hidden" name="version" value="32">`)
+}
+
+func TestServicePage_NoNewerTagRowWithoutSuggestion(t *testing.T) {
+	srv, _, _, _, _ := newTestEnv(t)
+
+	_, body := get(t, srv, "/services/1")
+
+	assert.NotContains(t, body, "Newer tag")
+}

internal/master/web/views.go

@@ -1,58 +1,242 @@
 package web
 
 import (
+	"cmp"
+	"slices"
+	"strconv"
 	"strings"
 	"time"
+	"unicode/utf8"
 
 	"github.com/shcizo/package-updater/internal/imageref"
 	"github.com/shcizo/package-updater/internal/master/jobs"
 	"github.com/shcizo/package-updater/internal/master/model"
 	"github.com/shcizo/package-updater/internal/master/poller"
+	"github.com/shcizo/package-updater/internal/master/tagver"
+	"github.com/shcizo/package-updater/internal/master/web/icons"
 	"github.com/shcizo/package-updater/internal/protocol"
 )
 
+// serviceGroup is one block of the overview: a machine, a Swarm cluster or
+// the services whose agent is unknown.
 type serviceGroup struct {
-	Title    string
+	Key       string // "agent-<id>", "swarm-<cluster id>" or "unknown"; the machine filter matches on it
-	Services []serviceRow
+	Title     string
+	TitleHint string // hover text of the title; the full cluster id
+	Kind      string // "Machine" or "Swarm cluster"
+	Meta      string // hostname, or the cluster's managers
+	Cluster   bool
+	Sub       bool // a machine that belongs to the cluster shown above it
+	// Offline says the agent (or, for a cluster, every manager) is not
+	// connected, so the rows show the last known state.
+	Offline      bool
+	OfflineLabel string     // what the banner says before "since"
+	OfflineSince *time.Time // when the agent was last seen; nil if unknown
+	Services     []serviceRow
 }
 
-// serviceRow is a service as shown in the overview table.
+// serviceRow is a service as shown in the overview.
 type serviceRow struct {
 	model.Service
 	AutoBlocked string // why the service cannot be set to auto; "" when it can
+
+	Name         string // the service's own name, without its stack
+	Stack        string // the Compose project; "" for Swarm and plain containers
+	Repo         string // image repository without the registry host
+	Tag          string // ":tag", or ""
+	Digest       string // "@sha256:<12 hex>…" when the image has no tag, else ""
+	AvailableTag string // the tag of AvailableRef, or ""
+	NewerTag     string // the tag of NewerRef while it is still newer than the followed tag, or ""
+	Icon         string // Simple Icons slug; "" when there is no icon
+	Letter       string // shown instead of an icon
+	LetterClass  string // lc-0 .. lc-3
+	StatusLabel  string
+	StatusClass  string
+	Search       string // what the search box matches against, lowercased
 }
 
-// groupServices groups machine-bound services per agent and Swarm services
+// newServiceRow fills the display fields of a row.
-// in one group, keeping the store's order (machines first, Swarm last).
+func newServiceRow(s model.Service, autoBlocked string) serviceRow {
-// autoBlocked is called for every service and returns the reason it cannot be
+	r := serviceRow{Service: s, AutoBlocked: autoBlocked}
-// set to auto, or "" when it can.
+	switch s.Kind {
-func groupServices(services []model.Service, autoBlocked func(model.Service) string) []serviceGroup {
+	case protocol.KindCompose:
-	var groups []serviceGroup
+		r.Name, r.Stack = s.ComposeService, s.ComposeProject
-	index := map[string]int{}
+	case protocol.KindSwarm:
+		r.Name = s.SwarmService
+	default:
+		r.Name = s.ContainerName
+	}
+	r.Repo, r.Tag, r.Digest = imageParts(s.ActualRef)
+	r.AvailableTag = imageref.Parse(s.AvailableRef).Tag
+	r.NewerTag = newerTag(s)
+	r.Icon = icons.ForImage(s.ActualRef)
+	r.Letter = "?"
+	if first, _ := utf8.DecodeRuneInString(r.Name); first != utf8.RuneError {
+		r.Letter = strings.ToUpper(string(first))
+	}
+	r.LetterClass = letterClass(r.Name)
+	switch s.Status {
+	case "running":
+		r.StatusLabel, r.StatusClass = "Running", "s-running"
+	case "degraded":
+		r.StatusLabel, r.StatusClass = "Degraded", "s-degraded"
+	case "stopped":
+		r.StatusLabel, r.StatusClass = "Stopped", "s-stopped"
+	default:
+		r.StatusLabel, r.StatusClass = "Unknown", "s-unknown"
+	}
+	r.Search = strings.ToLower(strings.Join([]string{r.Name, r.Stack, r.Repo, strings.TrimPrefix(r.Tag, ":"), s.AgentName}, " "))
+	return r
+}
+
+// imageParts splits an image reference for display: the repository without
+// its registry host, ":tag", and a shortened digest that is only given when
+// there is no tag.
+func imageParts(ref string) (repo, tag, digest string) {
+	r := imageref.Parse(ref)
+	repo = r.Repo
+	if host, rest, ok := strings.Cut(repo, "/"); ok && (imageref.IsRegistryHost(host)) {
+		repo = rest
+	}
+	if r.Tag != "" {
+		return repo, ":" + r.Tag, ""
+	}
+	if r.Digest != "" {
+		return repo, "", "@" + shortDigest(r.Digest)
+	}
+	return repo, "", ""
+}
+
+// shortDigest shortens "sha256:<hex>" to 12 hex characters and an ellipsis.
+func shortDigest(d string) string {
+	if hex, ok := strings.CutPrefix(d, "sha256:"); ok && len(hex) > 12 {
+		return "sha256:" + hex[:12] + "…"
+	}
+	return d
+}
+
+// letterClass picks one of four colours from the name, so a service keeps
+// its colour.
+func letterClass(name string) string {
+	sum := 0
+	for i := 0; i < len(name); i++ {
+		sum += int(name[i])
+	}
+	return "lc-" + strconv.Itoa(sum%4)
+}
+
+// groupServices groups the services into one block per Swarm cluster and per
+// machine. A cluster comes first, followed by the machines that belong to it;
+// the remaining machines follow by name and the services whose agent is
+// unknown come last. Agents without services get no block. online says
+// whether an agent is connected; autoBlocked returns the reason a service
+// cannot be set to auto, or "".
+func groupServices(services []model.Service, agents []model.Agent, online func(int64) bool, autoBlocked func(model.Service) string) []serviceGroup {
+	agentByID := map[int64]model.Agent{}
+	for _, a := range agents {
+		agentByID[a.ID] = a
+	}
+	var clusterIDs []string
+	clusterRows := map[string][]serviceRow{}
+	clusterRowAgents := map[string]map[int64]bool{} // agents named by a cluster's rows
+	machineRows := map[int64][]serviceRow{}
+	var unknownRows []serviceRow
 	for _, s := range services {
-		title := s.AgentName
+		row := newServiceRow(s, autoBlocked(s))
-		if s.Kind == protocol.KindSwarm {
+		_, known := agentByID[derefID(s.AgentID)]
-			title = "Swarm"
+		switch {
-		} else if title == "" {
+		case s.Kind == protocol.KindSwarm:
-			title = "Unknown agent"
+			if _, seen := clusterRows[s.SwarmClusterID]; !seen {
+				clusterIDs = append(clusterIDs, s.SwarmClusterID)
+				clusterRowAgents[s.SwarmClusterID] = map[int64]bool{}
+			}
+			clusterRows[s.SwarmClusterID] = append(clusterRows[s.SwarmClusterID], row)
+			if s.AgentID != nil {
+				clusterRowAgents[s.SwarmClusterID][*s.AgentID] = true
+			}
+		case s.AgentID != nil && known:
+			machineRows[*s.AgentID] = append(machineRows[*s.AgentID], row)
+		default:
+			unknownRows = append(unknownRows, row)
 		}
-		i, ok := index[title]
+	}
-		if !ok {
+
-			i = len(groups)
+	machineIDs := make([]int64, 0, len(machineRows))
-			index[title] = i
+	for id := range machineRows {
-			groups = append(groups, serviceGroup{Title: title})
+		machineIDs = append(machineIDs, id)
+	}
+	slices.SortFunc(machineIDs, func(a, b int64) int {
+		return cmp.Or(cmp.Compare(agentByID[a].Name, agentByID[b].Name), cmp.Compare(a, b))
+	})
+	machine := func(id int64, sub bool) serviceGroup {
+		a := agentByID[id]
+		return serviceGroup{
+			Key: "agent-" + strconv.FormatInt(id, 10), Title: a.Name, Kind: "Machine", Meta: a.Hostname, Sub: sub,
+			Offline: !online(id), OfflineLabel: "Agent offline", OfflineSince: a.LastSeenAt, Services: machineRows[id],
 		}
-		groups[i].Services = append(groups[i].Services, serviceRow{Service: s, AutoBlocked: autoBlocked(s)})
+	}
+
+	var groups []serviceGroup
+	placed := map[int64]bool{}
+	for _, cid := range clusterIDs {
+		title := "Swarm"
+		if len(clusterIDs) > 1 {
+			title += " " + cid[:min(len(cid), 8)]
+		}
+		g := serviceGroup{
+			Key: "swarm-" + cid, Title: title, TitleHint: cid, Kind: "Swarm cluster", Cluster: true,
+			Offline: true, OfflineLabel: "No manager online", Services: clusterRows[cid],
+		}
+		var names []string
+		for _, a := range agents {
+			isManager := a.SwarmManager && a.SwarmClusterID == cid
+			if a.RevokedAt != nil || !(isManager || clusterRowAgents[cid][a.ID]) {
+				continue
+			}
+			names = append(names, a.Name)
+			if online(a.ID) {
+				g.Offline = false
+			}
+			if a.LastSeenAt != nil && (g.OfflineSince == nil || a.LastSeenAt.After(*g.OfflineSince)) {
+				g.OfflineSince = a.LastSeenAt
+			}
+		}
+		slices.Sort(names)
+		if len(names) > 0 {
+			g.Meta = "managers " + strings.Join(names, ", ")
+		}
+		groups = append(groups, g)
+		for _, id := range machineIDs {
+			if cid != "" && agentByID[id].SwarmClusterID == cid && !placed[id] {
+				placed[id] = true
+				groups = append(groups, machine(id, true))
+			}
+		}
+	}
+	for _, id := range machineIDs {
+		if !placed[id] {
+			groups = append(groups, machine(id, false))
+		}
+	}
+	if len(unknownRows) > 0 {
+		groups = append(groups, serviceGroup{
+			Key: "unknown", Title: "Unknown agent", Kind: "Machine", Offline: true, OfflineLabel: "Agent offline", Services: unknownRows,
+		})
 	}
 	return groups
 }
 
+func derefID(id *int64) int64 {
+	if id == nil {
+		return 0
+	}
+	return *id
+}
+
 // shortRef shortens a digest in an image reference to 12 hex characters.
 func shortRef(ref string) string {
 	r := imageref.Parse(ref)
-	if hex, ok := strings.CutPrefix(r.Digest, "sha256:"); ok && len(hex) > 12 {
+	r.Digest = shortDigest(r.Digest)
-		r.Digest = "sha256:" + hex[:12] + "…"
-	}
 	return r.String()
 }
 
@@ -62,6 +246,26 @@ func followsNoTag(svc model.Service) bool {
 	return jobs.FollowedRef(svc.ActualRef) == ""
 }
 
+// newerTag returns the tag of the service's newer-tag suggestion, or "" when
+// there is none or it no longer holds. The poller writes the suggestion once
+// per cycle, so after the service moved to another tag the stored value can
+// be stale until the next cycle; checking again here hides it meanwhile. A
+// suggestion from another repo than the service follows now is hidden too.
+func newerTag(s model.Service) string {
+	if s.RemovedAt != nil || s.NewerRef == "" {
+		return ""
+	}
+	newer := imageref.Parse(s.NewerRef)
+	repo, followed := poller.Followed(s)
+	if newer.Repo != repo {
+		return ""
+	}
+	if _, ok := tagver.Newer(followed, []string{newer.Tag}); !ok {
+		return ""
+	}
+	return newer.Tag
+}
+
 // versionRow is an image version as shown on the service page.
 type versionRow struct {
 	model.ImageVersion
@@ -77,6 +281,15 @@ func versionRows(versions []model.ImageVersion, actualDigest string) []versionRo
 	return rows
 }
 
+// pollBlocked returns why the registry cannot be checked for svc, or "" when
+// it can.
+func pollBlocked(svc model.Service) string {
+	if svc.RemovedAt != nil {
+		return model.ReasonRemoved
+	}
+	return poller.SkipReason(svc)
+}
+
 // pollingStatus is the text of the "Polling" row on the service page and
 // whether it is a registry error.
 func pollingStatus(svc model.Service, interval time.Duration) (text string, failed bool) {
@@ -101,9 +314,48 @@ func shortDuration(d time.Duration) string {
 	return s
 }
 
+// agentRow is an agent as shown on the agents page.
 type agentRow struct {
 	model.Agent
-	Online bool
+	Online      bool
+	Status      string // Online, Offline, Never connected or Revoked
+	StatusClass string
+	Role        string // Manager, Worker or "—"
+	VersionOld  bool   // the agent runs another version than the master
+}
+
+// newAgentRow fills the display fields of a row. masterVersion is the
+// master's build version; "dev" and "" never mark an agent as old.
+func newAgentRow(a model.Agent, online bool, masterVersion string) agentRow {
+	r := agentRow{Agent: a, Online: online}
+	switch {
+	case a.RevokedAt != nil:
+		r.Status, r.StatusClass = "Revoked", "a-revoked"
+	case online:
+		r.Status, r.StatusClass = "Online", "a-on"
+	case a.LastSeenAt == nil:
+		r.Status, r.StatusClass = "Never connected", "a-never"
+	default:
+		r.Status, r.StatusClass = "Offline", "a-off"
+	}
+	switch {
+	case a.SwarmManager:
+		r.Role = "Manager"
+	case a.SwarmClusterID != "":
+		r.Role = "Worker"
+	default:
+		r.Role = "—"
+	}
+	r.VersionOld = a.AgentVersion != "" && masterVersion != "" && masterVersion != "dev" && a.AgentVersion != masterVersion
+	return r
+}
+
+// middle shortens a long secret for display: its first 8 and last 4 characters.
+func middle(s string) string {
+	if len(s) <= 14 {
+		return s
+	}
+	return s[:8] + "…" + s[len(s)-4:]
 }
 
 // jobDuration is how long a job ran, or has been running; "" before it starts.

internal/master/web/views_test.go

@@ -0,0 +1,235 @@
+package web
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/shcizo/package-updater/internal/master/model"
+	"github.com/shcizo/package-updater/internal/protocol"
+)
+
+func TestImageParts(t *testing.T) {
+	cases := []struct {
+		ref, repo, tag, digest string
+	}{
+		{"postgres:17.10", "postgres", ":17.10", ""},
+		{"postgres", "postgres", "", ""},
+		{"gitea/gitea:1.27.3", "gitea/gitea", ":1.27.3", ""},
+		{"gitea.shcizo.se/shcizo/slusk:v0.8.1", "shcizo/slusk", ":v0.8.1", ""},
+		{"localhost:5000/foo:v1", "foo", ":v1", ""},
+		{"localhost/foo", "foo", "", ""},
+		{"lscr.io/linuxserver/jellyfin@sha256:0f42497a69fa0441bfd5f9d6bba8694f2a656ec984e571d0ac04c6dd91250039", "linuxserver/jellyfin", "", "@sha256:0f42497a69fa…"},
+		{"r/x:1.0@sha256:0f42497a69fa0441", "r/x", ":1.0", ""}, // the digest is hidden when there is a tag
+		{"r/x@sha256:abc", "r/x", "", "@sha256:abc"},
+	}
+	for _, c := range cases {
+		t.Run(c.ref, func(t *testing.T) {
+			repo, tag, digest := imageParts(c.ref)
+			assert.Equal(t, c.repo, repo)
+			assert.Equal(t, c.tag, tag)
+			assert.Equal(t, c.digest, digest)
+		})
+	}
+}
+
+func TestLetterClass(t *testing.T) {
+	assert.Equal(t, "lc-0", letterClass(""))
+	assert.Equal(t, "lc-1", letterClass("a"), "97 mod 4")
+	assert.Equal(t, letterClass("gitea"), letterClass("gitea"), "a name always gets the same colour")
+	for _, name := range []string{"gitea", "authentik-server", "ø", "n8n"} {
+		assert.Regexp(t, `^lc-[0-3]$`, letterClass(name))
+	}
+}
+
+func TestNewAgentRow(t *testing.T) {
+	seen := ptr(time.Now())
+	cases := []struct {
+		name          string
+		agent         model.Agent
+		online        bool
+		masterVersion string
+		status, role  string
+		old           bool
+	}{
+		{"online manager", model.Agent{SwarmManager: true, AgentVersion: "v1"}, true, "v1", "Online", "Manager", false},
+		{"offline worker on another version", model.Agent{SwarmClusterID: "c", AgentVersion: "v0", LastSeenAt: seen}, false, "v1", "Offline", "Worker", true},
+		{"never connected", model.Agent{}, false, "v1", "Never connected", "—", false},
+		{"revoked wins over online", model.Agent{RevokedAt: seen, LastSeenAt: seen}, true, "v1", "Revoked", "—", false},
+		{"dev master marks nothing", model.Agent{AgentVersion: "v0"}, true, "dev", "Online", "—", false},
+		{"agent without version", model.Agent{}, true, "v1", "Online", "—", false},
+	}
+	for _, c := range cases {
+		t.Run(c.name, func(t *testing.T) {
+			row := newAgentRow(c.agent, c.online, c.masterVersion)
+			assert.Equal(t, c.status, row.Status)
+			assert.Equal(t, c.role, row.Role)
+			assert.Equal(t, c.old, row.VersionOld)
+		})
+	}
+}
+
+func TestMiddle(t *testing.T) {
+	assert.Equal(t, "short", middle("short"))
+	assert.Equal(t, "pua_7Hq2…Ge5A", middle("pua_7Hq2mX9vKc4LwT8nRb3ZyF6dJp1sGe5A"))
+}
+
+func TestGroupServices(t *testing.T) {
+	at := func(h, m int) *time.Time { return ptr(time.Date(2026, 10, 12, h, m, 0, 0, time.UTC)) }
+	id := func(n int64) *int64 { return &n }
+	compose := func(sid int64, agent *int64, name string) model.Service {
+		return model.Service{ID: sid, Kind: protocol.KindCompose, AgentID: agent, ComposeProject: "p", ComposeService: name, ActualRef: "r/" + name + ":1"}
+	}
+	swarm := func(sid int64, cluster string, agent *int64) model.Service {
+		return model.Service{ID: sid, Kind: protocol.KindSwarm, SwarmClusterID: cluster, AgentID: agent, SwarmService: "s" + cluster}
+	}
+	noBlock := func(model.Service) string { return "" }
+	onlineSet := func(ids ...int64) func(int64) bool {
+		return func(a int64) bool {
+			for _, i := range ids {
+				if i == a {
+					return true
+				}
+			}
+			return false
+		}
+	}
+	titles := func(groups []serviceGroup) []string {
+		var out []string
+		for _, g := range groups {
+			out = append(out, g.Title)
+		}
+		return out
+	}
+
+	t.Run("empty", func(t *testing.T) {
+		assert.Empty(t, groupServices(nil, []model.Agent{{ID: 1, Name: "a"}}, onlineSet(), noBlock))
+	})
+
+	t.Run("agents without services get no group", func(t *testing.T) {
+		groups := groupServices([]model.Service{compose(1, id(1), "web")},
+			[]model.Agent{{ID: 1, Name: "a"}, {ID: 2, Name: "idle"}}, onlineSet(1, 2), noBlock)
+		assert.Equal(t, []string{"a"}, titles(groups))
+		assert.Equal(t, "agent-1", groups[0].Key)
+		assert.Equal(t, "Machine", groups[0].Kind)
+		assert.False(t, groups[0].Offline)
+	})
+
+	t.Run("services without a known agent are offline and last", func(t *testing.T) {
+		groups := groupServices(
+			[]model.Service{compose(1, nil, "orphan"), compose(2, id(7), "ghost"), compose(3, id(1), "web")},
+			[]model.Agent{{ID: 1, Name: "a"}}, onlineSet(1), noBlock)
+		assert.Equal(t, []string{"a", "Unknown agent"}, titles(groups))
+		unknown := groups[1]
+		assert.Equal(t, "unknown", unknown.Key)
+		assert.True(t, unknown.Offline)
+		assert.Nil(t, unknown.OfflineSince)
+		assert.Len(t, unknown.Services, 2)
+	})
+
+	t.Run("an offline agent says since when", func(t *testing.T) {
+		groups := groupServices([]model.Service{compose(1, id(1), "web")},
+			[]model.Agent{{ID: 1, Name: "a", Hostname: "a.lan", LastSeenAt: at(13, 51)}}, onlineSet(), noBlock)
+		require.Len(t, groups, 1)
+		assert.True(t, groups[0].Offline)
+		assert.Equal(t, "Agent offline", groups[0].OfflineLabel)
+		assert.Equal(t, at(13, 51), groups[0].OfflineSince)
+		assert.Equal(t, "a.lan", groups[0].Meta)
+	})
+
+	t.Run("a cluster is online when any manager is", func(t *testing.T) {
+		agents := []model.Agent{
+			{ID: 1, Name: "m1", SwarmManager: true, SwarmClusterID: "c1", LastSeenAt: at(10, 0)},
+			{ID: 2, Name: "m2", SwarmManager: true, SwarmClusterID: "c1", LastSeenAt: at(11, 0)},
+		}
+		groups := groupServices([]model.Service{swarm(1, "c1", nil)}, agents, onlineSet(2), noBlock)
+		require.Len(t, groups, 1)
+		assert.False(t, groups[0].Offline)
+		assert.Equal(t, "Swarm", groups[0].Title)
+		assert.Equal(t, "c1", groups[0].TitleHint)
+		assert.Equal(t, "Swarm cluster", groups[0].Kind)
+		assert.Equal(t, "managers m1, m2", groups[0].Meta)
+		assert.Equal(t, "swarm-c1", groups[0].Key)
+	})
+
+	t.Run("a cluster with no manager online has been offline since the latest manager was seen", func(t *testing.T) {
+		agents := []model.Agent{
+			{ID: 1, Name: "m1", SwarmManager: true, SwarmClusterID: "c1", LastSeenAt: at(10, 0)},
+			{ID: 2, Name: "m2", SwarmManager: true, SwarmClusterID: "c1", LastSeenAt: at(11, 0)},
+		}
+		groups := groupServices([]model.Service{swarm(1, "c1", nil)}, agents, onlineSet(), noBlock)
+		assert.True(t, groups[0].Offline)
+		assert.Equal(t, "No manager online", groups[0].OfflineLabel)
+		assert.Equal(t, at(11, 0), groups[0].OfflineSince)
+	})
+
+	t.Run("a revoked manager does not count", func(t *testing.T) {
+		agents := []model.Agent{
+			{ID: 1, Name: "gone", SwarmManager: true, SwarmClusterID: "c1", LastSeenAt: at(12, 0), RevokedAt: at(12, 5)},
+			{ID: 2, Name: "m2", SwarmManager: true, SwarmClusterID: "c1", LastSeenAt: at(9, 0)},
+		}
+		groups := groupServices([]model.Service{swarm(1, "c1", nil)}, agents, onlineSet(1), noBlock)
+		assert.True(t, groups[0].Offline, "the revoked manager is not connected for this purpose")
+		assert.Equal(t, "managers m2", groups[0].Meta)
+		assert.Equal(t, at(9, 0), groups[0].OfflineSince)
+	})
+
+	t.Run("managers are found through the rows when no agent reports the cluster", func(t *testing.T) {
+		agents := []model.Agent{{ID: 4, Name: "reporter", LastSeenAt: at(8, 0)}}
+		groups := groupServices([]model.Service{swarm(1, "c1", id(4))}, agents, onlineSet(4), noBlock)
+		assert.False(t, groups[0].Offline)
+		assert.Equal(t, "managers reporter", groups[0].Meta)
+	})
+
+	t.Run("order is cluster, its machines, other machines by name, unknown", func(t *testing.T) {
+		agents := []model.Agent{
+			{ID: 1, Name: "zeta"},
+			{ID: 2, Name: "mgr", SwarmManager: true, SwarmClusterID: "c1"},
+			{ID: 3, Name: "alpha"},
+			{ID: 4, Name: "mgr2", SwarmManager: true, SwarmClusterID: "c1"},
+		}
+		services := []model.Service{
+			compose(1, id(1), "a"), compose(2, id(2), "b"), compose(3, id(3), "c"),
+			compose(4, id(4), "d"), compose(5, nil, "e"), swarm(6, "c1", nil),
+		}
+		groups := groupServices(services, agents, onlineSet(1, 2, 3, 4), noBlock)
+		assert.Equal(t, []string{"Swarm", "mgr", "mgr2", "alpha", "zeta", "Unknown agent"}, titles(groups))
+		assert.Equal(t, []bool{false, true, true, false, false, false}, []bool{groups[0].Sub, groups[1].Sub, groups[2].Sub, groups[3].Sub, groups[4].Sub, groups[5].Sub})
+	})
+
+	t.Run("two clusters carry their ids in the titles", func(t *testing.T) {
+		groups := groupServices([]model.Service{swarm(1, "abcdefghijkl", nil), swarm(2, "xyz", nil)}, nil, onlineSet(), noBlock)
+		assert.Equal(t, []string{"Swarm abcdefgh", "Swarm xyz"}, titles(groups))
+		assert.Equal(t, "abcdefghijkl", groups[0].TitleHint)
+	})
+
+	t.Run("the auto-blocked reason is kept on the row", func(t *testing.T) {
+		groups := groupServices([]model.Service{compose(1, id(1), "web")}, []model.Agent{{ID: 1, Name: "a"}}, onlineSet(1),
+			func(model.Service) string { return "because" })
+		assert.Equal(t, "because", groups[0].Services[0].AutoBlocked)
+	})
+}
+
+func TestNewerTag(t *testing.T) {
+	removed := time.Now()
+	tests := []struct {
+		name string
+		svc  model.Service
+		want string
+	}{
+		{"newer", model.Service{ActualRef: "gitea/gitea:1.27.3", NewerRef: "gitea/gitea:1.28.0"}, "1.28.0"},
+		{"bare image follows latest", model.Service{ActualRef: "gitea/gitea", NewerRef: "gitea/gitea:1.28.0"}, ""},
+		{"stale after the tag changed", model.Service{ActualRef: "gitea/gitea:1.28.0", NewerRef: "gitea/gitea:1.28.0"}, ""},
+		{"stale after the tag moved past it", model.Service{ActualRef: "gitea/gitea:1.29.0", NewerRef: "gitea/gitea:1.28.0"}, ""},
+		{"other repo", model.Service{ActualRef: "redis:7.2.4", NewerRef: "bitnami/redis:7.2.5"}, ""},
+		{"removed", model.Service{ActualRef: "gitea/gitea:1.27.3", NewerRef: "gitea/gitea:1.28.0", RemovedAt: &removed}, ""},
+		{"empty", model.Service{ActualRef: "gitea/gitea:1.27.3"}, ""},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equal(t, tt.want, newerTag(tt.svc))
+		})
+	}
+}

internal/master/webhook/webhook.go

@@ -15,6 +15,7 @@ import (
 	"log/slog"
 	"net/http"
 	"regexp"
+	"slices"
 	"strconv"
 	"strings"
 	"time"
@@ -181,6 +182,9 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 
 	resp := response{RequestID: requestID, Image: req.Image, Tag: req.Tag, Matched: len(services)}
 	pending := map[int]int64{} // index into resp.Results -> job ID
+	// An agent's own service goes last: jobs that reach an agent while it
+	// replaces itself are lost, and messages to one agent arrive in order.
+	slices.SortStableFunc(services, func(a, b model.Service) int { return selfRank(a) - selfRank(b) })
 	for _, svc := range services {
 		res, job := h.handle(work, svc, req.Tag, started, log)
 		resp.Results = append(resp.Results, res)
@@ -252,6 +256,14 @@ func (h *Handler) handle(ctx context.Context, svc model.Service, tag string, sta
 	return res, nil
 }
 
+// selfRank sorts an agent's own service after all others.
+func selfRank(svc model.Service) int {
+	if svc.Self {
+		return 1
+	}
+	return 0
+}
+
 func (h *Handler) setJob(res *result, job model.Job) {
 	res.JobID = job.ID
 	res.JobURL = h.cfg.PublicURL + "/jobs/" + strconv.FormatInt(job.ID, 10)

internal/master/webhook/webhook_test.go

@@ -470,14 +470,14 @@ func TestBusyIsNotAFailure(t *testing.T) {
 
 func TestRefusedIsNotAFailure(t *testing.T) {
 	e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
-	e.jobs.errFor[1] = &model.NotUpdatableError{Reason: model.ReasonSelf}
+	e.jobs.errFor[1] = &model.NotUpdatableError{Reason: model.ReasonContainer}
 
 	rec := e.post(`{"image":"` + slusk + `","tag":"2"}`)
 
 	assert.Equal(t, 200, rec.Code)
 	r := decode(t, rec).Results[0]
 	assert.Equal(t, statusRefused, r.Status)
-	assert.Equal(t, model.ReasonSelf, r.Error)
+	assert.Equal(t, model.ReasonContainer, r.Error)
 	assert.Zero(t, r.JobID)
 	assert.NotEmpty(t, e.store.avail)
 }
@@ -500,6 +500,21 @@ func TestMixedAutoAndManual(t *testing.T) {
 	assert.Len(t, e.store.avail, 3)
 }
 
+func TestAgentsOwnServiceIsStartedLast(t *testing.T) {
+	own := compose(1, model.ModeAuto, slusk+":1")
+	own.Self = true
+	e := newEnv(t, own, compose(2, model.ModeAuto, slusk+":1"))
+	e.jobs.jobFor[1] = model.Job{Status: model.JobSucceeded}
+	e.jobs.jobFor[2] = model.Job{Status: model.JobSucceeded}
+
+	e.post(`{"image":"` + slusk + `","tag":"2"}`)
+
+	calls := e.jobs.startCalls()
+	require.Len(t, calls, 2)
+	assert.Equal(t, int64(2), calls[0].svc.ID)
+	assert.Equal(t, int64(1), calls[1].svc.ID)
+}
+
 func TestClientCancelStopsWaitingButNotStarting(t *testing.T) {
 	e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
 	e.h.cfg.Timeout = time.Minute

internal/protocol/protocol.go

@@ -16,7 +16,7 @@ import (
 
 // Version is the protocol version this build speaks. Agents send it in
 // Hello. Phase-1 agents do not send the field, which reads as 0.
-const Version = 3
+const Version = 4
 
 // MinJobVersion is the lowest agent protocol version the master sends jobs to.
 const MinJobVersion = 2
@@ -26,6 +26,11 @@ const MinJobVersion = 2
 // with a digest.
 const MinPinnedRefVersion = 3
 
+// MinSelfUpdateVersion is the lowest agent protocol version the master sends a
+// job for the agent's own Compose service to. Older agents neither run the
+// update in a helper container nor resume it after restarting.
+const MinSelfUpdateVersion = 4
+
 // MaxJobOutput caps the job output an agent keeps and the master stores, in
 // bytes. Only the tail is kept. With JSON escaping (worst case 6 bytes per
 // byte) a JobResult carrying this much output still fits the 1 MiB read limit.
@@ -141,6 +146,10 @@ type JobStart struct {
 	SwarmClusterID string `json:"swarm_cluster_id,omitempty"`
 	SwarmService   string `json:"swarm_service,omitempty"`
 	Ref            string `json:"ref"`
+	// Self says the service is the agent's own. The agent runs such a job
+	// after every other queued job; whether it needs a helper container is
+	// decided by the agent itself.
+	Self bool `json:"self,omitempty"`
 }
 
 // JobProgress carries output produced since the previous JobProgress for the