package-updater/README.md

# package-updater

Updates self-hosted Docker Compose and Docker Swarm services from one place. A
**master** (web UI behind OIDC login, state in Postgres) shows every Compose stack,
Swarm service and container on every machine, and starts updates when someone clicks
*Update*. One **agent** per machine reports its inventory and runs the jobs. Both are in
the same image: `updater-master` and `updater-agent` (the default entrypoint).

Design: [master/agent spec](docs/superpowers/specs/2026-10-12-master-agent-design.md).
Phase 2 plan: [jobs and manual update](docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md).
Phase 3 plan: [webhook and auto mode](docs/superpowers/plans/2026-10-12-master-agent-phase-3-webhook.md).
Phase 4 plan: [registry polling and version history](docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md).
Phase 5 plan: [drift and merge](docs/superpowers/plans/2026-10-13-master-agent-phase-5-drift-merge.md).

## What an update does

Clicking *Update* on a service re-pulls the service's **current tag** (`repo:tag`). The
agent asks its Docker daemon for the tag's digest and deploys `repo:tag@digest`, so what
runs is exactly what was pulled. The log streams to the browser while it runs; job
history is kept in Postgres. A service without a tag (`repo@digest`), a plain container,
and a Swarm agent's own service cannot be updated, and the UI says why (see
[Self-update](#self-update) for the master and Compose agents). A bare image
(`image: postgres`, no tag and no digest) counts as `postgres:latest`; only a reference
with a digest and no tag (`repo@sha256:...`) "follows no tag". Compose services with a
`build:` section are shown but never updated ("built from source (build:)"). One update at
a time per service, and one job at a time per agent.

## Deploying the master

Use [`deploy/master.stack.yml`](deploy/master.stack.yml) as a starting point (Swarm stack;
needs Postgres and an OIDC provider). The master listens for the UI on `PORT` (default
`8080`, put it behind your reverse proxy) and for agents on `AGENT_PORT` (default `8090`,
pinned TLS, reached directly on the internal network). The example runs the master as
`nobody` (`65534`); it needs no root. The agent runs as root because access to the Docker
socket is root on the host anyway.

| Variable | Required | Default | Purpose |
|---|---|---|---|
| `UPDATER_API_KEY` | no | — | Bearer token for `POST /update`. Without it the webhook answers `503` and the master logs `webhook_disabled`. |
| `UPDATE_TIMEOUT` | no | `5m` | How long one `POST /update` waits for its auto jobs (Go duration). |
| `POLL_INTERVAL` | no | `6h` | Time between registry polling cycles (Go duration, must be positive). The first cycle runs one minute after start. |
| `DOCKER_CONFIG` | no | `~/.docker` | Directory with a `config.json` holding registry credentials for polling, readable by the master's user. Only inline `auths` entries work; see [Registry polling](#registry-polling). |

The master's other settings (`DATABASE_URL`, `PUBLIC_URL`, `AGENT_URL`, `OIDC_*`,
`ALLOWED_GROUP`, `AGENT_IMAGE`, `PORT`, `AGENT_PORT`, `LOG_LEVEL`) are shown in
[`deploy/master.stack.yml`](deploy/master.stack.yml).

## Webhook (CI)

CI calls the master after pushing an image. The Gitea action in
[`gitea-action/`](gitea-action/README.md) does this; any HTTP client works.

```
POST /update
Authorization: Bearer <UPDATER_API_KEY>
Content-Type: application/json

{"image": "registry.example.com/myapp", "tag": "1.5.0"}
```

`image` is matched against the image repo of every service on every agent (tag and digest
in the request are ignored for matching; matching is case-sensitive). `tag` is optional.
For every match the master records the version as *available* (the overview shows an
"update available" badge), including services that cannot be updated automatically
(`build:` services, plain containers and an agent's own Swarm service), so the badge shows for them too. Services
in **manual** mode stop there. Services in **auto**
mode also get an update job, and the request waits for those jobs to finish, at most
`UPDATE_TIMEOUT`.

**`tag` semantics.** The target of an auto job is `<service's image repo>:<tag>`, built
from the service's own repo and never from the request's `image` string. The tag must be
one that has been pushed. Compose services get pinned to it through
`compose.override.yaml`, as in any update. Omit `tag` to re-pull the tag the service
already follows (a bare image counts as `:latest`); a service that follows no tag
(`repo@digest`) is then not touched and is reported as `refused` ("no tag given and the
service follows no tag"), manual or auto.
With a `tag`, the "follows no tag" refusal does not apply; the other refusals (`build:`
services, plain containers, a Swarm agent's own service) still do.

Response (`200` or `500`):

```json
{"request_id": "…", "image": "registry.example.com/myapp", "tag": "1.5.0", "matched": 2,
 "results": [
  {"service_id": 12, "service": "myapp/web", "kind": "compose", "mode": "auto",
   "status": "succeeded", "job_id": 42, "job_url": "https://updater.example.com/jobs/42",
   "to_ref": "registry.example.com/myapp:1.5.0", "duration_ms": 8210},
  {"service_id": 15, "service": "myapp/worker", "kind": "compose", "mode": "manual",
   "status": "marked", "to_ref": "registry.example.com/myapp:1.5.0", "duration_ms": 3}]}
```

| `status` | Meaning | Counts as failure |
|---|---|---|
| `succeeded` | Auto job finished successfully. | no |
| `failed` | Auto job failed (for example agent offline), or the version could not be recorded. `error` says why. | yes |
| `timeout` | Job still running when `UPDATE_TIMEOUT` passed. The master leaves it running. | yes |
| `marked` | Manual service; the version was recorded only. | no |
| `busy` | The service already has a queued or running job; `job_id` is that job, which is not waited for. | no |
| `refused` | The service may not be updated, or there is no target (no `tag` given and the service follows no tag). `error` has the reason. | no |

| Code | When |
|---|---|
| `200` | No result is `failed` or `timeout`. |
| `400` | Body is not JSON, `image` is missing, or `tag` is not a valid Docker tag. |
| `401` | Missing or wrong bearer token. |
| `404` | No service uses the image (catches typos in CI configuration). |
| `500` | At least one result is `failed` or `timeout`. |
| `503` | `UPDATER_API_KEY` is not set on the master. |

A service that has **drifted** (see [Drift and moved services](#drift-and-moved-services))
is refused with its reason, in auto mode too, until someone restores or accepts it.

The webhook never creates services: the first deployment of a stack is done by hand. A
new service always starts in **manual** mode; switch it with the *Make auto* button on the
overview or the service page. Auto is refused for `build:` services, plain containers and
a Swarm agent's own service.

> **Warning:** auto mode deploys whatever CI pushes, without anyone looking at it. Using
> it on critical services is your own risk.

The request is open for as long as the jobs run, so the reverse proxy in front of the
master must allow a read timeout longer than `UPDATE_TIMEOUT`. If the client disconnects,
marking and starting jobs still complete; only the waiting stops.

## Registry polling

Besides the webhook, the master asks each registry whether the tag a service follows has
a new digest. Every `POLL_INTERVAL` it sends one `HEAD /v2/<repo>/manifests/<tag>` per
distinct followed `repo:tag` (a bare image counts as `:latest`; `repo:tag@digest` is polled
as `repo:tag`). It never downloads a manifest: Docker Hub counts a `GET` as a pull against
the rate limit and a `HEAD` as nothing
([source](https://docs.docker.com/docker-hub/usage/pulls/)), and Docker Hub also has a
separate abuse limit for many pulls from one IP address, so the poller stays HEAD-only. For a
multi-arch image the digest is the index digest, the same one `docker inspect` shows in
`RepoDigests`.

- **What counts as new.** A digest different from the one the service runs becomes the
  service's *available* version (the overview badge, source `poll`). A digest the master
  already suggested is not suggested again. A service in **auto** mode also gets an update
  job to `repo:tag` (the agent resolves the digest itself, so agents on protocol version 2
  keep working). The master remembers the newest digest it started or deployed for the tag
  in `services.last_tag_digest` (set when polling starts a job, when polling sees the
  service already running that digest, and when any plain tag job succeeds, whoever started
  it) and starts no job for that digest again, whether the job
  failed or the user rolled back afterwards. If the service already had a job when the
  digest was found, the next cycle tries again.
- **Newer tags.** For a service that follows a version-like tag (`1.27.3`, `v1.2.3`,
  `11.0-latest`; at least two numeric components), the master also lists the repo's tags
  once per cycle (`GET /v2/<repo>/tags/list`, which Docker Hub does not count as a pull) and
  suggests the highest tag with the same prefix, suffix and number of components and the
  same major number, for example `gitea:1.27.3` gives `1.28.0` but never `2.0.0`. The
  overview shows a "Newer tag" badge and the service page lists the tag under Versions, from
  where you deploy it. It is a suggestion only: nothing is deployed automatically, not even
  for **auto** services. Tags such as `latest`, `7` or `pg16` get no suggestion. A failed tag
  list keeps the old suggestion and is only logged (`poll_tags_failed`).
- **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and
  services without a registry digest (locally built images). The service page says which.
  Plain containers are polled, so their badge works, but jobs for them are refused.
- **Errors.** A failed lookup is stored for the services that follow that tag and shown as a
  "poll failed" badge on the overview and in the "Polling" row of the service page; other
  tags continue. A later success clears it.
- **Credentials.** The master reads `$DOCKER_CONFIG/config.json` (a file written by
  `docker login`), mounted read-only into the container; without it registries are accessed
  anonymously. The directory and file must be readable by the master's user (`65534` in the
  example stack) without being world-readable: `chown -R 65534:65534` the directory and keep
  `0700`/`0600`. Only inline `auths` entries are supported. A `credsStore` or `credHelpers`
  entry makes every lookup fail with `error getting credentials ... executable file not found
  in $PATH`, because the container has no credential helper binaries. The master logs
  `registry_credentials` at startup with the directory, `config_json_found` and
  `config_json_readable`, never the contents. If the file exists but cannot be opened it also
  logs a `registry_credentials_unreadable` warning with the error.
- **Requests go wherever the image name points.** The master sends the lookup to the
  registry host in the image name, including a `GET /v2/` ping that follows redirects.
  Anyone who can start containers on a managed host can therefore make the master send
  blind requests to internal hosts. The registry's answer is not shown: an error keeps only
  the HTTP status and the registry's error codes; any other failure shows only "timeout",
  "cannot reach registry" or "lookup failed". The full error goes to the master's log
  (`registry_lookup_failed`).

### Version history and rollback

The master records every version it sees (webhook, polling, and what agents report as
running) in `image_versions`. The service page lists the 10 most recently first seen
versions of the service's image repo, marks the one that is running, and has a *Deploy*
button on each row. A webhook announces a tag but not its digest, so such a row has no
digest until polling or an agent reports one; it is hidden when a row with the same tag
and a digest was first seen at or after it. A bare image (`postgres`) is recorded as tag
`latest`.

Picking a version starts a manual job to exactly that reference. Picking an older one is a
rollback, including to a digest-pinned reference. A version without a tag deploys
`repo@digest` and the service then follows no tag, so it is no longer polled or updatable
with the plain *Update* button until it is moved to a tagged version again. Picks of
digest-pinned or tagless versions need an agent speaking protocol version 3 (this
release); on an older agent the job fails with "agent too old to deploy a pinned version
(needs protocol 3)".

A rollback made before the first poll cycle after the upgrade has seen the service is not
protected, because the master has not yet recorded the digest it was running.

A service in **auto** mode stays auto after a rollback. Polling does not undo it: the
service stays on the older version until the registry shows a digest of the followed tag
that differs from the newest one the master deployed. The newer version it was rolled back
from stays marked as available. A webhook that names a tag still starts a job for an auto
service every time, so a CI push moves it forward again.

## Drift and moved services

A service has **drifted** when it runs something other than the version the last update
job deployed. A service the updater never touched has nothing to drift from. Drift is
decided by what runs, not by how the reference is written: the digest decides when both
sides know one, so `repo:tag@sha256:…` is run by a service reporting `repo:tag` with that
digest, and a different tag with the same digest is not drift. (The design spec compares
the two reference strings; this is deliberately looser.) Between a successful job and the
agent's next inventory report the service is not counted as drifted, because the report
has not arrived yet.

The master never corrects drift on its own, because a manual `docker stack deploy` of a
newer tag may be intentional. Polling and the webhook start no job for a drifted
service, in auto mode too; a person can still start one by hand. There are two ways out:

- **Restore desired** deploys the recorded version again (a manual job to the exact
  `repo:tag@digest`; an agent speaking protocol 3 is needed). If the service had drifted
  to another digest of the same tag, that digest is remembered, so polling does not
  deploy it again until the registry shows a newer one (unless `last_tag_digest` already
  holds back a newer digest, which then stays).
- **Accept running** records what runs now as the desired version. It starts no job and
  leaves the available suggestion and the update mode alone.

When a Compose service disappears from one agent and exactly one service with the same
project and service name appears on another, and the removed one has history (a desired
version, auto mode or jobs) while the new one has none, the service page suggests that it
moved. **Merge** moves the old row, with its history, mode and desired version, to the new
location and deletes the duplicate. Merging is never automatic. It can also be done by
hand for any removed service of the same kind (Compose or Swarm, names may differ), but
not for plain containers, an agent's own service, or a service with a queued or running job.
A merge keeps the old row's mode and desired version, so a manually merged target with its
own history may show drift right after the merge and can be accepted.

The restore, accept and merge buttons arrive with the UI follow-up (issue #28); the endpoints
(`POST /services/{id}/restore`, `/accept`, `/merge`) and the data for them are in place.

## Adding agents

Open the master's **Agents** page and add one. It shows a ready-to-run compose file with
the agent's token (shown once) and the master's key fingerprint; the same layout is in
[`deploy/agent.compose.yml`](deploy/agent.compose.yml).

| Variable | Required | Default | Purpose |
|---|---|---|---|
| `MASTER_URL` | yes | — | `wss://host:port/agent` |
| `MASTER_KEY` | yes | — | Master key fingerprint (`sha256:...`) shown in the UI. |
| `AGENT_TOKEN` | yes | — | Token shown once when the agent was added. |
| `SELF_ID` | no | hostname | Container ID or name of the agent itself; used to read its mounts. |
| `UPDATE_TIMEOUT` | no | `5m` | Per-job timeout (Go duration). |
| `DOCKER_CONFIG` | no | — | Directory with a `config.json` holding registry credentials. Only inline `auths` entries work; `credsStore`/`credHelpers` are not available in the container. |
| `LOG_LEVEL` | no | `info` | `debug` / `info` / `warn` / `error`. |

`docker compose` runs inside the agent with only `PATH`, `HOME` and the `DOCKER_*`
variables. Variables a stack uses for interpolation must come from the stack's own `.env`
file or an `--env-file` (which the agent honours), never from the agent container's
environment.

### Registry credentials

With `DOCKER_CONFIG` set, Compose jobs pull with the same credentials as `docker compose`.
For Swarm jobs the agent passes the matching registry's credentials with the service
update (same as `docker service update --with-registry-auth`), so Swarm stores them in the
cluster and hands them to the nodes that pull the image. Only the entry for the image's
registry host is ever sent.

### Mounting stack directories

There is no `STACKS_ROOT`. The agent may only touch Compose stacks in directories that
are bind-mounted into its own container **read-write at the same path as on the host**
(Compose labels hold host paths). Add one line per directory:

```yaml
volumes:
  - /var/run/docker.sock:/var/run/docker.sock
  - /home/shcizo/self-hosted:/home/shcizo/self-hosted
```

Read-only mounts and mounts whose source differs from the destination are refused with a
clear message. On Docker Desktop and OrbStack bind sources are translated host paths, so
Compose jobs fail there; that is a development-only limitation.

## Self-update

The master's own Compose service and each Compose agent's own service are updated like
any other service; the overview marks the agent's row "This agent".

- **Master.** The agent on the master's host runs the job. The `up -d` recreates the
  master, which ends the connection and any webhook request that waits for the job. The
  agent keeps the result and sends it again when the new master is up, so the job is
  recorded as finished. A client that waited gets an error and has to look at the job
  afterwards.
- **Agent.** `up -d` would stop the agent that runs it, so the agent starts a short-lived
  helper container from its own image (same mounts) and the helper runs `docker compose
  up -d <service>`. The helper carries `se.shcizo.package-updater.self-update.*` labels
  with the job ID, the pinned ref and the output so far. When the new agent starts it
  finds the helper by those labels, waits for it, adds its logs to the job output, removes
  it and reports the result under the old job ID. The old agent reports nothing after it
  is stopped.
- **Requirements.** The agent's own stack directory has to be mounted read-write at the
  same path (see below), like any stack it updates, and the agent must speak protocol 4
  (`agent too old to update itself` otherwise). The master and the agent need to be
  upgraded to a version with protocol 4 by hand once.
- **Order.** A job for an agent's own service runs after every other queued job on that
  agent. A master and an agent on the same host share an image, so one push starts both
  jobs, and the master's job has to finish before the agent replaces itself. The master
  also starts a job for an agent's own service after the other jobs of the same poll
  cycle or webhook request. Any other job that reaches an agent while it replaces itself
  is marked lost when the new agent connects without it.
- **Docker socket.** The helper runs on the default network, so `DOCKER_HOST` only works
  with a `docker.sock` bind mount. A TCP socket proxy that is reachable only on a Compose
  network does not work for self-update.
- **Swarm.** An agent's own Swarm service is refused: the agent cannot update the service
  it runs in.

## How Compose updates work, and their limits

The agent never edits your files. It writes one file next to the stack's base file,
`compose.override.yaml`, with a header line and an `image:` entry per updated service
(`repo:tag@digest`), then runs `docker compose pull` and `up -d` for that service with
the override passed explicitly.

- The agent only writes `compose.override.yaml` if it is absent or starts with its header
  line, and **refuses the job** if any other auto-loaded override exists
  (`compose.override.yml`, `docker-compose.override.yml`, `docker-compose.override.yaml`):
  Compose loads only one of them, so ours would silently disable yours.
- Stacks you start with explicit `-f` files do not load the override on a manual
  `docker compose up`; the agent's own jobs always pass it.
- **The override is never pruned.** If you delete a service from the stack file, its
  `image:` entry stays in `compose.override.yaml`, and a manual `docker compose up -d`
  would recreate it. Remove the entry by hand.

**Check before enabling.** List the config files of every running stack to spot existing
override files and stacks started with explicit `-f`:

```bash
docker ps --format '{{.Label "com.docker.compose.project.config_files"}}' | sort -u
```

## Swarm updates

The agent updates the service with `ServiceUpdate` and the pinned image. "Succeeded"
means the daemon accepted the new spec, not that the tasks converged; the inventory shows
`degraded` until they do. The agent doing the update must talk to a manager node (run it
on a manager, or point `DOCKER_HOST` at one).

## Upgrading from the single-instance updater

This release removes `cmd/server` and changes the image's default entrypoint to
`updater-agent`. A running old-style updater (default entrypoint, `UPDATER_API_KEY`) will
crash-loop the next time its image is pulled. Anything that pulls `:latest` for the old
instance triggers this: its own `/update` webhook, Watchtower or Shepherd.

Pinning means two things: pin the old instance to the last pre-phase-2 tag **and** remove
its `se.shcizo.auto-update` label, so nothing updates it to the new image. The master
serves `POST /update` again (see [Webhook](#webhook-ci)); point the Gitea action's
`endpoint` there and set `UPDATER_API_KEY` on the master.

Master and agents must be on this release together: a phase-1 agent connects but every
job sent to it fails "agent too old for updates".

## Development

```bash
go test ./...
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
go build ./cmd/master ./cmd/agent
docker build -t package-updater:dev .
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
```