package-updater/README.md

package-updater

Updates self-hosted Docker Compose and Docker Swarm services from one place. A master (web UI behind OIDC login, state in Postgres) shows every Compose stack, Swarm service and container on every machine, and starts updates when someone clicks Update. One agent per machine reports its inventory and runs the jobs. Both are in the same image: updater-master and updater-agent (the default entrypoint).

Design: master/agent spec. Phase 2 plan: jobs and manual update. Phase 3 plan: webhook and auto mode. Phase 4 plan: registry polling and version history. Phase 5 plan: drift and merge.

What an update does

Clicking Update on a service re-pulls the service's current tag (repo:tag). The agent asks its Docker daemon for the tag's digest and deploys repo:tag@digest, so what runs is exactly what was pulled. The log streams to the browser while it runs; job history is kept in Postgres. A service without a tag (repo@digest), a plain container, and the updater's own services cannot be updated, and the UI says why. A bare image (image: postgres, no tag and no digest) counts as postgres:latest; only a reference with a digest and no tag (repo@sha256:...) "follows no tag". Compose services with a build: section are shown but never updated ("built from source (build:)"). One update at a time per service, and one job at a time per agent.

Deploying the master

Use deploy/master.stack.yml as a starting point (Swarm stack; needs Postgres and an OIDC provider). The master listens for the UI on PORT (default 8080, put it behind your reverse proxy) and for agents on AGENT_PORT (default 8090, pinned TLS, reached directly on the internal network). The example runs the master as nobody (65534); it needs no root. The agent runs as root because access to the Docker socket is root on the host anyway.

Variable	Required	Default	Purpose
UPDATER_API_KEY	no	—	Bearer token for POST /update. Without it the webhook answers 503 and the master logs webhook_disabled.
UPDATE_TIMEOUT	no	5m	How long one POST /update waits for its auto jobs (Go duration).
POLL_INTERVAL	no	6h	Time between registry polling cycles (Go duration, must be positive). The first cycle runs one minute after start.
DOCKER_CONFIG	no	~/.docker	Directory with a config.json holding registry credentials for polling, readable by the master's user. Only inline auths entries work; see Registry polling.

The master's other settings (DATABASE_URL, PUBLIC_URL, AGENT_URL, OIDC_*, ALLOWED_GROUP, AGENT_IMAGE, PORT, AGENT_PORT, LOG_LEVEL) are shown in deploy/master.stack.yml.

Webhook (CI)

CI calls the master after pushing an image. The Gitea action in gitea-action/ does this; any HTTP client works.

POST /update
Authorization: Bearer <UPDATER_API_KEY>
Content-Type: application/json

{"image": "registry.example.com/myapp", "tag": "1.5.0"}

image is matched against the image repo of every service on every agent (tag and digest in the request are ignored for matching; matching is case-sensitive). tag is optional. For every match the master records the version as available (the overview shows an "update available" badge), including services that cannot be updated automatically (updater's own, build: and plain containers), so the badge shows for them too. Services in manual mode stop there. Services in auto mode also get an update job, and the request waits for those jobs to finish, at most UPDATE_TIMEOUT.

tag semantics. The target of an auto job is <service's image repo>:<tag>, built from the service's own repo and never from the request's image string. The tag must be one that has been pushed. Compose services get pinned to it through compose.override.yaml, as in any update. Omit tag to re-pull the tag the service already follows (a bare image counts as :latest); a service that follows no tag (repo@digest) is then not touched and is reported as refused ("no tag given and the service follows no tag"), manual or auto. With a tag, the "follows no tag" refusal does not apply; the other refusals (the updater's own services, build: services, plain containers) still do.

Response (200 or 500):

{"request_id": "…", "image": "registry.example.com/myapp", "tag": "1.5.0", "matched": 2,
 "results": [
  {"service_id": 12, "service": "myapp/web", "kind": "compose", "mode": "auto",
   "status": "succeeded", "job_id": 42, "job_url": "https://updater.example.com/jobs/42",
   "to_ref": "registry.example.com/myapp:1.5.0", "duration_ms": 8210},
  {"service_id": 15, "service": "myapp/worker", "kind": "compose", "mode": "manual",
   "status": "marked", "to_ref": "registry.example.com/myapp:1.5.0", "duration_ms": 3}]}

status	Meaning	Counts as failure
succeeded	Auto job finished successfully.	no
failed	Auto job failed (for example agent offline), or the version could not be recorded. error says why.	yes
timeout	Job still running when UPDATE_TIMEOUT passed. The master leaves it running.	yes
marked	Manual service; the version was recorded only.	no
busy	The service already has a queued or running job; job_id is that job, which is not waited for.	no
refused	The service may not be updated, or there is no target (no tag given and the service follows no tag). error has the reason.	no

Code	When
200	No result is failed or timeout.
400	Body is not JSON, image is missing, or tag is not a valid Docker tag.
401	Missing or wrong bearer token.
404	No service uses the image (catches typos in CI configuration).
500	At least one result is failed or timeout.
503	UPDATER_API_KEY is not set on the master.

A service that has drifted (see Drift and moved services) is refused with its reason, in auto mode too, until someone restores or accepts it.

The webhook never creates services: the first deployment of a stack is done by hand. A new service always starts in manual mode; switch it with the Make auto button on the overview or the service page. Auto is refused for the updater's own services, build: services and plain containers.

Warning: auto mode deploys whatever CI pushes, without anyone looking at it. Using it on critical services is your own risk.

The request is open for as long as the jobs run, so the reverse proxy in front of the master must allow a read timeout longer than UPDATE_TIMEOUT. If the client disconnects, marking and starting jobs still complete; only the waiting stops.

Registry polling

Besides the webhook, the master asks each registry whether the tag a service follows has a new digest. Every POLL_INTERVAL it sends one HEAD /v2/<repo>/manifests/<tag> per distinct followed repo:tag (a bare image counts as :latest; repo:tag@digest is polled as repo:tag). It never downloads a manifest: Docker Hub counts a GET as a pull against the rate limit and a HEAD as nothing (source), and Docker Hub also has a separate abuse limit for many pulls from one IP address, so the poller stays HEAD-only. For a multi-arch image the digest is the index digest, the same one docker inspect shows in RepoDigests.

• What counts as new. A digest different from the one the service runs becomes the service's available version (the overview badge, source poll). A digest the master already suggested is not suggested again. A service in auto mode also gets an update job to repo:tag (the agent resolves the digest itself, so agents on protocol version 2 keep working). The master remembers the newest digest it started or deployed for the tag in services.last_tag_digest (set when polling starts a job, when polling sees the service already running that digest, and when any plain tag job succeeds, whoever started it) and starts no job for that digest again, whether the job failed or the user rolled back afterwards. If the service already had a job when the digest was found, the next cycle tries again.
• What is not polled. Services that follow no tag (repo@digest), build: services, and services without a registry digest (locally built images). The service page says which. The updater's own services and plain containers are polled, so their badge works, but jobs for them are still refused.
• Errors. A failed lookup is stored for the services that follow that tag and shown as a "poll failed" badge on the overview and in the "Polling" row of the service page; other tags continue. A later success clears it.
• Credentials. The master reads $DOCKER_CONFIG/config.json (a file written by docker login), mounted read-only into the container; without it registries are accessed anonymously. The directory and file must be readable by the master's user (65534 in the example stack) without being world-readable: chown -R 65534:65534 the directory and keep 0700/0600. Only inline auths entries are supported. A credsStore or credHelpers entry makes every lookup fail with error getting credentials ... executable file not found in $PATH, because the container has no credential helper binaries. The master logs registry_credentials at startup with the directory, config_json_found and config_json_readable, never the contents. If the file exists but cannot be opened it also logs a registry_credentials_unreadable warning with the error.
• Requests go wherever the image name points. The master sends the lookup to the registry host in the image name, including a GET /v2/ ping that follows redirects. Anyone who can start containers on a managed host can therefore make the master send blind requests to internal hosts. The registry's answer is not shown: an error keeps only the HTTP status and the registry's error codes; any other failure shows only "timeout", "cannot reach registry" or "lookup failed". The full error goes to the master's log (registry_lookup_failed).

Version history and rollback

The master records every version it sees (webhook, polling, and what agents report as running) in image_versions. The service page lists the 10 most recently first seen versions of the service's image repo, marks the one that is running, and has a Deploy button on each row. A webhook announces a tag but not its digest, so such a row has no digest until polling or an agent reports one; it is hidden when a row with the same tag and a digest was first seen at or after it. A bare image (postgres) is recorded as tag latest.

Picking a version starts a manual job to exactly that reference. Picking an older one is a rollback, including to a digest-pinned reference. A version without a tag deploys repo@digest and the service then follows no tag, so it is no longer polled or updatable with the plain Update button until it is moved to a tagged version again. Picks of digest-pinned or tagless versions need an agent speaking protocol version 3 (this release); on an older agent the job fails with "agent too old to deploy a pinned version (needs protocol 3)".

A rollback made before the first poll cycle after the upgrade has seen the service is not protected, because the master has not yet recorded the digest it was running.

A service in auto mode stays auto after a rollback. Polling does not undo it: the service stays on the older version until the registry shows a digest of the followed tag that differs from the newest one the master deployed. The newer version it was rolled back from stays marked as available. A webhook that names a tag still starts a job for an auto service every time, so a CI push moves it forward again.

Drift and moved services

A service has drifted when it runs something other than the version the last update job deployed. A service the updater never touched has nothing to drift from. Drift is decided by what runs, not by how the reference is written: the digest decides when both sides know one, so repo:tag@sha256:… is run by a service reporting repo:tag with that digest, and a different tag with the same digest is not drift. (The design spec compares the two reference strings; this is deliberately looser.) Between a successful job and the agent's next inventory report the service is not counted as drifted, because the report has not arrived yet.

The master never corrects drift on its own, because a manual docker stack deploy of a newer tag may be intentional. Polling and the webhook start no job for a drifted service, in auto mode too; a person can still start one by hand. There are two ways out:

• Restore desired deploys the recorded version again (a manual job to the exact repo:tag@digest; an agent speaking protocol 3 is needed). If the service had drifted to another digest of the same tag, that digest is remembered, so polling does not deploy it again until the registry shows a newer one (unless last_tag_digest already holds back a newer digest, which then stays).
• Accept running records what runs now as the desired version. It starts no job and leaves the available suggestion and the update mode alone.

When a Compose service disappears from one agent and exactly one service with the same project and service name appears on another, and the removed one has history (a desired version, auto mode or jobs) while the new one has none, the service page suggests that it moved. Merge moves the old row, with its history, mode and desired version, to the new location and deletes the duplicate. Merging is never automatic. It can also be done by hand for any removed service of the same kind (Compose or Swarm, names may differ), but not for plain containers, the updater itself, or a service with a queued or running job. A merge keeps the old row's mode and desired version, so a manually merged target with its own history may show drift right after the merge and can be accepted.

The restore, accept and merge buttons arrive with the UI follow-up (issue #28); the endpoints (POST /services/{id}/restore, /accept, /merge) and the data for them are in place.

Adding agents

Open the master's Agents page and add one. It shows a ready-to-run compose file with the agent's token (shown once) and the master's key fingerprint; the same layout is in deploy/agent.compose.yml.

Variable	Required	Default	Purpose
MASTER_URL	yes	—	wss://host:port/agent
MASTER_KEY	yes	—	Master key fingerprint (sha256:...) shown in the UI.
AGENT_TOKEN	yes	—	Token shown once when the agent was added.
SELF_ID	no	hostname	Container ID or name of the agent itself; used to read its mounts.
UPDATE_TIMEOUT	no	5m	Per-job timeout (Go duration).
DOCKER_CONFIG	no	—	Directory with a config.json holding registry credentials. Only inline auths entries work; credsStore/credHelpers are not available in the container.
LOG_LEVEL	no	info	debug / info / warn / error.

docker compose runs inside the agent with only PATH, HOME and the DOCKER_* variables. Variables a stack uses for interpolation must come from the stack's own .env file or an --env-file (which the agent honours), never from the agent container's environment.

Registry credentials

With DOCKER_CONFIG set, Compose jobs pull with the same credentials as docker compose. For Swarm jobs the agent passes the matching registry's credentials with the service update (same as docker service update --with-registry-auth), so Swarm stores them in the cluster and hands them to the nodes that pull the image. Only the entry for the image's registry host is ever sent.

Mounting stack directories

There is no STACKS_ROOT. The agent may only touch Compose stacks in directories that are bind-mounted into its own container read-write at the same path as on the host (Compose labels hold host paths). Add one line per directory:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock
  - /home/shcizo/self-hosted:/home/shcizo/self-hosted

Read-only mounts and mounts whose source differs from the destination are refused with a clear message. On Docker Desktop and OrbStack bind sources are translated host paths, so Compose jobs fail there; that is a development-only limitation.

How Compose updates work, and their limits

The agent never edits your files. It writes one file next to the stack's base file, compose.override.yaml, with a header line and an image: entry per updated service (repo:tag@digest), then runs docker compose pull and up -d for that service with the override passed explicitly.

• The agent only writes compose.override.yaml if it is absent or starts with its header line, and refuses the job if any other auto-loaded override exists (compose.override.yml, docker-compose.override.yml, docker-compose.override.yaml): Compose loads only one of them, so ours would silently disable yours.
• Stacks you start with explicit -f files do not load the override on a manual docker compose up; the agent's own jobs always pass it.
• The override is never pruned. If you delete a service from the stack file, its image: entry stays in compose.override.yaml, and a manual docker compose up -d would recreate it. Remove the entry by hand.

Check before enabling. List the config files of every running stack to spot existing override files and stacks started with explicit -f:

docker ps --format '{{.Label "com.docker.compose.project.config_files"}}' | sort -u

Swarm updates

The agent updates the service with ServiceUpdate and the pinned image. "Succeeded" means the daemon accepted the new spec, not that the tasks converged; the inventory shows degraded until they do. The agent doing the update must talk to a manager node (run it on a manager, or point DOCKER_HOST at one).

Upgrading from the single-instance updater

This release removes cmd/server and changes the image's default entrypoint to updater-agent. A running old-style updater (default entrypoint, UPDATER_API_KEY) will crash-loop the next time its image is pulled. Anything that pulls :latest for the old instance triggers this: its own /update webhook, Watchtower or Shepherd.

Pinning means two things: pin the old instance to the last pre-phase-2 tag and remove its se.shcizo.auto-update label, so nothing updates it to the new image. The master serves POST /update again (see Webhook); point the Gitea action's endpoint there and set UPDATER_API_KEY on the master.

Master and agents must be on this release together: a phase-1 agent connects but every job sent to it fails "agent too old for updates".

Development

go test ./...
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
go build ./cmd/master ./cmd/agent
docker build -t package-updater:dev .
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/