Compare commits

..
Author SHA1 Message Date
shcizo 8fe75b2cbf fix(master): seed the tag digest from running services and hide registry error text (#9) 2026-10-13 08:38:17 +02:00
shcizo 3166874433 fix(master): remember the tag's deployed digest after any tag job (#9)
Renames last_auto_digest to last_tag_digest. FinishJob sets it to the pinned
digest after any successful job whose to_ref had no digest, so a rollback
holds however the update was started.
2026-10-13 08:33:17 +02:00
shcizo 53a2a23f1b docs: auto rollbacks, bare image as latest and registry request scope (#9) 2026-10-13 08:30:08 +02:00
shcizo 73ac8f613e refactor: tidy poller, web test helper and GoDoc line widths (#9) 2026-10-13 08:29:22 +02:00
shcizo e2f71521bb fix(deploy): make the registry credentials mount of the master stack opt-in (#9) 2026-10-13 08:29:22 +02:00
shcizo 7e4b927293 fix(master): keep rollbacks of auto services until a new digest (#9)
Adds services.last_auto_digest so the poller starts one automatic update per
registry digest, and re-reads the service before starting it. Also records a
bare image's running version as tag latest, only records well-formed
digests, and shows a digest-less version unless a digest row of its tag was
first seen at or after it.
2026-10-13 08:28:50 +02:00
shcizo f169eddc4c fix(master): drop registry response bodies from poll errors (#9) 2026-10-13 08:26:29 +02:00
shcizo 92e7646221 fix(master): say why a pinned pick fails on a protocol 2 agent (#9) 2026-10-13 08:25:21 +02:00
shcizo 2e87d98012 docs: registry polling, version history and phase 4 plan (#9) 2026-10-13 08:16:47 +02:00
shcizo 687e411052 test(e2e): pick a pinned version and poll an auto service end to end (#9) 2026-10-13 08:15:29 +02:00
shcizo 2390b6151e feat(master): POLL_INTERVAL, poller wiring and version picker in the UI (#9) 2026-10-13 08:14:43 +02:00
shcizo c6ea2d3710 feat(master): registry digest lookup and polling loop (#9) 2026-10-13 08:11:37 +02:00
shcizo d0726ca992 feat(master,agent): deploy digest-pinned refs and route them to protocol 3 agents (#9) 2026-10-13 08:07:45 +02:00
shcizo af61f17dee feat(master): record image versions and track poll suggestions in the store (#9) 2026-10-13 08:05:21 +02:00
shcizo 41a619b316 feat: master/agent phase 3 — webhook and auto mode (#8) (#22)
Docker Build / docker (push) Successful in 38s
## Summary
- `POST /update` on the master (outside OIDC, bearer `UPDATER_API_KEY`; unset → 503). Matches `image` against `image_repo` on all agents, marks `available_*` on every match, starts webhook-triggered jobs for `auto` services targeting `<image_repo>:<tag>` (followed tag when no tag is given), and waits for them up to `UPDATE_TIMEOUT` (default 5m). 404 on no match, 500 if any auto job failed or timed out, otherwise 200.
- Manual/auto toggle per service on the overview and service page (`POST /services/{id}/mode`, CSRF); auto is blocked for self, `build:` and container services with the reason shown. "Update available" badge and an "Available" row.
- `FinishJob` clears `available_*` only when the successful job was created for that exact ref, after it was reported. Migration `0003_webhook.sql`, phase-3 plan doc, README / gitea-action README / CLAUDE.md / deploy example updated. `action.yml` unchanged.

Closes #8

## Behaviour changes for CI
- Mixed results now return 500 (the old 207 is gone); no match returns 404 instead of 200.
- With a `tag`, Compose services get pinned to `<image_repo>:<tag>` via `compose.override.yaml`; the tag must be a pushed tag.
- `busy` (job already active) and `refused` (not updatable) are reported but return 200.
- The reverse proxy read timeout must exceed `UPDATE_TIMEOUT`.

## Test plan
- [x] `go test ./...` and `go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/`
- [x] `go test -tags integration ./internal/master/store/` against throwaway Postgres
- [x] UI verified in browser (mode toggle, disabled reason, badge via SSE, Available row)
- [ ] Manual: webhook with manual service → `marked` + badge
- [ ] Manual: webhook with auto service → waits, `succeeded`, badge cleared, override pinned
- [ ] Manual: wrong token 401, unknown image 404, offline agent 500
- [ ] Gitea action pointed at the master's `/update` in a real CI run

Reviewed-on: #22
2026-10-12 20:03:22 +00:00
shcizo 417ef315c2 Merge pull request 'feat: master/agent phase 2 — jobs and manual update (#7)' (#20) from feat/master-agent-jobs-7 into main
Docker Build / docker (push) Successful in 40s
Reviewed-on: #20
2026-10-12 17:58:48 +00:00
51 changed files with 5323 additions and 143 deletions
+46 -7
View File
@@ -22,7 +22,7 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
## Architecture (one-line per package) ## Architecture (one-line per package)
- `cmd/master` — wiring only: config → store → hub → jobs → web, two listeners (UI, agents over pinned TLS) - `cmd/master` — wiring only: config → store → hub → jobs → web + webhook, two listeners (UI, agents over pinned TLS)
- `cmd/agent` — wiring only: config → docker client → inventory → job runner → master connection - `cmd/agent` — wiring only: config → docker client → inventory → job runner → master connection
- `internal/protocol` — the WebSocket messages both sides speak (`Version`, `JobStart`, `JobResult`, ...) - `internal/protocol` — the WebSocket messages both sides speak (`Version`, `JobStart`, `JobResult`, ...)
- `internal/pin` — master key pair, certificate and fingerprint pinning - `internal/pin` — master key pair, certificate and fingerprint pinning
@@ -36,7 +36,10 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
- `internal/master/config` — env-var loading - `internal/master/config` — env-var loading
- `internal/master/store` — Postgres access and migrations - `internal/master/store` — Postgres access and migrations
- `internal/master/hub` — agent connections: auth, per-connection writer, routing, job reconcile on hello - `internal/master/hub` — agent connections: auth, per-connection writer, routing, job reconcile on hello
- `internal/master/registry` — HEAD-only digest lookup against a registry (go-containerregistry, `DOCKER_CONFIG` credentials)
- `internal/master/poller` — every `POLL_INTERVAL`: one lookup per followed `repo:tag`, sets the available version, starts auto jobs
- `internal/master/jobs` — decides whether a service may be updated and starts a job on the right agent - `internal/master/jobs` — decides whether a service may be updated and starts a job on the right agent
- `internal/master/webhook` — `POST /update`: records the available version, starts jobs for auto services, waits for them
- `internal/master/web`, `webauth` — HTMX UI behind OIDC login, SSE signals - `internal/master/web`, `webauth` — HTMX UI behind OIDC login, SSE signals
- `internal/master/events` — in-process publish/subscribe for UI refresh signals - `internal/master/events` — in-process publish/subscribe for UI refresh signals
- `internal/master/model` — domain types shared by the above - `internal/master/model` — domain types shared by the above
@@ -59,11 +62,34 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
- **Unknown message types are logged and ignored; malformed messages close the connection.** - **Unknown message types are logged and ignored; malformed messages close the connection.**
- **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not - **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not
list in `known_jobs`, and only then accepts jobs for that connection. list in `known_jobs`, and only then accepts jobs for that connection.
- **Auth**: agents by token over pinned TLS; people by OIDC. Every state-changing UI - **The webhook never creates services; new services always start `manual`.** The upsert
request is a POST with a CSRF token. does not touch `update_mode`.
- **Auth**: agents by token over pinned TLS; people by OIDC; CI by the `UPDATER_API_KEY`
bearer token on `POST /update` only. Every state-changing UI request is a POST with a
CSRF token.
## Gotchas ## Gotchas
- **Registry access is HEAD only** (`internal/master/registry`). Docker Hub counts a manifest
GET as a pull; the client never falls back to GET, and a test asserts this.
- **`image_versions` uses `''` instead of NULL** for a missing tag or digest, so
`UNIQUE (repo, tag, digest)` deduplicates. The webhook records `(repo, tag, '')`; that
row is hidden from `RecentVersions` when a row with the same tag and a digest was first
seen at or after it.
- **The UI picks a version by `image_versions.id`, never by a ref from the form**, and the row
must belong to the service's `image_repo`. A digest-pinned or tagless pick needs agent
protocol 3 (`protocol.MinPinnedRefVersion`); `Hub.Route` takes the minimum version.
- **Picking a tagless version (`repo@digest`) leaves the service following no tag:** it is no
longer polled and the plain Update button is refused.
- **`services.last_tag_digest`** is set by the poller when it starts an auto job and when it
sees the service already running the tag's digest, and by `FinishJob` after any successful
plain tag job (not pinned/tagless picks; the digest is the one the agent pinned). The poller
starts no job for that digest, nor for one `desired_ref` (`ref@digest`) already names, and
re-reads the service before `Start`; so a rollback of an auto service holds until the
registry shows a different digest. `ErrJobActive` and `NotUpdatableError` leave it unset so
the next cycle retries.
- **A bare image's running version is recorded with tag `latest`** (like `jobs.FollowedRef`);
`repo@digest` is recorded with tag `''`. Only `sha256:<64 hex>` digests are recorded.
- **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where - **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where
the first colon is a port, not a tag. See `internal/imageref`. the first colon is a port, not a tag. See `internal/imageref`.
- **Path safety uses `filepath.Rel` + ".." prefix check**, NOT `strings.HasPrefix`. - **Path safety uses `filepath.Rel` + ".." prefix check**, NOT `strings.HasPrefix`.
@@ -89,7 +115,19 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
- **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged. - **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged.
- **Swarm updates need manager-node API access** for the agent doing the update. - **Swarm updates need manager-node API access** for the agent doing the update.
- **The image's default entrypoint is `updater-agent`.** The master overrides it with - **The image's default entrypoint is `updater-agent`.** The master overrides it with
`updater-master`. There is no `POST /update` until phase 3. `updater-master`.
- **The webhook waits via a broker signal plus a store re-read, with a fallback poll**
(the broker drops signals for slow subscribers). Marking and starting jobs runs on
`context.WithoutCancel`, so a client that disconnects cannot leave a queued job no agent
knows about; only the waiting stops. The master never fails or cancels a timed-out job.
- **A webhook job's target is `<image_repo>:<request tag>`**, built from the service's own
repo, never from the request's image string. With an explicit target the "follows no tag"
refusal does not apply; removed, self, built and container still do.
- **`available_*` is cleared by `FinishJob`** on success only when the job's `to_ref`
(as created, without `@digest`) equals `available_ref`, `available_at <= job.created_at`
(that version is now running) and, when the job's `to_ref` carries a digest, that digest
equals `available_digest`. A rollback to an older digest of the tag therefore keeps the
newer suggestion, and a digest-pinned pick does not clear a digest-less (webhook) one.
- **Job output is capped at `protocol.MaxJobOutput` (128 KiB, tail).** NUL bytes are - **Job output is capped at `protocol.MaxJobOutput` (128 KiB, tail).** NUL bytes are
stripped by the hub because Postgres text rejects them. stripped by the hub because Postgres text rejects them.
@@ -98,10 +136,11 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md` (authoritative) - Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md` (authoritative)
- Phase 1 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md` - Phase 1 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md`
- Phase 2 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md` - Phase 2 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md`
- Original single-instance design (historical, its webhook is gone until phase 3): - Phase 3 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-3-webhook.md`
- Phase 4 plan: `docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md`
- Original single-instance design (historical):
`docs/superpowers/specs/2026-05-22-package-updater-design.md` `docs/superpowers/specs/2026-05-22-package-updater-design.md`
- Consumer-side CI integration: `gitea-action/` (calls `POST /update`, which no longer - Consumer-side CI integration: `gitea-action/` (calls the master's `POST /update`)
exists until phase 3)
## Conventions ## Conventions
+164 -8
View File
@@ -8,9 +8,8 @@ the same image: `updater-master` and `updater-agent` (the default entrypoint).
Design: [master/agent spec](docs/superpowers/specs/2026-10-12-master-agent-design.md). Design: [master/agent spec](docs/superpowers/specs/2026-10-12-master-agent-design.md).
Phase 2 plan: [jobs and manual update](docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md). Phase 2 plan: [jobs and manual update](docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md).
Phase 3 plan: [webhook and auto mode](docs/superpowers/plans/2026-10-12-master-agent-phase-3-webhook.md).
> **Webhook:** the old `POST /update` endpoint is gone. It returns in phase 3 on the Phase 4 plan: [registry polling and version history](docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md).
> master; until then the Gitea action in `gitea-action/` has nothing to call.
## What an update does ## What an update does
@@ -20,9 +19,9 @@ runs is exactly what was pulled. The log streams to the browser while it runs; j
history is kept in Postgres. A service without a tag (`repo@digest`), a plain container, history is kept in Postgres. A service without a tag (`repo@digest`), a plain container,
and the updater's own services cannot be updated, and the UI says why. A bare image and the updater's own services cannot be updated, and the UI says why. A bare image
(`image: postgres`, no tag and no digest) counts as `postgres:latest`; only a reference (`image: postgres`, no tag and no digest) counts as `postgres:latest`; only a reference
with a digest and no tag (`repo@sha256:...`) "follows no tag". Compose services with a `build:` section are shown but never updated with a digest and no tag (`repo@sha256:...`) "follows no tag". Compose services with a
("built from source (build:)"). One update at a `build:` section are shown but never updated ("built from source (build:)"). One update at
time per service, and one job at a time per agent. a time per service, and one job at a time per agent.
## Deploying the master ## Deploying the master
@@ -31,6 +30,162 @@ needs Postgres and an OIDC provider). The master listens for the UI on `PORT` (d
`8080`, put it behind your reverse proxy) and for agents on `AGENT_PORT` (default `8090`, `8080`, put it behind your reverse proxy) and for agents on `AGENT_PORT` (default `8090`,
pinned TLS, reached directly on the internal network). pinned TLS, reached directly on the internal network).
| Variable | Required | Default | Purpose |
|---|---|---|---|
| `UPDATER_API_KEY` | no | — | Bearer token for `POST /update`. Without it the webhook answers `503` and the master logs `webhook_disabled`. |
| `UPDATE_TIMEOUT` | no | `5m` | How long one `POST /update` waits for its auto jobs (Go duration). |
| `POLL_INTERVAL` | no | `6h` | Time between registry polling cycles (Go duration, must be positive). The first cycle runs one minute after start. |
| `DOCKER_CONFIG` | no | `~/.docker` | Directory with a `config.json` holding registry credentials for polling. Only inline `auths` entries work; see [Registry polling](#registry-polling). |
The master's other settings (`DATABASE_URL`, `PUBLIC_URL`, `AGENT_URL`, `OIDC_*`,
`ALLOWED_GROUP`, `AGENT_IMAGE`, `PORT`, `AGENT_PORT`, `LOG_LEVEL`) are shown in
[`deploy/master.stack.yml`](deploy/master.stack.yml).
## Webhook (CI)
CI calls the master after pushing an image. The Gitea action in
[`gitea-action/`](gitea-action/README.md) does this; any HTTP client works.
```
POST /update
Authorization: Bearer <UPDATER_API_KEY>
Content-Type: application/json
{"image": "registry.example.com/myapp", "tag": "1.5.0"}
```
`image` is matched against the image repo of every service on every agent (tag and digest
in the request are ignored for matching; matching is case-sensitive). `tag` is optional.
For every match the master records the version as *available* (the overview shows an
"update available" badge), including services that cannot be updated automatically
(updater's own, `build:` and plain containers), so the badge shows for them too. Services
in **manual** mode stop there. Services in **auto**
mode also get an update job, and the request waits for those jobs to finish, at most
`UPDATE_TIMEOUT`.
**`tag` semantics.** The target of an auto job is `<service's image repo>:<tag>`, built
from the service's own repo and never from the request's `image` string. The tag must be
one that has been pushed. Compose services get pinned to it through
`compose.override.yaml`, as in any update. Omit `tag` to re-pull the tag the service
already follows (a bare image counts as `:latest`); a service that follows no tag
(`repo@digest`) is then not touched and is reported as `refused` ("no tag given and the
service follows no tag"), manual or auto.
With a `tag`, the "follows no tag" refusal does not apply; the other refusals (the
updater's own services, `build:` services, plain containers) still do.
Response (`200` or `500`):
```json
{"request_id": "…", "image": "registry.example.com/myapp", "tag": "1.5.0", "matched": 2,
"results": [
{"service_id": 12, "service": "myapp/web", "kind": "compose", "mode": "auto",
"status": "succeeded", "job_id": 42, "job_url": "https://updater.example.com/jobs/42",
"to_ref": "registry.example.com/myapp:1.5.0", "duration_ms": 8210},
{"service_id": 15, "service": "myapp/worker", "kind": "compose", "mode": "manual",
"status": "marked", "to_ref": "registry.example.com/myapp:1.5.0", "duration_ms": 3}]}
```
| `status` | Meaning | Counts as failure |
|---|---|---|
| `succeeded` | Auto job finished successfully. | no |
| `failed` | Auto job failed (for example agent offline), or the version could not be recorded. `error` says why. | yes |
| `timeout` | Job still running when `UPDATE_TIMEOUT` passed. The master leaves it running. | yes |
| `marked` | Manual service; the version was recorded only. | no |
| `busy` | The service already has a queued or running job; `job_id` is that job, which is not waited for. | no |
| `refused` | The service may not be updated, or there is no target (no `tag` given and the service follows no tag). `error` has the reason. | no |
| Code | When |
|---|---|
| `200` | No result is `failed` or `timeout`. |
| `400` | Body is not JSON, `image` is missing, or `tag` is not a valid Docker tag. |
| `401` | Missing or wrong bearer token. |
| `404` | No service uses the image (catches typos in CI configuration). |
| `500` | At least one result is `failed` or `timeout`. |
| `503` | `UPDATER_API_KEY` is not set on the master. |
The webhook never creates services: the first deployment of a stack is done by hand. A
new service always starts in **manual** mode; switch it with the *Make auto* button on the
overview or the service page. Auto is refused for the updater's own services, `build:`
services and plain containers.
> **Warning:** auto mode deploys whatever CI pushes, without anyone looking at it. Using
> it on critical services is your own risk.
The request is open for as long as the jobs run, so the reverse proxy in front of the
master must allow a read timeout longer than `UPDATE_TIMEOUT`. If the client disconnects,
marking and starting jobs still complete; only the waiting stops.
## Registry polling
Besides the webhook, the master asks each registry whether the tag a service follows has
a new digest. Every `POLL_INTERVAL` it sends one `HEAD /v2/<repo>/manifests/<tag>` per
distinct followed `repo:tag` (a bare image counts as `:latest`; `repo:tag@digest` is polled
as `repo:tag`). It never downloads a manifest: Docker Hub counts a `GET` as a pull against
the rate limit and a `HEAD` as nothing
([source](https://docs.docker.com/docker-hub/usage/pulls/)), and Docker Hub also has a
separate abuse limit for many pulls from one IP address, so the poller stays HEAD-only. For a
multi-arch image the digest is the index digest, the same one `docker inspect` shows in
`RepoDigests`.
- **What counts as new.** A digest different from the one the service runs becomes the
service's *available* version (the overview badge, source `poll`). A digest the master
already suggested is not suggested again. A service in **auto** mode also gets an update
job to `repo:tag` (the agent resolves the digest itself, so agents on protocol version 2
keep working). The master remembers the newest digest it started or deployed for the tag
in `services.last_tag_digest` (set when polling starts a job, when polling sees the
service already running that digest, and when any plain tag job succeeds, whoever started
it) and starts no job for that digest again, whether the job
failed or the user rolled back afterwards. If the service already had a job when the
digest was found, the next cycle tries again.
- **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and
services without a registry digest (locally built images). The service page says which.
The updater's own services and plain containers are polled, so their badge works, but
jobs for them are still refused.
- **Errors.** A failed lookup is stored for the services that follow that tag and shown as a
"poll failed" badge on the overview and in the "Polling" row of the service page; other
tags continue. A later success clears it.
- **Credentials.** The master reads `$DOCKER_CONFIG/config.json` (a file written by
`docker login`), mounted read-only into the container; without it registries are accessed
anonymously. Only inline `auths` entries are supported. A `credsStore` or `credHelpers`
entry makes every lookup fail with `error getting credentials ... executable file not found
in $PATH`, because the container has no credential helper binaries. The master logs
`registry_credentials` at startup with the directory and whether `config.json` exists,
never its contents.
- **Requests go wherever the image name points.** The master sends the lookup to the
registry host in the image name, including a `GET /v2/` ping that follows redirects.
Anyone who can start containers on a managed host can therefore make the master send
blind requests to internal hosts. The registry's answer is not shown: an error keeps only
the HTTP status and the registry's error codes; any other failure shows only "timeout",
"cannot reach registry" or "lookup failed". The full error goes to the master's log
(`registry_lookup_failed`).
### Version history and rollback
The master records every version it sees (webhook, polling, and what agents report as
running) in `image_versions`. The service page lists the 10 most recently first seen
versions of the service's image repo, marks the one that is running, and has a *Deploy*
button on each row. A webhook announces a tag but not its digest, so such a row has no
digest until polling or an agent reports one; it is hidden when a row with the same tag
and a digest was first seen at or after it. A bare image (`postgres`) is recorded as tag
`latest`.
Picking a version starts a manual job to exactly that reference. Picking an older one is a
rollback, including to a digest-pinned reference. A version without a tag deploys
`repo@digest` and the service then follows no tag, so it is no longer polled or updatable
with the plain *Update* button until it is moved to a tagged version again. Picks of
digest-pinned or tagless versions need an agent speaking protocol version 3 (this
release); on an older agent the job fails with "agent too old to deploy a pinned version
(needs protocol 3)".
A rollback made before the first poll cycle after the upgrade has seen the service is not
protected, because the master has not yet recorded the digest it was running.
A service in **auto** mode stays auto after a rollback. Polling does not undo it: the
service stays on the older version until the registry shows a digest of the followed tag
that differs from the newest one the master deployed. The newer version it was rolled back
from stays marked as available. A webhook that names a tag still starts a job for an auto
service every time, so a CI push moves it forward again.
## Adding agents ## Adding agents
Open the master's **Agents** page and add one. It shows a ready-to-run compose file with Open the master's **Agents** page and add one. It shows a ready-to-run compose file with
@@ -115,8 +270,9 @@ crash-loop the next time its image is pulled. Anything that pulls `:latest` for
instance triggers this: its own `/update` webhook, Watchtower or Shepherd. instance triggers this: its own `/update` webhook, Watchtower or Shepherd.
Pinning means two things: pin the old instance to the last pre-phase-2 tag **and** remove Pinning means two things: pin the old instance to the last pre-phase-2 tag **and** remove
its `se.shcizo.auto-update` label, so nothing updates it to the new image. Or do not tag a its `se.shcizo.auto-update` label, so nothing updates it to the new image. The master
release until phase 3 moves `POST /update` to the master. serves `POST /update` again (see [Webhook](#webhook-ci)); point the Gitea action's
`endpoint` there and set `UPDATER_API_KEY` on the master.
Master and agents must be on this release together: a phase-1 agent connects but every Master and agents must be on this release together: a phase-1 agent connects but every
job sent to it fails "agent too old for updates". job sent to it fails "agent too old for updates".
+57 -4
View File
@@ -11,6 +11,7 @@ import (
"net/http" "net/http"
"os" "os"
"os/signal" "os/signal"
"path/filepath"
"strings" "strings"
"syscall" "syscall"
"time" "time"
@@ -20,9 +21,12 @@ import (
"github.com/shcizo/package-updater/internal/master/events" "github.com/shcizo/package-updater/internal/master/events"
"github.com/shcizo/package-updater/internal/master/hub" "github.com/shcizo/package-updater/internal/master/hub"
"github.com/shcizo/package-updater/internal/master/jobs" "github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/poller"
"github.com/shcizo/package-updater/internal/master/registry"
"github.com/shcizo/package-updater/internal/master/store" "github.com/shcizo/package-updater/internal/master/store"
"github.com/shcizo/package-updater/internal/master/web" "github.com/shcizo/package-updater/internal/master/web"
"github.com/shcizo/package-updater/internal/master/webauth" "github.com/shcizo/package-updater/internal/master/webauth"
"github.com/shcizo/package-updater/internal/master/webhook"
"github.com/shcizo/package-updater/internal/pin" "github.com/shcizo/package-updater/internal/pin"
) )
@@ -36,6 +40,29 @@ func main() {
} }
} }
// pollFirstDelay is how long the master runs before its first registry
// polling cycle, so agents have reported their inventory.
const pollFirstDelay = time.Minute
// pollWorkers is how many registry lookups run at once.
const pollWorkers = 4
// registryCredentialsDir returns the directory the registry client reads
// credentials from (DOCKER_CONFIG, else ~/.docker) and whether a config.json
// is in it. Only the path is logged, never the file.
func registryCredentialsDir() (dir string, found bool) {
dir = os.Getenv("DOCKER_CONFIG")
if dir == "" {
home, err := os.UserHomeDir()
if err != nil {
return "", false
}
dir = filepath.Join(home, ".docker")
}
_, err := os.Stat(filepath.Join(dir, "config.json"))
return dir, err == nil
}
func run() error { func run() error {
cfg, err := config.Load() cfg, err := config.Load()
if err != nil { if err != nil {
@@ -80,6 +107,7 @@ func run() error {
ui, err := web.New(st, agentHub, jobService, broker, web.Config{ ui, err := web.New(st, agentHub, jobService, broker, web.Config{
AgentURL: cfg.AgentURL, MasterKey: fingerprint, AgentImage: cfg.AgentImage, AgentURL: cfg.AgentURL, MasterKey: fingerprint, AgentImage: cfg.AgentImage,
PollInterval: cfg.PollInterval,
}, logger) }, logger)
if err != nil { if err != nil {
return err return err
@@ -93,6 +121,14 @@ func run() error {
} }
_, _ = w.Write([]byte("ok")) _, _ = w.Write([]byte("ok"))
}) })
// The webhook does its own bearer-token check, so it sits outside the
// OIDC login like /healthz.
mux.Handle("POST /update", webhook.New(st, jobService, broker, webhook.Config{
Token: cfg.UpdaterAPIKey, Timeout: cfg.UpdateTimeout, PublicURL: cfg.PublicURL,
}, logger))
if cfg.UpdaterAPIKey == "" {
logger.Warn("webhook_disabled", "reason", "UPDATER_API_KEY not set")
}
mux.HandleFunc("GET /auth/login", auth.Login) mux.HandleFunc("GET /auth/login", auth.Login)
mux.HandleFunc("GET /auth/callback", auth.Callback) mux.HandleFunc("GET /auth/callback", auth.Callback)
mux.Handle("POST /auth/logout", auth.Require(http.HandlerFunc(auth.Logout))) mux.Handle("POST /auth/logout", auth.Require(http.HandlerFunc(auth.Logout)))
@@ -103,8 +139,8 @@ func run() error {
// instead of holding Shutdown until its timeout. // instead of holding Shutdown until its timeout.
baseContext := func(net.Listener) context.Context { return ctx } baseContext := func(net.Listener) context.Context { return ctx }
// No WriteTimeout on either server: SSE streams and agent connections // No WriteTimeout on either server: SSE streams, agent connections and
// are long-lived by design. // POST /update (which waits up to UPDATE_TIMEOUT) are long-lived by design.
webSrv := &http.Server{ webSrv := &http.Server{
Addr: ":" + cfg.Port, Handler: mux, ReadHeaderTimeout: 10 * time.Second, Addr: ":" + cfg.Port, Handler: mux, ReadHeaderTimeout: 10 * time.Second,
BaseContext: baseContext, BaseContext: baseContext,
@@ -118,22 +154,39 @@ func run() error {
BaseContext: baseContext, BaseContext: baseContext,
} }
credDir, credFound := registryCredentialsDir()
logger.Info("registry_credentials", "docker_config", credDir, "config_json_found", credFound)
pollCtx, stopPolling := context.WithCancel(ctx)
defer stopPolling()
registryPoller := poller.New(st, jobService, registry.New("package-updater-master/"+version, logger), broker, poller.Config{
Interval: cfg.PollInterval, FirstDelay: pollFirstDelay, Workers: pollWorkers,
}, logger)
pollDone := make(chan struct{})
go func() {
defer close(pollDone)
registryPoller.Run(pollCtx)
}()
errCh := make(chan error, 2) errCh := make(chan error, 2)
go func() { errCh <- webSrv.ListenAndServe() }() go func() { errCh <- webSrv.ListenAndServe() }()
go func() { errCh <- agentSrv.ListenAndServeTLS("", "") }() go func() { errCh <- agentSrv.ListenAndServeTLS("", "") }()
logger.Info("listening", "web", webSrv.Addr, "agents", agentSrv.Addr) logger.Info("listening", "web", webSrv.Addr, "agents", agentSrv.Addr)
var serveErr error
select { select {
case <-ctx.Done(): case <-ctx.Done():
logger.Info("shutdown_signal") logger.Info("shutdown_signal")
case err := <-errCh: case err := <-errCh:
if !errors.Is(err, http.ErrServerClosed) { if !errors.Is(err, http.ErrServerClosed) {
return err serveErr = err
} }
} }
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel() defer cancel()
_ = webSrv.Shutdown(shutdownCtx) _ = webSrv.Shutdown(shutdownCtx)
_ = agentSrv.Shutdown(shutdownCtx) _ = agentSrv.Shutdown(shutdownCtx)
return nil // The poller writes to the store, which is closed when run returns.
stopPolling()
<-pollDone
return serveErr
} }
+19 -2
View File
@@ -1,4 +1,4 @@
# Example Swarm stack for the package-updater master (phase 2: inventory and manual updates). # Example Swarm stack for the package-updater master (inventory, updates and the CI webhook).
# Deploy on the manager: docker stack deploy -c master.stack.yml package-updater-master # Deploy on the manager: docker stack deploy -c master.stack.yml package-updater-master
# #
# Before deploying: # Before deploying:
@@ -6,19 +6,34 @@
# 2. In Authentik: create an OAuth2/OpenID provider + application with redirect URI # 2. In Authentik: create an OAuth2/OpenID provider + application with redirect URI
# https://updater.example.com/auth/callback, and a group "updater-admins". # https://updater.example.com/auth/callback, and a group "updater-admins".
# 3. docker secret create pu_oidc_client_secret - ; docker secret create pu_database_url - # 3. docker secret create pu_oidc_client_secret - ; docker secret create pu_database_url -
# 4. docker secret create pu_updater_api_key - (required by this example; it enables POST /update)
# To run without the webhook, remove the pu_updater_api_key lines (the UPDATER_API_KEY
# part of the entrypoint, the service secret and the secrets entry); the endpoint then answers 503.
# The reverse proxy must allow a read timeout longer than UPDATE_TIMEOUT (default 5m).
# 5. Optional, for registry polling of private images: put a config.json written by
# `docker login` (inline "auths" only; no credsStore) in /path/to/docker-config and
# uncomment DOCKER_CONFIG and the volumes entry below. The directory must exist on every
# manager node (the master is placed on a manager), or Swarm rejects the task. Without
# it registries are accessed anonymously.
services: services:
master: master:
image: gitea.shcizo.se/shcizo/package-updater:latest image: gitea.shcizo.se/shcizo/package-updater:latest
entrypoint: ["/bin/sh", "-c", "DATABASE_URL=$$(cat /run/secrets/pu_database_url) OIDC_CLIENT_SECRET=$$(cat /run/secrets/pu_oidc_client_secret) exec /usr/local/bin/updater-master"] entrypoint: ["/bin/sh", "-c", "DATABASE_URL=$$(cat /run/secrets/pu_database_url) OIDC_CLIENT_SECRET=$$(cat /run/secrets/pu_oidc_client_secret) UPDATER_API_KEY=$$(cat /run/secrets/pu_updater_api_key) exec /usr/local/bin/updater-master"]
environment: environment:
PUBLIC_URL: https://updater.example.com PUBLIC_URL: https://updater.example.com
AGENT_URL: wss://10.0.0.5:8090/agent AGENT_URL: wss://10.0.0.5:8090/agent
OIDC_ISSUER: https://auth.example.com/application/o/package-updater/ OIDC_ISSUER: https://auth.example.com/application/o/package-updater/
OIDC_CLIENT_ID: package-updater OIDC_CLIENT_ID: package-updater
ALLOWED_GROUP: updater-admins ALLOWED_GROUP: updater-admins
# UPDATE_TIMEOUT: 5m
# POLL_INTERVAL: 6h
# DOCKER_CONFIG: /docker-config # step 5
secrets: secrets:
- pu_database_url - pu_database_url
- pu_oidc_client_secret - pu_oidc_client_secret
- pu_updater_api_key
# volumes: # step 5
# - /path/to/docker-config:/docker-config:ro
ports: ports:
# Agents connect here directly over the internal network, not via the proxy. # Agents connect here directly over the internal network, not via the proxy.
- target: 8090 - target: 8090
@@ -35,6 +50,8 @@ secrets:
external: true external: true
pu_oidc_client_secret: pu_oidc_client_secret:
external: true external: true
pu_updater_api_key:
external: true
networks: networks:
proxy: proxy:
@@ -0,0 +1,148 @@
# Master/Agent Phase 3 — Webhook and Auto Mode Implementation Plan
> This plan documents what was built for issue #8 and where it deviates from the spec. Code in "Frozen contracts" is final; tasks give the contract and the test cases.
**Goal:** CI calls `POST /update` on the master after pushing an image. The master records the pushed version as *available* on every service that uses the image. Services in `auto` mode are updated at once and the request waits for the result, so CI sees a broken deploy. People switch a service between `manual` and `auto` in the UI.
**Architecture:** A new package `internal/master/webhook` is an `http.Handler` mounted on the master's mux outside the OIDC login; it does its own bearer-token check. It looks up services by image repo, writes `available_*` on each match, starts a job through `jobs.Service.Start` for the `auto` ones and waits for them. Waiting subscribes to the in-process broker before any job exists, reacts to `events.Jobs`, re-reads each pending job from Postgres on every signal and on a fallback ticker, and gives up at `UPDATE_TIMEOUT`. `FinishJob` clears `available_*` in the transaction that records a successful deploy.
**Tech Stack:** unchanged. No new dependencies.
**Spec:** `docs/superpowers/specs/2026-10-12-master-agent-design.md`, "Webhook" section. Phase 2 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-2-jobs.md` (its conventions still apply).
## Global Constraints
- `go test ./...` passes with no Docker daemon and no Postgres. Postgres tests carry `//go:build integration` and need `DATABASE_URL`. Run `go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/`.
- Interfaces are declared in the consuming package. Hand-written fakes, `stretchr/testify`, table-driven where there is more than one case.
- Every exported identifier gets a GoDoc comment. Conventional Commits; every commit message ends with ` (#8)`.
- The token (`UPDATER_API_KEY`) never appears in logs.
- Every state-changing UI request is a POST carrying the CSRF token (the mode toggle included).
- The HTTP server for the UI has no `WriteTimeout`, so a webhook request can wait up to `UPDATE_TIMEOUT`.
## Deliberate deviations from the spec in this phase
- **No `image_versions` table and no `available_digest` column.** Both belong to phase 4 (version history and the version picker). Phase 3 stores only `available_ref`, `available_source` (`webhook`) and `available_at` on `services`.
- **The old `207` is gone.** A request with both a working and a failed service answers `500` as a whole; there is no partial-success code. The Gitea action fails on any 4xx/5xx, so CI surfaces it.
- **`busy` and `refused` are not failures.** A service that already has an active job (`busy`, with that job's ID, not waited for) or may not be updated (`refused`, with the reason) does not make the request fail. Only `failed` and `timeout` do. A manual match is `marked`, also not a failure.
- **A webhook with a `tag` bypasses the "follows no tag" refusal.** The target is then explicit (`<service's image_repo>:<tag>`), so a `repo@digest` service can be moved to a tag. Removed, self, built and container refusals still apply. Without a `tag`, the target is the followed tag (bare image = `:latest`); a service that follows no tag is left alone (`available_*` untouched) and is reported `refused` ("no tag given and the service follows no tag"), manual or auto. Because of this, the UI allows `auto` for a service that follows no tag (`jobs.AutoAllowed`), unlike the Update button (`jobs.Updatable`).
- **The target ref is built from the service's own repo.** `imageref.Parse(req.Image).Repo` only selects services; it never reaches the job, so a request cannot point a service at another registry.
- **Token is optional on the master.** Unset `UPDATER_API_KEY` means `POST /update` answers `503` and startup logs `webhook_disabled`; the master still runs.
- **`tag` is validated** against Docker's tag grammar (`400 invalid tag`). The spec does not say; without it a tag such as `v1@sha256:x` or one with spaces would end up in the override file and fail on the agent.
- **An oversized body (over 64 KiB) answers `400 invalid JSON body`**, not `413`.
- **`UPDATE_TIMEOUT` exists on both sides** with the same name and parsing: on the agent it bounds one job, on the master it bounds one webhook request. The master never fails or cancels a job that outlives the request; the agent's own timeout does.
- **Marking and starting jobs runs on `context.WithoutCancel(r.Context())`.** A client that disconnects stops only the waiting; otherwise a queued job could be left that no agent knows about.
## Review Focus
| Failure case | Test that covers it |
|---|---|
| Wrong, missing or malformed token never reaches the lookup; unset key answers 503. | webhook `TestAuth` |
| An unknown or differently-cased image answers 404 and starts nothing. | webhook `TestNoMatch` |
| The image string is normalised (tag, digest, registry with port) before matching. | webhook `TestImageIsNormalisedBeforeMatching` |
| The auto job's target is built from the service's repo and the request tag, never the request's image string. | webhook `TestAutoJobTargetIsBuiltFromServiceRepo` |
| A job that fails at start (agent offline) is reported at once, not waited for. | webhook `TestJobFailedAtStartIsNotWaitedFor` |
| A dropped broker signal is recovered by the fallback poll. | webhook `TestDroppedSignalIsRecoveredByPolling` |
| A timed-out job is reported `timeout` and left untouched. | webhook `TestTimeout` |
| A client that disconnects does not abort marking or starting. | webhook `TestClientCancelStopsWaitingButNotStarting` |
| The webhook waits for the real agent's result and answers 200 / 500. | e2e `TestWebhookWaitsForAutoJob`, `TestWebhookFailedAutoJobIs500` |
| The available ref is cleared only by a successful job whose `to_ref` equals it and that was created at or after the suggestion; a different ref, a newer suggestion and a failed job keep it. | store `TestFinishJob_ClearsAvailableOnlyWhenThatVersionWasDeployed` |
| Re-reporting inventory never resets a service's update mode; new services are manual. | store `TestUpsertKeepsUpdateMode` |
| `auto` is refused for removed, self, built and container services; `manual` is always allowed. | jobs `TestAutoAllowed`; web `TestSetMode_AutoRefusedShows409`, `TestSetMode_ManualAlwaysAllowed` |
## Frozen contracts
### C1. Migration `internal/master/store/migrations/0003_webhook.sql` (complete)
```sql
ALTER TABLE services ADD COLUMN available_ref text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN available_source text NOT NULL DEFAULT ''
CHECK (available_source IN ('', 'webhook'));
ALTER TABLE services ADD COLUMN available_at timestamptz;
CREATE INDEX services_by_image_repo ON services (image_repo) WHERE removed_at IS NULL;
```
### C2. `internal/master/model` additions
`Service` gains `AvailableRef string`, `AvailableSource string`, `AvailableAt *time.Time`. Constants `ModeManual`, `ModeAuto`, `SourceWebhook`. `type Available struct { Ref, Source string; At time.Time }`.
### C3. Store methods
```go
ServicesByRepo(ctx, repo string) ([]model.Service, error) // non-removed, same order as ListServices
SetAvailable(ctx, id int64, a model.Available) error
SetUpdateMode(ctx, id int64, mode string) error // model.ErrNotFound for unknown or removed
```
`FinishJob` additionally reads the job's `created_at` and, on success, clears `available_*` in the same transaction when the job's `to_ref` (as created, without `@digest`) equals `available_ref` and `available_at <= created_at`.
### C4. `internal/master/jobs`
```go
type StartRequest struct {
Trigger model.JobTrigger
RequestedBy string
Ref string // empty: the service's followed tag; set: used as is
}
func (s *Service) Start(ctx, svc model.Service, req StartRequest) (model.Job, error)
func (s *Service) AutoAllowed(svc model.Service) error
func FollowedRef(actualRef string) string // "repo:tag", or "" when no tag is followed
```
`Start` has the return contract `Trigger` had: `*model.NotUpdatableError` with no job, `model.ErrJobActive` with the existing job, and an already-failed job for offline / too-old / start failures. `Trigger(ctx, id, by)` is `GetService` plus `Start` with `TriggerManual`.
### C5. `internal/master/webhook`
```go
type Config struct { Token string; Timeout time.Duration; PublicURL string }
func New(store Store, jobs Jobs, broker Broker, cfg Config, log *slog.Logger) *Handler
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)
```
Request `{"image","tag"}`; response `{"request_id","image","tag","matched","results":[{"service_id","service","kind","mode","status","job_id","job_url","to_ref","error","duration_ms"}]}`. Statuses `succeeded | failed | timeout | marked | busy | refused`; HTTP `500` when any result is `failed` or `timeout`, else `200`. Other codes: `400`, `401`, `404`, `503`. See the README for the full table.
### C6. Master configuration
`UPDATER_API_KEY` (optional) and `UPDATE_TIMEOUT` (default `5m`, positive Go duration, same error text style as the agent's).
### C7. Web
`POST /services/{id}/mode` with `mode=manual|auto` and optional `return=overview`. `303` to the service page (or `/`), `400` for a bad mode, `404` for an unknown or removed service, `409` re-rendering the service page with "Cannot enable auto: <reason>." Publishes `events.Services`.
## Tasks
### Task 1: Store and model
Migration C1, model C2, store C3. Integration tests (`//go:build integration`): `ServicesByRepo` excludes removed rows and other repos; `SetAvailable` round trip; `SetUpdateMode` including `ErrNotFound`; new service is manual and upsert/snapshot keep the mode; `FinishJob` clears the same ref, keeps a different ref, keeps newer, keeps on failure.
### Task 2: Jobs service
Extract `Start` from `Trigger`; `updatable(svc, explicitRef)`; `AutoAllowed`; `FollowedRef`. Tests: webhook trigger with explicit ref is recorded; explicit ref bypasses only the no-tag refusal; `AutoAllowed` table; existing `TestTrigger*` stay green.
### Task 3: Config, webhook, wiring
Config C6 with tests. Handler C5 with fakes and the real `events.Broker`. `cmd/master/main.go` mounts `POST /update` directly on the mux and logs `webhook_disabled` when the key is empty.
### Task 4: Web
Store gains `SetUpdateMode`, Jobs gains `AutoAllowed`. Overview: mode column with a toggle form (hidden `csrf_token`, `mode`, `return=overview`; disabled with the reason as visible text when auto is not allowed) and an "update available" badge. Service page: Available row, Mode row, toggle form, 409 error. The forms are inside the `services` template that the SSE fragment re-renders, so they survive a refresh.
### Task 5: End-to-end
`internal/e2e/webhook_test.go`: `TestWebhookWaitsForAutoJob` (real in-process master and agent, gate executor, the handler on an `httptest` server) and `TestWebhookFailedAutoJobIs500`. The in-memory `memStore` gains `ServicesByRepo`, `SetAvailable` and the same clearing rule as the SQL.
### Task 6: Documentation
README (webhook section, master variables, warning about `auto`), `gitea-action/README.md` (do not change `action.yml`), `deploy/master.stack.yml` (`UPDATER_API_KEY` as a Docker secret), `CLAUDE.md`, this plan.
## Final verification
```bash
gofmt -l . ; go vet ./...
go build ./cmd/master ./cmd/agent && rm -f master agent
go test ./...
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/ # when a Postgres is available
```
Manual smoke test: set `UPDATER_API_KEY`, make a throwaway Compose service `auto`, then `curl -X POST -H "Authorization: Bearer $KEY" -d '{"image":"<repo>","tag":"<pushed tag>"}' https://<master>/update`. Expect `200` with `succeeded` after the job finishes, the job visible in the UI with trigger `webhook`, and the "update available" badge gone.
@@ -0,0 +1,311 @@
# Master/Agent Phase 4 — Registry Polling and Version History Implementation Plan
> This plan documents what to build for issue #9 and where it deviates from the spec. Code in "Frozen contracts" is final; tasks give the contract and the test cases.
**Goal:** The master records every version it sees (webhook, registry polling, running inventory) in `image_versions`. Every `POLL_INTERVAL` (default 6h) it checks each distinct followed `repo:tag` against its registry with one HEAD request. A new digest becomes the service's available suggestion, and `auto` services get a job. The Service page lists the 10 most recent versions of the service's repo. Picking one starts a manual job to exactly that reference; picking an older one is a rollback, including to a digest-pinned or tagless reference.
**Architecture:**
- New package `internal/master/registry`: thin wrapper around `go-containerregistry`'s `remote.Head` with `authn.DefaultKeychain` (reads `DOCKER_CONFIG`/`config.json`).
- New package `internal/master/poller`: owns the cycle (list services, group by followed ref, run HEADs with bounded concurrency, then set available / start auto jobs / store poll error per service). `cmd/master` runs it as a background goroutine on the signal context.
- Store: `image_versions` written in `SetAvailable` (webhook and poll, same transaction) and in `upsertService` (running inventory, inside the existing snapshot/upsert transactions).
- Jobs gains `TriggerVersion`, which takes a version ID, never a ref from the form. `Hub.Route` gains a minimum protocol version so a pinned or tagless ref only goes to agents that understand it.
- Agent: resolver passes digest-pinned refs through unchanged. Protocol version goes to 3.
**Tech Stack:** adds `github.com/google/go-containerregistry` (`pkg/name`, `pkg/v1/remote`, `pkg/authn`; `pkg/registry` and `pkg/v1/random` in tests only).
**Spec:** `docs/superpowers/specs/2026-10-12-master-agent-design.md`, sections "Versions: what the system has seen", "Data model", "Update", "Failure handling", "Registry polling", "UI". Phase 3 plan conventions still apply.
## Global Constraints
- `go test ./...` passes with no Docker, no Postgres and no network. Registry tests use `httptest` plus `ggcr/pkg/registry`. Postgres tests carry `//go:build integration`.
- Interfaces declared at the consumer (`poller.Store`, `poller.Jobs`, `poller.Registry`). Hand-written fakes and testify, table-driven where there is more than one case.
- GoDoc on every exported identifier. Conventional Commits, each ending with ` (#9)`.
- Registry credentials never appear in logs or in `last_poll_error`. Error text stored truncated to 500 bytes.
- Registry traffic is HEAD only: never `GET /v2/<repo>/manifests/...` (Docker Hub counts GETs as pulls). A test asserts this.
- A pick from the UI is a POST with the CSRF token and a version ID. The ref is built server-side from the `image_versions` row, and the row must belong to the service's `image_repo`.
- The poller never overlaps itself. Next cycle is scheduled after the previous one finishes.
- `go.mod` `go` directive must stay `1.26.3` after `go get`.
## Deliberate deviations from the spec in this phase
1. **Webhook records `(repo, tag, '')`, no digest.** The master does not resolve the digest at webhook time (no registry round-trip/credentials in the CI path). The digest arrives later from polling or running inventory. The picker hides a digest-less row when a row with the same `(repo, tag)` and a digest was first seen at or after it.
2. **Empty string instead of NULL** for missing tag or digest in `image_versions`, so `UNIQUE (repo, tag, digest)` works. A CHECK forbids both being empty.
3. **`first_seen_at` is never bumped.** Re-observation is `ON CONFLICT DO NOTHING`; `source` is the first source that saw it. "10 most recent" = "10 most recently first seen". A new digest of a tag is a new row at the top; old digest stays below (enables rollback of moving tags).
4. **Not polled:** services that follow no tag, `build:` services, services with empty `actual_digest`. UI shows the reason. Self and container services ARE polled (badge informative; jobs still refused by `jobs`).
5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, the badge is not written again (no `available_at` bump). The auto job is decided separately: it starts only if the service is auto, `actual_digest != D`, `D != last_tag_digest` and `desired_ref != ref@D`, on a fresh read of the row. `last_tag_digest` is set when a job row was created (not on `ErrJobActive`/`NotUpdatableError`, which retry next cycle). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears. The poller also sets `last_tag_digest` to D when `actual_digest == D` (seeding services that predate the column). `FinishJob` also sets `last_tag_digest` to the pinned digest after any successful job whose `to_ref` as created has no digest (manual, webhook or poll); pinned and tagless picks leave it, so a rollback holds whoever started the update.
6. **FinishJob clearing rule gets a digest condition** (C3). Rollback to an older digest of the same tag no longer clears the newer-version badge.
7. **Poll auto jobs use `repo:tag`, not `repo:tag@D`.** v2 agents keep working for poll and webhook jobs. Only UI picks of digest-pinned or tagless versions need a v3 agent.
8. **First cycle runs 1 minute after start**, then every `POLL_INTERVAL` measured from the end of the previous cycle.
9. **No `last_polled_at` column** (YAGNI).
10. **Overview gets two new badges: "follows no tag" and "poll failed"** (error as `title`). Service page gets a "Polling" row. "Drifted" is not in this phase.
11. **Credentials from `DOCKER_CONFIG`** (a directory), same name and meaning as on the agent. Only inline `auths` entries supported in the container.
## Implementation notes (differences found while building)
- The plan's test names `TestResolve` and `TestComposeWritesPinnedRef` became `TestResolver`
(extended), a subtest of `TestComposeExecutor` and `TestSwarmUsesPinnedRef`.
- Tasks 2 and 3 are one commit: changing `Hub.Route`'s signature breaks `jobs` and the e2e hub
wrapper until both are updated.
- `Poller.Cycle` runs the registry lookups concurrently and then records the results
sequentially (on `context.WithoutCancel`).
- `web.Config` gained `PollInterval` for the "every <interval>" text.
- `authn.DefaultKeychain` with a `credsStore` whose helper binary is missing makes the lookup fail
with `error getting credentials - err: exec: "docker-credential-...": executable file not found in $PATH`
(covered by `TestCredsStoreWithoutHelperFailsTheLookup`); it does not fall back to anonymous access.
## Review Focus
| Failure case | Test that covers it |
|---|---|
| Registry access is HEAD only; no manifest GET ever happens. | registry `TestDigestUsesHeadOnly` |
| Multi-arch: digest returned is the index digest (same as `RepoDigests` / Swarm spec). | registry `TestDigestOfIndexIsIndexDigest` |
| Credentials from `$DOCKER_CONFIG/config.json` are sent; anonymous works without the file. | registry `TestCredentialsFromDockerConfig`, `TestAnonymousWithoutConfig` |
| One HEAD per distinct followed ref (`repo:tag`, `repo:tag@d`, bare → `:latest`). | poller `TestOneHeadPerDistinctRef` |
| No tag / built / no digest are skipped, no HEAD. | poller `TestSkipped` |
| Error on one ref goes into its services' `last_poll_error`; others continue; later success clears it. | poller `TestErrorIsPerRef`, `TestErrorClearedOnSuccess` |
| Already-suggested digest neither re-suggested nor re-started. | poller `TestSameSuggestionIsIdempotent` |
| Auto gets exactly one `TriggerPoll` job to `repo:tag`; manual only gets marked. | poller `TestNewDigest` table |
| `ErrJobActive` / `NotUpdatableError` from `Start` do not abort the cycle. | poller `TestStartErrorsDoNotAbortCycle` |
| Concurrency within limit; cancellation stops the cycle; marking/starting on `WithoutCancel`. | poller `TestConcurrencyLimit`, `TestRunStopsOnCancel` |
| Form cannot deploy an arbitrary ref: unknown version ID → 404; version of another repo → 404, no job. | jobs `TestTriggerVersion_*`, web `TestUpdateService_Version*` |
| Pinned/tagless refs routed only to protocol ≥ 3 agents; old agents → failed "agent too old to deploy a pinned version (needs protocol 3)" job (`ErrAgentTooOldForPinned`, which also satisfies `errors.Is(err, ErrAgentTooOld)`). | jobs `TestTriggerVersion_RequiresPinnedProtocol`, hub `TestRoute` |
| Agent deploys a pinned ref verbatim, no registry resolve. | updater `TestResolve`, `TestComposeWritesPinnedRef`, `TestSwarmUsesPinnedRef` |
| Rollback to older digest of same tag keeps newer suggestion; pick of suggested digest clears it. | store `TestFinishJob_ClearsAvailableOnlyWhenThatVersionWasDeployed` (extended) |
| Running versions recorded only with a digest; re-snapshot does not duplicate or reorder. | store `TestSnapshotRecordsRunningVersions` |
| Digest-less webhook row hidden once the tag's digest is known; order, limit, repo filter. | store `TestRecentVersions` |
| `available_source = 'poll'` passes the CHECK. | store `TestSetAvailable_RecordsVersion` |
| E2E: pick of a pinned version reaches the agent as that exact ref. | e2e `TestPickPinnedVersion` |
## Frozen contracts
### C1. Migration `internal/master/store/migrations/0004_polling.sql`
```sql
ALTER TABLE services DROP CONSTRAINT services_available_source_check;
ALTER TABLE services ADD CONSTRAINT services_available_source_check
CHECK (available_source IN ('', 'webhook', 'poll'));
ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_tag_digest text NOT NULL DEFAULT '';
-- Every version the master has seen. '' means "no tag" / "digest unknown";
-- never NULL, so the unique constraint deduplicates.
CREATE TABLE image_versions (
id bigserial PRIMARY KEY,
repo text NOT NULL,
tag text NOT NULL DEFAULT '',
digest text NOT NULL DEFAULT '',
source text NOT NULL CHECK (source IN ('webhook', 'poll', 'running')),
first_seen_at timestamptz NOT NULL,
CONSTRAINT image_versions_tag_or_digest CHECK (tag <> '' OR digest <> ''),
CONSTRAINT image_versions_unique UNIQUE (repo, tag, digest)
);
CREATE INDEX image_versions_recent ON image_versions (repo, first_seen_at DESC, id DESC);
```
Confirm the existing constraint name in `0003_webhook.sql` before relying on it.
### C2. `internal/master/model`
```go
// Service gains:
AvailableDigest string // digest of AvailableRef when known (poll); "" from the webhook
LastPollError string // last registry error for this service's followed ref; "" when fine
const SourcePoll = "poll" // alongside SourceWebhook
const SourceRunning = "running" // image_versions only
type Available struct { Ref, Digest, Source string; At time.Time } // Digest added
// ImageVersion is one row of image_versions.
type ImageVersion struct {
ID int64
Repo, Tag, Digest, Source string
FirstSeenAt time.Time
}
// Ref returns "repo:tag@digest", "repo:tag" or "repo@digest".
func (v ImageVersion) Ref() string
```
### C3. Store
```go
// SetAvailable sets available_ref/_digest/_source/_at and, in the same
// transaction, inserts image_versions(repo, tag parsed from a.Ref, a.Digest,
// a.Source, a.At) ON CONFLICT DO NOTHING.
SetAvailable(ctx, id int64, a model.Available) error
// RecentVersions returns up to limit versions of repo, newest first_seen_at
// first (ties: id DESC). It leaves out a digest-less row when a row with the
// same repo and tag and a digest was first seen at or after it.
RecentVersions(ctx, repo string, limit int) ([]model.ImageVersion, error)
// GetImageVersion returns one version or model.ErrNotFound.
GetImageVersion(ctx, id int64) (model.ImageVersion, error)
// SetPollError sets last_poll_error on the given services ("" clears).
SetPollError(ctx, ids []int64, msg string) error
```
`RecentVersions` SQL:
```sql
SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions v
WHERE v.repo = $1
AND NOT (v.digest = '' AND EXISTS (
SELECT 1 FROM image_versions d
WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> ''
AND d.first_seen_at >= v.first_seen_at))
ORDER BY v.first_seen_at DESC, v.id DESC
LIMIT $2
```
`upsertService` (running inventory): after the service upsert, on the same `execer`, only when `svc.Digest` matches `^sha256:[0-9a-f]{64}$`:
```sql
INSERT INTO image_versions (repo, tag, digest, source, first_seen_at)
VALUES ($1, $2, $3, 'running', $4) ON CONFLICT DO NOTHING
```
`repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image (no tag, no digest in the name) is recorded with `tag = 'latest'`, like `jobs.FollowedRef`, so it dedupes with the poll row; `repo@digest` is recorded with `tag = ''`. Covers `ApplySnapshot` and `UpsertService` with no hub change.
`serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error, s.last_tag_digest`. `SetLastTagDigest(ctx, id, digest)` sets the last column; inventory writes never touch it.
`FinishJob` clearing rule (replaces phase 3's). With `requestedRef, requestedDigest, _ := strings.Cut(to_ref_as_created, "@")`:
```sql
available_ref = $4 AND available_at <= $3
AND ($5 = '' OR available_digest = $5)
```
`$5` = `requestedDigest`. A digest-pinned job does NOT clear a digest-less (webhook) suggestion: a rollback to `main@old` must not hide that a newer `main` was announced. The badge then stays until a plain tag job succeeds; that is accepted. Clearing sets `available_ref=''`, `available_digest=''`, `available_source=''`, `available_at=NULL`.
### C4. `internal/master/registry`
```go
// Client looks up tag digests with HEAD requests only.
type Client struct{ /* keychain authn.Keychain, userAgent string */ }
// New uses authn.DefaultKeychain ($DOCKER_CONFIG/config.json, else ~/.docker/config.json).
func New(userAgent string) *Client
// Digest returns the digest ref ("repo:tag", any digest suffix ignored) points
// at: the index digest for multi-arch images. HEAD /v2/<repo>/manifests/<tag> only.
func (c *Client) Digest(ctx context.Context, ref string) (string, error)
```
Implementation: `name.ParseReference(repo+":"+tag)`; `remote.Head(r, remote.WithContext(ctx), remote.WithAuthFromKeychain(c.keychain), remote.WithUserAgent(c.userAgent))`; return `desc.Digest.String()`. Never fall back to GET if `Docker-Content-Digest` is missing.
### C5. `internal/master/poller`
```go
type Store interface {
ListServices(ctx context.Context) ([]model.Service, error)
GetService(ctx context.Context, id int64) (model.Service, error)
SetAvailable(ctx context.Context, id int64, a model.Available) error
SetLastTagDigest(ctx context.Context, id int64, digest string) error
SetPollError(ctx context.Context, ids []int64, msg string) error
}
type Jobs interface {
Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error)
}
type Registry interface { Digest(ctx context.Context, ref string) (string, error) }
type Publisher interface { Publish(t events.Topic) }
type Config struct {
Interval time.Duration // POLL_INTERVAL
FirstDelay time.Duration // 1m in production; small in tests
Workers int // 4
}
func New(st Store, j Jobs, reg Registry, pub Publisher, cfg Config, log *slog.Logger) *Poller
// Run polls until ctx is done. Blocking.
func (p *Poller) Run(ctx context.Context)
// Cycle runs one poll over all services (exported for tests and e2e).
func (p *Poller) Cycle(ctx context.Context)
// SkipReason says why svc is not polled: "follows no tag", "built from
// source", "no registry digest", or "" when it is polled.
func SkipReason(svc model.Service) string
```
Cycle semantics, per distinct `ref = jobs.FollowedRef(svc.ActualRef)` over non-removed, non-skipped services:
1. `D, err := reg.Digest(ctx, ref)` with a 30s timeout per call.
2. On error: `SetPollError(ids whose LastPollError != msg, msg)`, log `poll_failed` (ref, error).
3. On success: `SetPollError(ids with LastPollError != "", "")`. Then per service:
- Suggestion: `svc.ActualDigest == D` → nothing; `svc.AvailableRef == ref && svc.AvailableDigest == D` → no write; otherwise `SetAvailable({ref, D, poll, now})`.
- Auto job, independent of the suggestion: if `UpdateMode == auto && ActualDigest != D && D != LastTagDigest && DesiredRef != ref+"@"+D`, re-read the service (`GetService`), re-check the same condition and removed/mode on the fresh row, then `Start(TriggerPoll, Ref: ref)`. If a job row was created, `SetLastTagDigest(D)`. `ErrJobActive` and `NotUpdatableError` are logged at debug and ignored, leaving `last_tag_digest` unset so the next cycle retries.
Steps 2 and 3 use `context.WithoutCancel(ctx)`. At the end, publish `events.Services` if anything was written; log `poll_cycle` with refs, ok, failed, suggested, jobs, skipped, duration_ms.
### C6. `internal/master/jobs` and hub
```go
// Hub.Route gains the minimum protocol version for this job:
Route(svc model.Service, minProtocol int) (model.Route, error)
// Store gains:
GetImageVersion(ctx context.Context, id int64) (model.ImageVersion, error)
// TriggerVersion starts a manual job to exactly version versionID of the
// service's repo. model.ErrNotFound for an unknown service, an unknown
// version, or a version of another repo. Otherwise what Start returns.
func (s *Service) TriggerVersion(ctx context.Context, serviceID, versionID int64, requestedBy string) (model.Job, error)
```
- Ref = `svc.ImageRepo` + `":"+tag` if tag set + `"@"+digest` if digest set.
- Calls `Start` with `TriggerManual` and explicit ref ("follows no tag" refusal does not apply; removed, self, built, container still do).
- `Start` passes `protocol.MinPinnedRefVersion` to `Route` when the ref has a digest or no tag, otherwise `protocol.MinJobVersion`.
### C7. Protocol and agent
```go
const Version = 3
const MinJobVersion = 2 // unchanged: repo:tag jobs
const MinPinnedRefVersion = 3 // jobs whose Ref has a digest or no tag
```
`JobStart` doc: "Ref is `repo:tag`, `repo:tag@digest` or `repo@digest`. For `repo:tag` the agent resolves the tag's current digest; a ref with a digest is deployed as is."
`Resolver.Resolve`: digest present → return ref unchanged, no `DistributionInspect`; tag present → resolve as today; neither → error "has no tag to resolve" (unchanged). Compose writes the ref verbatim into the override, Swarm sets it as the image. Executors print `pinned <ref>` instead of `resolved <ref>` when nothing was resolved.
### C8. Master configuration
- `POLL_INTERVAL`: default `6h`, positive Go duration, error `"POLL_INTERVAL must be a positive duration such as 6h, got %q"`.
- `DOCKER_CONFIG`: not parsed by `config`; `go-containerregistry` reads it. `main` logs `registry_credentials` with the path and whether `config.json` exists, never its contents.
### C9. Web
- `POST /services/{id}/update` gains optional form field `version` (an `image_versions.id`): empty → `Trigger` as today; non-numeric → `400 invalid version`; otherwise `TriggerVersion`. Results: `303` to the job (also on `ErrJobActive`), `404` on `ErrNotFound`, `409` re-render "Cannot update: <reason>."
- `web.Store` gains `RecentVersions`; `web.Jobs` gains `TriggerVersion`.
- Service page: "Versions" table, up to 10 rows: ref (shortRef, full ref as title), source, first seen, "running" marker when digest equals `ActualDigest`, a POST form with `csrf_token` and `version`. Forms shown only when `AutoBlocked == ""`. Rows with empty tag note "the service will follow no tag afterwards". When the service is `auto`, header notes auto mode will move it forward at the next new version. A "Polling" row shows `SkipReason`, or `last_poll_error`, or "every <interval>".
- Overview: badge "follows no tag" when `FollowedRef(ActualRef) == ""`; badge "poll failed" (title = error) when `LastPollError != ""`. Template func `followsNoTag`.
## Tasks
### Task 1: Store and model (C1, C2, C3)
Migration, model fields, store methods, `upsertService` running versions, `FinishJob` rule. Integration tests: SetAvailable writes digest/source poll and records version, second call keeps first_seen_at; TestSnapshotRecordsRunningVersions (tag+digest, tagless digest, no digest skipped, re-snapshot idempotent); TestRecentVersions (order, limit 10, repo filter, digest-less row hidden only when a digest row for the same tag was first seen at or after it); GetImageVersion ErrNotFound; SetPollError sets/clears, upsert does not touch last_poll_error/available_*; FinishJob table extended (tag job clears poll suggestion; pick tag@old keeps suggestion tag@new; pick tag@new clears; webhook suggestion (digest '') cleared by matching tag; tagless pick keeps it). Update the e2e `memStore` in the same commit.
### Task 2: Protocol, agent and hub routing (C7, C6 Route)
Bump protocol, MinPinnedRefVersion, docs; Resolver pass-through; compose/swarm output line; `hub.Route(svc, minProtocol)`. Tests: TestResolve table (tag resolves; tag@d and @d pass through with zero inspect calls; bare errors); compose pinned ref verbatim in override; swarm spec image equals pinned ref; route rows v2 agent + min 3 → ErrAgentTooOld, v3 → ok, swarm fallback to v3 manager when reporting manager is v2.
### Task 3: Jobs (C6)
`Start` picks min version from the ref; `TriggerVersion`. Tests: version of other repo → ErrNotFound, no CreateJob; ref construction table; min version passed to fake hub (plain Trigger → 2, pinned → 3); explicit ref bypasses no-tag but not removed/self/built/container; existing tests green.
### Task 4: Registry client and poller (C4, C5)
Registry tests (local httptest + ggcr/pkg/registry, no network): random.Index digest equals index digest; recording handler shows only HEAD on /manifests/; basic-auth handler succeeds with temp DOCKER_CONFIG (t.Setenv) and fails without; 404 → error. Verify behaviour of DefaultKeychain when config.json has credsStore but no helper binary; document. Poller tests per Review Focus, plus Run test (first cycle after delay, no overlap, returns on cancel).
### Task 5: Config, wiring, web (C8, C9)
Config POLL_INTERVAL with tests; add to setAll list. `cmd/master/main.go`: registry.New, poller, `go p.Run(ctx)` with done channel, wait for it after HTTP shutdown and before st.Close; log registry_credentials. Web handlers/templates per C9 with tests (picker ≤10 rows with CSRF and version IDs; hidden for built/self/container/removed; POST with version → TriggerVersion, without → Trigger; 400/404/409; overview badges; Polling row).
### Task 6: End-to-end
`internal/e2e/versions_test.go`: TestPickPinnedVersion; TestPollStartsAutoJob (poller.Cycle with fake registry → one TriggerPoll job to repo:tag; second Cycle starts nothing). `currentHub.Route` new signature.
### Task 7: Documentation
README (POLL_INTERVAL, DOCKER_CONFIG, "Registry polling" section: polled/skipped, credentials only inline auths, Docker Hub note, version picker & rollback, auto caveat, agent protocol 3 for pinned/tagless picks). Spec: replace "Believed not to count against Docker Hub pull limits for HEAD requests — verify during implementation" with verified statement + source link (docs.docker.com/docker-hub/usage/pulls: "Using GET emulates a real pull and counts towards the limit. Using HEAD won't."; note the separate unquantified abuse rate limit). `deploy/master.stack.yml`: mount config dir read-only, set DOCKER_CONFIG. CLAUDE.md: architecture entries for registry and poller; gotchas (HEAD only; digest-less webhook row hidden once digest known; image_versions uses '' not NULL; picks by version ID and pinned/tagless need protocol 3; FinishJob rule includes digest (replace existing bullet); rollback to repo@digest makes the service follow no tag); phase 4 plan reference. Save this plan as `docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md`.
## Final verification
```bash
gofmt -l . ; go vet ./...
go build ./cmd/master ./cmd/agent && rm -f master agent
go test ./...
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
```
@@ -247,8 +247,12 @@ them. An agent recreating itself would kill the job it is running.
- Mechanism: `HEAD /v2/<repo>/manifests/<tag>`; the returned digest is compared with - Mechanism: `HEAD /v2/<repo>/manifests/<tag>`; the returned digest is compared with
`actual_digest`. Use `go-containerregistry` — it handles Docker Hub's token auth and `actual_digest`. Use `go-containerregistry` — it handles Docker Hub's token auth and
multi-arch indexes. multi-arch indexes.
- Interval `POLL_INTERVAL`, default 6h. Believed not to count against Docker Hub pull - Interval `POLL_INTERVAL`, default 6h. Docker Hub documents that HEAD requests do not
limits for HEAD requests — verify during implementation. count against the pull limit ("Using GET emulates a real pull and counts towards the
limit. Using HEAD won't."; verified against
https://docs.docker.com/docker-hub/usage/pulls/). The same page mentions a separate,
unquantified abuse rate limit for many pulls from one IP address, so the poller stays
HEAD-only and never downloads a manifest.
- Credentials: a Docker `config.json` (as written by `docker login`) mounted into the - Credentials: a Docker `config.json` (as written by `docker login`) mounted into the
master. No credential UI, no encryption in Postgres. master. No credential UI, no encryption in Postgres.
+15 -4
View File
@@ -1,6 +1,6 @@
# Deploy via package-updater (composite action) # Deploy via package-updater (composite action)
Notifies `package-updater` to `docker compose pull` + `up -d` for the matching service(s) after a CI build. Notifies the `package-updater` master after a CI build. The master records the pushed version on every matching service and, for services in auto mode, updates them and waits for the result.
## Usage ## Usage
@@ -20,7 +20,7 @@ jobs:
token: ${{ secrets.UPDATER_TOKEN }} token: ${{ secrets.UPDATER_TOKEN }}
``` ```
`UPDATER_TOKEN` should be set as an organisation-level secret in Gitea so all repos share it. `endpoint` is the master's `/update`, and `UPDATER_TOKEN` is the master's `UPDATER_API_KEY`. It should be set as an organisation-level secret in Gitea so all repos share it.
## Inputs ## Inputs
@@ -28,9 +28,20 @@ jobs:
|---|---|---|---| |---|---|---|---|
| `endpoint` | yes | — | Full URL to `/update` | | `endpoint` | yes | — | Full URL to `/update` |
| `image` | yes | — | Image reference without tag | | `image` | yes | — | Image reference without tag |
| `tag` | no | `""` | Tag that was just pushed (logged for audit) | | `tag` | no | `""` | Tag that was just pushed. Auto services are updated to `<their image repo>:<tag>`, so it must exist in the registry. Omit it to re-pull the tag each service already follows. |
| `token` | yes | — | Bearer token configured in package-updater | | `token` | yes | — | Bearer token configured in package-updater |
## Failure modes ## Failure modes
The step exits non-zero if `package-updater` returns HTTP 4xx or 5xx. This is intentional — the workflow surfaces the deploy failure to whoever pushed. The step exits non-zero on any HTTP 4xx or 5xx. This is intentional: the workflow surfaces the deploy failure to whoever pushed.
| Code | Meaning |
|---|---|
| `200` | Every match was marked, updated, busy or refused. Manual services are only marked, which is not an error. A service that follows no tag is `refused` when the request has no `tag`. |
| `401` | Wrong or missing token. |
| `404` | No service uses the image. Check `image` for typos; matching is case-sensitive. |
| `500` | At least one auto job failed or timed out (`UPDATE_TIMEOUT` on the master). |
| `503` | The master has no `UPDATER_API_KEY`. |
A request that matches both a working and a failed service now fails as a whole (`500`); the old single-instance updater's `207` no longer exists.
The master waits for auto jobs, so the request can take minutes; raise any step timeout in your workflow accordingly.
+7 -2
View File
@@ -7,6 +7,7 @@ require (
github.com/coreos/go-oidc/v3 v3.21.0 github.com/coreos/go-oidc/v3 v3.21.0
github.com/distribution/reference v0.6.0 github.com/distribution/reference v0.6.0
github.com/docker/docker v28.5.2+incompatible github.com/docker/docker v28.5.2+incompatible
github.com/google/go-containerregistry v0.22.1
github.com/jackc/pgx/v5 v5.10.0 github.com/jackc/pgx/v5 v5.10.0
github.com/opencontainers/go-digest v1.0.0 github.com/opencontainers/go-digest v1.0.0
github.com/stretchr/testify v1.11.1 github.com/stretchr/testify v1.11.1
@@ -21,6 +22,8 @@ require (
github.com/containerd/errdefs/pkg v0.3.0 // indirect github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect github.com/containerd/log v0.1.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect github.com/davecgh/go-spew v1.1.1 // indirect
github.com/docker/cli v29.7.2+incompatible // indirect
github.com/docker/docker-credential-helpers v0.9.3 // indirect
github.com/docker/go-connections v0.7.0 // indirect github.com/docker/go-connections v0.7.0 // indirect
github.com/docker/go-units v0.5.0 // indirect github.com/docker/go-units v0.5.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect
@@ -30,6 +33,7 @@ require (
github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.19.2 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/sys/atomicwriter v0.1.0 // indirect github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect github.com/moby/term v0.5.2 // indirect
@@ -37,14 +41,15 @@ require (
github.com/opencontainers/image-spec v1.1.1 // indirect github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pkg/errors v0.9.1 // indirect github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 // indirect
go.opentelemetry.io/otel v1.43.0 // indirect go.opentelemetry.io/otel v1.43.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 // indirect
go.opentelemetry.io/otel/metric v1.43.0 // indirect go.opentelemetry.io/otel/metric v1.43.0 // indirect
go.opentelemetry.io/otel/trace v1.43.0 // indirect go.opentelemetry.io/otel/trace v1.43.0 // indirect
golang.org/x/sync v0.20.0 // indirect golang.org/x/sync v0.22.0 // indirect
golang.org/x/sys v0.42.0 // indirect golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.35.0 // indirect golang.org/x/text v0.35.0 // indirect
golang.org/x/time v0.15.0 // indirect golang.org/x/time v0.15.0 // indirect
gotest.tools/v3 v3.5.2 // indirect gotest.tools/v3 v3.5.2 // indirect
+18 -6
View File
@@ -21,8 +21,12 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/docker/cli v29.7.2+incompatible h1:dlkwallR8XqfeVnA2ELEhdwvb4lsSwuB4IgsG8Q9cLY=
github.com/docker/cli v29.7.2+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM= github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM=
github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker-credential-helpers v0.9.3 h1:gAm/VtF9wgqJMoxzT3Gj5p4AqIjCBS4wrsOh9yRqcz8=
github.com/docker/docker-credential-helpers v0.9.3/go.mod h1:x+4Gbw9aGmChi3qTLZj8Dfn0TD20M/fuWy0E5+WDeCo=
github.com/docker/go-connections v0.7.0 h1:6SsRfJddP22WMrCkj19x9WKjEDTB+ahsdiGYf0mN39c= github.com/docker/go-connections v0.7.0 h1:6SsRfJddP22WMrCkj19x9WKjEDTB+ahsdiGYf0mN39c=
github.com/docker/go-connections v0.7.0/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q= github.com/docker/go-connections v0.7.0/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
@@ -38,6 +42,8 @@ github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/go-containerregistry v0.22.1 h1:RZuuSYhTvlDvtsK+NkutoCZ//C0X2ebLK8X8l3ULs84=
github.com/google/go-containerregistry v0.22.1/go.mod h1:bJR35SK8XgisYmhg/FMQ/5RK0S/XrOAqLBV5/LR2XE0=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs= github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs=
@@ -50,6 +56,8 @@ github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -74,8 +82,8 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
@@ -101,18 +109,22 @@ go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09
go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0= go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g= go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g=
go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk= go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk=
golang.org/x/mod v0.39.0 h1:UF5zwQdCRRUpHfyPwr7d4UrGiVeldIsogtzWVnczL74=
golang.org/x/mod v0.39.0/go.mod h1:bvIbwjQ0HUFFf5AKukeeYQG4ZBUG9yxQbR9aEweIwYY=
golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0= golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=
golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw= golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo= golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8= golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA= golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 h1:VPWxll4HlMw1Vs/qXtN7BvhZqsS9cdAittCNvVENElA= google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 h1:VPWxll4HlMw1Vs/qXtN7BvhZqsS9cdAittCNvVENElA=
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9/go.mod h1:7QBABkRtR8z+TEnmXTqIqwJLlzrZKVfAUm7tY3yGv0M= google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9/go.mod h1:7QBABkRtR8z+TEnmXTqIqwJLlzrZKVfAUm7tY3yGv0M=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 h1:m8qni9SQFH0tJc1X0vmnpw/0t+AImlSvp30sEupozUg= google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 h1:m8qni9SQFH0tJc1X0vmnpw/0t+AImlSvp30sEupozUg=
+5 -5
View File
@@ -91,10 +91,10 @@ func NewComposeExecutor(cli ComposeDocker, resolve *Resolver, selfID string, run
return &ComposeExecutor{cli: cli, resolve: resolve, selfID: selfID, run: run, build: NewBuildChecker(cli, selfID, runOut)} return &ComposeExecutor{cli: cli, resolve: resolve, selfID: selfID, run: run, build: NewBuildChecker(cli, selfID, runOut)}
} }
// Execute pins job.Ref to its current digest, writes it to the stack's // Execute pins job.Ref to its current digest (a ref that has one is used as
// override file and recreates the service. Working directory and config // is), writes it to the stack's override file and recreates the service.
// files come from the service's container labels on this machine, never from // Working directory and config files come from the service's container
// the job. It returns the pinned ref. // labels on this machine, never from the job. It returns the pinned ref.
func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) { func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) {
project, service := job.ComposeProject, job.ComposeService project, service := job.ComposeProject, job.ComposeService
cts, err := e.cli.ContainerList(ctx, container.ListOptions{ cts, err := e.cli.ContainerList(ctx, container.ListOptions{
@@ -150,7 +150,7 @@ func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, ou
if err != nil { if err != nil {
return "", err return "", err
} }
fmt.Fprintf(out, "resolved %s\n", pinned) fmt.Fprintln(out, resolveNote(job.Ref, pinned))
if _, err := WriteOverride(dir, service, pinned); err != nil { if _, err := WriteOverride(dir, service, pinned); err != nil {
return "", err return "", err
+18
View File
@@ -108,6 +108,24 @@ func TestComposeExecutor(t *testing.T) {
assert.Equal(t, pinned, serviceImage(doc, "web")) assert.Equal(t, pinned, serviceImage(doc, "web"))
assert.Contains(t, out.String(), "resolved "+pinned) assert.Contains(t, out.String(), "resolved "+pinned)
}) })
t.Run("a pinned ref is written verbatim and not resolved", func(t *testing.T) {
t.Cleanup(cleanOverride)
for _, ref := range []string{"gitea.shcizo.se/shcizo/slusk:1.3@" + d, "gitea.shcizo.se/shcizo/slusk@" + d} {
rec := &recorder{}
dist := &fakeDist{digest: "sha256:other"}
var out strings.Builder
pinnedJob := job
pinnedJob.Ref = ref
got, err := setup(fakeCompose{rw, []container.Summary{ct}}, dist, rec).Execute(context.Background(), pinnedJob, &out)
require.NoError(t, err)
assert.Equal(t, ref, got)
assert.Empty(t, dist.refs, "no registry lookup")
_, doc := readOverride(t, stack)
assert.Equal(t, ref, serviceImage(doc, "web"))
assert.Contains(t, out.String(), "pinned "+ref)
assert.NotContains(t, out.String(), "resolved")
}
})
t.Run("built from source is refused before anything is written", func(t *testing.T) { t.Run("built from source is refused before anything is written", func(t *testing.T) {
t.Cleanup(cleanOverride) t.Cleanup(cleanOverride)
rec := &recorder{} rec := &recorder{}
+17 -4
View File
@@ -16,7 +16,8 @@ type DistributionClient interface {
} }
// Resolver turns "repo:tag" into "repo:tag@digest" by asking the local // Resolver turns "repo:tag" into "repo:tag@digest" by asking the local
// daemon, which contacts the registry. // daemon, which contacts the registry. A ref that already has a digest is
// passed through.
type Resolver struct { type Resolver struct {
cli DistributionClient cli DistributionClient
auth *RegistryAuth auth *RegistryAuth
@@ -27,13 +28,15 @@ func NewResolver(cli DistributionClient, auth *RegistryAuth) *Resolver {
return &Resolver{cli: cli, auth: auth} return &Resolver{cli: cli, auth: auth}
} }
// Resolve returns ref pinned to the digest its tag currently points at. It // Resolve returns ref pinned to the digest its tag currently points at. A ref
// fails for a ref without a tag or with a digest already. // that already has a digest ("repo:tag@digest" or "repo@digest") is returned
// unchanged without asking the daemon. It fails for a ref with neither tag nor
// digest.
func (r *Resolver) Resolve(ctx context.Context, ref string) (string, error) { func (r *Resolver) Resolve(ctx context.Context, ref string) (string, error) {
p := imageref.Parse(ref) p := imageref.Parse(ref)
switch { switch {
case p.Digest != "": case p.Digest != "":
return "", fmt.Errorf("image %q already has a digest", ref) return ref, nil
case p.Tag == "": case p.Tag == "":
return "", fmt.Errorf("image %q has no tag to resolve", ref) return "", fmt.Errorf("image %q has no tag to resolve", ref)
} }
@@ -46,3 +49,13 @@ func (r *Resolver) Resolve(ctx context.Context, ref string) (string, error) {
} }
return ref + "@" + info.Descriptor.Digest.String(), nil return ref + "@" + info.Descriptor.Digest.String(), nil
} }
// resolveNote is the job-output line that says which ref is deployed:
// "resolved" when pinned came from resolving a tag, "pinned" when ref already
// carried its digest.
func resolveNote(ref, pinned string) string {
if pinned == ref {
return "pinned " + pinned
}
return "resolved " + pinned
}
+11 -4
View File
@@ -21,14 +21,21 @@ func TestResolver(t *testing.T) {
assert.Equal(t, []string{"gitea.shcizo.se/shcizo/slusk:1.4"}, dist.refs) assert.Equal(t, []string{"gitea.shcizo.se/shcizo/slusk:1.4"}, dist.refs)
assert.Equal(t, []string{"enc-gitea"}, dist.auths) assert.Equal(t, []string{"enc-gitea"}, dist.auths)
}) })
t.Run("digest or missing tag is refused without asking the daemon", func(t *testing.T) { t.Run("a ref with a digest passes through without asking the daemon", func(t *testing.T) {
for _, ref := range []string{"repo@sha256:x", "repo:1@sha256:x", "repo"} { for _, ref := range []string{"repo@sha256:x", "repo:1@sha256:x", "localhost:5000/repo:1@sha256:x"} {
dist := &fakeDist{digest: d} dist := &fakeDist{digest: d}
_, err := NewResolver(dist, nil).Resolve(context.Background(), ref) got, err := NewResolver(dist, nil).Resolve(context.Background(), ref)
assert.Error(t, err, ref) require.NoError(t, err, ref)
assert.Equal(t, ref, got)
assert.Empty(t, dist.refs) assert.Empty(t, dist.refs)
} }
}) })
t.Run("a bare ref is refused without asking the daemon", func(t *testing.T) {
dist := &fakeDist{digest: d}
_, err := NewResolver(dist, nil).Resolve(context.Background(), "repo")
assert.ErrorContains(t, err, "has no tag to resolve")
assert.Empty(t, dist.refs)
})
t.Run("daemon error is wrapped", func(t *testing.T) { t.Run("daemon error is wrapped", func(t *testing.T) {
cause := errors.New("unauthorized") cause := errors.New("unauthorized")
_, err := NewResolver(&fakeDist{err: cause}, nil).Resolve(context.Background(), "repo:1") _, err := NewResolver(&fakeDist{err: cause}, nil).Resolve(context.Background(), "repo:1")
+6 -6
View File
@@ -35,11 +35,11 @@ func NewSwarmExecutor(cli SwarmClient, resolve *Resolver, auth *RegistryAuth) *S
} }
// Execute checks that the local daemon is a manager of the job's cluster, // Execute checks that the local daemon is a manager of the job's cluster,
// pins job.Ref to its current digest and updates the service to it. The // pins job.Ref to its current digest (a ref that has one is used as is) and
// update is addressed by the ID and version of the service that was // updates the service to it. The update is addressed by the ID and version of
// inspected, so it changes exactly that object. Success means the daemon // the service that was inspected, so it changes exactly that object. Success
// accepted the new spec, not that the tasks have converged. It returns the // means the daemon accepted the new spec, not that the tasks have converged.
// pinned ref. // It returns the pinned ref.
func (e *SwarmExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) { func (e *SwarmExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) {
info, err := e.cli.Info(ctx) info, err := e.cli.Info(ctx)
if err != nil { if err != nil {
@@ -71,7 +71,7 @@ func (e *SwarmExecutor) Execute(ctx context.Context, job protocol.JobStart, out
if err != nil { if err != nil {
return "", err return "", err
} }
fmt.Fprintf(out, "resolved %s\n", pinned) fmt.Fprintln(out, resolveNote(job.Ref, pinned))
spec := svc.Spec spec := svc.Spec
spec.TaskTemplate.ContainerSpec.Image = pinned spec.TaskTemplate.ContainerSpec.Image = pinned
+16
View File
@@ -77,6 +77,22 @@ func swarmSvc(name string) swarm.Service {
} }
} }
func TestSwarmUsesPinnedRef(t *testing.T) {
const ref = "gitea/gitea:1.26@sha256:abababababababababababababababababababababababababababababababab"
job := protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: ref}
fake := &fakeSwarm{info: managerOf("c1"), svc: swarmSvc("gitea_gitea")}
dist := &fakeDist{digest: "sha256:other"}
var out bytes.Buffer
got, err := NewSwarmExecutor(fake, NewResolver(dist, nil), nil).Execute(context.Background(), job, &out)
require.NoError(t, err)
assert.Equal(t, ref, got)
assert.Equal(t, ref, fake.updSpec.TaskTemplate.ContainerSpec.Image)
assert.Empty(t, dist.refs, "no registry lookup")
assert.Contains(t, out.String(), "pinned "+ref)
}
func TestSwarmExecutor(t *testing.T) { func TestSwarmExecutor(t *testing.T) {
job := protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: "gitea/gitea:1.27"} job := protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: "gitea/gitea:1.27"}
workerInfo := managerOf("c1") workerInfo := managerOf("c1")
+102 -1
View File
@@ -11,6 +11,8 @@ import (
"net" "net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"regexp"
"sort"
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
@@ -30,6 +32,9 @@ import (
"github.com/shcizo/package-updater/internal/protocol" "github.com/shcizo/package-updater/internal/protocol"
) )
// digestPattern is the digest format the real store records.
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
type memStore struct { type memStore struct {
mu sync.Mutex mu sync.Mutex
agents map[string]model.Agent agents map[string]model.Agent
@@ -43,6 +48,63 @@ type memStore struct {
jobs map[int64]*model.Job jobs map[int64]*model.Job
nextJob int64 nextJob int64
finishApplied int // FinishJob calls that returned true finishApplied int // FinishJob calls that returned true
versions map[int64]model.ImageVersion
}
// seedVersion adds an image_versions row that GetImageVersion returns.
func (m *memStore) seedVersion(v model.ImageVersion) {
m.mu.Lock()
defer m.mu.Unlock()
if m.versions == nil {
m.versions = map[int64]model.ImageVersion{}
}
m.versions[v.ID] = v
}
func (m *memStore) GetImageVersion(_ context.Context, id int64) (model.ImageVersion, error) {
m.mu.Lock()
defer m.mu.Unlock()
v, ok := m.versions[id]
if !ok {
return model.ImageVersion{}, model.ErrNotFound
}
return v, nil
}
func (m *memStore) ListServices(_ context.Context) ([]model.Service, error) {
m.mu.Lock()
defer m.mu.Unlock()
var out []model.Service
for _, svc := range m.svcRows {
if svc.RemovedAt == nil {
out = append(out, svc)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out, nil
}
func (m *memStore) SetPollError(_ context.Context, ids []int64, msg string) error {
m.mu.Lock()
defer m.mu.Unlock()
for _, id := range ids {
svc := m.svcRows[id]
svc.LastPollError = msg
m.svcRows[id] = svc
}
return nil
}
func (m *memStore) SetLastTagDigest(_ context.Context, id int64, digest string) error {
m.mu.Lock()
defer m.mu.Unlock()
svc, ok := m.svcRows[id]
if !ok {
return model.ErrNotFound
}
svc.LastTagDigest = digest
m.svcRows[id] = svc
return nil
} }
// seedService adds a service row that jobs can be triggered for. // seedService adds a service row that jobs can be triggered for.
@@ -65,6 +127,33 @@ func (m *memStore) GetService(_ context.Context, id int64) (model.Service, error
return svc, nil return svc, nil
} }
// ServicesByRepo returns the seeded services whose image repo equals repo,
// ordered by ID.
func (m *memStore) ServicesByRepo(_ context.Context, repo string) ([]model.Service, error) {
m.mu.Lock()
defer m.mu.Unlock()
var out []model.Service
for _, svc := range m.svcRows {
if svc.ImageRepo == repo && svc.RemovedAt == nil {
out = append(out, svc)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out, nil
}
func (m *memStore) SetAvailable(_ context.Context, id int64, a model.Available) error {
m.mu.Lock()
defer m.mu.Unlock()
svc, ok := m.svcRows[id]
if !ok {
return model.ErrNotFound
}
svc.AvailableRef, svc.AvailableDigest, svc.AvailableSource, svc.AvailableAt = a.Ref, a.Digest, a.Source, &a.At
m.svcRows[id] = svc
return nil
}
func (m *memStore) CreateJob(_ context.Context, nj model.NewJob) (model.Job, error) { func (m *memStore) CreateJob(_ context.Context, nj model.NewJob) (model.Job, error) {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
@@ -146,6 +235,7 @@ func (m *memStore) FinishJob(_ context.Context, agentID, jobID int64, f model.Jo
return false, nil return false, nil
} }
j.Status, j.Error, j.Output, j.OutputTruncated, j.FinishedAt = f.Status, f.Error, f.Output, f.OutputTruncated, &f.FinishedAt j.Status, j.Error, j.Output, j.OutputTruncated, j.FinishedAt = f.Status, f.Error, f.Output, f.OutputTruncated, &f.FinishedAt
requestedRef := j.ToRef
if f.Ref != "" { if f.Ref != "" {
j.ToRef = f.Ref j.ToRef = f.Ref
} }
@@ -155,6 +245,15 @@ func (m *memStore) FinishJob(_ context.Context, agentID, jobID int64, f model.Jo
if f.Status == model.JobSucceeded { if f.Status == model.JobSucceeded {
svc := m.svcRows[j.ServiceID] svc := m.svcRows[j.ServiceID]
svc.DesiredRef = f.Ref svc.DesiredRef = f.Ref
if _, d, _ := strings.Cut(f.Ref, "@"); !strings.Contains(requestedRef, "@") && digestPattern.MatchString(d) {
svc.LastTagDigest = d
}
// Same rule as the SQL: the job was created for the suggested ref after it was reported.
requested, digest, _ := strings.Cut(requestedRef, "@")
if svc.AvailableAt != nil && svc.AvailableRef == requested && !svc.AvailableAt.After(j.CreatedAt) &&
(digest == "" || svc.AvailableDigest == digest) {
svc.AvailableRef, svc.AvailableDigest, svc.AvailableSource, svc.AvailableAt = "", "", "", nil
}
m.svcRows[j.ServiceID] = svc m.svcRows[j.ServiceID] = svc
} }
m.finishApplied++ m.finishApplied++
@@ -248,7 +347,9 @@ type currentHub struct{ p atomic.Pointer[hub.Hub] }
func (c *currentHub) ServeHTTP(w http.ResponseWriter, r *http.Request) { c.p.Load().ServeHTTP(w, r) } func (c *currentHub) ServeHTTP(w http.ResponseWriter, r *http.Request) { c.p.Load().ServeHTTP(w, r) }
func (c *currentHub) Disconnect(id int64) { c.p.Load().Disconnect(id) } func (c *currentHub) Disconnect(id int64) { c.p.Load().Disconnect(id) }
func (c *currentHub) Route(svc model.Service) (model.Route, error) { return c.p.Load().Route(svc) } func (c *currentHub) Route(svc model.Service, minProtocol int) (model.Route, error) {
return c.p.Load().Route(svc, minProtocol)
}
func (c *currentHub) StartJob(r model.Route, id int64, s protocol.JobStart) error { func (c *currentHub) StartJob(r model.Route, id int64, s protocol.JobStart) error {
return c.p.Load().StartJob(r, id, s) return c.p.Load().StartJob(r, id, s)
} }
+12 -6
View File
@@ -32,6 +32,9 @@ const (
lostMessage = "lost: the agent did not know this job when it reconnected" lostMessage = "lost: the agent did not know this job when it reconnected"
) )
// e2eDigest is the digest the fake executor pins a tag to.
const e2eDigest = "sha256:" + "e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0e2e0"
// gateExecutor writes a line of output, then blocks until release is closed // gateExecutor writes a line of output, then blocks until release is closed
// or the job is cancelled. // or the job is cancelled.
type gateExecutor struct { type gateExecutor struct {
@@ -52,7 +55,10 @@ func (g *gateExecutor) Execute(ctx context.Context, job protocol.JobStart, out i
return "", ctx.Err() return "", ctx.Err()
} }
_, _ = io.WriteString(out, "recreated\n") _, _ = io.WriteString(out, "recreated\n")
return job.Ref + "@sha256:e2e", nil if strings.Contains(job.Ref, "@") {
return job.Ref, nil
}
return job.Ref + "@" + e2eDigest, nil
} }
// jobAgent is a connected agent with a real runner and its two lifetimes: the // jobAgent is a connected agent with a real runner and its two lifetimes: the
@@ -102,7 +108,7 @@ func newJobEnv(t *testing.T) *jobEnv {
t.Helper() t.Helper()
store := &memStore{agents: map[string]model.Agent{agenttoken.Hash("pua_ok"): {ID: agentID, Name: "node-2"}}} store := &memStore{agents: map[string]model.Agent{agenttoken.Hash("pua_ok"): {ID: agentID, Name: "node-2"}}}
store.seedService(model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", store.seedService(model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",
AgentID: ptr(agentID), ActualRef: slusk + ":1.4.2@sha256:old"}) AgentID: ptr(agentID), ImageRepo: slusk, ActualRef: slusk + ":1.4.2@sha256:old"})
m := startMaster(t, store) m := startMaster(t, store)
exec := newGateExecutor() exec := newGateExecutor()
env := &jobEnv{store: store, m: m, exec: exec, env := &jobEnv{store: store, m: m, exec: exec,
@@ -119,7 +125,7 @@ func (e *jobEnv) waitRoutable(t *testing.T) {
t.Helper() t.Helper()
svc, err := e.store.GetService(context.Background(), 1) svc, err := e.store.GetService(context.Background(), 1)
require.NoError(t, err) require.NoError(t, err)
require.Eventually(t, func() bool { _, err := e.m.hub.Route(svc); return err == nil }, waitFor, pollEvery) require.Eventually(t, func() bool { _, err := e.m.hub.Route(svc, protocol.MinJobVersion); return err == nil }, waitFor, pollEvery)
} }
func (e *jobEnv) job(t *testing.T, id int64) model.Job { func (e *jobEnv) job(t *testing.T, id int64) model.Job {
@@ -159,12 +165,12 @@ func TestManualJobFlowsToAgentAndBack(t *testing.T) {
j := e.trigger(t) j := e.trigger(t)
done := e.waitStatus(t, j.ID, model.JobSucceeded) done := e.waitStatus(t, j.ID, model.JobSucceeded)
assert.Equal(t, slusk+":1.4.2@sha256:e2e", done.ToRef) assert.Equal(t, slusk+":1.4.2@"+e2eDigest, done.ToRef)
assert.Contains(t, done.Output, "pulling "+slusk+":1.4.2") assert.Contains(t, done.Output, "pulling "+slusk+":1.4.2")
assert.Contains(t, done.Output, "recreated") assert.Contains(t, done.Output, "recreated")
svc, err := e.store.GetService(context.Background(), 1) svc, err := e.store.GetService(context.Background(), 1)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, slusk+":1.4.2@sha256:e2e", svc.DesiredRef) assert.Equal(t, slusk+":1.4.2@"+e2eDigest, svc.DesiredRef)
require.Eventually(t, func() bool { return len(e.agent.runner.Known()) == 0 }, waitFor, pollEvery, "the result must be acked") require.Eventually(t, func() bool { return len(e.agent.runner.Known()) == 0 }, waitFor, pollEvery, "the result must be acked")
} }
@@ -240,7 +246,7 @@ func TestPhaseOneAgentGetsNoJobs(t *testing.T) {
require.NoError(t, wsjson.Write(ctx, c, snap)) require.NoError(t, wsjson.Write(ctx, c, snap))
require.Eventually(t, func() bool { require.Eventually(t, func() bool {
_, err := m.hub.Route(svcRow) _, err := m.hub.Route(svcRow, protocol.MinJobVersion)
return errors.Is(err, model.ErrAgentTooOld) return errors.Is(err, model.ErrAgentTooOld)
}, waitFor, pollEvery) }, waitFor, pollEvery)
+133
View File
@@ -0,0 +1,133 @@
package e2e
import (
"context"
"io"
"log/slog"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/master/poller"
"github.com/shcizo/package-updater/internal/protocol"
)
// jobCount is the number of jobs created so far.
func (m *memStore) jobCount() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.jobs)
}
// fixedRegistry answers every lookup from a fixed table.
type fixedRegistry struct {
digests map[string]string
calls int
}
func (f *fixedRegistry) Digest(_ context.Context, ref string) (string, error) {
f.calls++
return f.digests[ref], nil
}
func TestPickPinnedVersion(t *testing.T) {
env := newJobEnv(t)
env.store.seedVersion(model.ImageVersion{ID: 5, Repo: slusk, Tag: "1.3.0", Digest: "sha256:old130", Source: model.SourceRunning})
job, err := env.jobs.TriggerVersion(context.Background(), 1, 5, "alice")
require.NoError(t, err)
assert.Equal(t, slusk+":1.3.0@sha256:old130", job.ToRef)
select {
case ref := <-env.exec.started:
assert.Equal(t, slusk+":1.3.0@sha256:old130", ref, "the agent receives exactly the picked version")
case <-time.After(waitFor):
t.Fatal("the agent never received the job")
}
close(env.exec.release)
done := env.waitStatus(t, job.ID, model.JobSucceeded)
assert.Equal(t, model.TriggerManual, done.Trigger)
assert.Equal(t, "alice", done.RequestedBy)
}
func TestPickVersionOfAnotherRepoIsRefused(t *testing.T) {
env := newJobEnv(t)
env.store.seedVersion(model.ImageVersion{ID: 6, Repo: "evil/other", Tag: "1", Digest: "sha256:x", Source: model.SourcePoll})
_, err := env.jobs.TriggerVersion(context.Background(), 1, 6, "alice")
assert.ErrorIs(t, err, model.ErrNotFound)
assert.Zero(t, env.store.jobCount(), "no job was created")
}
func TestPollStartsAutoJob(t *testing.T) {
env := newJobEnv(t)
env.store.seedService(model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",
AgentID: ptr(agentID), ImageRepo: slusk, ActualRef: slusk + ":1.4.2@sha256:old", ActualDigest: "sha256:old",
UpdateMode: model.ModeAuto})
reg := &fixedRegistry{digests: map[string]string{slusk + ":1.4.2": "sha256:new"}}
p := poller.New(env.store, env.jobs, reg, env.m.broker, poller.Config{Workers: 1}, slog.New(slog.NewTextHandler(io.Discard, nil)))
p.Cycle(context.Background())
select {
case ref := <-env.exec.started:
assert.Equal(t, slusk+":1.4.2", ref, "the agent gets repo:tag and resolves the digest itself")
case <-time.After(waitFor):
t.Fatal("the agent never received the job")
}
require.Equal(t, 1, env.store.jobCount())
job := env.job(t, 1)
assert.Equal(t, model.TriggerPoll, job.Trigger)
assert.Equal(t, slusk+":1.4.2", job.ToRef)
svc, err := env.store.GetService(context.Background(), 1)
require.NoError(t, err)
assert.Equal(t, slusk+":1.4.2", svc.AvailableRef)
assert.Equal(t, "sha256:new", svc.AvailableDigest)
assert.Equal(t, model.SourcePoll, svc.AvailableSource)
assert.Equal(t, "sha256:new", svc.LastTagDigest)
p.Cycle(context.Background())
assert.Equal(t, 2, reg.calls, "the registry is asked again")
assert.Equal(t, 1, env.store.jobCount(), "the same digest does not start another job")
close(env.exec.release)
env.waitStatus(t, job.ID, model.JobSucceeded)
}
// An update started by the webhook, then a rollback by hand: polling must not
// redo the update while the registry still shows the digest that was deployed.
func TestPollKeepsARollbackAfterAWebhookUpdate(t *testing.T) {
env := newJobEnv(t)
ctx := context.Background()
close(env.exec.release)
env.store.seedService(model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",
AgentID: ptr(agentID), ImageRepo: slusk, ActualRef: slusk + ":main@sha256:old", ActualDigest: "sha256:old",
UpdateMode: model.ModeAuto})
env.store.seedVersion(model.ImageVersion{ID: 5, Repo: slusk, Tag: "main", Digest: "sha256:old", Source: model.SourceRunning})
svc, err := env.store.GetService(ctx, 1)
require.NoError(t, err)
hook, err := env.jobs.Start(ctx, svc, jobs.StartRequest{Trigger: model.TriggerWebhook, Ref: slusk + ":main"})
require.NoError(t, err)
env.waitStatus(t, hook.ID, model.JobSucceeded)
svc, err = env.store.GetService(ctx, 1)
require.NoError(t, err)
assert.Equal(t, e2eDigest, svc.LastTagDigest, "the deployed digest of the tag is remembered")
pick, err := env.jobs.TriggerVersion(ctx, 1, 5, "alice")
require.NoError(t, err)
env.waitStatus(t, pick.ID, model.JobSucceeded)
jobsBefore := env.store.jobCount()
reg := &fixedRegistry{digests: map[string]string{slusk + ":main": e2eDigest}}
p := poller.New(env.store, env.jobs, reg, env.m.broker, poller.Config{Workers: 1}, slog.New(slog.NewTextHandler(io.Discard, nil)))
p.Cycle(ctx)
assert.Equal(t, 1, reg.calls)
assert.Equal(t, jobsBefore, env.store.jobCount(), "the rollback stays")
}
+129
View File
@@ -0,0 +1,129 @@
package e2e
import (
"context"
"encoding/json"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/master/webhook"
"github.com/shcizo/package-updater/internal/protocol"
)
// webhookServer mounts the webhook handler on a test server, wired to the
// job environment's store, jobs service and broker.
func (e *jobEnv) webhookServer(t *testing.T) *httptest.Server {
t.Helper()
h := webhook.New(e.store, e.jobs, e.m.broker, webhook.Config{
Token: "ci-key", Timeout: 5 * time.Second, PublicURL: "https://updater.example.com",
}, slog.New(slog.NewTextHandler(io.Discard, nil)))
srv := httptest.NewServer(h)
t.Cleanup(srv.Close)
return srv
}
func (e *jobEnv) setAuto(id int64) {
e.store.mu.Lock()
defer e.store.mu.Unlock()
svc := e.store.svcRows[id]
svc.UpdateMode = model.ModeAuto
e.store.svcRows[id] = svc
}
type webhookResponse struct {
Matched int `json:"matched"`
Results []struct {
Status string `json:"status"`
JobID int64 `json:"job_id"`
ToRef string `json:"to_ref"`
Error string `json:"error"`
} `json:"results"`
}
func callWebhook(t *testing.T, srv *httptest.Server, body string) (int, webhookResponse) {
t.Helper()
req, err := http.NewRequest(http.MethodPost, srv.URL, strings.NewReader(body))
require.NoError(t, err)
req.Header.Set("Authorization", "Bearer ci-key")
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer resp.Body.Close()
var out webhookResponse
require.NoError(t, json.NewDecoder(resp.Body).Decode(&out))
return resp.StatusCode, out
}
func TestWebhookWaitsForAutoJob(t *testing.T) {
e := newJobEnv(t)
e.setAuto(1)
srv := e.webhookServer(t)
type reply struct {
code int
body webhookResponse
}
done := make(chan reply, 1)
go func() {
code, body := callWebhook(t, srv, `{"image":"`+slusk+`:ignored","tag":"2.0.0"}`)
done <- reply{code, body}
}()
e.waitStarted(t)
select {
case <-done:
t.Fatal("the webhook answered before the job finished")
case <-time.After(100 * time.Millisecond):
}
close(e.exec.release)
var r reply
select {
case r = <-done:
case <-time.After(waitFor):
t.Fatal("the webhook never answered")
}
require.Equal(t, http.StatusOK, r.code)
require.Len(t, r.body.Results, 1)
assert.Equal(t, "succeeded", r.body.Results[0].Status)
assert.Equal(t, slusk+":2.0.0", r.body.Results[0].ToRef)
job := e.job(t, r.body.Results[0].JobID)
assert.Equal(t, model.TriggerWebhook, job.Trigger)
assert.Equal(t, model.JobSucceeded, job.Status)
assert.Equal(t, slusk+":2.0.0@"+e2eDigest, job.ToRef)
svc, err := e.store.GetService(context.Background(), 1)
require.NoError(t, err)
assert.Empty(t, svc.AvailableRef, "the job deployed the suggestion, so it is cleared")
}
func TestWebhookFailedAutoJobIs500(t *testing.T) {
e := newJobEnv(t)
e.setAuto(1)
srv := e.webhookServer(t)
e.agent.stopConn() // no agent connection: the job fails at once with "agent offline"
require.Eventually(t, func() bool {
svc, _ := e.store.GetService(context.Background(), 1)
_, err := e.m.hub.Route(svc, protocol.MinJobVersion)
return err != nil
}, waitFor, pollEvery)
code, body := callWebhook(t, srv, `{"image":"`+slusk+`","tag":"2.0.0"}`)
assert.Equal(t, http.StatusInternalServerError, code)
require.Len(t, body.Results, 1)
assert.Equal(t, "failed", body.Results[0].Status)
assert.Equal(t, "agent offline", body.Results[0].Error)
assert.Equal(t, model.JobFailed, e.job(t, body.Results[0].JobID).Status)
svc, err := e.store.GetService(context.Background(), 1)
require.NoError(t, err)
assert.Equal(t, slusk+":2.0.0", svc.AvailableRef, "a failed job keeps the suggestion")
}
+34
View File
@@ -5,6 +5,7 @@ import (
"fmt" "fmt"
"os" "os"
"strings" "strings"
"time"
) )
// Config holds the master's runtime configuration. // Config holds the master's runtime configuration.
@@ -20,8 +21,24 @@ type Config struct {
Port string Port string
AgentPort string AgentPort string
LogLevel string LogLevel string
// UpdaterAPIKey is the bearer token for POST /update (UPDATER_API_KEY).
// Empty disables the webhook.
UpdaterAPIKey string
// UpdateTimeout bounds how long one webhook request waits for its auto
// jobs (UPDATE_TIMEOUT, default 5m).
UpdateTimeout time.Duration
// PollInterval is the time between registry polling cycles
// (POLL_INTERVAL, default 6h).
PollInterval time.Duration
} }
// defaultUpdateTimeout is used when UPDATE_TIMEOUT is unset.
const defaultUpdateTimeout = 5 * time.Minute
// defaultPollInterval is used when POLL_INTERVAL is unset.
const defaultPollInterval = 6 * time.Hour
// Load reads and validates the master configuration, failing fast on // Load reads and validates the master configuration, failing fast on
// anything missing so a half-configured master never starts. // anything missing so a half-configured master never starts.
func Load() (*Config, error) { func Load() (*Config, error) {
@@ -37,6 +54,9 @@ func Load() (*Config, error) {
Port: getenvDefault("PORT", "8080"), Port: getenvDefault("PORT", "8080"),
AgentPort: getenvDefault("AGENT_PORT", "8090"), AgentPort: getenvDefault("AGENT_PORT", "8090"),
LogLevel: getenvDefault("LOG_LEVEL", "info"), LogLevel: getenvDefault("LOG_LEVEL", "info"),
UpdaterAPIKey: os.Getenv("UPDATER_API_KEY"),
UpdateTimeout: defaultUpdateTimeout,
PollInterval: defaultPollInterval,
} }
for _, r := range []struct{ name, value string }{ for _, r := range []struct{ name, value string }{
{"DATABASE_URL", cfg.DatabaseURL}, {"DATABASE_URL", cfg.DatabaseURL},
@@ -57,6 +77,20 @@ func Load() (*Config, error) {
if !strings.HasPrefix(cfg.AgentURL, "wss://") { if !strings.HasPrefix(cfg.AgentURL, "wss://") {
return nil, fmt.Errorf("AGENT_URL must start with wss://") return nil, fmt.Errorf("AGENT_URL must start with wss://")
} }
if v := os.Getenv("UPDATE_TIMEOUT"); v != "" {
d, err := time.ParseDuration(v)
if err != nil || d <= 0 {
return nil, fmt.Errorf("UPDATE_TIMEOUT must be a positive duration such as 5m, got %q", v)
}
cfg.UpdateTimeout = d
}
if v := os.Getenv("POLL_INTERVAL"); v != "" {
d, err := time.ParseDuration(v)
if err != nil || d <= 0 {
return nil, fmt.Errorf("POLL_INTERVAL must be a positive duration such as 6h, got %q", v)
}
cfg.PollInterval = d
}
return cfg, nil return cfg, nil
} }
+45 -1
View File
@@ -2,6 +2,7 @@ package config
import ( import (
"testing" "testing"
"time"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -21,7 +22,7 @@ func setAll(t *testing.T) {
for k, v := range required { for k, v := range required {
t.Setenv(k, v) t.Setenv(k, v)
} }
for _, k := range []string{"PORT", "AGENT_PORT", "AGENT_IMAGE", "LOG_LEVEL"} { for _, k := range []string{"PORT", "AGENT_PORT", "AGENT_IMAGE", "LOG_LEVEL", "UPDATER_API_KEY", "UPDATE_TIMEOUT", "POLL_INTERVAL"} {
t.Setenv(k, "") t.Setenv(k, "")
} }
} }
@@ -35,6 +36,49 @@ func TestLoad_Defaults(t *testing.T) {
assert.Equal(t, "gitea.shcizo.se/shcizo/package-updater:latest", cfg.AgentImage) assert.Equal(t, "gitea.shcizo.se/shcizo/package-updater:latest", cfg.AgentImage)
assert.Equal(t, "info", cfg.LogLevel) assert.Equal(t, "info", cfg.LogLevel)
assert.Equal(t, "updater-admins", cfg.AllowedGroup) assert.Equal(t, "updater-admins", cfg.AllowedGroup)
assert.Empty(t, cfg.UpdaterAPIKey)
assert.Equal(t, 5*time.Minute, cfg.UpdateTimeout)
assert.Equal(t, 6*time.Hour, cfg.PollInterval)
}
func TestLoad_PollInterval(t *testing.T) {
setAll(t)
t.Setenv("POLL_INTERVAL", "30m")
cfg, err := Load()
require.NoError(t, err)
assert.Equal(t, 30*time.Minute, cfg.PollInterval)
}
func TestLoad_InvalidPollInterval(t *testing.T) {
for _, v := range []string{"abc", "0s", "-1h", "6"} {
t.Run(v, func(t *testing.T) {
setAll(t)
t.Setenv("POLL_INTERVAL", v)
_, err := Load()
assert.ErrorContains(t, err, "POLL_INTERVAL must be a positive duration such as 6h, got")
})
}
}
func TestLoad_WebhookSettings(t *testing.T) {
setAll(t)
t.Setenv("UPDATER_API_KEY", "s3cret")
t.Setenv("UPDATE_TIMEOUT", "90s")
cfg, err := Load()
require.NoError(t, err)
assert.Equal(t, "s3cret", cfg.UpdaterAPIKey)
assert.Equal(t, 90*time.Second, cfg.UpdateTimeout)
}
func TestLoad_InvalidUpdateTimeout(t *testing.T) {
for _, v := range []string{"abc", "0s", "-5m", "30"} {
t.Run(v, func(t *testing.T) {
setAll(t)
t.Setenv("UPDATE_TIMEOUT", v)
_, err := Load()
assert.ErrorContains(t, err, "UPDATE_TIMEOUT must be a positive duration")
})
}
} }
func TestLoad_EachRequiredVariable(t *testing.T) { func TestLoad_EachRequiredVariable(t *testing.T) {
+2 -2
View File
@@ -43,7 +43,7 @@ func waitRoute(t *testing.T, h *Hub) model.Route {
var r model.Route var r model.Route
require.Eventually(t, func() bool { require.Eventually(t, func() bool {
var err error var err error
r, err = h.Route(svc) r, err = h.Route(svc, protocol.MinJobVersion)
return err == nil return err == nil
}, 2*time.Second, 5*time.Millisecond) }, 2*time.Second, 5*time.Millisecond)
return r return r
@@ -61,7 +61,7 @@ func TestHub_ReconcilesBeforeRoutable(t *testing.T) {
send(t, c, protocol.TypeHello, protocol.Hello{Hostname: "node-2", ProtocolVersion: 2, KnownJobs: []string{"7", "9", "x"}}) send(t, c, protocol.TypeHello, protocol.Hello{Hostname: "node-2", ProtocolVersion: 2, KnownJobs: []string{"7", "9", "x"}})
svc := model.Service{Kind: protocol.KindCompose, AgentID: ptr(node2.ID)} svc := model.Service{Kind: protocol.KindCompose, AgentID: ptr(node2.ID)}
time.Sleep(50 * time.Millisecond) // hello is now blocked inside FailLostJobs time.Sleep(50 * time.Millisecond) // hello is now blocked inside FailLostJobs
_, err = h.Route(svc) _, err = h.Route(svc, protocol.MinJobVersion)
assert.ErrorIs(t, err, model.ErrAgentOffline) assert.ErrorIs(t, err, model.ErrAgentOffline)
close(store.lostGate) close(store.lostGate)
+11 -6
View File
@@ -14,9 +14,11 @@ type candidate struct {
hello protocol.Hello hello protocol.Hello
} }
// Route picks the connection a job for svc goes to. It returns // Route picks the connection a job for svc goes to, among agents that speak
// model.ErrAgentOffline or model.ErrAgentTooOld when there is none. // at least protocol version minProtocol. It returns model.ErrAgentOffline or
func (h *Hub) Route(svc model.Service) (model.Route, error) { // model.ErrAgentTooOld (model.ErrAgentTooOldForPinned when minProtocol asks
// for a pinned ref) when there is none.
func (h *Hub) Route(svc model.Service, minProtocol int) (model.Route, error) {
h.mu.Lock() h.mu.Lock()
conns := make([]candidate, 0, len(h.conns)) conns := make([]candidate, 0, len(h.conns))
for id, c := range h.conns { for id, c := range h.conns {
@@ -25,7 +27,7 @@ func (h *Hub) Route(svc model.Service) (model.Route, error) {
} }
} }
h.mu.Unlock() h.mu.Unlock()
return route(svc, conns) return route(svc, conns, minProtocol)
} }
// route picks the connection a job for svc goes to. Compose services and // route picks the connection a job for svc goes to. Compose services and
@@ -33,7 +35,7 @@ func (h *Hub) Route(svc model.Service) (model.Route, error) {
// run on the reporting agent if it qualifies, else on any other connected // run on the reporting agent if it qualifies, else on any other connected
// manager of the same cluster (lowest agent ID first). Too-old wins over // manager of the same cluster (lowest agent ID first). Too-old wins over
// offline only when a connection would otherwise have qualified. // offline only when a connection would otherwise have qualified.
func route(svc model.Service, conns []candidate) (model.Route, error) { func route(svc model.Service, conns []candidate, minProtocol int) (model.Route, error) {
sort.Slice(conns, func(a, b int) bool { return conns[a].agentID < conns[b].agentID }) sort.Slice(conns, func(a, b int) bool { return conns[a].agentID < conns[b].agentID })
tooOld := false tooOld := false
try := func(c candidate) bool { try := func(c candidate) bool {
@@ -42,7 +44,7 @@ func route(svc model.Service, conns []candidate) (model.Route, error) {
return false return false
} }
} }
if c.hello.ProtocolVersion < protocol.MinJobVersion { if c.hello.ProtocolVersion < minProtocol {
tooOld = true tooOld = true
return false return false
} }
@@ -63,6 +65,9 @@ func route(svc model.Service, conns []candidate) (model.Route, error) {
} }
} }
if tooOld { if tooOld {
if minProtocol >= protocol.MinPinnedRefVersion {
return model.Route{}, model.ErrAgentTooOldForPinned
}
return model.Route{}, model.ErrAgentTooOld return model.Route{}, model.ErrAgentTooOld
} }
return model.Route{}, model.ErrAgentOffline return model.Route{}, model.ErrAgentOffline
+24 -15
View File
@@ -12,37 +12,46 @@ import (
func TestRoute(t *testing.T) { func TestRoute(t *testing.T) {
composeSvc := model.Service{Kind: protocol.KindCompose, AgentID: ptr(int64(2))} composeSvc := model.Service{Kind: protocol.KindCompose, AgentID: ptr(int64(2))}
swarmSvc := model.Service{Kind: protocol.KindSwarm, AgentID: ptr(int64(1)), SwarmClusterID: "c1"} swarmSvc := model.Service{Kind: protocol.KindSwarm, AgentID: ptr(int64(1)), SwarmClusterID: "c1"}
v2 := protocol.Hello{ProtocolVersion: protocol.Version} v2 := protocol.Hello{ProtocolVersion: 2}
mgrV2 := protocol.Hello{ProtocolVersion: protocol.Version, SwarmManager: true, SwarmClusterID: "c1"} v3 := protocol.Hello{ProtocolVersion: 3}
mgrV2 := protocol.Hello{ProtocolVersion: 2, SwarmManager: true, SwarmClusterID: "c1"}
mgrV3 := protocol.Hello{ProtocolVersion: 3, SwarmManager: true, SwarmClusterID: "c1"}
jobs, pinned := protocol.MinJobVersion, protocol.MinPinnedRefVersion
tests := []struct { tests := []struct {
name string name string
svc model.Service svc model.Service
conns []candidate conns []candidate
min int
want model.Route want model.Route
err error err error
}{ }{
{"compose on own agent", composeSvc, []candidate{{2, 20, v2}}, model.Route{AgentID: 2, ConnID: 20}, nil}, {"compose on own agent", composeSvc, []candidate{{2, 20, v2}}, jobs, model.Route{AgentID: 2, ConnID: 20}, nil},
{"compose, own agent too old", composeSvc, []candidate{{2, 20, protocol.Hello{}}}, model.Route{}, model.ErrAgentTooOld}, {"compose, own agent too old", composeSvc, []candidate{{2, 20, protocol.Hello{}}}, jobs, model.Route{}, model.ErrAgentTooOld},
{"compose, own agent missing", composeSvc, nil, model.Route{}, model.ErrAgentOffline}, {"compose, own agent missing", composeSvc, nil, jobs, model.Route{}, model.ErrAgentOffline},
{"compose, only another agent online", composeSvc, []candidate{{5, 50, v2}}, model.Route{}, model.ErrAgentOffline}, {"compose, only another agent online", composeSvc, []candidate{{5, 50, v2}}, jobs, model.Route{}, model.ErrAgentOffline},
{"swarm on reporting agent", swarmSvc, []candidate{{1, 10, mgrV2}, {3, 30, mgrV2}}, model.Route{AgentID: 1, ConnID: 10}, nil}, {"compose, v2 agent and a pinned ref", composeSvc, []candidate{{2, 20, v2}}, pinned, model.Route{}, model.ErrAgentTooOldForPinned},
{"swarm, reporting agent offline, other manager", swarmSvc, []candidate{{3, 30, mgrV2}}, model.Route{AgentID: 3, ConnID: 30}, nil}, {"compose, v3 agent and a pinned ref", composeSvc, []candidate{{2, 20, v3}}, pinned, model.Route{AgentID: 2, ConnID: 20}, nil},
{"swarm, lowest agent id first", swarmSvc, []candidate{{4, 40, mgrV2}, {3, 30, mgrV2}}, model.Route{AgentID: 3, ConnID: 30}, nil}, {"swarm on reporting agent", swarmSvc, []candidate{{1, 10, mgrV2}, {3, 30, mgrV2}}, jobs, model.Route{AgentID: 1, ConnID: 10}, nil},
{"swarm, reporting agent offline, other manager", swarmSvc, []candidate{{3, 30, mgrV2}}, jobs, model.Route{AgentID: 3, ConnID: 30}, nil},
{"swarm, lowest agent id first", swarmSvc, []candidate{{4, 40, mgrV2}, {3, 30, mgrV2}}, jobs, model.Route{AgentID: 3, ConnID: 30}, nil},
{"swarm, other manager in another cluster", swarmSvc, {"swarm, other manager in another cluster", swarmSvc,
[]candidate{{3, 30, protocol.Hello{ProtocolVersion: 2, SwarmManager: true, SwarmClusterID: "c2"}}}, model.Route{}, model.ErrAgentOffline}, []candidate{{3, 30, protocol.Hello{ProtocolVersion: 2, SwarmManager: true, SwarmClusterID: "c2"}}}, jobs, model.Route{}, model.ErrAgentOffline},
{"swarm, only a v0 manager of the cluster", swarmSvc, {"swarm, only a v0 manager of the cluster", swarmSvc,
[]candidate{{1, 10, protocol.Hello{SwarmManager: true, SwarmClusterID: "c1"}}}, model.Route{}, model.ErrAgentTooOld}, []candidate{{1, 10, protocol.Hello{SwarmManager: true, SwarmClusterID: "c1"}}}, jobs, model.Route{}, model.ErrAgentTooOld},
{"swarm, v0 manager but a v2 manager exists", swarmSvc, {"swarm, v0 manager but a v2 manager exists", swarmSvc,
[]candidate{{1, 10, protocol.Hello{SwarmManager: true, SwarmClusterID: "c1"}}, {3, 30, mgrV2}}, model.Route{AgentID: 3, ConnID: 30}, nil}, []candidate{{1, 10, protocol.Hello{SwarmManager: true, SwarmClusterID: "c1"}}, {3, 30, mgrV2}}, jobs, model.Route{AgentID: 3, ConnID: 30}, nil},
{"swarm, worker of the cluster", swarmSvc, {"swarm, worker of the cluster", swarmSvc,
[]candidate{{1, 10, protocol.Hello{ProtocolVersion: 2, SwarmClusterID: "c1"}}}, model.Route{}, model.ErrAgentOffline}, []candidate{{1, 10, protocol.Hello{ProtocolVersion: 2, SwarmClusterID: "c1"}}}, jobs, model.Route{}, model.ErrAgentOffline},
{"swarm, manager with empty cluster id", model.Service{Kind: protocol.KindSwarm}, {"swarm, manager with empty cluster id", model.Service{Kind: protocol.KindSwarm},
[]candidate{{1, 10, protocol.Hello{ProtocolVersion: 2, SwarmManager: true}}}, model.Route{}, model.ErrAgentOffline}, []candidate{{1, 10, protocol.Hello{ProtocolVersion: 2, SwarmManager: true}}}, jobs, model.Route{}, model.ErrAgentOffline},
{"swarm, pinned ref falls back to a v3 manager when the reporting one is v2", swarmSvc,
[]candidate{{1, 10, mgrV2}, {3, 30, mgrV3}}, pinned, model.Route{AgentID: 3, ConnID: 30}, nil},
{"swarm, pinned ref and only v2 managers", swarmSvc, []candidate{{1, 10, mgrV2}}, pinned, model.Route{}, model.ErrAgentTooOldForPinned},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
got, err := route(tt.svc, tt.conns) got, err := route(tt.svc, tt.conns, tt.min)
assert.ErrorIs(t, err, tt.err) assert.ErrorIs(t, err, tt.err)
assert.Equal(t, tt.want, got) assert.Equal(t, tt.want, got)
}) })
+85 -13
View File
@@ -20,13 +20,15 @@ type Store interface {
CreateJob(ctx context.Context, j model.NewJob) (model.Job, error) CreateJob(ctx context.Context, j model.NewJob) (model.Job, error)
FailJob(ctx context.Context, id int64, msg string, at time.Time) (bool, error) FailJob(ctx context.Context, id int64, msg string, at time.Time) (bool, error)
GetJob(ctx context.Context, id int64) (model.Job, error) GetJob(ctx context.Context, id int64) (model.Job, error)
GetImageVersion(ctx context.Context, id int64) (model.ImageVersion, error)
} }
// Hub routes jobs to agent connections. *hub.Hub satisfies it. // Hub routes jobs to agent connections. *hub.Hub satisfies it.
type Hub interface { type Hub interface {
// Route picks the connection for svc: model.ErrAgentOffline or // Route picks the connection for svc among agents speaking at least
// protocol version minProtocol: model.ErrAgentOffline or
// model.ErrAgentTooOld when there is none. // model.ErrAgentTooOld when there is none.
Route(svc model.Service) (model.Route, error) Route(svc model.Service, minProtocol int) (model.Route, error)
// StartJob sends job.start over exactly the routed connection. // StartJob sends job.start over exactly the routed connection.
// model.ErrAgentOffline if that connection is gone or was replaced. // model.ErrAgentOffline if that connection is gone or was replaced.
StartJob(r model.Route, jobID int64, start protocol.JobStart) error StartJob(r model.Route, jobID int64, start protocol.JobStart) error
@@ -68,9 +70,34 @@ func target(actualRef string) (repo, tag string, ok bool) {
return ref.Repo, "", false return ref.Repo, "", false
} }
// FollowedRef returns "repo:tag" for the tag a service with this actual ref
// follows (a bare image follows "latest"), or "" when it follows no tag.
func FollowedRef(actualRef string) string {
repo, tag, ok := target(actualRef)
if !ok {
return ""
}
return repo + ":" + tag
}
// Updatable returns nil or a *model.NotUpdatableError. It looks only at the // Updatable returns nil or a *model.NotUpdatableError. It looks only at the
// service row, so the UI can call it for every page render. // service row, so the UI can call it for every page render. It answers for an
// update to the service's own followed tag (the Update button).
func (s *Service) Updatable(svc model.Service) error { func (s *Service) Updatable(svc model.Service) error {
return s.updatable(svc, false)
}
// AutoAllowed returns nil or a *model.NotUpdatableError saying why the
// service can never be updated automatically. Unlike Updatable it does not
// refuse a service that follows no tag: a webhook that names a tag can still
// update it.
func (s *Service) AutoAllowed(svc model.Service) error {
return s.updatable(svc, true)
}
// updatable checks the service row. explicitRef means the caller supplies the
// target ref, so a service that follows no tag is fine.
func (s *Service) updatable(svc model.Service, explicitRef bool) error {
switch { switch {
case svc.RemovedAt != nil: case svc.RemovedAt != nil:
return &model.NotUpdatableError{Reason: model.ReasonRemoved} return &model.NotUpdatableError{Reason: model.ReasonRemoved}
@@ -81,32 +108,77 @@ func (s *Service) Updatable(svc model.Service) error {
case svc.Kind == protocol.KindContainer: case svc.Kind == protocol.KindContainer:
return &model.NotUpdatableError{Reason: model.ReasonContainer} return &model.NotUpdatableError{Reason: model.ReasonContainer}
} }
if _, _, ok := target(svc.ActualRef); !ok { if _, _, ok := target(svc.ActualRef); !ok && !explicitRef {
return &model.NotUpdatableError{Reason: model.ReasonNoTag} return &model.NotUpdatableError{Reason: model.ReasonNoTag}
} }
return nil return nil
} }
// StartRequest says who asks for an update and what to deploy.
type StartRequest struct {
Trigger model.JobTrigger
RequestedBy string // OIDC subject for manual jobs; "" otherwise
// Ref is the ref to deploy ("repo:tag", "repo:tag@digest" or
// "repo@digest"). Empty means the service's followed tag, derived from
// its actual ref.
Ref string
}
// Trigger starts a manual update of the service to its current tag. It // Trigger starts a manual update of the service to its current tag. It
// returns the job, which may already be failed ("agent offline", "agent too // returns what Start returns, and model.ErrNotFound for an unknown service.
// old for updates"); a *model.NotUpdatableError when the service cannot be
// updated (no job created); the active job and model.ErrJobActive when one
// is queued or running; model.ErrNotFound for an unknown service.
func (s *Service) Trigger(ctx context.Context, serviceID int64, requestedBy string) (model.Job, error) { func (s *Service) Trigger(ctx context.Context, serviceID int64, requestedBy string) (model.Job, error) {
svc, err := s.store.GetService(ctx, serviceID) svc, err := s.store.GetService(ctx, serviceID)
if err != nil { if err != nil {
return model.Job{}, err return model.Job{}, err
} }
if err := s.Updatable(svc); err != nil { return s.Start(ctx, svc, StartRequest{Trigger: model.TriggerManual, RequestedBy: requestedBy})
}
// TriggerVersion starts a manual update of the service to exactly the image
// version versionID, which must be a version of the service's own repo; the
// ref is built from the stored version, never taken from the caller. It
// returns model.ErrNotFound for an unknown service, an unknown version or a
// version of another repo, and otherwise what Start returns. A version
// without a tag leaves the service following no tag.
func (s *Service) TriggerVersion(ctx context.Context, serviceID, versionID int64, requestedBy string) (model.Job, error) {
svc, err := s.store.GetService(ctx, serviceID)
if err != nil {
return model.Job{}, err return model.Job{}, err
} }
repo, tag, _ := target(svc.ActualRef) // Updatable checked the tag v, err := s.store.GetImageVersion(ctx, versionID)
ref := repo + ":" + tag if err != nil {
return model.Job{}, err
}
if v.Repo != svc.ImageRepo {
return model.Job{}, model.ErrNotFound
}
return s.Start(ctx, svc, StartRequest{Trigger: model.TriggerManual, RequestedBy: requestedBy, Ref: v.Ref()})
}
r, routeErr := s.hub.Route(svc) // Start starts an update of svc. It returns the job, which may already be
// failed ("agent offline", "agent too old for updates", or "agent too old to
// deploy a pinned version" when the ref has a digest or no tag and the agent
// speaks protocol version 2); a
// *model.NotUpdatableError when the service cannot be updated (no job
// created); the active job and model.ErrJobActive when one is queued or
// running.
func (s *Service) Start(ctx context.Context, svc model.Service, req StartRequest) (model.Job, error) {
if err := s.updatable(svc, req.Ref != ""); err != nil {
return model.Job{}, err
}
ref := req.Ref
if ref == "" {
ref = FollowedRef(svc.ActualRef) // updatable checked the tag
}
minProtocol := protocol.MinJobVersion
if p := imageref.Parse(ref); p.Digest != "" || p.Tag == "" {
minProtocol = protocol.MinPinnedRefVersion
}
r, routeErr := s.hub.Route(svc, minProtocol)
nj := model.NewJob{ nj := model.NewJob{
ServiceID: svc.ID, FromRef: svc.ActualRef, ToRef: ref, ServiceID: svc.ID, FromRef: svc.ActualRef, ToRef: ref,
Trigger: model.TriggerManual, RequestedBy: requestedBy, CreatedAt: s.now(), Trigger: req.Trigger, RequestedBy: req.RequestedBy, CreatedAt: s.now(),
} }
if routeErr == nil { if routeErr == nil {
nj.AgentID = &r.AgentID nj.AgentID = &r.AgentID
+254 -5
View File
@@ -21,6 +21,7 @@ type fakeStore struct {
jobs map[int64]model.Job jobs map[int64]model.Job
created []model.NewJob created []model.NewJob
active *model.Job // returned with ErrJobActive when set active *model.Job // returned with ErrJobActive when set
versions map[int64]model.ImageVersion
revokedAgents map[int64]bool // CreateJob returns ErrAgentOffline for these revokedAgents map[int64]bool // CreateJob returns ErrAgentOffline for these
} }
@@ -54,6 +55,14 @@ func (f *fakeStore) FailJob(_ context.Context, id int64, msg string, at time.Tim
return true, nil return true, nil
} }
func (f *fakeStore) GetImageVersion(_ context.Context, id int64) (model.ImageVersion, error) {
v, ok := f.versions[id]
if !ok {
return model.ImageVersion{}, model.ErrNotFound
}
return v, nil
}
func (f *fakeStore) GetJob(_ context.Context, id int64) (model.Job, error) { return f.jobs[id], nil } func (f *fakeStore) GetJob(_ context.Context, id int64) (model.Job, error) { return f.jobs[id], nil }
type startCall struct { type startCall struct {
@@ -63,13 +72,24 @@ type startCall struct {
} }
type fakeHub struct { type fakeHub struct {
route model.Route route model.Route
routeErr error routeErr error
startErr error startErr error
starts []startCall starts []startCall
agentProtocol int // when set, Route fails with ErrAgentTooOld for a higher minProtocol
mins []int // minProtocol of every Route call
} }
func (h *fakeHub) Route(model.Service) (model.Route, error) { return h.route, h.routeErr } func (h *fakeHub) Route(_ model.Service, minProtocol int) (model.Route, error) {
h.mins = append(h.mins, minProtocol)
if h.agentProtocol != 0 && minProtocol > h.agentProtocol {
if minProtocol >= protocol.MinPinnedRefVersion {
return model.Route{}, model.ErrAgentTooOldForPinned
}
return model.Route{}, model.ErrAgentTooOld
}
return h.route, h.routeErr
}
func (h *fakeHub) StartJob(r model.Route, id int64, s protocol.JobStart) error { func (h *fakeHub) StartJob(r model.Route, id int64, s protocol.JobStart) error {
h.starts = append(h.starts, startCall{r, id, s}) h.starts = append(h.starts, startCall{r, id, s})
@@ -242,3 +262,232 @@ func TestTrigger_NotUpdatableCreatesNoJob(t *testing.T) {
assert.Empty(t, h.starts) assert.Empty(t, h.starts)
assert.Empty(t, pub.topics) assert.Empty(t, pub.topics)
} }
func TestStart_WebhookTriggerWithExplicitRef(t *testing.T) {
st := &fakeStore{services: map[int64]model.Service{1: composeSvc}, jobs: map[int64]model.Job{}}
h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}}
s, _ := newService(st, h)
job, err := s.Start(context.Background(), composeSvc, StartRequest{
Trigger: model.TriggerWebhook, Ref: "gitea.shcizo.se/shcizo/slusk:2.0.0"})
require.NoError(t, err)
assert.Equal(t, model.JobQueued, job.Status)
require.Len(t, st.created, 1)
assert.Equal(t, model.TriggerWebhook, st.created[0].Trigger)
assert.Empty(t, st.created[0].RequestedBy)
assert.Equal(t, "gitea.shcizo.se/shcizo/slusk:2.0.0", st.created[0].ToRef)
require.Len(t, h.starts, 1)
assert.Equal(t, "gitea.shcizo.se/shcizo/slusk:2.0.0", h.starts[0].start.Ref)
}
func TestStart_ExplicitRefBypassesOnlyTheNoTagRefusal(t *testing.T) {
tests := []struct {
name string
svc model.Service
reason string // "" means the job starts
}{
{"digest only", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, ""},
{"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved},
{"self", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, model.ReasonSelf},
{"built", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Built: true}, model.ReasonBuilt},
{"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tt.svc.ID = 1
st := &fakeStore{services: map[int64]model.Service{1: tt.svc}, jobs: map[int64]model.Job{}}
h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}}
s, _ := newService(st, h)
_, err := s.Start(context.Background(), tt.svc, StartRequest{Trigger: model.TriggerWebhook, Ref: "x/y:2"})
if tt.reason == "" {
require.NoError(t, err)
assert.Len(t, st.created, 1)
return
}
var nu *model.NotUpdatableError
require.True(t, errors.As(err, &nu))
assert.Equal(t, tt.reason, nu.Reason)
assert.Empty(t, st.created)
})
}
}
func TestStart_WithoutRefStillRefusesNoTag(t *testing.T) {
svc := model.Service{ID: 1, Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}
s, _ := newService(&fakeStore{jobs: map[int64]model.Job{}}, &fakeHub{})
_, err := s.Start(context.Background(), svc, StartRequest{Trigger: model.TriggerWebhook})
var nu *model.NotUpdatableError
require.True(t, errors.As(err, &nu))
assert.Equal(t, model.ReasonNoTag, nu.Reason)
}
func TestAutoAllowed(t *testing.T) {
tests := []struct {
name string
svc model.Service
reason string
}{
{"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved},
{"self", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, model.ReasonSelf},
{"agent image repo", model.Service{Kind: protocol.KindCompose, ActualRef: selfImage}, model.ReasonSelf},
{"built", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Built: true}, model.ReasonBuilt},
{"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer},
{"digest without tag is allowed", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, ""},
{"tag", model.Service{Kind: protocol.KindCompose, ActualRef: "postgres:16"}, ""},
}
s, _ := newService(&fakeStore{}, &fakeHub{})
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := s.AutoAllowed(tt.svc)
if tt.reason == "" {
assert.NoError(t, err)
return
}
var nu *model.NotUpdatableError
require.True(t, errors.As(err, &nu))
assert.Equal(t, tt.reason, nu.Reason)
})
}
}
func TestFollowedRef(t *testing.T) {
assert.Equal(t, "postgres:16", FollowedRef("postgres:16@sha256:abc"))
assert.Equal(t, "postgres:latest", FollowedRef("postgres"))
assert.Equal(t, "localhost:5000/foo:latest", FollowedRef("localhost:5000/foo"))
assert.Empty(t, FollowedRef("redis@sha256:abc"))
}
func TestStart_RoutesWithTheMinimumProtocolOfTheRef(t *testing.T) {
tests := []struct {
name string
ref string // "" means the followed tag
want int
}{
{"followed tag", "", protocol.MinJobVersion},
{"repo:tag", "x/y:2", protocol.MinJobVersion},
{"repo:tag@digest", "x/y:2@sha256:a", protocol.MinPinnedRefVersion},
{"repo@digest", "x/y@sha256:a", protocol.MinPinnedRefVersion},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
st := &fakeStore{services: map[int64]model.Service{1: composeSvc}, jobs: map[int64]model.Job{}}
h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}}
s, _ := newService(st, h)
_, err := s.Start(context.Background(), composeSvc, StartRequest{Trigger: model.TriggerManual, Ref: tt.ref})
require.NoError(t, err)
assert.Equal(t, []int{tt.want}, h.mins)
})
}
}
func TestTriggerVersion_Ref(t *testing.T) {
const repo = "gitea.shcizo.se/shcizo/slusk"
noTag := model.Service{ID: 4, Kind: protocol.KindCompose, ComposeProject: "n", ComposeService: "n",
AgentID: ptr(int64(7)), ImageRepo: repo, ActualRef: repo + "@sha256:d1"}
svc := composeSvc
svc.ImageRepo = repo
tests := []struct {
name string
svc model.Service
version model.ImageVersion
wantRef string
wantMin int
}{
{"tag and digest", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"}, repo + ":1.3.0@sha256:old", protocol.MinPinnedRefVersion},
{"tag only", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "main"}, repo + ":main", protocol.MinJobVersion},
{"digest only", svc, model.ImageVersion{ID: 10, Repo: repo, Digest: "sha256:old"}, repo + "@sha256:old", protocol.MinPinnedRefVersion},
{"service that follows no tag", noTag, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"}, repo + ":1.3.0@sha256:old", protocol.MinPinnedRefVersion},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
st := &fakeStore{
services: map[int64]model.Service{tt.svc.ID: tt.svc},
jobs: map[int64]model.Job{},
versions: map[int64]model.ImageVersion{10: tt.version},
}
h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}}
s, _ := newService(st, h)
job, err := s.TriggerVersion(context.Background(), tt.svc.ID, 10, "sub-1")
require.NoError(t, err)
assert.Equal(t, model.JobQueued, job.Status)
require.Len(t, st.created, 1)
assert.Equal(t, model.TriggerManual, st.created[0].Trigger)
assert.Equal(t, "sub-1", st.created[0].RequestedBy)
assert.Equal(t, tt.wantRef, st.created[0].ToRef)
require.Len(t, h.starts, 1)
assert.Equal(t, tt.wantRef, h.starts[0].start.Ref)
assert.Equal(t, []int{tt.wantMin}, h.mins)
})
}
}
func TestTriggerVersion_NotFound(t *testing.T) {
const repo = "gitea.shcizo.se/shcizo/slusk"
svc := composeSvc
svc.ImageRepo = repo
tests := []struct {
name string
serviceID int64
versionID int64
}{
{"unknown service", 9, 10},
{"unknown version", 1, 99},
{"version of another repo", 1, 11},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
st := &fakeStore{
services: map[int64]model.Service{1: svc},
jobs: map[int64]model.Job{},
versions: map[int64]model.ImageVersion{
10: {ID: 10, Repo: repo, Tag: "1"},
11: {ID: 11, Repo: "evil/other", Tag: "1"},
},
}
h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}}
s, pub := newService(st, h)
_, err := s.TriggerVersion(context.Background(), tt.serviceID, tt.versionID, "sub-1")
assert.ErrorIs(t, err, model.ErrNotFound)
assert.Empty(t, st.created, "no job")
assert.Empty(t, h.starts)
assert.Empty(t, pub.topics)
})
}
}
func TestTriggerVersion_RequiresPinnedProtocol(t *testing.T) {
const repo = "gitea.shcizo.se/shcizo/slusk"
svc := composeSvc
svc.ImageRepo = repo
st := &fakeStore{
services: map[int64]model.Service{1: svc},
jobs: map[int64]model.Job{},
versions: map[int64]model.ImageVersion{
10: {ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"},
11: {ID: 11, Repo: repo, Tag: "main"},
},
}
h := &fakeHub{route: model.Route{AgentID: 7, ConnID: 70}, agentProtocol: 2}
s, _ := newService(st, h)
pinned, err := s.TriggerVersion(context.Background(), 1, 10, "sub-1")
require.NoError(t, err)
assert.Equal(t, model.JobFailed, pinned.Status)
assert.Equal(t, "agent too old to deploy a pinned version (needs protocol 3)", pinned.Error)
assert.Empty(t, h.starts, "nothing sent to the old agent")
plain, err := s.TriggerVersion(context.Background(), 1, 11, "sub-1")
require.NoError(t, err)
assert.Equal(t, model.JobQueued, plain.Status)
}
+13 -1
View File
@@ -15,8 +15,20 @@ var (
// ErrAgentTooOld means the only agent that could run the job speaks a // ErrAgentTooOld means the only agent that could run the job speaks a
// protocol version without jobs. // protocol version without jobs.
ErrAgentTooOld = errors.New("agent too old for updates") ErrAgentTooOld = errors.New("agent too old for updates")
// ErrAgentTooOldForPinned means the only agent that could run the job
// speaks protocol version 2, which cannot deploy a digest-pinned or
// tagless ref. errors.Is(err, ErrAgentTooOld) is true for it as well.
ErrAgentTooOldForPinned error = agentTooOldForPinned{}
) )
type agentTooOldForPinned struct{}
func (agentTooOldForPinned) Error() string {
return "agent too old to deploy a pinned version (needs protocol 3)"
}
func (agentTooOldForPinned) Is(target error) bool { return target == ErrAgentTooOld }
// NotUpdatableError says why a service cannot be updated. Reason is shown // NotUpdatableError says why a service cannot be updated. Reason is shown
// in the UI as is. // in the UI as is.
type NotUpdatableError struct { type NotUpdatableError struct {
@@ -46,7 +58,7 @@ const (
JobFailed JobStatus = "failed" JobFailed JobStatus = "failed"
) )
// JobTrigger says what created a job. Only TriggerManual is used in phase 2. // JobTrigger says what created a job. TriggerManual comes from the UI, TriggerWebhook from POST /update.
type JobTrigger string type JobTrigger string
// Job triggers. // Job triggers.
+60 -5
View File
@@ -8,6 +8,7 @@ import (
"strings" "strings"
"time" "time"
"github.com/shcizo/package-updater/internal/imageref"
"github.com/shcizo/package-updater/internal/protocol" "github.com/shcizo/package-updater/internal/protocol"
) )
@@ -51,11 +52,65 @@ type Service struct {
ActualDigest string ActualDigest string
Status string Status string
Self bool Self bool
Built bool // Compose service with a build: section; never updated Built bool // Compose service with a build: section; never updated
UpdateMode string UpdateMode string // ModeManual or ModeAuto
FirstSeenAt time.Time // AvailableRef is the newest ref a webhook or poll reported for this
UpdatedAt time.Time // service; "" if none, or if a later job already deployed it.
RemovedAt *time.Time AvailableRef string
AvailableDigest string // digest of AvailableRef when known (poll); "" from the webhook
AvailableSource string // SourceWebhook or SourcePoll; "" when AvailableRef is empty
AvailableAt *time.Time
LastPollError string // last registry error for the followed ref; "" when fine
// LastTagDigest is the newest digest of the followed tag that was deployed
// or started: set when the poller starts an auto job, when it sees the service
// already running that digest, and when any plain tag job succeeds (the
// digest the agent pinned); "" if never. The poller does
// not start another job for the same digest, so a rollback sticks until the
// registry shows a different one.
LastTagDigest string
FirstSeenAt time.Time
UpdatedAt time.Time
RemovedAt *time.Time
}
// Update modes of a service.
const (
ModeManual = "manual"
ModeAuto = "auto"
)
// Sources of an available ref and of an ImageVersion.
const (
SourceWebhook = "webhook"
SourcePoll = "poll"
// SourceRunning is used by image_versions only: an agent reported the
// version as running.
SourceRunning = "running"
)
// Available is a newer ref reported for a service.
type Available struct {
Ref string
Digest string // "" when unknown (webhook)
Source string // SourceWebhook or SourcePoll
At time.Time
}
// ImageVersion is one version of an image repo the master has seen. Tag and
// Digest are "" when the version has no tag or its digest is unknown; they
// are never both empty.
type ImageVersion struct {
ID int64
Repo string
Tag string
Digest string
Source string // SourceWebhook, SourcePoll or SourceRunning: the first source that saw it
FirstSeenAt time.Time
}
// Ref returns "repo:tag@digest", "repo:tag" or "repo@digest".
func (v ImageVersion) Ref() string {
return imageref.Ref{Repo: v.Repo, Tag: v.Tag, Digest: v.Digest}.String()
} }
// DisplayName is the human-facing name of the service. // DisplayName is the human-facing name of the service.
+18
View File
@@ -40,6 +40,24 @@ func TestDisplayName(t *testing.T) {
} }
} }
func TestImageVersionRef(t *testing.T) {
cases := []struct {
name string
v ImageVersion
want string
}{
{"tag and digest", ImageVersion{Repo: "r/x", Tag: "1", Digest: "sha256:a"}, "r/x:1@sha256:a"},
{"tag only", ImageVersion{Repo: "r/x", Tag: "1"}, "r/x:1"},
{"digest only", ImageVersion{Repo: "r/x", Digest: "sha256:a"}, "r/x@sha256:a"},
{"registry port", ImageVersion{Repo: "localhost:5000/x", Tag: "1"}, "localhost:5000/x:1"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
assert.Equal(t, c.want, c.v.Ref())
})
}
}
func TestJob_Active(t *testing.T) { func TestJob_Active(t *testing.T) {
cases := []struct { cases := []struct {
status JobStatus status JobStatus
+320
View File
@@ -0,0 +1,320 @@
// Package poller checks the registry for new digests of the tags the
// services follow. A new digest becomes the service's available version, and
// a service in auto mode gets an update job.
package poller
import (
"context"
"errors"
"log/slog"
"strings"
"sync"
"time"
"github.com/shcizo/package-updater/internal/master/events"
"github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model"
)
// lookupTimeout bounds one registry lookup.
const lookupTimeout = 30 * time.Second
// maxErrorBytes caps the registry error stored in services.last_poll_error.
const maxErrorBytes = 500
// Store is the storage the poller needs. *store.Store satisfies it.
type Store interface {
ListServices(ctx context.Context) ([]model.Service, error)
GetService(ctx context.Context, id int64) (model.Service, error)
SetAvailable(ctx context.Context, id int64, a model.Available) error
SetLastTagDigest(ctx context.Context, id int64, digest string) error
SetPollError(ctx context.Context, ids []int64, msg string) error
}
// Jobs starts updates. *jobs.Service satisfies it.
type Jobs interface {
Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error)
}
// Registry looks up the digest a "repo:tag" ref points at. *registry.Client
// satisfies it.
type Registry interface {
Digest(ctx context.Context, ref string) (string, error)
}
// Publisher receives change signals. *events.Broker satisfies it.
type Publisher interface{ Publish(t events.Topic) }
// Config holds the poller settings.
type Config struct {
Interval time.Duration // time between the end of a cycle and the start of the next (POLL_INTERVAL)
FirstDelay time.Duration // wait before the first cycle; 1 minute in production
Workers int // registry lookups in flight at once; 4 in production
}
// Poller runs polling cycles.
type Poller struct {
store Store
jobs Jobs
reg Registry
pub Publisher
cfg Config
log *slog.Logger
now func() time.Time
}
// New returns a Poller.
func New(st Store, j Jobs, reg Registry, pub Publisher, cfg Config, log *slog.Logger) *Poller {
if cfg.Workers < 1 {
cfg.Workers = 1
}
return &Poller{store: st, jobs: j, reg: reg, pub: pub, cfg: cfg, log: log, now: time.Now}
}
// SkipReason says why svc is not polled: "follows no tag", "built from
// source", "no registry digest", or "" when it is polled.
func SkipReason(svc model.Service) string {
switch {
case jobs.FollowedRef(svc.ActualRef) == "":
return model.ReasonNoTag
case svc.Built:
return "built from source"
case svc.ActualDigest == "":
return "no registry digest"
}
return ""
}
// Run polls until ctx is done. The first cycle starts Config.FirstDelay after
// the call and each following one Config.Interval after the previous cycle
// ended, so cycles never overlap. It blocks.
func (p *Poller) Run(ctx context.Context) {
timer := time.NewTimer(p.cfg.FirstDelay)
defer timer.Stop()
for {
select {
case <-ctx.Done():
return
case <-timer.C:
}
p.Cycle(ctx)
timer.Reset(p.cfg.Interval)
}
}
// lookup is the registry's answer for one followed ref.
type lookup struct {
ref string
digest string
err error
}
// cycleStats is what one cycle did, for the poll_cycle log line.
type cycleStats struct {
refs, ok, failed, suggested, jobs, skipped int
wrote bool
}
// Cycle runs one poll over all services. It looks up every distinct followed
// ref once, with at most Config.Workers lookups at a time, then records the
// results. Writing results and starting jobs is not cut short by ctx ending
// during the lookups, so a job that was marked is always started.
func (p *Poller) Cycle(ctx context.Context) {
began := p.now()
svcs, err := p.store.ListServices(ctx)
if err != nil {
p.log.Error("poll_list_failed", "error", err)
return
}
var stats cycleStats
byRef := map[string][]model.Service{}
var refs []string
for _, svc := range svcs {
if svc.RemovedAt != nil {
continue
}
if SkipReason(svc) != "" {
stats.skipped++
continue
}
ref := jobs.FollowedRef(svc.ActualRef)
if _, seen := byRef[ref]; !seen {
refs = append(refs, ref)
}
byRef[ref] = append(byRef[ref], svc)
}
stats.refs = len(refs)
write := context.WithoutCancel(ctx)
for _, l := range p.lookupAll(ctx, refs) {
if l.err != nil && ctx.Err() != nil {
continue // cancelled mid-lookup: not the registry's fault
}
p.record(write, l, byRef[l.ref], &stats)
}
if stats.wrote {
p.pub.Publish(events.Services)
}
p.log.Info("poll_cycle", "refs", stats.refs, "ok", stats.ok, "failed", stats.failed,
"suggested", stats.suggested, "jobs", stats.jobs, "skipped", stats.skipped,
"duration_ms", p.now().Sub(began).Milliseconds())
}
// lookupAll asks the registry for every ref, Config.Workers at a time. It
// returns the answers in ref order and stops starting new lookups once ctx
// is done; refs never looked up are missing from the result.
func (p *Poller) lookupAll(ctx context.Context, refs []string) []lookup {
results := make([]*lookup, len(refs))
next := make(chan int)
var wg sync.WaitGroup
for w := 0; w < p.cfg.Workers; w++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := range next {
lctx, cancel := context.WithTimeout(ctx, lookupTimeout)
d, err := p.reg.Digest(lctx, refs[i])
cancel()
results[i] = &lookup{ref: refs[i], digest: d, err: err}
}
}()
}
feed:
for i := range refs {
select {
case next <- i:
case <-ctx.Done():
break feed
}
}
close(next)
wg.Wait()
var out []lookup
for _, r := range results {
if r != nil {
out = append(out, *r)
}
}
return out
}
// record stores one lookup result for the services that follow l.ref.
func (p *Poller) record(ctx context.Context, l lookup, svcs []model.Service, stats *cycleStats) {
if l.err != nil {
stats.failed++
msg := truncate(l.err.Error())
p.log.Warn("poll_failed", "ref", l.ref, "error", msg)
p.setPollError(ctx, svcs, msg, stats, func(s model.Service) bool { return s.LastPollError != msg })
return
}
stats.ok++
p.setPollError(ctx, svcs, "", stats, func(s model.Service) bool { return s.LastPollError != "" })
for _, svc := range svcs {
if !p.suggest(ctx, svc, l, stats) {
continue
}
if autoWanted(svc, l.ref, l.digest) {
p.startAuto(ctx, svc.ID, l.ref, l.digest, stats)
}
}
}
// suggest marks l.digest as the available version of svc unless the service
// already runs it or already has exactly that suggestion. A service that
// runs l.digest also gets it as last_tag_digest. It returns false only when
// writing the suggestion failed.
func (p *Poller) suggest(ctx context.Context, svc model.Service, l lookup, stats *cycleStats) bool {
if svc.ActualDigest == l.digest && svc.LastTagDigest != l.digest {
// Seed last_tag_digest for a service that was not updated by a tag job
// (it predates the column or was updated by hand), so that a later
// rollback is not undone.
if err := p.store.SetLastTagDigest(ctx, svc.ID, l.digest); err != nil {
p.log.Error("poll_set_last_tag_digest_failed", "service_id", svc.ID, "error", err)
}
}
if svc.ActualDigest == l.digest || (svc.AvailableRef == l.ref && svc.AvailableDigest == l.digest) {
return true
}
err := p.store.SetAvailable(ctx, svc.ID, model.Available{Ref: l.ref, Digest: l.digest, Source: model.SourcePoll, At: p.now()})
if err != nil {
p.log.Error("poll_set_available_failed", "service_id", svc.ID, "ref", l.ref, "error", err)
return false
}
stats.wrote = true
stats.suggested++
return true
}
// autoWanted reports whether an update of svc to ref at digest should be
// started automatically: the service is in auto mode, not removed, does not
// already run that digest, was not already updated automatically to it
// (otherwise a rollback to an older version would be undone at the next
// cycle), and its last job did not already deploy it (desired_ref holds the
// digest until the agent reports the new inventory).
func autoWanted(svc model.Service, ref, digest string) bool {
return svc.RemovedAt == nil &&
svc.UpdateMode == model.ModeAuto &&
svc.ActualDigest != digest &&
svc.LastTagDigest != digest &&
svc.DesiredRef != ref+"@"+digest
}
// startAuto starts the update job of an auto service to ref and remembers
// digest as services.last_tag_digest, so the same digest is not started
// again, for instance after the user rolled back. It re-reads the service
// first, because a cycle can run for a long time and the user may have
// switched the service to manual meanwhile. A service that already has a
// job or cannot be updated is not an error for the cycle, and last_tag_digest
// is left alone so the next cycle tries again.
func (p *Poller) startAuto(ctx context.Context, id int64, ref, digest string, stats *cycleStats) {
svc, err := p.store.GetService(ctx, id)
if err != nil {
p.log.Warn("poll_get_service_failed", "service_id", id, "error", err)
return
}
if !autoWanted(svc, ref, digest) {
return
}
_, err = p.jobs.Start(ctx, svc, jobs.StartRequest{Trigger: model.TriggerPoll, Ref: ref})
var notUpdatable *model.NotUpdatableError
switch {
case err == nil:
stats.jobs++
if err := p.store.SetLastTagDigest(ctx, id, digest); err != nil {
p.log.Error("poll_set_last_tag_digest_failed", "service_id", id, "error", err)
}
case errors.Is(err, model.ErrJobActive), errors.As(err, &notUpdatable):
p.log.Debug("poll_job_not_started", "service_id", id, "ref", ref, "reason", err.Error())
default:
p.log.Warn("poll_job_failed", "service_id", id, "ref", ref, "error", err)
}
}
// setPollError sets msg as last_poll_error on the services for which
// changed returns true; with none, it writes nothing.
func (p *Poller) setPollError(ctx context.Context, svcs []model.Service, msg string, stats *cycleStats, changed func(model.Service) bool) {
var ids []int64
for _, svc := range svcs {
if changed(svc) {
ids = append(ids, svc.ID)
}
}
if len(ids) == 0 {
return
}
if err := p.store.SetPollError(ctx, ids, msg); err != nil {
p.log.Error("poll_set_error_failed", "error", err)
return
}
stats.wrote = true
}
// truncate cuts msg to maxErrorBytes without leaving a broken UTF-8
// sequence, and drops NUL bytes, which Postgres text cannot hold.
func truncate(msg string) string {
msg = strings.ReplaceAll(msg, "\x00", "")
if len(msg) > maxErrorBytes {
msg = strings.ToValidUTF8(msg[:maxErrorBytes], "")
}
return msg
}
+651
View File
@@ -0,0 +1,651 @@
package poller
import (
"context"
"errors"
"fmt"
"io"
"log/slog"
"sort"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/events"
"github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model"
)
type availableCall struct {
id int64
a model.Available
}
type pollErrorCall struct {
ids []int64
msg string
}
type lastTagCall struct {
id int64
digest string
}
type fakeStore struct {
mu sync.Mutex
services []model.Service
fresh map[int64]model.Service // what GetService returns instead of the listed row
lastTag []lastTagCall
listErr error
available []availableCall
pollErrors []pollErrorCall
ctxErrs []error // ctx.Err() seen by every write
}
func (f *fakeStore) ListServices(context.Context) ([]model.Service, error) {
return f.services, f.listErr
}
func (f *fakeStore) GetService(_ context.Context, id int64) (model.Service, error) {
f.mu.Lock()
defer f.mu.Unlock()
if s, ok := f.fresh[id]; ok {
return s, nil
}
for _, s := range f.services {
if s.ID == id {
return s, nil
}
}
return model.Service{}, model.ErrNotFound
}
// SetLastTagDigest records the call and, like the database, changes the
// row the next cycle lists.
func (f *fakeStore) SetLastTagDigest(_ context.Context, id int64, digest string) error {
f.mu.Lock()
defer f.mu.Unlock()
f.lastTag = append(f.lastTag, lastTagCall{id, digest})
for i := range f.services {
if f.services[i].ID == id {
f.services[i].LastTagDigest = digest
}
}
return nil
}
func (f *fakeStore) SetAvailable(ctx context.Context, id int64, a model.Available) error {
f.mu.Lock()
defer f.mu.Unlock()
f.ctxErrs = append(f.ctxErrs, ctx.Err())
f.available = append(f.available, availableCall{id, a})
return nil
}
func (f *fakeStore) SetPollError(ctx context.Context, ids []int64, msg string) error {
f.mu.Lock()
defer f.mu.Unlock()
f.ctxErrs = append(f.ctxErrs, ctx.Err())
f.pollErrors = append(f.pollErrors, pollErrorCall{ids, msg})
return nil
}
type startCall struct {
svcID int64
req jobs.StartRequest
}
type fakeJobs struct {
mu sync.Mutex
starts []startCall
errFor map[int64]error // by service ID
ctxErrs []error
}
func (f *fakeJobs) Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.ctxErrs = append(f.ctxErrs, ctx.Err())
f.starts = append(f.starts, startCall{svc.ID, req})
return model.Job{ID: int64(len(f.starts))}, f.errFor[svc.ID]
}
type fakeRegistry struct {
mu sync.Mutex
digests map[string]string // ref -> digest
errs map[string]error // ref -> error
calls []string
inFlight int
maxSeen int
delay time.Duration
started chan struct{} // receives a value whenever a lookup starts, if set
release chan struct{} // lookups wait for it to be closed, if set
onResult func() // called after a lookup computed its answer
}
func (f *fakeRegistry) Digest(ctx context.Context, ref string) (string, error) {
f.mu.Lock()
f.calls = append(f.calls, ref)
f.inFlight++
if f.inFlight > f.maxSeen {
f.maxSeen = f.inFlight
}
f.mu.Unlock()
defer func() {
f.mu.Lock()
f.inFlight--
f.mu.Unlock()
}()
if f.started != nil {
f.started <- struct{}{}
}
if f.release != nil {
select {
case <-f.release:
case <-ctx.Done():
return "", ctx.Err()
}
}
if f.delay > 0 {
time.Sleep(f.delay)
}
if f.onResult != nil {
f.onResult()
}
if err := f.errs[ref]; err != nil {
return "", err
}
return f.digests[ref], nil
}
func (f *fakeRegistry) sortedCalls() []string {
f.mu.Lock()
defer f.mu.Unlock()
out := append([]string(nil), f.calls...)
sort.Strings(out)
return out
}
type fakePub struct {
mu sync.Mutex
topics []events.Topic
}
func (p *fakePub) Publish(t events.Topic) {
p.mu.Lock()
defer p.mu.Unlock()
p.topics = append(p.topics, t)
}
const (
repoA = "gitea.shcizo.se/shcizo/slusk"
repoB = "gitea/gitea"
)
// svc builds a polled service whose running version has digest sha256:old.
func svc(id int64, actualRef string) model.Service {
return model.Service{ID: id, Kind: "compose", ActualRef: actualRef, ActualDigest: "sha256:old", UpdateMode: model.ModeManual}
}
func newPoller(st *fakeStore, j *fakeJobs, reg *fakeRegistry, cfg Config) (*Poller, *fakePub) {
pub := &fakePub{}
if cfg.Workers == 0 {
cfg.Workers = 4
}
p := New(st, j, reg, pub, cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
p.now = func() time.Time { return time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC) }
return p, pub
}
func TestSkipReason(t *testing.T) {
tests := []struct {
name string
svc model.Service
want string
}{
{"tag and digest", model.Service{ActualRef: "r:1", ActualDigest: "sha256:a"}, ""},
{"bare image follows latest", model.Service{ActualRef: "r", ActualDigest: "sha256:a"}, ""},
{"digest only follows no tag", model.Service{ActualRef: "r@sha256:a", ActualDigest: "sha256:a"}, model.ReasonNoTag},
{"built", model.Service{ActualRef: "r:1", ActualDigest: "sha256:a", Built: true}, "built from source"},
{"no digest", model.Service{ActualRef: "r:1"}, "no registry digest"},
{"self and container are polled", model.Service{ActualRef: "r:1", ActualDigest: "sha256:a", Self: true, Kind: "container"}, ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, SkipReason(tt.svc))
})
}
}
func TestOneHeadPerDistinctRef(t *testing.T) {
st := &fakeStore{services: []model.Service{
svc(1, repoA+":1.4"),
svc(2, repoA+":1.4@sha256:old"),
svc(3, repoB),
svc(4, repoB+":latest"),
svc(5, repoB+":1.27"),
}}
reg := &fakeRegistry{digests: map[string]string{}}
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Equal(t, []string{repoA + ":1.4", repoB + ":1.27", repoB + ":latest"}, reg.sortedCalls())
}
func TestSkipped(t *testing.T) {
removed := svc(4, "r:1")
removed.RemovedAt = ptr(time.Now())
st := &fakeStore{services: []model.Service{
{ID: 1, ActualRef: "r@sha256:a", ActualDigest: "sha256:a"},
{ID: 2, ActualRef: "r:1", ActualDigest: "sha256:a", Built: true},
{ID: 3, ActualRef: "r:1"},
removed,
}}
reg := &fakeRegistry{}
j := &fakeJobs{}
p, pub := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
assert.Empty(t, reg.calls, "no lookups")
assert.Empty(t, st.available)
assert.Empty(t, st.pollErrors)
assert.Empty(t, j.starts)
assert.Empty(t, pub.topics)
}
func TestNewDigest(t *testing.T) {
ref := repoA + ":1.4"
tests := []struct {
name string
mode string
wantStart bool
}{
{"auto gets one poll job to repo:tag", model.ModeAuto, true},
{"manual is only marked", model.ModeManual, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := svc(1, ref+"@sha256:old")
s.UpdateMode = tt.mode
st := &fakeStore{services: []model.Service{s}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{}
p, pub := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
require.Len(t, st.available, 1)
assert.Equal(t, int64(1), st.available[0].id)
assert.Equal(t, model.Available{Ref: ref, Digest: "sha256:new", Source: model.SourcePoll,
At: time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}, st.available[0].a)
if tt.wantStart {
assert.Equal(t, []startCall{{1, jobs.StartRequest{Trigger: model.TriggerPoll, Ref: ref}}}, j.starts)
assert.Equal(t, []lastTagCall{{1, "sha256:new"}}, st.lastTag)
} else {
assert.Empty(t, j.starts)
assert.Empty(t, st.lastTag)
}
assert.Equal(t, []events.Topic{events.Services}, pub.topics)
})
}
}
func TestSameSuggestionIsIdempotent(t *testing.T) {
ref := repoA + ":1.4"
running := svc(1, ref+"@sha256:old")
running.UpdateMode = model.ModeAuto
running.ActualDigest = "sha256:new" // already on the registry's digest
suggested := svc(2, ref)
suggested.UpdateMode = model.ModeAuto
suggested.AvailableRef, suggested.AvailableDigest = ref, "sha256:new"
suggested.LastTagDigest = "sha256:new" // its job was started in an earlier cycle
staleSuggestion := svc(3, ref)
staleSuggestion.AvailableRef, staleSuggestion.AvailableDigest = ref, "sha256:older"
st := &fakeStore{services: []model.Service{running, suggested, staleSuggestion}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
require.Len(t, st.available, 1, "only the service with an older suggestion is updated")
assert.Equal(t, int64(3), st.available[0].id)
assert.Empty(t, j.starts, "no new job for an already suggested digest")
}
func TestStartErrorsDoNotAbortCycle(t *testing.T) {
ref := repoA + ":1.4"
var svcs []model.Service
for id := int64(1); id <= 3; id++ {
s := svc(id, ref)
s.UpdateMode = model.ModeAuto
svcs = append(svcs, s)
}
st := &fakeStore{services: svcs}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{errFor: map[int64]error{
1: model.ErrJobActive,
2: &model.NotUpdatableError{Reason: model.ReasonSelf},
3: errors.New("database down"),
}}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
assert.Len(t, st.available, 3)
assert.Len(t, j.starts, 3)
assert.Empty(t, st.lastTag, "a job that was not created is retried next cycle")
}
func TestRollbackOfAutoServiceIsNotUndone(t *testing.T) {
ref := repoA + ":main"
// Updated to sha256:new automatically, then rolled back to sha256:old by hand.
s := svc(1, ref+"@sha256:old")
s.UpdateMode = model.ModeAuto
s.LastTagDigest = "sha256:new"
s.DesiredRef = ref + "@sha256:old"
st := &fakeStore{services: []model.Service{s}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
p.Cycle(context.Background())
assert.Empty(t, j.starts, "same registry digest: the rollback stays")
assert.Equal(t, "sha256:new", st.available[0].a.Digest, "the newer version is still offered")
reg.digests[ref] = "sha256:newer"
p.Cycle(context.Background())
assert.Equal(t, []startCall{{1, jobs.StartRequest{Trigger: model.TriggerPoll, Ref: ref}}}, j.starts, "a new digest moves it again")
assert.Equal(t, []lastTagCall{{1, "sha256:newer"}}, st.lastTag)
}
func TestActiveJobIsRetriedNextCycle(t *testing.T) {
ref := repoA + ":1.4"
s := svc(1, ref)
s.UpdateMode = model.ModeAuto
st := &fakeStore{services: []model.Service{s}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{errFor: map[int64]error{1: model.ErrJobActive}}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
require.Len(t, j.starts, 1)
assert.Empty(t, st.lastTag)
j.errFor = nil
p.Cycle(context.Background())
assert.Len(t, j.starts, 2)
assert.Equal(t, []lastTagCall{{1, "sha256:new"}}, st.lastTag)
p.Cycle(context.Background())
assert.Len(t, j.starts, 2, "started once the job exists")
}
func TestJobThatAlreadyDeployedTheDigestIsNotStartedAgain(t *testing.T) {
ref := repoA + ":1.4"
// FinishJob set desired_ref; the agent has not reported the new digest yet.
s := svc(1, ref+"@sha256:old")
s.UpdateMode = model.ModeAuto
s.DesiredRef = ref + "@sha256:new"
st := &fakeStore{services: []model.Service{s}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
assert.Empty(t, j.starts)
}
func TestServiceSwitchedToManualDuringTheCycleIsNotStarted(t *testing.T) {
ref := repoA + ":1.4"
s := svc(1, ref)
s.UpdateMode = model.ModeAuto
manual := s
manual.UpdateMode = model.ModeManual
st := &fakeStore{services: []model.Service{s}, fresh: map[int64]model.Service{1: manual}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
assert.Len(t, st.available, 1, "still suggested")
assert.Empty(t, j.starts)
assert.Empty(t, st.lastTag)
}
func TestErrorIsPerRef(t *testing.T) {
bad, good := repoA+":1.4", repoB+":1.27"
badOld := svc(1, bad)
badSame := svc(2, bad)
badSame.LastPollError = "head: 401 unauthorized"
goodSvc := svc(3, good)
st := &fakeStore{services: []model.Service{badOld, badSame, goodSvc}}
reg := &fakeRegistry{
errs: map[string]error{bad: errors.New("head: 401 unauthorized")},
digests: map[string]string{good: "sha256:new"},
}
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Equal(t, []pollErrorCall{{[]int64{1}, "head: 401 unauthorized"}}, st.pollErrors,
"only the service whose stored error differs is written")
require.Len(t, st.available, 1, "the other ref is still processed")
assert.Equal(t, int64(3), st.available[0].id)
assert.Equal(t, []events.Topic{events.Services}, pub.topics)
}
func TestErrorTextIsTruncated(t *testing.T) {
ref := repoA + ":1.4"
st := &fakeStore{services: []model.Service{svc(1, ref)}}
reg := &fakeRegistry{errs: map[string]error{ref: errors.New(strings.Repeat("é", 400) + "\x00")}}
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
require.Len(t, st.pollErrors, 1)
msg := st.pollErrors[0].msg
assert.LessOrEqual(t, len(msg), maxErrorBytes)
assert.Equal(t, strings.Repeat("é", 250), msg, "cut on a character boundary, NUL removed")
}
func TestErrorClearedOnSuccess(t *testing.T) {
ref := repoA + ":1.4"
failing := svc(1, ref)
failing.LastPollError = "head: boom"
fine := svc(2, ref)
failing.ActualDigest, fine.ActualDigest = "sha256:new", "sha256:new"
st := &fakeStore{services: []model.Service{failing, fine}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Equal(t, []pollErrorCall{{[]int64{1}, ""}}, st.pollErrors)
assert.Empty(t, st.available)
assert.Equal(t, []events.Topic{events.Services}, pub.topics)
}
func TestNothingWrittenNothingPublished(t *testing.T) {
ref := repoA + ":1.4"
s := svc(1, ref)
s.ActualDigest = "sha256:new"
st := &fakeStore{services: []model.Service{s}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Empty(t, st.pollErrors)
assert.Empty(t, st.available)
assert.Empty(t, pub.topics)
}
func TestListErrorEndsTheCycle(t *testing.T) {
st := &fakeStore{listErr: errors.New("db down")}
reg := &fakeRegistry{}
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Empty(t, reg.calls)
}
func TestConcurrencyLimit(t *testing.T) {
const workers = 3
var svcs []model.Service
for i := 0; i < 8; i++ {
svcs = append(svcs, svc(int64(i+1), fmt.Sprintf("%s%d:1", repoA, i)))
}
st := &fakeStore{services: svcs}
reg := &fakeRegistry{release: make(chan struct{}), started: make(chan struct{}, 8)}
p, _ := newPoller(st, &fakeJobs{}, reg, Config{Workers: workers})
done := make(chan struct{})
go func() { p.Cycle(context.Background()); close(done) }()
for i := 0; i < workers; i++ {
select {
case <-reg.started:
case <-time.After(2 * time.Second):
t.Fatalf("only %d lookups started, want %d", i, workers)
}
}
select {
case <-reg.started:
t.Fatal("more lookups in flight than Workers")
case <-time.After(100 * time.Millisecond):
}
close(reg.release)
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Cycle did not finish")
}
assert.Len(t, reg.calls, 8)
}
func TestCycleStopsOnCancel(t *testing.T) {
var svcs []model.Service
for i := 0; i < 6; i++ {
svcs = append(svcs, svc(int64(i+1), fmt.Sprintf("%s%d:1", repoA, i)))
}
st := &fakeStore{services: svcs}
reg := &fakeRegistry{release: make(chan struct{}), started: make(chan struct{}, 6)}
p, pub := newPoller(st, &fakeJobs{}, reg, Config{Workers: 2})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { p.Cycle(ctx); close(done) }()
<-reg.started
<-reg.started
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Cycle did not return after cancel")
}
assert.Len(t, reg.calls, 2, "no new lookups after cancel")
assert.Empty(t, st.pollErrors, "cancelled lookups are not recorded as registry errors")
assert.Empty(t, st.available)
assert.Empty(t, pub.topics)
}
func TestMarkedJobsAreStartedEvenIfCancelledDuringTheCycle(t *testing.T) {
ref := repoA + ":1.4"
s := svc(1, ref)
s.UpdateMode = model.ModeAuto
st := &fakeStore{services: []model.Service{s}}
ctx, cancel := context.WithCancel(context.Background())
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}, onResult: cancel}
j := &fakeJobs{}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(ctx)
require.Len(t, st.available, 1)
require.Len(t, j.starts, 1)
assert.NoError(t, st.ctxErrs[0], "writes do not see the cancelled context")
assert.NoError(t, j.ctxErrs[0])
}
func TestRun(t *testing.T) {
ref := repoA + ":1.4"
st := &fakeStore{services: []model.Service{svc(1, ref)}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:old"}, delay: 30 * time.Millisecond}
p, _ := newPoller(st, &fakeJobs{}, reg, Config{Interval: 5 * time.Millisecond, FirstDelay: 60 * time.Millisecond, Workers: 1})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
began := time.Now()
go func() { p.Run(ctx); close(done) }()
require.Eventually(t, func() bool {
reg.mu.Lock()
defer reg.mu.Unlock()
return len(reg.calls) >= 3
}, 5*time.Second, 5*time.Millisecond)
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Run did not return after cancel")
}
assert.GreaterOrEqual(t, time.Since(began), 60*time.Millisecond+2*30*time.Millisecond, "first cycle waits FirstDelay")
assert.Equal(t, 1, reg.maxSeen, "cycles never overlap")
}
func TestRunReturnsDuringFirstDelay(t *testing.T) {
p, _ := newPoller(&fakeStore{}, &fakeJobs{}, &fakeRegistry{}, Config{Interval: time.Hour, FirstDelay: time.Hour})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { p.Run(ctx); close(done) }()
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Run did not return after cancel")
}
}
func ptr[T any](v T) *T { return &v }
func TestSeedsTheTagDigestOfARunningService(t *testing.T) {
ref := repoA + ":main"
s := svc(1, ref+"@sha256:new")
s.UpdateMode = model.ModeAuto
s.ActualDigest = "sha256:new" // updated outside the updater: last_tag_digest is empty
st := &fakeStore{services: []model.Service{s}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
assert.Equal(t, []lastTagCall{{1, "sha256:new"}}, st.lastTag)
assert.Empty(t, j.starts)
// Rolled back by hand to the older digest.
st.services[0].ActualRef, st.services[0].ActualDigest = ref+"@sha256:old", "sha256:old"
st.services[0].DesiredRef = ref + "@sha256:old"
p.Cycle(context.Background())
assert.Empty(t, j.starts, "the rollback stays")
assert.Len(t, st.lastTag, 1, "nothing more to seed")
}
+114
View File
@@ -0,0 +1,114 @@
// Package registry looks up the digest a tag points at in a container
// registry, using HEAD requests only.
package registry
import (
"context"
"errors"
"fmt"
"log/slog"
"net"
"net/http"
"net/url"
"regexp"
"strings"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/remote"
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
"github.com/shcizo/package-updater/internal/imageref"
)
// Client looks up tag digests with HEAD requests only. Docker Hub counts a
// GET of a manifest as a pull against the rate limit and a HEAD as nothing,
// so Client never issues a manifest GET.
type Client struct {
keychain authn.Keychain
userAgent string
log *slog.Logger
}
// New returns a Client that finds credentials the way the Docker CLI does:
// in $DOCKER_CONFIG/config.json, else ~/.docker/config.json. Registries
// without an entry are accessed anonymously. userAgent is added to the
// User-Agent header of every request. log receives the full error of a
// failed lookup, which Digest itself returns only in a sanitised form.
func New(userAgent string, log *slog.Logger) *Client {
return &Client{keychain: authn.DefaultKeychain, userAgent: userAgent, log: log}
}
// Digest returns the digest that "repo:tag" points at, as the registry
// reports it: for a multi-arch image that is the digest of the index, the
// same one `docker inspect` shows in RepoDigests. A digest suffix in ref is
// ignored. It sends HEAD /v2/<repo>/manifests/<tag> and nothing else to the
// manifest endpoint; it does not fall back to GET when the registry omits
// the digest header.
func (c *Client) Digest(ctx context.Context, ref string) (string, error) {
p := imageref.Parse(ref)
if p.Tag == "" {
return "", fmt.Errorf("image %q has no tag", ref)
}
r, err := name.ParseReference(p.Repo + ":" + p.Tag)
if err != nil {
return "", fmt.Errorf("parse %s: %w", ref, err)
}
// Credentials are resolved here, not by ggcr, so that a broken credential
// helper keeps its own message: that text comes from the local config.
auth, err := c.keychain.Resolve(r.Context())
if err != nil {
return "", fmt.Errorf("lookup %s: %w", r, err)
}
desc, err := remote.Head(r,
remote.WithContext(ctx),
remote.WithAuth(auth),
remote.WithUserAgent(c.userAgent))
if err != nil {
var terr *transport.Error
if errors.As(err, &terr) {
return "", fmt.Errorf("lookup %s: %s", r, describe(terr))
}
c.log.Warn("registry_lookup_failed", "ref", ref, "error", err)
return "", fmt.Errorf("lookup %s: %s", r, classify(err))
}
return desc.Digest.String(), nil
}
// classify returns a fixed text for an error that is not a registry answer.
// ggcr copies registry-controlled text (token responses, authentication
// challenges, redirect URLs, certificate names) into such errors, so none of
// it is passed on.
func classify(err error) string {
var netErr net.Error
var urlErr *url.Error
switch {
case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled):
return "timeout"
case errors.As(err, &netErr), errors.As(err, &urlErr):
return "cannot reach registry"
}
return "lookup failed"
}
// errorCodePattern matches the registry error codes of the distribution
// spec ("MANIFEST_UNKNOWN"). Anything else is dropped from error text.
var errorCodePattern = regexp.MustCompile(`^[A-Z_]{1,40}$`)
// describe returns the HTTP status and the structured error codes of a
// registry error, without the messages, details or raw body that ggcr copies
// into its error text. The registry host comes from an image name, so
// whatever it answers must not end up in the UI.
func describe(e *transport.Error) string {
msg := fmt.Sprintf("registry answered %d %s", e.StatusCode, http.StatusText(e.StatusCode))
var codes []string
for _, d := range e.Errors {
if errorCodePattern.MatchString(string(d.Code)) {
codes = append(codes, string(d.Code))
}
}
if len(codes) > 0 {
msg += " (" + strings.Join(codes, ", ") + ")"
}
return msg
}
+336
View File
@@ -0,0 +1,336 @@
package registry
import (
"context"
"encoding/base64"
"errors"
"fmt"
"io"
"log"
"log/slog"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
ggcrregistry "github.com/google/go-containerregistry/pkg/registry"
"github.com/google/go-containerregistry/pkg/v1/random"
"github.com/google/go-containerregistry/pkg/v1/remote"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
var discardLog = slog.New(slog.NewTextHandler(io.Discard, nil))
func newClient() *Client { return New("test", discardLog) }
// request is one request the test registry received.
type request struct{ method, path string }
// testRegistry is an in-memory registry that records every request.
type testRegistry struct {
srv *httptest.Server
host string // "127.0.0.1:port", the registry part of a ref
mu sync.Mutex
requests []request
}
// newTestRegistry starts a registry. When user is set every request needs
// those basic-auth credentials.
func newTestRegistry(t *testing.T, user, pass string) *testRegistry {
t.Helper()
backend := ggcrregistry.New(ggcrregistry.Logger(log.New(io.Discard, "", 0)))
tr := &testRegistry{}
tr.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
tr.mu.Lock()
tr.requests = append(tr.requests, request{r.Method, r.URL.Path})
tr.mu.Unlock()
if user != "" {
if u, p, ok := r.BasicAuth(); !ok || u != user || p != pass {
w.Header().Set("WWW-Authenticate", `Basic realm="test"`)
w.WriteHeader(http.StatusUnauthorized)
return
}
}
backend.ServeHTTP(w, r)
}))
t.Cleanup(tr.srv.Close)
tr.host = strings.TrimPrefix(tr.srv.URL, "http://")
return tr
}
// pushIndex stores a random multi-arch index as repo:tag and returns its
// digest. user and pass are used when the registry requires them.
func (tr *testRegistry) pushIndex(t *testing.T, repo, tag string, opts ...remote.Option) string {
t.Helper()
idx, err := random.Index(64, 1, 2)
require.NoError(t, err)
ref, err := name.ParseReference(tr.host + "/" + repo + ":" + tag)
require.NoError(t, err)
require.NoError(t, remote.WriteIndex(ref, idx, opts...))
d, err := idx.Digest()
require.NoError(t, err)
return d.String()
}
// manifestRequests returns the requests that touched a manifest.
func (tr *testRegistry) manifestRequests() []request {
tr.mu.Lock()
defer tr.mu.Unlock()
var out []request
for _, r := range tr.requests {
if strings.Contains(r.path, "/manifests/") {
out = append(out, r)
}
}
return out
}
// reset forgets the requests received so far.
func (tr *testRegistry) reset() {
tr.mu.Lock()
defer tr.mu.Unlock()
tr.requests = nil
}
// dockerConfig points DOCKER_CONFIG at a fresh directory holding config.json
// with the given content; "" leaves the directory empty.
func dockerConfig(t *testing.T, configJSON string) {
t.Helper()
dir := t.TempDir()
if configJSON != "" {
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte(configJSON), 0o600))
}
t.Setenv("DOCKER_CONFIG", dir)
}
func TestDigestOfIndexIsIndexDigest(t *testing.T) {
dockerConfig(t, "")
reg := newTestRegistry(t, "", "")
want := reg.pushIndex(t, "team/app", "1.2")
got, err := newClient().Digest(context.Background(), reg.host+"/team/app:1.2")
require.NoError(t, err)
assert.Equal(t, want, got)
}
func TestDigestIgnoresDigestSuffix(t *testing.T) {
dockerConfig(t, "")
reg := newTestRegistry(t, "", "")
want := reg.pushIndex(t, "team/app", "1.2")
got, err := newClient().Digest(context.Background(), reg.host+"/team/app:1.2@sha256:"+strings.Repeat("0", 64))
require.NoError(t, err)
assert.Equal(t, want, got)
}
func TestDigestUsesHeadOnly(t *testing.T) {
dockerConfig(t, "")
reg := newTestRegistry(t, "", "")
reg.pushIndex(t, "team/app", "1.2")
reg.reset()
_, err := newClient().Digest(context.Background(), reg.host+"/team/app:1.2")
require.NoError(t, err)
assert.Equal(t, []request{{http.MethodHead, "/v2/team/app/manifests/1.2"}}, reg.manifestRequests())
for _, r := range reg.requests {
if !strings.Contains(r.path, "/manifests/") {
assert.Equal(t, request{http.MethodGet, "/v2/"}, r, "the only other request is the API ping")
}
}
}
func TestDigestSendsUserAgent(t *testing.T) {
dockerConfig(t, "")
var ua string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ua = r.Header.Get("User-Agent")
w.WriteHeader(http.StatusNotFound)
}))
t.Cleanup(srv.Close)
_, _ = New("package-updater-test/1", discardLog).Digest(context.Background(), strings.TrimPrefix(srv.URL, "http://")+"/a/b:1")
assert.Contains(t, ua, "package-updater-test/1")
}
func TestCredentialsFromDockerConfig(t *testing.T) {
reg := newTestRegistry(t, "bot", "s3cret")
want := reg.pushIndex(t, "team/app", "1.2", remote.WithAuth(&authn.Basic{Username: "bot", Password: "s3cret"}))
auth := base64.StdEncoding.EncodeToString([]byte("bot:s3cret"))
dockerConfig(t, `{"auths":{"`+reg.host+`":{"auth":"`+auth+`"}}}`)
got, err := newClient().Digest(context.Background(), reg.host+"/team/app:1.2")
require.NoError(t, err)
assert.Equal(t, want, got)
}
func TestWrongCredentialsFail(t *testing.T) {
reg := newTestRegistry(t, "bot", "s3cret")
reg.pushIndex(t, "team/app", "1.2", remote.WithAuth(&authn.Basic{Username: "bot", Password: "s3cret"}))
auth := base64.StdEncoding.EncodeToString([]byte("bot:wrong"))
dockerConfig(t, `{"auths":{"`+reg.host+`":{"auth":"`+auth+`"}}}`)
_, err := newClient().Digest(context.Background(), reg.host+"/team/app:1.2")
require.Error(t, err)
assert.NotContains(t, err.Error(), "wrong", "credentials do not leak into errors")
assert.NotContains(t, err.Error(), auth)
}
func TestAnonymousWithoutConfig(t *testing.T) {
dockerConfig(t, "")
open := newTestRegistry(t, "", "")
want := open.pushIndex(t, "team/app", "1.2")
protected := newTestRegistry(t, "bot", "s3cret")
got, err := newClient().Digest(context.Background(), open.host+"/team/app:1.2")
require.NoError(t, err)
assert.Equal(t, want, got, "anonymous access works without a config file")
_, err = newClient().Digest(context.Background(), protected.host+"/team/app:1.2")
assert.Error(t, err, "a registry that needs credentials fails without them")
}
func TestUnknownTagIsAnError(t *testing.T) {
dockerConfig(t, "")
reg := newTestRegistry(t, "", "")
reg.pushIndex(t, "team/app", "1.2")
_, err := newClient().Digest(context.Background(), reg.host+"/team/app:nope")
assert.Error(t, err)
}
func TestRefWithoutTagIsAnError(t *testing.T) {
_, err := newClient().Digest(context.Background(), "team/app@sha256:"+strings.Repeat("a", 64))
assert.ErrorContains(t, err, "has no tag")
}
// A config.json whose credsStore names a helper that is not installed makes
// the lookup fail for every registry, including ones that would work
// anonymously. The master container has no credential helpers, so only
// inline "auths" entries are supported there.
func TestCredsStoreWithoutHelperFailsTheLookup(t *testing.T) {
reg := newTestRegistry(t, "", "")
reg.pushIndex(t, "team/app", "1.2")
dockerConfig(t, `{"credsStore":"no-such-helper-for-package-updater-tests"}`)
_, err := newClient().Digest(context.Background(), reg.host+"/team/app:1.2")
assert.ErrorContains(t, err, "error getting credentials")
assert.ErrorContains(t, err, "docker-credential-no-such-helper-for-package-updater-tests")
}
// A registry that leaves out Docker-Content-Digest must not make the client
// retry with a manifest download, which Docker Hub counts as a pull.
func TestMissingDigestHeaderDoesNotFallBackToDownload(t *testing.T) {
dockerConfig(t, "")
var mu sync.Mutex
var methods []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.URL.Path, "/manifests/") {
return
}
mu.Lock()
methods = append(methods, r.Method)
mu.Unlock()
w.Header().Set("Content-Type", "application/vnd.oci.image.index.v1+json")
w.Header().Set("Content-Length", "10")
}))
t.Cleanup(srv.Close)
_, err := newClient().Digest(context.Background(), strings.TrimPrefix(srv.URL, "http://")+"/a/b:1")
assert.Error(t, err)
assert.Equal(t, []string{http.MethodHead}, methods)
}
// ggcr copies up to 64 KiB of the registry's response into its errors. The
// registry host comes from an image name, so that text must not reach the
// stored poll error.
func TestRegistryErrorBodyDoesNotLeak(t *testing.T) {
dockerConfig(t, "")
tests := []struct {
name string
body string
want string
}{
{"raw body", "internal-secret-body", "registry answered 400 Bad Request"},
{"structured errors", `{"errors":[{"code":"MANIFEST_UNKNOWN","message":"internal-secret-body","detail":"internal-secret-body"},{"code":"internal-secret-body"}]}`,
"registry answered 400 Bad Request (MANIFEST_UNKNOWN)"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// HEAD answers carry no body, so the leak is in the GET /v2/ ping.
w.WriteHeader(http.StatusBadRequest)
_, _ = w.Write([]byte(tt.body))
}))
t.Cleanup(srv.Close)
host := strings.TrimPrefix(srv.URL, "http://")
_, err := newClient().Digest(context.Background(), host+"/a/b:1")
require.Error(t, err)
assert.NotContains(t, err.Error(), "internal-secret-body")
assert.Equal(t, "lookup "+host+"/a/b:1: "+tt.want, strings.Replace(err.Error(), "index.docker.io/", "", 1))
})
}
}
// ggcr puts the token server's answer into its error when the answer has no
// token, and the token server is named by the registry's challenge.
func TestNonRegistryErrorsAreFixedText(t *testing.T) {
dockerConfig(t, "")
tokenSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"detail":"internal-secret-body"}`))
}))
t.Cleanup(tokenSrv.Close)
reg := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("WWW-Authenticate", `Bearer realm="`+tokenSrv.URL+`/internal-secret-realm",service="internal-secret-service"`)
w.WriteHeader(http.StatusUnauthorized)
}))
t.Cleanup(reg.Close)
host := strings.TrimPrefix(reg.URL, "http://")
_, err := newClient().Digest(context.Background(), host+"/a/b:1")
require.Error(t, err)
assert.NotContains(t, err.Error(), "internal-secret")
assert.Equal(t, "lookup "+host+"/a/b:1: lookup failed", err.Error())
}
func TestClassify(t *testing.T) {
refused := &url.Error{Op: "Get", URL: "http://internal-secret-host/v2/", Err: errors.New("connection refused")}
tests := []struct {
name string
err error
want string
}{
{"deadline", fmt.Errorf("x: %w", context.DeadlineExceeded), "timeout"},
{"canceled", context.Canceled, "timeout"},
{"url error", refused, "cannot reach registry"},
{"net error", &net.DNSError{Name: "internal-secret-host"}, "cannot reach registry"},
{"deadline inside url error", &url.Error{Op: "Get", URL: "http://x", Err: context.DeadlineExceeded}, "timeout"},
{"anything else", errors.New("internal-secret"), "lookup failed"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, classify(tt.err))
})
}
}
+32 -8
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"strings"
"time" "time"
"github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5"
@@ -187,7 +188,12 @@ func (s *Store) JobProgress(ctx context.Context, agentID, jobID int64, chunk str
// agentID: status, error, output (replacing what progress assembled), // agentID: status, error, output (replacing what progress assembled),
// started_at (the agent's, or the existing one if the agent sent none), // started_at (the agent's, or the existing one if the agent sent none),
// finished_at; to_ref = f.Ref when set. On success it also sets the // finished_at; to_ref = f.Ref when set. On success it also sets the
// service's desired_ref = f.Ref, in the same transaction. Returns false if no // service's desired_ref = f.Ref and clears its available ref when that is
// the ref the job was created for and was reported no later than the job was
// created (that version is now running), in the same transaction. A job
// created for "repo:tag@digest" clears only a suggestion with that same
// digest, so rolling back to an older digest keeps a newer suggestion (and a
// suggestion without a digest, such as a webhook's). Returns false if no
// such active job belongs to agentID. // such active job belongs to agentID.
func (s *Store) FinishJob(ctx context.Context, agentID, jobID int64, f model.JobFinish) (bool, error) { func (s *Store) FinishJob(ctx context.Context, agentID, jobID int64, f model.JobFinish) (bool, error) {
tx, err := s.pool.Begin(ctx) tx, err := s.pool.Begin(ctx)
@@ -203,14 +209,18 @@ func (s *Store) FinishJob(ctx context.Context, agentID, jobID int64, f model.Job
startedAt = &f.StartedAt startedAt = &f.StartedAt
} }
var serviceID int64 var serviceID int64
var createdAt time.Time
var requestedRef string
// requested is the job's to_ref as it was created, before f.Ref overwrites it.
err = tx.QueryRow(ctx, ` err = tx.QueryRow(ctx, `
UPDATE update_jobs SET UPDATE update_jobs j SET
status = $3, error = $4, output = $5, output_truncated = $6, status = $3, error = $4, output = $5, output_truncated = $6,
to_ref = CASE WHEN $7::text <> '' THEN $7::text ELSE to_ref END, to_ref = CASE WHEN $7::text <> '' THEN $7::text ELSE j.to_ref END,
started_at = COALESCE($8, started_at), finished_at = $9 started_at = COALESCE($8, j.started_at), finished_at = $9
WHERE id = $1 AND agent_id = $2 AND status IN ('queued', 'running') FROM (SELECT id, to_ref FROM update_jobs WHERE id = $1) requested
RETURNING service_id`, WHERE j.id = requested.id AND j.agent_id = $2 AND j.status IN ('queued', 'running')
jobID, agentID, string(f.Status), f.Error, f.Output, f.OutputTruncated, f.Ref, startedAt, f.FinishedAt).Scan(&serviceID) RETURNING j.service_id, j.created_at, requested.to_ref`,
jobID, agentID, string(f.Status), f.Error, f.Output, f.OutputTruncated, f.Ref, startedAt, f.FinishedAt).Scan(&serviceID, &createdAt, &requestedRef)
if errors.Is(err, pgx.ErrNoRows) { if errors.Is(err, pgx.ErrNoRows) {
return false, nil return false, nil
} }
@@ -218,7 +228,21 @@ func (s *Store) FinishJob(ctx context.Context, agentID, jobID int64, f model.Job
return false, fmt.Errorf("finish job: %w", err) return false, fmt.Errorf("finish job: %w", err)
} }
if f.Status == model.JobSucceeded { if f.Status == model.JobSucceeded {
if _, err := tx.Exec(ctx, `UPDATE services SET desired_ref = $2 WHERE id = $1`, serviceID, f.Ref); err != nil { requestedRef, requestedDigest, _ := strings.Cut(requestedRef, "@")
// A plain tag job, whoever started it, deployed the tag at the digest
// the agent pinned; remember it. Pinned and tagless picks do not.
tagDigest := ""
if _, d, _ := strings.Cut(f.Ref, "@"); requestedDigest == "" && digestPattern.MatchString(d) {
tagDigest = d
}
if _, err := tx.Exec(ctx, `
UPDATE services SET desired_ref = $2,
last_tag_digest = CASE WHEN $6 <> '' THEN $6 ELSE last_tag_digest END,
available_ref = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_ref END,
available_digest = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_digest END,
available_source = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_source END,
available_at = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN NULL ELSE available_at END
WHERE id = $1`, serviceID, f.Ref, createdAt, requestedRef, requestedDigest, tagDigest); err != nil {
return false, fmt.Errorf("finish job: desired ref: %w", err) return false, fmt.Errorf("finish job: desired ref: %w", err)
} }
} }
@@ -0,0 +1,6 @@
ALTER TABLE services ADD COLUMN available_ref text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN available_source text NOT NULL DEFAULT ''
CHECK (available_source IN ('', 'webhook'));
ALTER TABLE services ADD COLUMN available_at timestamptz;
CREATE INDEX services_by_image_repo ON services (image_repo) WHERE removed_at IS NULL;
@@ -0,0 +1,20 @@
ALTER TABLE services DROP CONSTRAINT services_available_source_check;
ALTER TABLE services ADD CONSTRAINT services_available_source_check
CHECK (available_source IN ('', 'webhook', 'poll'));
ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_tag_digest text NOT NULL DEFAULT '';
-- Every version the master has seen. '' means "no tag" / "digest unknown";
-- never NULL, so the unique constraint deduplicates.
CREATE TABLE image_versions (
id bigserial PRIMARY KEY,
repo text NOT NULL,
tag text NOT NULL DEFAULT '',
digest text NOT NULL DEFAULT '',
source text NOT NULL CHECK (source IN ('webhook', 'poll', 'running')),
first_seen_at timestamptz NOT NULL,
CONSTRAINT image_versions_tag_or_digest CHECK (tag <> '' OR digest <> ''),
CONSTRAINT image_versions_unique UNIQUE (repo, tag, digest)
);
CREATE INDEX image_versions_recent ON image_versions (repo, first_seen_at DESC, id DESC);
+152 -6
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"regexp"
"time" "time"
"github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5"
@@ -56,7 +57,11 @@ ON CONFLICT (location_key) DO UPDATE SET
updated_at = EXCLUDED.updated_at, updated_at = EXCLUDED.updated_at,
removed_at = NULL` removed_at = NULL`
// digestPattern is the form of a digest worth recording as a version.
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
func upsertService(ctx context.Context, db execer, agentID int64, clusterID string, svc protocol.Service, at time.Time) error { func upsertService(ctx context.Context, db execer, agentID int64, clusterID string, svc protocol.Service, at time.Time) error {
image := imageref.Parse(svc.Image)
clusterCol := "" clusterCol := ""
if svc.Kind == protocol.KindSwarm { if svc.Kind == protocol.KindSwarm {
clusterCol = clusterID clusterCol = clusterID
@@ -65,11 +70,34 @@ func upsertService(ctx context.Context, db execer, agentID int64, clusterID stri
string(svc.Kind), model.LocationKey(agentID, clusterID, svc.Key()), agentID, string(svc.Kind), model.LocationKey(agentID, clusterID, svc.Key()), agentID,
svc.ComposeProject, svc.ComposeService, svc.WorkingDir, svc.ConfigFiles, svc.Built, svc.ComposeProject, svc.ComposeService, svc.WorkingDir, svc.ConfigFiles, svc.Built,
clusterCol, svc.SwarmService, svc.ContainerName, clusterCol, svc.SwarmService, svc.ContainerName,
imageref.Parse(svc.Image).Repo, svc.Image, svc.Digest, image.Repo, svc.Image, svc.Digest,
svc.Status, svc.Self, at) svc.Status, svc.Self, at)
if err != nil { if err != nil {
return fmt.Errorf("upsert service %s: %w", svc.Key(), err) return fmt.Errorf("upsert service %s: %w", svc.Key(), err)
} }
if digestPattern.MatchString(svc.Digest) {
// A bare image runs "latest", as jobs.FollowedRef sees it, so it
// shares its row with the polled one. repo@digest has no tag.
if image.Tag == "" && image.Digest == "" {
image.Tag = "latest"
}
if err := insertVersion(ctx, db, image.Repo, image.Tag, svc.Digest, model.SourceRunning, at); err != nil {
return err
}
}
return nil
}
// insertVersion records a version in image_versions unless the same
// (repo, tag, digest) is already there; the first sighting keeps its source
// and first_seen_at.
func insertVersion(ctx context.Context, db execer, repo, tag, digest, source string, at time.Time) error {
if _, err := db.Exec(ctx, `
INSERT INTO image_versions (repo, tag, digest, source, first_seen_at)
VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING`,
repo, tag, digest, source, at); err != nil {
return fmt.Errorf("record version %s: %w", imageref.Ref{Repo: repo, Tag: tag, Digest: digest}, err)
}
return nil return nil
} }
@@ -149,7 +177,8 @@ func (s *Store) RemoveService(ctx context.Context, agentID int64, clusterID, age
const serviceColumns = `s.id, s.kind, s.location_key, s.agent_id, COALESCE(a.name, ''), const serviceColumns = `s.id, s.kind, s.location_key, s.agent_id, COALESCE(a.name, ''),
s.compose_project, s.compose_service, s.working_dir, s.config_files, s.built, s.swarm_cluster_id, s.compose_project, s.compose_service, s.working_dir, s.config_files, s.built, s.swarm_cluster_id,
s.swarm_service, s.container_name, s.image_repo, s.desired_ref, s.actual_ref, s.actual_digest, s.swarm_service, s.container_name, s.image_repo, s.desired_ref, s.actual_ref, s.actual_digest,
s.status, s.self, s.update_mode, s.first_seen_at, s.updated_at, s.removed_at` s.status, s.self, s.update_mode, s.available_ref, s.available_digest, s.available_source, s.available_at,
s.last_poll_error, s.last_tag_digest, s.first_seen_at, s.updated_at, s.removed_at`
func scanService(row pgx.Row) (model.Service, error) { func scanService(row pgx.Row) (model.Service, error) {
var m model.Service var m model.Service
@@ -157,7 +186,8 @@ func scanService(row pgx.Row) (model.Service, error) {
err := row.Scan(&m.ID, &kind, &m.LocationKey, &m.AgentID, &m.AgentName, err := row.Scan(&m.ID, &kind, &m.LocationKey, &m.AgentID, &m.AgentName,
&m.ComposeProject, &m.ComposeService, &m.WorkingDir, &m.ConfigFiles, &m.Built, &m.SwarmClusterID, &m.ComposeProject, &m.ComposeService, &m.WorkingDir, &m.ConfigFiles, &m.Built, &m.SwarmClusterID,
&m.SwarmService, &m.ContainerName, &m.ImageRepo, &m.DesiredRef, &m.ActualRef, &m.ActualDigest, &m.SwarmService, &m.ContainerName, &m.ImageRepo, &m.DesiredRef, &m.ActualRef, &m.ActualDigest,
&m.Status, &m.Self, &m.UpdateMode, &m.FirstSeenAt, &m.UpdatedAt, &m.RemovedAt) &m.Status, &m.Self, &m.UpdateMode, &m.AvailableRef, &m.AvailableDigest, &m.AvailableSource, &m.AvailableAt,
&m.LastPollError, &m.LastTagDigest, &m.FirstSeenAt, &m.UpdatedAt, &m.RemovedAt)
m.Kind = protocol.Kind(kind) m.Kind = protocol.Kind(kind)
return m, err return m, err
} }
@@ -165,13 +195,23 @@ func scanService(row pgx.Row) (model.Service, error) {
// ListServices returns all services not marked removed: machine-bound // ListServices returns all services not marked removed: machine-bound
// services grouped by agent name first, Swarm services last. // services grouped by agent name first, Swarm services last.
func (s *Store) ListServices(ctx context.Context) ([]model.Service, error) { func (s *Store) ListServices(ctx context.Context) ([]model.Service, error) {
return s.queryServices(ctx, "list services", "", nil)
}
// ServicesByRepo returns the services not marked removed whose image repo
// equals repo (case-sensitive), in the same order as ListServices.
func (s *Store) ServicesByRepo(ctx context.Context, repo string) ([]model.Service, error) {
return s.queryServices(ctx, "services by repo", "AND s.image_repo = $1", []any{repo})
}
func (s *Store) queryServices(ctx context.Context, what, extraWhere string, args []any) ([]model.Service, error) {
rows, err := s.pool.Query(ctx, ` rows, err := s.pool.Query(ctx, `
SELECT `+serviceColumns+` SELECT `+serviceColumns+`
FROM services s LEFT JOIN agents a ON a.id = s.agent_id FROM services s LEFT JOIN agents a ON a.id = s.agent_id
WHERE s.removed_at IS NULL WHERE s.removed_at IS NULL `+extraWhere+`
ORDER BY s.kind = 'swarm', COALESCE(a.name, ''), s.location_key`) ORDER BY s.kind = 'swarm', COALESCE(a.name, ''), s.location_key`, args...)
if err != nil { if err != nil {
return nil, fmt.Errorf("list services: %w", err) return nil, fmt.Errorf("%s: %w", what, err)
} }
defer rows.Close() defer rows.Close()
var out []model.Service var out []model.Service
@@ -185,6 +225,112 @@ func (s *Store) ListServices(ctx context.Context) ([]model.Service, error) {
return out, rows.Err() return out, rows.Err()
} }
// SetAvailable records a newer ref for the service, replacing any earlier
// suggestion, and records the version in image_versions in the same
// transaction (the tag is parsed from a.Ref).
func (s *Store) SetAvailable(ctx context.Context, id int64, a model.Available) error {
tx, err := s.pool.Begin(ctx)
if err != nil {
return fmt.Errorf("set available: begin: %w", err)
}
defer func() { _ = tx.Rollback(ctx) }()
if _, err := tx.Exec(ctx, `
UPDATE services SET available_ref = $2, available_digest = $3, available_source = $4, available_at = $5
WHERE id = $1`, id, a.Ref, a.Digest, a.Source, a.At); err != nil {
return fmt.Errorf("set available: %w", err)
}
ref := imageref.Parse(a.Ref)
if err := insertVersion(ctx, tx, ref.Repo, ref.Tag, a.Digest, a.Source, a.At); err != nil {
return err
}
if err := tx.Commit(ctx); err != nil {
return fmt.Errorf("set available: commit: %w", err)
}
return nil
}
// SetPollError sets last_poll_error on the given services; msg "" clears it.
func (s *Store) SetPollError(ctx context.Context, ids []int64, msg string) error {
if len(ids) == 0 {
return nil
}
if _, err := s.pool.Exec(ctx,
`UPDATE services SET last_poll_error = $2 WHERE id = ANY($1)`, ids, msg); err != nil {
return fmt.Errorf("set poll error: %w", err)
}
return nil
}
// SetLastTagDigest sets last_tag_digest; see model.Service.LastTagDigest. The
// poller calls it, and FinishJob sets the same column after a tag job.
// Inventory writes never change it.
func (s *Store) SetLastTagDigest(ctx context.Context, id int64, digest string) error {
if _, err := s.pool.Exec(ctx,
`UPDATE services SET last_tag_digest = $2 WHERE id = $1`, id, digest); err != nil {
return fmt.Errorf("set last tag digest: %w", err)
}
return nil
}
// RecentVersions returns up to limit versions of repo, most recently first
// seen first. A version without a digest is left out when a version with the
// same tag and a digest was first seen at or after it.
func (s *Store) RecentVersions(ctx context.Context, repo string, limit int) ([]model.ImageVersion, error) {
rows, err := s.pool.Query(ctx, `
SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions v
WHERE v.repo = $1
AND NOT (v.digest = '' AND EXISTS (
SELECT 1 FROM image_versions d
WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> ''
AND d.first_seen_at >= v.first_seen_at))
ORDER BY v.first_seen_at DESC, v.id DESC
LIMIT $2`, repo, limit)
if err != nil {
return nil, fmt.Errorf("recent versions: %w", err)
}
defer rows.Close()
var out []model.ImageVersion
for rows.Next() {
var v model.ImageVersion
if err := rows.Scan(&v.ID, &v.Repo, &v.Tag, &v.Digest, &v.Source, &v.FirstSeenAt); err != nil {
return nil, fmt.Errorf("scan version: %w", err)
}
out = append(out, v)
}
return out, rows.Err()
}
// GetImageVersion returns one version. Returns model.ErrNotFound if the ID
// does not exist.
func (s *Store) GetImageVersion(ctx context.Context, id int64) (model.ImageVersion, error) {
var v model.ImageVersion
err := s.pool.QueryRow(ctx, `
SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions WHERE id = $1`, id,
).Scan(&v.ID, &v.Repo, &v.Tag, &v.Digest, &v.Source, &v.FirstSeenAt)
if errors.Is(err, pgx.ErrNoRows) {
return model.ImageVersion{}, model.ErrNotFound
}
if err != nil {
return model.ImageVersion{}, fmt.Errorf("get image version: %w", err)
}
return v, nil
}
// SetUpdateMode sets the service's update mode (model.ModeManual or
// model.ModeAuto). Returns model.ErrNotFound if the service does not exist
// or is marked removed.
func (s *Store) SetUpdateMode(ctx context.Context, id int64, mode string) error {
tag, err := s.pool.Exec(ctx,
`UPDATE services SET update_mode = $2 WHERE id = $1 AND removed_at IS NULL`, id, mode)
if err != nil {
return fmt.Errorf("set update mode: %w", err)
}
if tag.RowsAffected() == 0 {
return model.ErrNotFound
}
return nil
}
// GetService returns one service, including a removed one (RemovedAt set). // GetService returns one service, including a removed one (RemovedAt set).
// Returns model.ErrNotFound if the ID does not exist. // Returns model.ErrNotFound if the ID does not exist.
func (s *Store) GetService(ctx context.Context, id int64) (model.Service, error) { func (s *Store) GetService(ctx context.Context, id int64) (model.Service, error) {
@@ -17,9 +17,9 @@ import (
var ( var (
t0 = time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC) t0 = time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
slusk = protocol.Service{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", WorkingDir: "/srv/slusk", Image: "gitea.shcizo.se/shcizo/slusk:1.4.2", Digest: "sha256:d1", Status: "running"} slusk = protocol.Service{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", WorkingDir: "/srv/slusk", Image: "gitea.shcizo.se/shcizo/slusk:1.4.2", Digest: digD1, Status: "running"}
adhoc = protocol.Service{Kind: protocol.KindContainer, ContainerName: "adhoc", Image: "redis:alpine", Status: "stopped"} adhoc = protocol.Service{Kind: protocol.KindContainer, ContainerName: "adhoc", Image: "redis:alpine", Status: "stopped"}
gitea = protocol.Service{Kind: protocol.KindSwarm, SwarmService: "gitea_gitea", Image: "gitea/gitea:1.27.3@sha256:87a6", Digest: "sha256:87a6", Status: "running"} gitea = protocol.Service{Kind: protocol.KindSwarm, SwarmService: "gitea_gitea", Image: "gitea/gitea:1.27.3@sha256:87a6", Digest: digGitea, Status: "running"}
worker = protocol.Hello{Hostname: "node-2"} worker = protocol.Hello{Hostname: "node-2"}
manager = protocol.Hello{Hostname: "swarm-mgr", SwarmManager: true, SwarmClusterID: "c1"} manager = protocol.Hello{Hostname: "swarm-mgr", SwarmManager: true, SwarmClusterID: "c1"}
) )
@@ -53,7 +53,7 @@ func TestApplySnapshot_InsertsAsManualWithAgentName(t *testing.T) {
assert.Equal(t, "node-2", got[0].AgentName) assert.Equal(t, "node-2", got[0].AgentName)
assert.Equal(t, "gitea.shcizo.se/shcizo/slusk", got[0].ImageRepo) assert.Equal(t, "gitea.shcizo.se/shcizo/slusk", got[0].ImageRepo)
assert.Equal(t, "gitea.shcizo.se/shcizo/slusk:1.4.2", got[0].ActualRef) assert.Equal(t, "gitea.shcizo.se/shcizo/slusk:1.4.2", got[0].ActualRef)
assert.Equal(t, "sha256:d1", got[0].ActualDigest) assert.Equal(t, digD1, got[0].ActualDigest)
assert.Equal(t, "compose|"+itoa(id)+"|slusk|web", got[0].LocationKey) assert.Equal(t, "compose|"+itoa(id)+"|slusk|web", got[0].LocationKey)
} }
@@ -0,0 +1,216 @@
//go:build integration
package store
import (
"context"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/protocol"
)
const sluskRepo = "gitea.shcizo.se/shcizo/slusk"
// Digests of the fixture services; only well-formed ones are recorded as versions.
var (
digD1 = "sha256:" + strings.Repeat("d", 63) + "1"
digGitea = "sha256:" + strings.Repeat("a", 63) + "1"
digPg = "sha256:" + strings.Repeat("b", 63) + "1"
)
func versionsOf(t *testing.T, s *Store, repo string) []model.ImageVersion {
t.Helper()
got, err := s.RecentVersions(context.Background(), repo, 100)
require.NoError(t, err)
return got
}
func TestSetAvailable_RecordsVersion(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
_, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":1.5.0", Digest: "sha256:n1", Source: model.SourcePoll, At: t0}))
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, sluskRepo+":1.5.0", got.AvailableRef)
assert.Equal(t, "sha256:n1", got.AvailableDigest)
assert.Equal(t, model.SourcePoll, got.AvailableSource)
vs := versionsOf(t, s, sluskRepo)
var found model.ImageVersion
for _, v := range vs {
if v.Digest == "sha256:n1" {
found = v
}
}
require.NotZero(t, found.ID)
assert.Equal(t, "1.5.0", found.Tag)
assert.Equal(t, model.SourcePoll, found.Source)
// Seeing the same version again keeps the first sighting.
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":1.5.0", Digest: "sha256:n1", Source: model.SourceWebhook, At: t0.Add(time.Hour)}))
again, err := s.GetImageVersion(ctx, found.ID)
require.NoError(t, err)
assert.Equal(t, model.SourcePoll, again.Source)
assert.True(t, t0.Equal(again.FirstSeenAt))
assert.Len(t, versionsOf(t, s, sluskRepo), len(vs))
}
func TestSnapshotRecordsRunningVersions(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
id := newAgent(t, s, "node-2")
tagless := protocol.Service{Kind: protocol.KindSwarm, SwarmService: "g_g", Image: "gitea/gitea@" + digGitea, Digest: digGitea, Status: "running"}
bare := protocol.Service{Kind: protocol.KindContainer, ContainerName: "pg", Image: "postgres", Digest: digPg, Status: "running"}
nodigest := protocol.Service{Kind: protocol.KindContainer, ContainerName: "nd", Image: "nodigest:1", Status: "running"}
hello := protocol.Hello{Hostname: "node-2", SwarmManager: true, SwarmClusterID: "c1"}
svcs := []protocol.Service{slusk, tagless, bare, nodigest}
require.NoError(t, s.ApplySnapshot(ctx, id, hello, svcs, t0))
require.NoError(t, s.ApplySnapshot(ctx, id, hello, svcs, t0.Add(time.Hour)))
got := versionsOf(t, s, sluskRepo)
require.Len(t, got, 1)
assert.Equal(t, "1.4.2", got[0].Tag)
assert.Equal(t, digD1, got[0].Digest)
assert.Equal(t, model.SourceRunning, got[0].Source)
assert.True(t, t0.Equal(got[0].FirstSeenAt), "re-snapshot does not bump first_seen_at")
tl := versionsOf(t, s, "gitea/gitea")
require.Len(t, tl, 1)
assert.Equal(t, "", tl[0].Tag)
assert.Equal(t, digGitea, tl[0].Digest)
b := versionsOf(t, s, "postgres")
require.Len(t, b, 1)
assert.Equal(t, "latest", b[0].Tag, "a bare image runs latest, like the followed ref")
assert.Empty(t, versionsOf(t, s, "nodigest"))
}
func TestSnapshotRecordsOnlyWellFormedDigests(t *testing.T) {
s := openTestStore(t)
id := newAgent(t, s, "node-2")
bad := protocol.Service{Kind: protocol.KindContainer, ContainerName: "bad", Image: "bad:1", Digest: "sha256:short", Status: "running"}
upper := protocol.Service{Kind: protocol.KindContainer, ContainerName: "up", Image: "up:1", Digest: "sha256:" + strings.Repeat("A", 64), Status: "running"}
require.NoError(t, s.ApplySnapshot(context.Background(), id, protocol.Hello{Hostname: "node-2"}, []protocol.Service{bad, upper}, t0))
assert.Empty(t, versionsOf(t, s, "bad"))
assert.Empty(t, versionsOf(t, s, "up"))
}
func TestRecentVersions(t *testing.T) {
ctx := context.Background()
t.Run("order, limit and repo filter", func(t *testing.T) {
s := openTestStore(t)
_, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: "other/repo:x", Digest: "sha256:ox", Source: model.SourcePoll, At: t0}))
// 12 versions of the service's repo, one per minute.
for i := 0; i < 12; i++ {
tag := "t" + string(rune('a'+i))
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":" + tag, Digest: "sha256:" + tag, Source: model.SourcePoll, At: t0.Add(time.Duration(i+1) * time.Minute)}))
}
got, err := s.RecentVersions(ctx, sluskRepo, 10)
require.NoError(t, err)
require.Len(t, got, 10)
assert.Equal(t, "tl", got[0].Tag, "newest first")
assert.Equal(t, "tc", got[9].Tag)
for _, v := range got {
assert.Equal(t, sluskRepo, v.Repo)
}
})
t.Run("digest-less row is hidden once its tag has a digest", func(t *testing.T) {
s := openTestStore(t)
_, svc := jobFixture(t, s, "a")
add := func(tag, digest string, at time.Time) {
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":" + tag, Digest: digest, Source: model.SourceWebhook, At: at}))
}
add("main", "", t0.Add(time.Minute))
add("solo", "", t0.Add(2*time.Minute))
require.Len(t, versionsOf(t, s, sluskRepo), 3, "running 1.4.2 plus two digest-less rows")
add("main", "sha256:m1", t0.Add(3*time.Minute))
var refs []string
for _, v := range versionsOf(t, s, sluskRepo) {
refs = append(refs, v.Ref())
}
assert.Equal(t, []string{sluskRepo + ":main@sha256:m1", sluskRepo + ":solo", sluskRepo + ":1.4.2@" + digD1}, refs)
})
t.Run("digest-less row newer than the digest row of its tag is shown", func(t *testing.T) {
s := openTestStore(t)
_, svc := jobFixture(t, s, "a")
add := func(tag, digest string, at time.Time) {
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":" + tag, Digest: digest, Source: model.SourceWebhook, At: at}))
}
add("main", "sha256:m1", t0.Add(time.Minute))
add("main", "", t0.Add(2*time.Minute))
var refs []string
for _, v := range versionsOf(t, s, sluskRepo) {
refs = append(refs, v.Ref())
}
assert.Equal(t, []string{sluskRepo + ":main", sluskRepo + ":main@sha256:m1", sluskRepo + ":1.4.2@" + digD1}, refs)
})
}
func TestGetImageVersion_NotFound(t *testing.T) {
s := openTestStore(t)
_, err := s.GetImageVersion(context.Background(), 99999)
assert.ErrorIs(t, err, model.ErrNotFound)
}
func TestSetPollError(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
agent, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":1.5.0", Digest: "sha256:n1", Source: model.SourcePoll, At: t0}))
require.NoError(t, s.SetPollError(ctx, []int64{svc}, "boom"))
require.NoError(t, s.SetPollError(ctx, nil, "ignored"))
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, "boom", got.LastPollError)
require.NoError(t, s.UpsertService(ctx, agent, "", slusk, t0.Add(time.Hour)))
got, err = s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, "boom", got.LastPollError, "inventory writes do not touch the poll error")
assert.Equal(t, "sha256:n1", got.AvailableDigest, "nor the suggestion")
require.NoError(t, s.SetPollError(ctx, []int64{svc}, ""))
got, err = s.GetService(ctx, svc)
require.NoError(t, err)
assert.Empty(t, got.LastPollError)
}
func TestSetLastTagDigest(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
agent, svc := jobFixture(t, s, "a")
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Empty(t, got.LastTagDigest)
require.NoError(t, s.SetLastTagDigest(ctx, svc, "sha256:n1"))
require.NoError(t, s.UpsertService(ctx, agent, "", slusk, t0.Add(time.Hour)))
got, err = s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, "sha256:n1", got.LastTagDigest, "inventory writes do not touch it")
list, err := s.ListServices(ctx)
require.NoError(t, err)
require.Len(t, list, 1)
assert.Equal(t, "sha256:n1", list[0].LastTagDigest)
}
@@ -0,0 +1,208 @@
//go:build integration
package store
import (
"context"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/protocol"
)
func TestServicesByRepo(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
id := newAgent(t, s, "node-2")
other := protocol.Service{Kind: protocol.KindCompose, ComposeProject: "o", ComposeService: "o", Image: "gitea.shcizo.se/shcizo/Slusk:1", Status: "running"}
gone := protocol.Service{Kind: protocol.KindCompose, ComposeProject: "g", ComposeService: "g", Image: "gitea.shcizo.se/shcizo/slusk:0.9", Status: "running"}
require.NoError(t, s.ApplySnapshot(ctx, id, worker, []protocol.Service{slusk, other, gone, adhoc}, t0))
require.NoError(t, s.RemoveService(ctx, id, "", gone.Key(), t0))
got, err := s.ServicesByRepo(ctx, "gitea.shcizo.se/shcizo/slusk")
require.NoError(t, err)
assert.Equal(t, []string{"slusk/web"}, names(got), "removed rows and rows of other (case-sensitive) repos are excluded")
assert.Equal(t, "node-2", got[0].AgentName)
none, err := s.ServicesByRepo(ctx, "nope")
require.NoError(t, err)
assert.Empty(t, none)
}
func TestSetAvailable_RoundTrip(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
_, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: "r:2", Source: model.SourceWebhook, At: t0}))
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, "r:2", got.AvailableRef)
assert.Equal(t, model.SourceWebhook, got.AvailableSource)
require.NotNil(t, got.AvailableAt)
assert.True(t, t0.Equal(*got.AvailableAt))
}
func TestSetUpdateMode(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
agent, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetUpdateMode(ctx, svc, model.ModeAuto))
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, model.ModeAuto, got.UpdateMode)
require.NoError(t, s.SetUpdateMode(ctx, svc, model.ModeManual))
got, _ = s.GetService(ctx, svc)
assert.Equal(t, model.ModeManual, got.UpdateMode)
assert.ErrorIs(t, s.SetUpdateMode(ctx, 99999, model.ModeAuto), model.ErrNotFound)
require.NoError(t, s.RemoveService(ctx, agent, "", slusk.Key(), t0))
assert.ErrorIs(t, s.SetUpdateMode(ctx, svc, model.ModeAuto), model.ErrNotFound)
}
func TestUpsertKeepsUpdateMode(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
agent, svc := jobFixture(t, s, "a")
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, model.ModeManual, got.UpdateMode, "a new service starts manual")
require.NoError(t, s.SetUpdateMode(ctx, svc, model.ModeAuto))
changed := slusk
changed.Image = "gitea.shcizo.se/shcizo/slusk:1.5.0"
require.NoError(t, s.UpsertService(ctx, agent, "", changed, t0.Add(time.Minute)))
require.NoError(t, s.ApplySnapshot(ctx, agent, worker, []protocol.Service{changed}, t0.Add(2*time.Minute)))
got, err = s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, model.ModeAuto, got.UpdateMode)
}
func TestFinishJob_ClearsAvailableOnlyWhenThatVersionWasDeployed(t *testing.T) {
succeed := model.JobFinish{Status: model.JobSucceeded, Ref: "r:2@sha256:d", StartedAt: t0, FinishedAt: t0}
tests := []struct {
name string
availableRef string
availableDigest string
availableAt time.Time // the job is created at t0
toRef string // the job's to_ref as created
finish model.JobFinish
wantCleared bool
}{
{"same ref, suggestion before the job", "r:2", "", t0.Add(-time.Minute), "r:2", succeed, true},
{"same ref, suggestion at the same instant", "r:2", "", t0, "r:2", succeed, true},
{"same ref, suggestion after the job was created", "r:2", "", t0.Add(time.Minute), "r:2", succeed, false},
{"different ref keeps the suggestion", "r:3", "", t0.Add(-time.Minute), "r:2", succeed, false},
{"failed job keeps the suggestion", "r:2", "", t0.Add(-time.Minute), "r:2",
model.JobFinish{Status: model.JobFailed, Error: "boom", StartedAt: t0, FinishedAt: t0}, false},
{"tag job clears a poll suggestion with a digest", "r:2", "sha256:new", t0.Add(-time.Minute), "r:2", succeed, true},
{"pick of the suggested digest clears it", "r:2", "sha256:new", t0.Add(-time.Minute), "r:2@sha256:new", succeed, true},
{"rollback to an older digest keeps the newer suggestion", "r:2", "sha256:new", t0.Add(-time.Minute), "r:2@sha256:old", succeed, false},
{"digest pick keeps a suggestion without digest", "r:2", "", t0.Add(-time.Minute), "r:2@sha256:old", succeed, false},
{"tagless pick keeps the suggestion", "r:2", "sha256:new", t0.Add(-time.Minute), "r@sha256:old", succeed, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
agent, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: tt.availableRef, Digest: tt.availableDigest, Source: model.SourcePoll, At: tt.availableAt}))
nj := newJob(svc, agent)
nj.ToRef = tt.toRef
j, err := s.CreateJob(ctx, nj)
require.NoError(t, err)
ok, err := s.FinishJob(ctx, agent, j.ID, tt.finish)
require.NoError(t, err)
require.True(t, ok)
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
if tt.wantCleared {
assert.Empty(t, got.AvailableRef)
assert.Empty(t, got.AvailableDigest)
assert.Empty(t, got.AvailableSource)
assert.Nil(t, got.AvailableAt)
} else {
assert.Equal(t, tt.availableRef, got.AvailableRef)
assert.Equal(t, tt.availableDigest, got.AvailableDigest)
assert.Equal(t, model.SourcePoll, got.AvailableSource)
assert.NotNil(t, got.AvailableAt)
}
})
}
}
func TestFinishJob_WithoutAvailableLeavesItEmpty(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
agent, svc := jobFixture(t, s, "a")
j, err := s.CreateJob(ctx, newJob(svc, agent))
require.NoError(t, err)
ok, err := s.FinishJob(ctx, agent, j.ID, model.JobFinish{Status: model.JobSucceeded, Ref: "r:1@sha256:d", StartedAt: t0, FinishedAt: t0})
require.NoError(t, err)
require.True(t, ok)
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Empty(t, got.AvailableRef)
assert.Nil(t, got.AvailableAt)
assert.Equal(t, "r:1@sha256:d", got.DesiredRef)
}
func TestFinishJob_RemembersTheDeployedDigestOfATagJob(t *testing.T) {
good := "sha256:" + strings.Repeat("a", 64)
old := "sha256:" + strings.Repeat("b", 64)
ok := func(ref string) model.JobFinish {
return model.JobFinish{Status: model.JobSucceeded, Ref: ref, StartedAt: t0, FinishedAt: t0}
}
tests := []struct {
name string
trigger model.JobTrigger
toRef string
finish model.JobFinish
want string
}{
{"webhook tag job", model.TriggerWebhook, "r:2", ok("r:2@" + good), good},
{"manual tag job", model.TriggerManual, "r:2", ok("r:2@" + good), good},
{"poll tag job", model.TriggerPoll, "r:2", ok("r:2@" + good), good},
{"pinned pick leaves it", model.TriggerManual, "r:2@" + old, ok("r:2@" + old), "sha256:before"},
{"tagless pick leaves it", model.TriggerManual, "r@" + old, ok("r@" + old), "sha256:before"},
{"failed job leaves it", model.TriggerWebhook, "r:2",
model.JobFinish{Status: model.JobFailed, Error: "boom", Ref: "r:2@" + good, StartedAt: t0, FinishedAt: t0}, "sha256:before"},
{"malformed digest leaves it", model.TriggerWebhook, "r:2", ok("r:2@sha256:d"), "sha256:before"},
{"no pinned ref leaves it", model.TriggerWebhook, "r:2", ok(""), "sha256:before"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
agent, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetLastTagDigest(ctx, svc, "sha256:before"))
nj := newJob(svc, agent)
nj.ToRef, nj.Trigger = tt.toRef, tt.trigger
j, err := s.CreateJob(ctx, nj)
require.NoError(t, err)
done, err := s.FinishJob(ctx, agent, j.ID, tt.finish)
require.NoError(t, err)
require.True(t, done)
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, tt.want, got.LastTagDigest)
})
}
}
+102 -9
View File
@@ -26,12 +26,20 @@ var assets embed.FS
// serviceJobsLimit is how many jobs the service page lists. // serviceJobsLimit is how many jobs the service page lists.
const serviceJobsLimit = 20 const serviceJobsLimit = 20
// versionsLimit is how many image versions the service page lists.
const versionsLimit = 10
var agentNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`) var agentNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`)
// Store is the storage the UI reads and writes. *store.Store satisfies it. // Store is the storage the UI reads and writes. *store.Store satisfies it.
type Store interface { type Store interface {
ListServices(ctx context.Context) ([]model.Service, error) ListServices(ctx context.Context) ([]model.Service, error)
GetService(ctx context.Context, id int64) (model.Service, error) GetService(ctx context.Context, id int64) (model.Service, error)
// RecentVersions returns up to limit versions of repo the master has
// seen, most recently first seen first.
RecentVersions(ctx context.Context, repo string, limit int) ([]model.ImageVersion, error)
// SetUpdateMode returns model.ErrNotFound for an unknown or removed service.
SetUpdateMode(ctx context.Context, id int64, mode string) error
GetJob(ctx context.Context, id int64) (model.Job, error) GetJob(ctx context.Context, id int64) (model.Job, error)
ListJobs(ctx context.Context, f model.JobFilter) ([]model.Job, error) ListJobs(ctx context.Context, f model.JobFilter) ([]model.Job, error)
ListAgents(ctx context.Context) ([]model.Agent, error) ListAgents(ctx context.Context) ([]model.Agent, error)
@@ -46,11 +54,15 @@ type Hub interface {
Disconnect(agentID int64) Disconnect(agentID int64)
} }
// Jobs starts updates and explains why a service cannot be updated. // Jobs starts updates and explains why a service cannot be updated or set
// *jobs.Service satisfies it. // to auto mode. *jobs.Service satisfies it.
type Jobs interface { type Jobs interface {
Updatable(svc model.Service) error Updatable(svc model.Service) error
AutoAllowed(svc model.Service) error
Trigger(ctx context.Context, serviceID int64, requestedBy string) (model.Job, error) Trigger(ctx context.Context, serviceID int64, requestedBy string) (model.Job, error)
// TriggerVersion updates to exactly the image version versionID of the
// service's repo; model.ErrNotFound for an unknown service or version.
TriggerVersion(ctx context.Context, serviceID, versionID int64, requestedBy string) (model.Job, error)
} }
// Broker delivers change signals to open streams and accepts signals from // Broker delivers change signals to open streams and accepts signals from
@@ -60,11 +72,12 @@ type Broker interface {
Publish(t events.Topic) Publish(t events.Topic)
} }
// Config holds values shown in the "add agent" snippet. // Config holds values shown in the "add agent" snippet and on the service page.
type Config struct { type Config struct {
AgentURL string // what agents dial, e.g. wss://10.0.0.5:8090/agent AgentURL string // what agents dial, e.g. wss://10.0.0.5:8090/agent
MasterKey string // the master's fingerprint MasterKey string // the master's fingerprint
AgentImage string AgentImage string
PollInterval time.Duration // shown as "every <interval>" on the service page
} }
// Server renders the UI. // Server renders the UI.
@@ -82,7 +95,7 @@ type Server struct {
// New parses the templates. Each page is its own template set so every page // New parses the templates. Each page is its own template set so every page
// can define its own "content" block. // can define its own "content" block.
func New(store Store, hub Hub, jobs Jobs, broker Broker, cfg Config, log *slog.Logger) (*Server, error) { func New(store Store, hub Hub, jobs Jobs, broker Broker, cfg Config, log *slog.Logger) (*Server, error) {
funcs := template.FuncMap{"shortRef": shortRef, "duration": jobDuration} funcs := template.FuncMap{"shortRef": shortRef, "duration": jobDuration, "followsNoTag": followsNoTag}
pages := map[string]*template.Template{} pages := map[string]*template.Template{}
for _, page := range []string{"overview.html", "agents.html", "agent_created.html", "service.html", "jobs.html", "job.html"} { for _, page := range []string{"overview.html", "agents.html", "agent_created.html", "service.html", "jobs.html", "job.html"} {
t, err := template.New(page).Funcs(funcs).ParseFS(assets, "templates/layout.html", "templates/"+page) t, err := template.New(page).Funcs(funcs).ParseFS(assets, "templates/layout.html", "templates/"+page)
@@ -104,6 +117,7 @@ func (s *Server) Register(mux *http.ServeMux, protect func(http.Handler) http.Ha
route("GET /services/{id}", s.servicePage) route("GET /services/{id}", s.servicePage)
route("GET /fragments/services/{id}/jobs", s.serviceJobsFragment) route("GET /fragments/services/{id}/jobs", s.serviceJobsFragment)
route("POST /services/{id}/update", s.updateService) route("POST /services/{id}/update", s.updateService)
route("POST /services/{id}/mode", s.setMode)
route("GET /jobs", s.jobsPage) route("GET /jobs", s.jobsPage)
route("GET /fragments/jobs", s.jobsFragment) route("GET /fragments/jobs", s.jobsFragment)
route("GET /jobs/{id}", s.jobPage) route("GET /jobs/{id}", s.jobPage)
@@ -127,6 +141,10 @@ type pageData struct {
Service model.Service Service model.Service
ConfigFiles []string ConfigFiles []string
Reason string // why Service cannot be updated; "" when it can Reason string // why Service cannot be updated; "" when it can
AutoBlocked string // why Service cannot be set to auto; "" when it can
Versions []versionRow
Polling string // what the service page says about registry polling
PollFailed bool // Polling is the last registry error
Jobs []model.Job Jobs []model.Job
Job model.Job Job model.Job
} }
@@ -150,7 +168,7 @@ func (s *Server) serviceData(r *http.Request) (pageData, error) {
if err != nil { if err != nil {
return data, err return data, err
} }
data.Groups = groupServices(services) data.Groups = groupServices(services, s.autoBlocked)
return data, nil return data, nil
} }
@@ -172,6 +190,15 @@ func (s *Server) servicesFragment(w http.ResponseWriter, r *http.Request) {
s.render(w, http.StatusOK, "overview.html", "services", data) s.render(w, http.StatusOK, "overview.html", "services", data)
} }
// autoBlocked returns the reason svc cannot be set to auto mode, or "".
func (s *Server) autoBlocked(svc model.Service) string {
var nu *model.NotUpdatableError
if err := s.jobs.AutoAllowed(svc); errors.As(err, &nu) {
return nu.Reason
}
return ""
}
// pathID parses the {id} path value. On failure it has already answered 400. // pathID parses the {id} path value. On failure it has already answered 400.
func pathID(w http.ResponseWriter, r *http.Request) (int64, bool) { func pathID(w http.ResponseWriter, r *http.Request) (int64, bool) {
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64) id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
@@ -207,6 +234,13 @@ func (s *Server) serviceDetail(r *http.Request, id int64) (pageData, error) {
} else if err != nil { } else if err != nil {
return data, err return data, err
} }
data.AutoBlocked = s.autoBlocked(svc)
versions, err := s.store.RecentVersions(r.Context(), svc.ImageRepo, versionsLimit)
if err != nil {
return data, err
}
data.Versions = versionRows(versions, svc.ActualDigest)
data.Polling, data.PollFailed = pollingStatus(svc, s.cfg.PollInterval)
data.Jobs, err = s.store.ListJobs(r.Context(), model.JobFilter{ServiceID: id, Limit: serviceJobsLimit}) data.Jobs, err = s.store.ListJobs(r.Context(), model.JobFilter{ServiceID: id, Limit: serviceJobsLimit})
return data, err return data, err
} }
@@ -245,7 +279,18 @@ func (s *Server) updateService(w http.ResponseWriter, r *http.Request) {
return return
} }
sess, _ := webauth.SessionFrom(r.Context()) sess, _ := webauth.SessionFrom(r.Context())
job, err := s.jobs.Trigger(r.Context(), id, sess.Subject) var job model.Job
var err error
if v := r.PostFormValue("version"); v == "" {
job, err = s.jobs.Trigger(r.Context(), id, sess.Subject)
} else {
versionID, perr := strconv.ParseInt(v, 10, 64)
if perr != nil {
http.Error(w, "invalid version", http.StatusBadRequest)
return
}
job, err = s.jobs.TriggerVersion(r.Context(), id, versionID, sess.Subject)
}
var nu *model.NotUpdatableError var nu *model.NotUpdatableError
switch { switch {
case err == nil || errors.Is(err, model.ErrJobActive): case err == nil || errors.Is(err, model.ErrJobActive):
@@ -263,6 +308,54 @@ func (s *Server) updateService(w http.ResponseWriter, r *http.Request) {
} }
} }
// setMode switches a service between manual and auto. Manual is always
// allowed; auto only when jobs.AutoAllowed says so.
func (s *Server) setMode(w http.ResponseWriter, r *http.Request) {
id, ok := pathID(w, r)
if !ok {
return
}
mode := r.PostFormValue("mode")
if mode != model.ModeManual && mode != model.ModeAuto {
http.Error(w, "invalid mode", http.StatusBadRequest)
return
}
if mode == model.ModeAuto {
svc, err := s.store.GetService(r.Context(), id)
if err == nil && svc.RemovedAt != nil {
err = model.ErrNotFound
}
if err != nil {
s.failLookup(w, err)
return
}
var nu *model.NotUpdatableError
if err := s.jobs.AutoAllowed(svc); errors.As(err, &nu) {
data, derr := s.serviceDetail(r, id)
if derr != nil {
s.failLookup(w, derr)
return
}
data.Error = "Cannot enable auto: " + nu.Reason + "."
s.render(w, http.StatusConflict, "service.html", "layout", data)
return
} else if err != nil {
s.fail(w, err)
return
}
}
if err := s.store.SetUpdateMode(r.Context(), id, mode); err != nil {
s.failLookup(w, err)
return
}
s.broker.Publish(events.Services)
if r.PostFormValue("return") == "overview" {
http.Redirect(w, r, "/", http.StatusSeeOther)
return
}
http.Redirect(w, r, "/services/"+strconv.FormatInt(id, 10), http.StatusSeeOther)
}
func (s *Server) jobsData(r *http.Request) (pageData, error) { func (s *Server) jobsData(r *http.Request) (pageData, error) {
data := s.base(r, "Jobs") data := s.base(r, "Jobs")
var err error var err error
+170
View File
@@ -34,6 +34,17 @@ type fakeStore struct {
created map[string]string // name → token hash created map[string]string // name → token hash
revokedID int64 revokedID int64
revokeFailed []int64 // job IDs RevokeAgent reports as failed revokeFailed []int64 // job IDs RevokeAgent reports as failed
versions []model.ImageVersion
}
func (f *fakeStore) RecentVersions(_ context.Context, repo string, limit int) ([]model.ImageVersion, error) {
var out []model.ImageVersion
for _, v := range f.versions {
if v.Repo == repo && len(out) < limit {
out = append(out, v)
}
}
return out, nil
} }
func (f *fakeStore) ListServices(context.Context) ([]model.Service, error) { return f.services, nil } func (f *fakeStore) ListServices(context.Context) ([]model.Service, error) { return f.services, nil }
@@ -46,6 +57,29 @@ func (f *fakeStore) GetService(_ context.Context, id int64) (model.Service, erro
return model.Service{}, model.ErrNotFound return model.Service{}, model.ErrNotFound
} }
func (f *fakeStore) SetUpdateMode(_ context.Context, id int64, mode string) error {
f.mu.Lock()
defer f.mu.Unlock()
for i := range f.services {
if f.services[i].ID == id && f.services[i].RemovedAt == nil {
f.services[i].UpdateMode = mode
return nil
}
}
return model.ErrNotFound
}
func (f *fakeStore) modeOf(id int64) string {
f.mu.Lock()
defer f.mu.Unlock()
for _, s := range f.services {
if s.ID == id {
return s.UpdateMode
}
}
return ""
}
func (f *fakeStore) GetJob(_ context.Context, id int64) (model.Job, error) { func (f *fakeStore) GetJob(_ context.Context, id int64) (model.Job, error) {
for _, j := range f.jobs { for _, j := range f.jobs {
if j.ID == id { if j.ID == id {
@@ -99,19 +133,35 @@ func (f *fakeStore) revoked() int64 {
// fakeJobs returns scripted results and records who asked for the update. // fakeJobs returns scripted results and records who asked for the update.
type fakeJobs struct { type fakeJobs struct {
updatable error updatable error
autoErr error
job model.Job job model.Job
err error err error
mu sync.Mutex mu sync.Mutex
requestedBy string requestedBy string
triggered []int64 // version IDs passed to TriggerVersion
triggerSvc []int64 // service IDs passed to TriggerVersion
plainCalls int // Trigger calls
} }
func (f *fakeJobs) Updatable(model.Service) error { return f.updatable } func (f *fakeJobs) Updatable(model.Service) error { return f.updatable }
func (f *fakeJobs) AutoAllowed(model.Service) error { return f.autoErr }
func (f *fakeJobs) Trigger(_ context.Context, _ int64, by string) (model.Job, error) { func (f *fakeJobs) Trigger(_ context.Context, _ int64, by string) (model.Job, error) {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
f.requestedBy = by f.requestedBy = by
f.plainCalls++
return f.job, f.err
}
func (f *fakeJobs) TriggerVersion(_ context.Context, serviceID, versionID int64, by string) (model.Job, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.requestedBy = by
f.triggerSvc = append(f.triggerSvc, serviceID)
f.triggered = append(f.triggered, versionID)
return f.job, f.err return f.job, f.err
} }
@@ -151,7 +201,15 @@ func newTestServer(t *testing.T) (*httptest.Server, *fakeStore, *fakeHub, *event
return srv, store, hub, broker return srv, store, hub, broker
} }
// newTestEnv starts the UI on fake storage, with polling off.
func newTestEnv(t *testing.T) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) { func newTestEnv(t *testing.T) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) {
t.Helper()
return newTestEnvPolling(t, 0)
}
// newTestEnvPolling is newTestEnv with pollInterval as the polling interval
// the service page shows.
func newTestEnvPolling(t *testing.T, pollInterval time.Duration) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) {
t.Helper() t.Helper()
agentID := int64(2) agentID := int64(2)
store := &fakeStore{ store := &fakeStore{
@@ -175,6 +233,7 @@ func newTestEnv(t *testing.T) (*httptest.Server, *fakeStore, *fakeHub, *events.B
jobs := &fakeJobs{} jobs := &fakeJobs{}
s, err := New(store, hub, jobs, broker, Config{ s, err := New(store, hub, jobs, broker, Config{
AgentURL: "wss://10.0.0.5:8090/agent", MasterKey: "sha256:abc", AgentImage: "gitea.shcizo.se/shcizo/package-updater:latest", AgentURL: "wss://10.0.0.5:8090/agent", MasterKey: "sha256:abc", AgentImage: "gitea.shcizo.se/shcizo/package-updater:latest",
PollInterval: pollInterval,
}, slog.New(slog.NewTextHandler(io.Discard, nil))) }, slog.New(slog.NewTextHandler(io.Discard, nil)))
require.NoError(t, err) require.NoError(t, err)
mux := http.NewServeMux() mux := http.NewServeMux()
@@ -504,3 +563,114 @@ func TestJob_NotFound(t *testing.T) {
assert.Equal(t, http.StatusNotFound, code) assert.Equal(t, http.StatusNotFound, code)
} }
} }
func postMode(t *testing.T, srv *httptest.Server, id string, form url.Values) (*http.Response, string) {
t.Helper()
form.Set("csrf_token", "csrf")
resp, err := noRedirectClient().PostForm(srv.URL+"/services/"+id+"/mode", form)
require.NoError(t, err)
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
return resp, string(body)
}
func TestSetMode_AutoRedirectsToServiceAndPublishes(t *testing.T) {
srv, store, _, broker, _ := newTestEnv(t)
sub, cancel := broker.Subscribe()
defer cancel()
resp, _ := postMode(t, srv, "1", url.Values{"mode": {"auto"}})
assert.Equal(t, http.StatusSeeOther, resp.StatusCode)
assert.Equal(t, "/services/1", resp.Header.Get("Location"))
assert.Equal(t, "auto", store.modeOf(1))
assert.Equal(t, events.Services, <-sub)
}
func TestSetMode_ReturnToOverview(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
resp, _ := postMode(t, srv, "1", url.Values{"mode": {"auto"}, "return": {"overview"}})
assert.Equal(t, http.StatusSeeOther, resp.StatusCode)
assert.Equal(t, "/", resp.Header.Get("Location"))
assert.Equal(t, "auto", store.modeOf(1))
}
func TestSetMode_AutoRefusedShows409(t *testing.T) {
srv, store, _, _, jobs := newTestEnv(t)
jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonBuilt}
resp, body := postMode(t, srv, "1", url.Values{"mode": {"auto"}})
assert.Equal(t, http.StatusConflict, resp.StatusCode)
assert.Contains(t, body, "Cannot enable auto: "+model.ReasonBuilt+".")
assert.Equal(t, "manual", store.modeOf(1))
}
func TestSetMode_ManualAlwaysAllowed(t *testing.T) {
srv, store, _, _, jobs := newTestEnv(t)
store.services[0].UpdateMode = "auto"
jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonBuilt}
resp, _ := postMode(t, srv, "1", url.Values{"mode": {"manual"}})
assert.Equal(t, http.StatusSeeOther, resp.StatusCode)
assert.Equal(t, "manual", store.modeOf(1))
}
func TestSetMode_InvalidModeAndUnknownService(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services = append(store.services, model.Service{ID: 5, Kind: protocol.KindCompose, UpdateMode: "manual", RemovedAt: ptr(time.Now())})
resp, _ := postMode(t, srv, "1", url.Values{"mode": {"yolo"}})
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
resp, _ = postMode(t, srv, "1", url.Values{})
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
resp, _ = postMode(t, srv, "99", url.Values{"mode": {"auto"}})
assert.Equal(t, http.StatusNotFound, resp.StatusCode)
resp, _ = postMode(t, srv, "99", url.Values{"mode": {"manual"}})
assert.Equal(t, http.StatusNotFound, resp.StatusCode)
resp, _ = postMode(t, srv, "5", url.Values{"mode": {"auto"}})
assert.Equal(t, http.StatusNotFound, resp.StatusCode)
resp, _ = postMode(t, srv, "abc", url.Values{"mode": {"auto"}})
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
assert.Equal(t, "manual", store.modeOf(1))
}
func TestOverview_ShowsModeToggleAndAvailableBadge(t *testing.T) {
srv, store, _, _, jobs := newTestEnv(t)
store.services[0].AvailableRef = "gitea.shcizo.se/shcizo/slusk:1.5.0"
store.services[1].UpdateMode = "auto"
jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonSelf}
for _, path := range []string{"/", "/fragments/services"} {
_, body := get(t, srv, path)
assert.Contains(t, body, `action="/services/1/mode"`, path)
assert.Contains(t, body, `name="csrf_token" value="csrf"`, path)
assert.Contains(t, body, `name="return" value="overview"`, path)
assert.Contains(t, body, `class="badge badge-update" title="gitea.shcizo.se/shcizo/slusk:1.5.0"`, path)
assert.Contains(t, body, `<input type="hidden" name="mode" value="manual">`, "auto services toggle to manual: "+path)
assert.Contains(t, body, `disabled>Make auto</button> <span class="muted">`+model.ReasonSelf+`</span>`, "blocked manual services get a disabled button: "+path)
}
}
func TestServicePage_ShowsAvailableAndModeForm(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
at := time.Date(2026, 10, 12, 12, 30, 0, 0, time.UTC)
store.services[0].AvailableRef = "gitea.shcizo.se/shcizo/slusk:1.5.0"
store.services[0].AvailableSource = "webhook"
store.services[0].AvailableAt = &at
_, body := get(t, srv, "/services/1")
assert.Contains(t, body, "gitea.shcizo.se/shcizo/slusk:1.5.0")
assert.Contains(t, body, "via webhook, 2026-10-12 12:30")
assert.Contains(t, body, `action="/services/1/mode"`)
assert.Contains(t, body, `<input type="hidden" name="mode" value="auto">`)
}
func TestServicePage_AutoBlockedDisablesToggle(t *testing.T) {
srv, _, _, _, jobs := newTestEnv(t)
jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonContainer}
_, body := get(t, srv, "/services/1")
assert.Contains(t, body, "Auto unavailable: "+model.ReasonContainer+".")
}
+4
View File
@@ -28,3 +28,7 @@ form.inline { display: flex; gap: 8px; margin-bottom: 16px; }
table.details { width: auto; } table.details { width: auto; }
table.details th { padding-right: 24px; } table.details th { padding-right: 24px; }
pre.log { white-space: pre-wrap; } pre.log { white-space: pre-wrap; }
.badge-update { color: var(--warn); border-color: var(--warn); }
.badge-error { color: var(--bad); border-color: var(--bad); }
form.mode { display: inline; margin-left: 8px; }
form.mode button { font-size: 12px; padding: 0 6px; }
+11 -2
View File
@@ -14,10 +14,19 @@
<tbody> <tbody>
{{range .Services}} {{range .Services}}
<tr> <tr>
<td><a href="/services/{{.ID}}">{{.DisplayName}}</a>{{if .Self}} <span class="badge">managed outside</span>{{end}}</td> <td><a href="/services/{{.ID}}">{{.DisplayName}}</a>{{if .Self}} <span class="badge">managed outside</span>{{end}}{{if .AvailableRef}} <span class="badge badge-update" title="{{.AvailableRef}}">update available</span>{{end}}{{if followsNoTag .Service}} <span class="badge">follows no tag</span>{{end}}{{if .LastPollError}} <span class="badge badge-error" title="{{.LastPollError}}">poll failed</span>{{end}}</td>
<td><code>{{shortRef .ActualRef}}</code></td> <td><code>{{shortRef .ActualRef}}</code></td>
<td class="status status-{{.Status}}">{{.Status}}</td> <td class="status status-{{.Status}}">{{.Status}}</td>
<td>{{.UpdateMode}}</td> <td>{{.UpdateMode}}
<form method="post" action="/services/{{.ID}}/mode" class="mode">
<input type="hidden" name="csrf_token" value="{{$.CSRF}}">
<input type="hidden" name="return" value="overview">
{{if eq .UpdateMode "auto"}}<input type="hidden" name="mode" value="manual">
<button type="submit">Make manual</button>
{{else}}<input type="hidden" name="mode" value="auto">
<button type="submit"{{if .AutoBlocked}} disabled{{end}}>Make auto</button>{{if .AutoBlocked}} <span class="muted">{{.AutoBlocked}}</span>{{end}}{{end}}
</form>
</td>
</tr> </tr>
{{end}} {{end}}
</tbody> </tbody>
@@ -8,6 +8,9 @@
{{if .Service.ActualDigest}}<tr><th>Digest</th><td><code>{{.Service.ActualDigest}}</code></td></tr>{{end}} {{if .Service.ActualDigest}}<tr><th>Digest</th><td><code>{{.Service.ActualDigest}}</code></td></tr>{{end}}
<tr><th>Status</th><td class="status status-{{.Service.Status}}">{{.Service.Status}}</td></tr> <tr><th>Status</th><td class="status status-{{.Service.Status}}">{{.Service.Status}}</td></tr>
{{if .Service.DesiredRef}}<tr><th>Desired image</th><td><code>{{shortRef .Service.DesiredRef}}</code></td></tr>{{end}} {{if .Service.DesiredRef}}<tr><th>Desired image</th><td><code>{{shortRef .Service.DesiredRef}}</code></td></tr>{{end}}
{{if .Service.AvailableRef}}<tr><th>Available</th><td><code>{{.Service.AvailableRef}}</code> <span class="muted">via {{.Service.AvailableSource}}{{with .Service.AvailableAt}}, {{.Format "2006-01-02 15:04"}}{{end}}</span></td></tr>{{end}}
<tr><th>Mode</th><td>{{.Service.UpdateMode}}</td></tr>
<tr><th>Polling</th><td{{if .PollFailed}} class="error"{{end}}>{{.Polling}}</td></tr>
{{if .ConfigFiles}}<tr><th>Config files</th><td>{{range .ConfigFiles}}<code>{{.}}</code><br>{{end}}</td></tr>{{end}} {{if .ConfigFiles}}<tr><th>Config files</th><td>{{range .ConfigFiles}}<code>{{.}}</code><br>{{end}}</td></tr>{{end}}
</table> </table>
{{if .Reason}} {{if .Reason}}
@@ -19,6 +22,39 @@
</form> </form>
<p class="muted">Pulls the current tag again and redeploys the service.</p> <p class="muted">Pulls the current tag again and redeploys the service.</p>
{{end}} {{end}}
<form method="post" action="/services/{{.Service.ID}}/mode">
<input type="hidden" name="csrf_token" value="{{.CSRF}}">
{{if eq .Service.UpdateMode "auto"}}<input type="hidden" name="mode" value="manual">
<button type="submit">Make manual</button>
<p class="muted">The webhook will only mark new versions as available.</p>
{{else}}<input type="hidden" name="mode" value="auto">
<button type="submit"{{if .AutoBlocked}} disabled{{end}}>Make auto</button>
<p class="muted">{{if .AutoBlocked}}Auto unavailable: {{.AutoBlocked}}.{{else}}The webhook will update this service as soon as CI pushes a new image.{{end}}</p>
{{end}}
</form>
<h2>Versions</h2>
{{if eq .Service.UpdateMode "auto"}}<p class="warning">This service is in auto mode: a version picked here is replaced again when a newer version is announced or found by polling.</p>{{end}}
<table>
<thead><tr><th>Version</th><th>Seen via</th><th>First seen</th><th></th></tr></thead>
<tbody>
{{range .Versions}}
<tr>
<td><code title="{{.Ref}}">{{shortRef .Ref}}</code>{{if .Running}} <span class="badge">running</span>{{end}}{{if not .Tag}} <span class="muted">the service will follow no tag afterwards</span>{{end}}</td>
<td>{{.Source}}</td>
<td>{{.FirstSeenAt.Format "2006-01-02 15:04"}}</td>
{{/* AutoBlocked is the same check that decides whether the service can be updated with an explicit ref, which is what a pick is. */}}
<td>{{if not $.AutoBlocked}}
<form method="post" action="/services/{{$.Service.ID}}/update">
<input type="hidden" name="csrf_token" value="{{$.CSRF}}">
<input type="hidden" name="version" value="{{.ID}}">
<button type="submit">Deploy</button>
</form>{{end}}</td>
</tr>
{{else}}
<tr><td colspan="4">No versions seen yet.</td></tr>
{{end}}
</tbody>
</table>
<h2>Update history</h2> <h2>Update history</h2>
<div id="history" hx-get="/fragments/services/{{.Service.ID}}/jobs" hx-trigger="sse:jobs" hx-swap="innerHTML"> <div id="history" hx-get="/fragments/services/{{.Service.ID}}/jobs" hx-trigger="sse:jobs" hx-swap="innerHTML">
{{template "history" .}} {{template "history" .}}
+221
View File
@@ -0,0 +1,221 @@
package web
import (
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/model"
)
const sluskRepo = "gitea.shcizo.se/shcizo/slusk"
func TestServicePage_VersionPicker(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].ImageRepo = sluskRepo
store.services[0].ActualDigest = "sha256:cur"
first := time.Date(2026, 10, 12, 9, 5, 0, 0, time.UTC)
store.versions = []model.ImageVersion{
{ID: 31, Repo: sluskRepo, Tag: "1.5.0", Digest: "sha256:" + strings.Repeat("a", 40), Source: "poll", FirstSeenAt: first},
{ID: 32, Repo: sluskRepo, Tag: "1.4.2", Digest: "sha256:cur", Source: "running", FirstSeenAt: first},
{ID: 33, Repo: sluskRepo, Digest: "sha256:" + strings.Repeat("b", 40), Source: "running", FirstSeenAt: first},
{ID: 34, Repo: sluskRepo, Tag: "main", Source: "webhook", FirstSeenAt: first},
{ID: 40, Repo: "other/repo", Tag: "x", Source: "poll", FirstSeenAt: first},
}
_, body := get(t, srv, "/services/1")
for _, id := range []string{"31", "32", "33", "34"} {
assert.Contains(t, body, `<input type="hidden" name="version" value="`+id+`">`, id)
}
assert.NotContains(t, body, `name="version" value="40"`, "versions of other repos are not listed")
assert.Equal(t, 4, strings.Count(body, `name="version"`))
assert.Contains(t, body, `name="csrf_token" value="csrf"`)
assert.Contains(t, body, `title="`+sluskRepo+`:1.5.0@sha256:`+strings.Repeat("a", 40)+`"`, "full ref as title")
assert.Contains(t, body, sluskRepo+":1.5.0@sha256:"+strings.Repeat("a", 12), "short ref as text")
assert.Equal(t, 1, strings.Count(body, `<span class="badge">running</span>`), "only the running digest is marked")
assert.Equal(t, 1, strings.Count(body, "the service will follow no tag afterwards"), "only the tagless row")
assert.Contains(t, body, "2026-10-12 09:05")
assert.NotContains(t, body, "auto mode", "no auto warning for a manual service")
}
func TestServicePage_VersionPickerShowsAtMostTenRows(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].ImageRepo = sluskRepo
for i := int64(1); i <= 12; i++ {
store.versions = append(store.versions, model.ImageVersion{ID: i, Repo: sluskRepo, Tag: "t", Digest: "sha256:d", Source: "poll"})
}
_, body := get(t, srv, "/services/1")
assert.Equal(t, 10, strings.Count(body, `name="version"`))
}
func TestServicePage_VersionPickerWarnsForAutoServices(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].UpdateMode = "auto"
_, body := get(t, srv, "/services/1")
assert.Contains(t, body, "This service is in auto mode")
}
func TestServicePage_VersionPickerHiddenWhenServiceCannotBeUpdated(t *testing.T) {
for _, reason := range []string{model.ReasonSelf, model.ReasonContainer, model.ReasonRemoved, model.ReasonBuilt} {
t.Run(reason, func(t *testing.T) {
srv, store, _, _, jobs := newTestEnv(t)
store.services[0].ImageRepo = sluskRepo
store.versions = []model.ImageVersion{{ID: 31, Repo: sluskRepo, Tag: "1.5.0", Source: "poll"}}
jobs.autoErr = &model.NotUpdatableError{Reason: reason}
_, body := get(t, srv, "/services/1")
assert.Contains(t, body, "<h2>Versions</h2>")
assert.Contains(t, body, "1.5.0", "the history is still shown")
assert.NotContains(t, body, `name="version"`)
})
}
}
func TestServicePage_VersionPickerOffersVersionsToAServiceThatFollowsNoTag(t *testing.T) {
srv, store, _, _, jobs := newTestEnv(t)
store.services[0].ImageRepo = sluskRepo
store.versions = []model.ImageVersion{{ID: 31, Repo: sluskRepo, Tag: "1.5.0", Digest: "sha256:a", Source: "poll"}}
jobs.updatable = &model.NotUpdatableError{Reason: model.ReasonNoTag} // autoErr stays nil: an explicit ref is fine
_, body := get(t, srv, "/services/1")
assert.Contains(t, body, `name="version" value="31"`)
assert.NotContains(t, body, `<button type="submit">Update</button>`, "the plain Update button stays hidden")
}
func TestServicePage_PollingRow(t *testing.T) {
tests := []struct {
name string
edit func(*model.Service)
want string
failed bool
}{
{"polled", func(s *model.Service) { s.ActualDigest = "sha256:cur" }, "every 6h", false},
{"poll error", func(s *model.Service) { s.ActualDigest = "sha256:cur"; s.LastPollError = "head x: 401 <b>" }, "last check failed: head x: 401 &lt;b&gt;", true},
{"built", func(s *model.Service) { s.ActualDigest = "sha256:cur"; s.Built = true }, "not polled: built from source", false},
{"no digest", func(s *model.Service) {}, "not polled: no registry digest", false},
{"no tag", func(s *model.Service) { s.ActualRef = "r@sha256:a"; s.ActualDigest = "sha256:a" }, "not polled: follows no tag", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
srv, store, _, _, _ := newTestEnvPolling(t, 6*time.Hour)
tt.edit(&store.services[0])
_, body := get(t, srv, "/services/1")
assert.Contains(t, body, "<th>Polling</th><td")
assert.Contains(t, body, tt.want)
assert.Equal(t, tt.failed, strings.Contains(body, `<td class="error">`+tt.want))
})
}
}
func TestShortDuration(t *testing.T) {
tests := []struct {
d time.Duration
want string
}{
{6 * time.Hour, "6h"},
{90 * time.Minute, "1h30m"},
{30 * time.Minute, "30m"},
{90 * time.Second, "1m30s"},
{45 * time.Second, "45s"},
{6*time.Hour + 5*time.Second, "6h0m5s"},
}
for _, tt := range tests {
t.Run(tt.want, func(t *testing.T) {
assert.Equal(t, tt.want, shortDuration(tt.d))
})
}
}
func postVersion(t *testing.T, srv *httptest.Server, version string) (*http.Response, string) {
t.Helper()
resp, err := noRedirectClient().PostForm(srv.URL+"/services/1/update", url.Values{"csrf_token": {"csrf"}, "version": {version}})
require.NoError(t, err)
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
return resp, string(body)
}
func TestUpdateService_VersionStartsThatVersion(t *testing.T) {
srv, _, _, _, jobs := newTestEnv(t)
jobs.job = model.Job{ID: 42}
resp, _ := postVersion(t, srv, "31")
assert.Equal(t, http.StatusSeeOther, resp.StatusCode)
assert.Equal(t, "/jobs/42", resp.Header.Get("Location"))
assert.Equal(t, []int64{31}, jobs.triggered)
assert.Equal(t, []int64{1}, jobs.triggerSvc)
assert.Equal(t, "alice", jobs.triggeredBy())
assert.Zero(t, jobs.plainCalls)
}
func TestUpdateService_WithoutVersionTriggersTheFollowedTag(t *testing.T) {
srv, _, _, _, jobs := newTestEnv(t)
jobs.job = model.Job{ID: 42}
resp, _ := postVersion(t, srv, "")
assert.Equal(t, http.StatusSeeOther, resp.StatusCode)
assert.Equal(t, 1, jobs.plainCalls)
assert.Empty(t, jobs.triggered)
}
func TestUpdateService_VersionErrors(t *testing.T) {
tests := []struct {
name string
version string
err error
wantCode int
wantBody string
wantJob string // Location header
}{
{"not a number", "abc", nil, http.StatusBadRequest, "invalid version", ""},
{"a ref instead of an id", "repo:evil@sha256:x", nil, http.StatusBadRequest, "invalid version", ""},
{"unknown version or other repo", "77", model.ErrNotFound, http.StatusNotFound, "not found", ""},
{"not updatable", "31", &model.NotUpdatableError{Reason: model.ReasonBuilt}, http.StatusConflict, "Cannot update: " + model.ReasonBuilt + ".", ""},
{"job already active", "31", model.ErrJobActive, http.StatusSeeOther, "", "/jobs/41"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
srv, _, _, _, jobs := newTestEnv(t)
jobs.err, jobs.job = tt.err, model.Job{ID: 41}
resp, body := postVersion(t, srv, tt.version)
assert.Equal(t, tt.wantCode, resp.StatusCode)
assert.Contains(t, body, tt.wantBody)
assert.Equal(t, tt.wantJob, resp.Header.Get("Location"))
if tt.wantCode == http.StatusBadRequest {
assert.Empty(t, jobs.triggered, "no job for a malformed version")
}
})
}
}
func TestOverview_ShowsFollowsNoTagAndPollFailedBadges(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].LastPollError = `head x: "boom" <script>`
for _, path := range []string{"/", "/fragments/services"} {
_, body := get(t, srv, path)
assert.Equal(t, 1, strings.Count(body, "follows no tag"), "only the digest-only Swarm service: "+path)
assert.Contains(t, body, `class="badge badge-error" title="head x: &#34;boom&#34; &lt;script&gt;">poll failed</span>`, path)
assert.Equal(t, 1, strings.Count(body, "poll failed"), path)
}
}
+58 -3
View File
@@ -5,18 +5,28 @@ import (
"time" "time"
"github.com/shcizo/package-updater/internal/imageref" "github.com/shcizo/package-updater/internal/imageref"
"github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model" "github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/master/poller"
"github.com/shcizo/package-updater/internal/protocol" "github.com/shcizo/package-updater/internal/protocol"
) )
type serviceGroup struct { type serviceGroup struct {
Title string Title string
Services []model.Service Services []serviceRow
}
// serviceRow is a service as shown in the overview table.
type serviceRow struct {
model.Service
AutoBlocked string // why the service cannot be set to auto; "" when it can
} }
// groupServices groups machine-bound services per agent and Swarm services // groupServices groups machine-bound services per agent and Swarm services
// in one group, keeping the store's order (machines first, Swarm last). // in one group, keeping the store's order (machines first, Swarm last).
func groupServices(services []model.Service) []serviceGroup { // autoBlocked is called for every service and returns the reason it cannot be
// set to auto, or "" when it can.
func groupServices(services []model.Service, autoBlocked func(model.Service) string) []serviceGroup {
var groups []serviceGroup var groups []serviceGroup
index := map[string]int{} index := map[string]int{}
for _, s := range services { for _, s := range services {
@@ -32,7 +42,7 @@ func groupServices(services []model.Service) []serviceGroup {
index[title] = i index[title] = i
groups = append(groups, serviceGroup{Title: title}) groups = append(groups, serviceGroup{Title: title})
} }
groups[i].Services = append(groups[i].Services, s) groups[i].Services = append(groups[i].Services, serviceRow{Service: s, AutoBlocked: autoBlocked(s)})
} }
return groups return groups
} }
@@ -46,6 +56,51 @@ func shortRef(ref string) string {
return r.String() return r.String()
} }
// followsNoTag says the service's image reference has a digest but no tag,
// so there is no tag to poll or update.
func followsNoTag(svc model.Service) bool {
return jobs.FollowedRef(svc.ActualRef) == ""
}
// versionRow is an image version as shown on the service page.
type versionRow struct {
model.ImageVersion
Running bool // the service runs exactly this digest
}
// versionRows marks the version the service runs now, by digest.
func versionRows(versions []model.ImageVersion, actualDigest string) []versionRow {
rows := make([]versionRow, len(versions))
for i, v := range versions {
rows[i] = versionRow{ImageVersion: v, Running: v.Digest != "" && v.Digest == actualDigest}
}
return rows
}
// pollingStatus is the text of the "Polling" row on the service page and
// whether it is a registry error.
func pollingStatus(svc model.Service, interval time.Duration) (text string, failed bool) {
switch {
case poller.SkipReason(svc) != "":
return "not polled: " + poller.SkipReason(svc), false
case svc.LastPollError != "":
return "last check failed: " + svc.LastPollError, true
}
return "every " + shortDuration(interval), false
}
// shortDuration formats d without trailing zero units: "6h", "1h30m", "90s".
func shortDuration(d time.Duration) string {
s := d.String()
if strings.HasSuffix(s, "m0s") {
s = s[:len(s)-2]
}
if strings.HasSuffix(s, "h0m") {
s = s[:len(s)-2]
}
return s
}
type agentRow struct { type agentRow struct {
model.Agent model.Agent
Online bool Online bool
+350
View File
@@ -0,0 +1,350 @@
// Package webhook implements POST /update, the endpoint CI calls after
// pushing an image. It records the pushed tag as "available" on every service
// that uses the image, starts an update for the services in auto mode and
// waits for those updates to finish.
package webhook
import (
"context"
"crypto/rand"
"crypto/sha256"
"crypto/subtle"
"encoding/hex"
"encoding/json"
"errors"
"log/slog"
"net/http"
"regexp"
"strconv"
"strings"
"time"
"github.com/shcizo/package-updater/internal/imageref"
"github.com/shcizo/package-updater/internal/logging"
"github.com/shcizo/package-updater/internal/master/events"
"github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model"
)
// maxBodyBytes caps the request body; a valid request is a few hundred bytes.
const maxBodyBytes = 64 << 10
// defaultPollEvery is how often pending jobs are re-read from the store when
// no broker signal arrives (signals to a slow subscriber can be dropped).
const defaultPollEvery = 5 * time.Second
// finalReadTimeout bounds the last read of pending jobs after the wait ends.
const finalReadTimeout = 2 * time.Second
// tagPattern is Docker's tag grammar. It keeps spaces, colons and "@" out of
// the ref the agent is asked to deploy.
var tagPattern = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$`)
// requestIDPattern limits a caller-supplied X-Request-ID to what is safe to
// echo in a header and write to the log.
var requestIDPattern = regexp.MustCompile(`^[A-Za-z0-9._-]{1,128}$`)
// Result statuses.
const (
statusSucceeded = "succeeded"
statusFailed = "failed"
statusTimeout = "timeout"
statusMarked = "marked"
statusBusy = "busy"
statusRefused = "refused"
)
// Store is the storage the webhook needs. *store.Store satisfies it.
type Store interface {
ServicesByRepo(ctx context.Context, repo string) ([]model.Service, error)
SetAvailable(ctx context.Context, id int64, a model.Available) error
GetJob(ctx context.Context, id int64) (model.Job, error)
}
// Jobs starts updates. *jobs.Service satisfies it.
type Jobs interface {
Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error)
}
// Broker delivers change signals. *events.Broker satisfies it.
type Broker interface {
Subscribe() (<-chan events.Topic, func())
Publish(t events.Topic)
}
// Config holds the webhook settings.
type Config struct {
Token string // bearer token; empty disables the webhook
Timeout time.Duration // bounds the whole wait of one request
PublicURL string // base of job_url, without trailing slash
}
// Handler serves POST /update.
type Handler struct {
store Store
jobs Jobs
broker Broker
cfg Config
log *slog.Logger
now func() time.Time
pollEvery time.Duration
}
// New returns the handler for POST /update. It does its own bearer-token
// check, so mount it outside the OIDC login.
func New(store Store, jobs Jobs, broker Broker, cfg Config, log *slog.Logger) *Handler {
return &Handler{store: store, jobs: jobs, broker: broker, cfg: cfg, log: log,
now: time.Now, pollEvery: defaultPollEvery}
}
type request struct {
Image string `json:"image"`
Tag string `json:"tag"`
}
type result struct {
ServiceID int64 `json:"service_id"`
Service string `json:"service"`
Kind string `json:"kind"`
Mode string `json:"mode"`
Status string `json:"status"`
JobID int64 `json:"job_id,omitempty"`
JobURL string `json:"job_url,omitempty"`
ToRef string `json:"to_ref,omitempty"`
Error string `json:"error,omitempty"`
DurationMS int64 `json:"duration_ms"`
}
type response struct {
RequestID string `json:"request_id"`
Image string `json:"image"`
Tag string `json:"tag,omitempty"`
Matched int `json:"matched"`
Results []result `json:"results"`
}
// ServeHTTP handles one webhook call. See the README for the contract.
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
started := h.now()
requestID := r.Header.Get("X-Request-ID")
if !requestIDPattern.MatchString(requestID) {
requestID = newRequestID()
}
w.Header().Set("X-Request-ID", requestID)
if h.cfg.Token == "" {
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "webhook disabled: UPDATER_API_KEY not set"})
return
}
if !h.authorized(r) {
writeJSON(w, http.StatusUnauthorized, map[string]string{"error": "unauthorized"})
return
}
var req request
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid JSON body"})
return
}
if req.Image == "" {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "image is required"})
return
}
if req.Tag != "" && !tagPattern.MatchString(req.Tag) {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid tag"})
return
}
log := logging.FromContext(logging.WithRequestID(r.Context(), requestID), h.log)
repo := imageref.Parse(req.Image).Repo
// A client that disconnects must not abort marking or starting: that
// could leave a queued job no agent knows about.
work := context.WithoutCancel(r.Context())
services, err := h.store.ServicesByRepo(work, repo)
if err != nil {
log.Error("webhook_lookup_failed", "image", req.Image, "error", err)
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "internal error"})
return
}
if len(services) == 0 {
writeJSON(w, http.StatusNotFound, map[string]string{
"error": "no service uses image " + repo, "request_id": requestID})
return
}
// Subscribe before any job exists so no completion signal is missed.
sub, cancel := h.broker.Subscribe()
defer cancel()
resp := response{RequestID: requestID, Image: req.Image, Tag: req.Tag, Matched: len(services)}
pending := map[int]int64{} // index into resp.Results -> job ID
for _, svc := range services {
res, job := h.handle(work, svc, req.Tag, started, log)
resp.Results = append(resp.Results, res)
if job != nil {
pending[len(resp.Results)-1] = job.ID
}
}
h.broker.Publish(events.Services)
h.wait(r.Context(), sub, &resp, pending, started, log)
status := http.StatusOK
counts := map[string]int{}
for _, res := range resp.Results {
counts[res.Status]++
if res.Status == statusFailed || res.Status == statusTimeout {
status = http.StatusInternalServerError
}
}
log.Info("webhook_request", "image", req.Image, "tag", req.Tag, "matched", resp.Matched,
"succeeded", counts[statusSucceeded], "failed", counts[statusFailed], "timeout", counts[statusTimeout],
"marked", counts[statusMarked], "busy", counts[statusBusy], "refused", counts[statusRefused])
writeJSON(w, status, resp)
}
// handle records the available ref on svc and, in auto mode, starts an update.
// It returns the result and, if the job is still active, that job.
func (h *Handler) handle(ctx context.Context, svc model.Service, tag string, started time.Time, log *slog.Logger) (result, *model.Job) {
res := result{ServiceID: svc.ID, Service: svc.DisplayName(), Kind: string(svc.Kind), Mode: svc.UpdateMode}
// The ref is built from the service's own repo, never from the request
// string, so a request cannot point a service at another registry.
var ref string
if tag != "" {
ref = svc.ImageRepo + ":" + tag
} else {
ref = jobs.FollowedRef(svc.ActualRef)
}
if ref == "" {
return h.finish(res, statusRefused, "no tag given and the service follows no tag", started), nil
}
res.ToRef = ref
err := h.store.SetAvailable(ctx, svc.ID, model.Available{Ref: ref, Source: model.SourceWebhook, At: h.now()})
if err != nil {
log.Error("webhook_set_available_failed", "service_id", svc.ID, "error", err)
return h.finish(res, statusFailed, "could not record available version", started), nil
}
if svc.UpdateMode != model.ModeAuto {
return h.finish(res, statusMarked, "", started), nil
}
job, err := h.jobs.Start(ctx, svc, jobs.StartRequest{Trigger: model.TriggerWebhook, Ref: ref})
var notUpdatable *model.NotUpdatableError
switch {
case errors.As(err, &notUpdatable):
return h.finish(res, statusRefused, notUpdatable.Reason, started), nil
case errors.Is(err, model.ErrJobActive):
h.setJob(&res, job)
return h.finish(res, statusBusy, err.Error(), started), nil
case err != nil:
log.Error("webhook_start_failed", "service_id", svc.ID, "error", err)
return h.finish(res, statusFailed, "could not start job", started), nil
}
h.setJob(&res, job)
if job.Active() {
return res, &job
}
h.resolveJob(&res, job, started)
return res, nil
}
func (h *Handler) setJob(res *result, job model.Job) {
res.JobID = job.ID
res.JobURL = h.cfg.PublicURL + "/jobs/" + strconv.FormatInt(job.ID, 10)
}
func (h *Handler) finish(res result, status, msg string, started time.Time) result {
res.Status, res.Error = status, msg
res.DurationMS = h.now().Sub(started).Milliseconds()
return res
}
// resolveJob sets the result from a job that is no longer active.
func (h *Handler) resolveJob(res *result, job model.Job, started time.Time) {
if job.Status == model.JobSucceeded {
*res = h.finish(*res, statusSucceeded, "", started)
return
}
*res = h.finish(*res, statusFailed, job.Error, started)
}
// wait blocks until every pending job has finished, the timeout passes or
// the client goes away. Jobs still pending then get status "timeout"; the
// master never cancels or fails them.
func (h *Handler) wait(parent context.Context, sub <-chan events.Topic, resp *response,
pending map[int]int64, started time.Time, log *slog.Logger) {
if len(pending) == 0 {
return
}
ctx, cancel := context.WithTimeout(parent, h.cfg.Timeout)
defer cancel()
ticker := time.NewTicker(h.pollEvery)
defer ticker.Stop()
for len(pending) > 0 {
select {
case <-ctx.Done():
// A job may have finished while the timer fired: look once more,
// on a context that outlives the request.
final, stop := context.WithTimeout(context.WithoutCancel(parent), finalReadTimeout)
h.readPending(final, resp, pending, started, log)
stop()
for i := range pending {
resp.Results[i] = h.finish(resp.Results[i], statusTimeout, "still running when the request timed out", started)
}
return
case t := <-sub:
if t != events.Jobs {
continue
}
case <-ticker.C:
}
h.readPending(ctx, resp, pending, started, log)
}
}
// readPending re-reads every pending job and resolves the ones that are done.
func (h *Handler) readPending(ctx context.Context, resp *response, pending map[int]int64,
started time.Time, log *slog.Logger) {
for i, id := range pending {
job, err := h.store.GetJob(ctx, id)
if err != nil {
log.Warn("webhook_get_job_failed", "job_id", id, "error", err)
continue
}
if !job.Active() {
h.resolveJob(&resp.Results[i], job, started)
delete(pending, i)
}
}
}
// authorized reports whether the request carries the configured bearer token.
// Both sides are hashed so the comparison takes the same time whatever the
// length of the supplied token.
func (h *Handler) authorized(r *http.Request) bool {
const prefix = "Bearer "
got, ok := strings.CutPrefix(r.Header.Get("Authorization"), prefix)
if !ok || got == "" {
return false
}
a, b := sha256.Sum256([]byte(got)), sha256.Sum256([]byte(h.cfg.Token))
return subtle.ConstantTimeCompare(a[:], b[:]) == 1
}
func writeJSON(w http.ResponseWriter, status int, body any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(body)
}
// newRequestID returns a random 128-bit hex ID.
func newRequestID() string {
var b [16]byte
_, _ = rand.Read(b[:])
return hex.EncodeToString(b[:])
}
+517
View File
@@ -0,0 +1,517 @@
package webhook
import (
"context"
"encoding/json"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/events"
"github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/protocol"
)
const token = "s3cret"
type availCall struct {
id int64
a model.Available
}
type fakeStore struct {
mu sync.Mutex
services []model.Service // returned for any repo that equals their ImageRepo
jobs map[int64]model.Job
avail []availCall
repos []string
}
func (f *fakeStore) ServicesByRepo(_ context.Context, repo string) ([]model.Service, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.repos = append(f.repos, repo)
var out []model.Service
for _, s := range f.services {
if s.ImageRepo == repo {
out = append(out, s)
}
}
return out, nil
}
func (f *fakeStore) SetAvailable(_ context.Context, id int64, a model.Available) error {
f.mu.Lock()
defer f.mu.Unlock()
f.avail = append(f.avail, availCall{id, a})
return nil
}
func (f *fakeStore) GetJob(_ context.Context, id int64) (model.Job, error) {
f.mu.Lock()
defer f.mu.Unlock()
return f.jobs[id], nil
}
func (f *fakeStore) setJob(j model.Job) {
f.mu.Lock()
defer f.mu.Unlock()
f.jobs[j.ID] = j
}
type startCall struct {
svc model.Service
req jobs.StartRequest
ctx context.Context
}
// fakeJobs hands out job IDs and applies per-service behaviour.
type fakeJobs struct {
mu sync.Mutex
store *fakeStore
calls []startCall
errFor map[int64]error // returned together with errJob[id] if set
jobFor map[int64]model.Job // status of the job returned by Start
next int64
}
func (f *fakeJobs) Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.calls = append(f.calls, startCall{svc, req, ctx})
if err, ok := f.errFor[svc.ID]; ok {
return f.jobFor[svc.ID], err
}
f.next++
j := model.Job{ID: f.next, ServiceID: svc.ID, Status: model.JobQueued, Trigger: req.Trigger, ToRef: req.Ref}
if o, ok := f.jobFor[svc.ID]; ok {
o.ID = j.ID
j = o
}
f.store.mu.Lock()
f.store.jobs[j.ID] = j
f.store.mu.Unlock()
return j, nil
}
func (f *fakeJobs) startCalls() []startCall {
f.mu.Lock()
defer f.mu.Unlock()
return append([]startCall(nil), f.calls...)
}
type env struct {
store *fakeStore
jobs *fakeJobs
broker *events.Broker
h *Handler
}
func newEnv(t *testing.T, services ...model.Service) *env {
t.Helper()
st := &fakeStore{services: services, jobs: map[int64]model.Job{}}
fj := &fakeJobs{store: st, errFor: map[int64]error{}, jobFor: map[int64]model.Job{}}
b := events.NewBroker()
h := New(st, fj, b, Config{Token: token, Timeout: 2 * time.Second, PublicURL: "https://updater.example.com"},
slog.New(slog.NewTextHandler(io.Discard, nil)))
h.now = func() time.Time { return time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC) }
return &env{store: st, jobs: fj, broker: b, h: h}
}
func (e *env) post(body string, headers ...string) *httptest.ResponseRecorder {
return e.postCtx(context.Background(), body, headers...)
}
func (e *env) postCtx(ctx context.Context, body string, headers ...string) *httptest.ResponseRecorder {
req := httptest.NewRequest(http.MethodPost, "/update", strings.NewReader(body)).WithContext(ctx)
req.Header.Set("Authorization", "Bearer "+token)
for i := 0; i+1 < len(headers); i += 2 {
req.Header.Set(headers[i], headers[i+1])
}
rec := httptest.NewRecorder()
e.h.ServeHTTP(rec, req)
return rec
}
func decode(t *testing.T, rec *httptest.ResponseRecorder) response {
t.Helper()
var r response
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &r), rec.Body.String())
return r
}
func compose(id int64, mode, actualRef string) model.Service {
return model.Service{ID: id, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web" + string(rune('a'+id)),
ImageRepo: "gitea.shcizo.se/shcizo/slusk", ActualRef: actualRef, UpdateMode: mode}
}
const slusk = "gitea.shcizo.se/shcizo/slusk"
func TestAuth(t *testing.T) {
tests := []struct {
name string
serverKey string
header string
wantStatus int
wantBody string
}{
{"missing header", token, "", 401, `{"error":"unauthorized"}`},
{"wrong scheme", token, "Basic " + token, 401, `{"error":"unauthorized"}`},
{"wrong token", token, "Bearer nope", 401, `{"error":"unauthorized"}`},
{"empty bearer", token, "Bearer ", 401, `{"error":"unauthorized"}`},
{"key unset", "", "Bearer " + token, 503, `{"error":"webhook disabled: UPDATER_API_KEY not set"}`},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
e := newEnv(t, compose(1, model.ModeManual, slusk+":1"))
e.h.cfg.Token = tt.serverKey
req := httptest.NewRequest(http.MethodPost, "/update", strings.NewReader(`{"image":"`+slusk+`"}`))
if tt.header != "" {
req.Header.Set("Authorization", tt.header)
}
rec := httptest.NewRecorder()
e.h.ServeHTTP(rec, req)
assert.Equal(t, tt.wantStatus, rec.Code)
assert.JSONEq(t, tt.wantBody, rec.Body.String())
assert.Empty(t, e.store.repos, "nothing may be looked up before auth")
})
}
}
func TestBadRequest(t *testing.T) {
tests := []struct {
name, body, wantError string
}{
{"invalid JSON", `{`, "invalid JSON body"},
{"empty body", ``, "invalid JSON body"},
{"empty image", `{"image":""}`, "image is required"},
{"missing image", `{"tag":"v1"}`, "image is required"},
{"oversized", `{"image":"` + strings.Repeat("a", 70<<10) + `"}`, "invalid JSON body"},
{"tag with digest", `{"image":"` + slusk + `","tag":"v1@sha256:abc"}`, "invalid tag"},
{"tag with space", `{"image":"` + slusk + `","tag":"v 1"}`, "invalid tag"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1"))
rec := e.post(tt.body)
assert.Equal(t, 400, rec.Code)
assert.JSONEq(t, `{"error":"`+tt.wantError+`"}`, rec.Body.String())
assert.Empty(t, e.jobs.startCalls())
})
}
}
func TestNoMatch(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1"))
for _, image := range []string{"gitea.shcizo.se/shcizo/other", "gitea.shcizo.se/shcizo/Slusk"} {
rec := e.post(`{"image":"` + image + `"}`)
assert.Equal(t, 404, rec.Code, image)
assert.Contains(t, rec.Body.String(), "no service uses image "+image)
}
assert.Empty(t, e.jobs.startCalls())
assert.Empty(t, e.store.avail)
}
func TestImageIsNormalisedBeforeMatching(t *testing.T) {
svc := compose(1, model.ModeManual, slusk+":1")
local := compose(2, model.ModeManual, "localhost:5000/foo:v1")
local.ImageRepo = "localhost:5000/foo"
for _, tt := range []struct{ image, wantRepo string }{
{slusk, slusk},
{slusk + ":2.0", slusk},
{slusk + "@sha256:abc", slusk},
{slusk + ":2.0@sha256:abc", slusk},
{"localhost:5000/foo:v1", "localhost:5000/foo"},
} {
e := newEnv(t, svc, local)
rec := e.post(`{"image":"` + tt.image + `"}`)
assert.Equal(t, 200, rec.Code, tt.image)
assert.Equal(t, []string{tt.wantRepo}, e.store.repos, tt.image)
assert.Equal(t, 1, decode(t, rec).Matched, tt.image)
}
}
func TestManualServiceIsMarkedNotUpdated(t *testing.T) {
e := newEnv(t, compose(1, model.ModeManual, slusk+":1.4.2@sha256:old"))
sub, cancel := e.broker.Subscribe()
defer cancel()
rec := e.post(`{"image":"` + slusk + `:ignored-in-ref","tag":"1.5.0"}`)
assert.Equal(t, 200, rec.Code)
assert.Empty(t, e.jobs.startCalls())
require.Len(t, e.store.avail, 1)
assert.Equal(t, int64(1), e.store.avail[0].id)
assert.Equal(t, model.Available{Ref: slusk + ":1.5.0", Source: model.SourceWebhook, At: e.h.now()}, e.store.avail[0].a)
resp := decode(t, rec)
assert.Equal(t, 1, resp.Matched)
require.Len(t, resp.Results, 1)
r := resp.Results[0]
assert.Equal(t, statusMarked, r.Status)
assert.Equal(t, "slusk/webb", r.Service)
assert.Equal(t, "compose", r.Kind)
assert.Equal(t, "manual", r.Mode)
assert.Equal(t, slusk+":1.5.0", r.ToRef)
assert.Zero(t, r.JobID)
assert.NotEmpty(t, resp.RequestID)
assert.Equal(t, resp.RequestID, rec.Header().Get("X-Request-ID"))
assert.Equal(t, events.Services, <-sub)
}
func TestRequestIDIsEchoed(t *testing.T) {
e := newEnv(t, compose(1, model.ModeManual, slusk+":1"))
rec := e.post(`{"image":"`+slusk+`"}`, "X-Request-ID", "ci-123")
assert.Equal(t, "ci-123", decode(t, rec).RequestID)
assert.Equal(t, "ci-123", rec.Header().Get("X-Request-ID"))
}
func TestAvailableRefWithoutTag(t *testing.T) {
tests := []struct {
name string
actualRef string
wantRef string // "" means SetAvailable must not be called
}{
{"followed tag", slusk + ":1.4.2@sha256:old", slusk + ":1.4.2"},
{"bare image follows latest", slusk, slusk + ":latest"},
{"digest only", slusk + "@sha256:old", ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
e := newEnv(t, compose(1, model.ModeManual, tt.actualRef))
rec := e.post(`{"image":"` + slusk + `"}`)
assert.Equal(t, 200, rec.Code)
r := decode(t, rec).Results[0]
if tt.wantRef == "" {
assert.Empty(t, e.store.avail)
assert.Equal(t, statusRefused, r.Status)
assert.Equal(t, "no tag given and the service follows no tag", r.Error)
return
}
assert.Equal(t, statusMarked, r.Status)
require.Len(t, e.store.avail, 1)
assert.Equal(t, tt.wantRef, e.store.avail[0].a.Ref)
})
}
}
func TestAutoDigestOnlyServiceWithoutTagIsRefused(t *testing.T) {
svc := compose(1, model.ModeAuto, slusk+"@sha256:old")
e := newEnv(t, svc)
rec := e.post(`{"image":"` + slusk + `"}`)
assert.Equal(t, 200, rec.Code)
assert.Empty(t, e.store.avail)
assert.Empty(t, e.jobs.startCalls(), "no target ref: nothing to start")
r := decode(t, rec).Results[0]
assert.Equal(t, statusRefused, r.Status)
assert.Equal(t, "no tag given and the service follows no tag", r.Error)
}
func TestValidTags(t *testing.T) {
for _, tag := range []string{"latest", "1.2.3-rc.1", "v1_2", strings.Repeat("a", 128)} {
e := newEnv(t, compose(1, model.ModeManual, slusk+":1"))
rec := e.post(`{"image":"` + slusk + `","tag":"` + tag + `"}`)
assert.Equal(t, 200, rec.Code, tag)
require.Len(t, e.store.avail, 1, tag)
assert.Equal(t, slusk+":"+tag, e.store.avail[0].a.Ref)
}
e := newEnv(t, compose(1, model.ModeManual, slusk+":1"))
rec := e.post(`{"image":"` + slusk + `","tag":"` + strings.Repeat("a", 129) + `"}`)
assert.Equal(t, 400, rec.Code)
}
func TestUnsafeRequestIDIsReplaced(t *testing.T) {
for _, id := range []string{"has space", "bad/slash", strings.Repeat("a", 129)} {
e := newEnv(t, compose(1, model.ModeManual, slusk+":1"))
rec := e.post(`{"image":"`+slusk+`"}`, "X-Request-ID", id)
got := rec.Header().Get("X-Request-ID")
assert.NotEqual(t, id, got)
assert.Regexp(t, `^[0-9a-f]{32}$`, got)
}
e := newEnv(t, compose(1, model.ModeManual, slusk+":1"))
rec := e.post(`{"image":"`+slusk+`"}`, "X-Request-ID", strings.Repeat("a", 128))
assert.Equal(t, strings.Repeat("a", 128), rec.Header().Get("X-Request-ID"))
}
func TestJobFinishedJustBeforeTimeoutIsNotReportedAsTimeout(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.h.cfg.Timeout = 200 * time.Millisecond
e.h.pollEvery = time.Hour // only the final read after the deadline can see the result
go func() {
assert.Eventually(t, func() bool { return len(e.jobs.startCalls()) == 1 }, time.Second, time.Millisecond)
e.store.setJob(model.Job{ID: 1, Status: model.JobSucceeded}) // no Publish
}()
rec := e.post(`{"image":"` + slusk + `"}`)
assert.Equal(t, 200, rec.Code)
assert.Equal(t, statusSucceeded, decode(t, rec).Results[0].Status)
}
func TestAutoJobTargetIsBuiltFromServiceRepo(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2@sha256:old"))
e.jobs.jobFor[1] = model.Job{Status: model.JobSucceeded}
rec := e.post(`{"image":"` + slusk + `:other","tag":"2.0.0"}`)
assert.Equal(t, 200, rec.Code)
calls := e.jobs.startCalls()
require.Len(t, calls, 1)
assert.Equal(t, slusk+":2.0.0", calls[0].req.Ref)
assert.Equal(t, model.TriggerWebhook, calls[0].req.Trigger)
assert.Empty(t, calls[0].req.RequestedBy)
require.Len(t, e.store.avail, 1)
assert.Equal(t, slusk+":2.0.0", e.store.avail[0].a.Ref)
}
func TestAutoJobSucceeded(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
go func() {
assert.Eventually(t, func() bool { return len(e.jobs.startCalls()) == 1 }, time.Second, time.Millisecond)
e.store.setJob(model.Job{ID: 1, Status: model.JobSucceeded})
e.broker.Publish(events.JobTopic(1)) // progress topic is ignored
e.broker.Publish(events.Jobs)
}()
rec := e.post(`{"image":"` + slusk + `","tag":"1.5.0"}`)
assert.Equal(t, 200, rec.Code)
r := decode(t, rec).Results[0]
assert.Equal(t, statusSucceeded, r.Status)
assert.Equal(t, int64(1), r.JobID)
assert.Equal(t, "https://updater.example.com/jobs/1", r.JobURL)
assert.Equal(t, "auto", r.Mode)
}
func TestAutoJobFailedIs500(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
go func() {
assert.Eventually(t, func() bool { return len(e.jobs.startCalls()) == 1 }, time.Second, time.Millisecond)
e.store.setJob(model.Job{ID: 1, Status: model.JobFailed, Error: "pull access denied"})
e.broker.Publish(events.Jobs)
}()
rec := e.post(`{"image":"` + slusk + `"}`)
assert.Equal(t, 500, rec.Code)
r := decode(t, rec).Results[0]
assert.Equal(t, statusFailed, r.Status)
assert.Equal(t, "pull access denied", r.Error)
}
func TestJobFailedAtStartIsNotWaitedFor(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.jobs.jobFor[1] = model.Job{Status: model.JobFailed, Error: "agent offline"}
e.h.cfg.Timeout = time.Minute
begin := time.Now()
rec := e.post(`{"image":"` + slusk + `"}`)
assert.Less(t, time.Since(begin), 5*time.Second)
assert.Equal(t, 500, rec.Code)
r := decode(t, rec).Results[0]
assert.Equal(t, statusFailed, r.Status)
assert.Equal(t, "agent offline", r.Error)
assert.NotZero(t, r.JobID)
}
func TestTimeout(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.h.cfg.Timeout = 50 * time.Millisecond
rec := e.post(`{"image":"` + slusk + `"}`)
assert.Equal(t, 500, rec.Code)
r := decode(t, rec).Results[0]
assert.Equal(t, statusTimeout, r.Status)
assert.Equal(t, int64(1), r.JobID)
assert.Equal(t, model.JobQueued, e.store.jobs[1].Status, "the master must not touch a timed-out job")
}
func TestDroppedSignalIsRecoveredByPolling(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.h.pollEvery = 5 * time.Millisecond
go func() {
assert.Eventually(t, func() bool { return len(e.jobs.startCalls()) == 1 }, time.Second, time.Millisecond)
e.store.setJob(model.Job{ID: 1, Status: model.JobSucceeded}) // no Publish
}()
rec := e.post(`{"image":"` + slusk + `"}`)
assert.Equal(t, 200, rec.Code)
assert.Equal(t, statusSucceeded, decode(t, rec).Results[0].Status)
}
func TestBusyIsNotAFailure(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.jobs.errFor[1] = model.ErrJobActive
e.jobs.jobFor[1] = model.Job{ID: 42, Status: model.JobRunning}
e.h.cfg.Timeout = time.Minute
rec := e.post(`{"image":"` + slusk + `"}`)
assert.Equal(t, 200, rec.Code)
r := decode(t, rec).Results[0]
assert.Equal(t, statusBusy, r.Status)
assert.Equal(t, int64(42), r.JobID)
assert.Equal(t, "https://updater.example.com/jobs/42", r.JobURL)
assert.NotEmpty(t, e.store.avail, "the available ref is still recorded")
}
func TestRefusedIsNotAFailure(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.jobs.errFor[1] = &model.NotUpdatableError{Reason: model.ReasonSelf}
rec := e.post(`{"image":"` + slusk + `","tag":"2"}`)
assert.Equal(t, 200, rec.Code)
r := decode(t, rec).Results[0]
assert.Equal(t, statusRefused, r.Status)
assert.Equal(t, model.ReasonSelf, r.Error)
assert.Zero(t, r.JobID)
assert.NotEmpty(t, e.store.avail)
}
func TestMixedAutoAndManual(t *testing.T) {
e := newEnv(t,
compose(1, model.ModeManual, slusk+":1"),
compose(2, model.ModeAuto, slusk+":1"),
compose(3, model.ModeAuto, slusk+":1"))
e.jobs.jobFor[2] = model.Job{Status: model.JobSucceeded}
e.jobs.jobFor[3] = model.Job{Status: model.JobFailed, Error: "boom"}
rec := e.post(`{"image":"` + slusk + `","tag":"2"}`)
assert.Equal(t, 500, rec.Code)
resp := decode(t, rec)
assert.Equal(t, 3, resp.Matched)
got := []string{resp.Results[0].Status, resp.Results[1].Status, resp.Results[2].Status}
assert.Equal(t, []string{statusMarked, statusSucceeded, statusFailed}, got)
assert.Len(t, e.store.avail, 3)
}
func TestClientCancelStopsWaitingButNotStarting(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.h.cfg.Timeout = time.Minute
ctx, cancel := context.WithCancel(context.Background())
cancel() // the client is already gone when the handler runs
begin := time.Now()
rec := e.postCtx(ctx, `{"image":"`+slusk+`"}`)
assert.Less(t, time.Since(begin), 5*time.Second)
assert.Equal(t, statusTimeout, decode(t, rec).Results[0].Status)
calls := e.jobs.startCalls()
require.Len(t, calls, 1, "the job must still be started")
assert.NoError(t, calls[0].ctx.Err(), "Start must run on a context the client cannot cancel")
}
+10 -3
View File
@@ -16,11 +16,16 @@ import (
// Version is the protocol version this build speaks. Agents send it in // Version is the protocol version this build speaks. Agents send it in
// Hello. Phase-1 agents do not send the field, which reads as 0. // Hello. Phase-1 agents do not send the field, which reads as 0.
const Version = 2 const Version = 3
// MinJobVersion is the lowest agent protocol version the master sends jobs to. // MinJobVersion is the lowest agent protocol version the master sends jobs to.
const MinJobVersion = 2 const MinJobVersion = 2
// MinPinnedRefVersion is the lowest agent protocol version the master sends
// jobs whose Ref has a digest or no tag to. Version-2 agents refuse a Ref
// with a digest.
const MinPinnedRefVersion = 3
// MaxJobOutput caps the job output an agent keeps and the master stores, in // MaxJobOutput caps the job output an agent keeps and the master stores, in
// bytes. Only the tail is kept. With JSON escaping (worst case 6 bytes per // bytes. Only the tail is kept. With JSON escaping (worst case 6 bytes per
// byte) a JobResult carrying this much output still fits the 1 MiB read limit. // byte) a JobResult carrying this much output still fits the 1 MiB read limit.
@@ -123,8 +128,10 @@ type Remove struct {
Key string `json:"key"` Key string `json:"key"`
} }
// JobStart asks the agent to deploy Ref to one service. Ref is "repo:tag"; // JobStart asks the agent to deploy Ref to one service. Ref is "repo:tag",
// the agent resolves the tag's current digest and deploys "repo:tag@digest". // "repo:tag@digest" or "repo@digest". For "repo:tag" the agent resolves the
// tag's current digest and deploys "repo:tag@digest"; a Ref with a digest is
// deployed as is.
// The agent reads the Compose working directory and config files from its // The agent reads the Compose working directory and config files from its
// own Docker daemon, not from this message. // own Docker daemon, not from this message.
type JobStart struct { type JobStart struct {