Compare commits

..
5 Commits
55 changed files with 2935 additions and 195 deletions
+24 -7
View File
@@ -36,8 +36,9 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
- `internal/master/config` — env-var loading - `internal/master/config` — env-var loading
- `internal/master/store` — Postgres access and migrations - `internal/master/store` — Postgres access and migrations
- `internal/master/hub` — agent connections: auth, per-connection writer, routing, job reconcile on hello - `internal/master/hub` — agent connections: auth, per-connection writer, routing, job reconcile on hello
- `internal/master/registry` — HEAD-only digest lookup against a registry (go-containerregistry, `DOCKER_CONFIG` credentials) - `internal/master/registry` — HEAD-only digest lookup and tag listing against a registry (go-containerregistry, `DOCKER_CONFIG` credentials)
- `internal/master/poller` — every `POLL_INTERVAL`: one lookup per followed `repo:tag`, sets the available version, starts auto jobs - `internal/master/tagver` — which tag is a newer version of a followed tag (pure functions)
- `internal/master/poller` — every `POLL_INTERVAL`: one lookup per followed `repo:tag`, sets the available version, starts auto jobs; one tag list per repo, sets the newer tag suggestion; `PollService` does the same for one service (UI Check now)
- `internal/master/jobs` — decides whether a service may be updated and starts a job on the right agent - `internal/master/jobs` — decides whether a service may be updated and starts a job on the right agent
- `internal/master/webhook` — `POST /update`: records the available version, starts jobs for auto services, waits for them - `internal/master/webhook` — `POST /update`: records the available version, starts jobs for auto services, waits for them
- `internal/master/web/icons` — embedded Simple Icons path data; slug derived from the image ref; regenerate with `go generate ./internal/master/web/icons` - `internal/master/web/icons` — embedded Simple Icons path data; slug derived from the image ref; regenerate with `go generate ./internal/master/web/icons`
@@ -58,8 +59,12 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
the agent's own container whose source equals its destination.** Mounts are read per the agent's own container whose source equals its destination.** Mounts are read per
job (`ContainerInspect(SELF_ID)`) and checked after symlink resolution. This replaces job (`ContainerInspect(SELF_ID)`) and checked after symlink resolution. This replaces
the old `STACKS_ROOT`. the old `STACKS_ROOT`.
- **Self-protection on the master**: a service is refused when its image repo equals the - **Self-update** (an agent's own Compose service): only `up -d` runs in the helper
repo of `AGENT_IMAGE` or the row has the `self` flag. The agent does not re-check. container; the new agent resumes the job (`updater.ResumeSelfUpdates`) before the runner
and connection start; the old agent records no result once SIGTERM cancels it. Jobs for
the agent's own service run last on the agent and the master starts them last in a
batch. An agent's own Swarm service is refused (`ReasonSelfSwarm`). Protocol 4 is needed,
also for any service with the `AGENT_IMAGE` repo.
- **Unknown message types are logged and ignored; malformed messages close the connection.** - **Unknown message types are logged and ignored; malformed messages close the connection.**
- **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not - **Reconcile before routable**: the hub fails a reconnecting agent's jobs it does not
list in `known_jobs`, and only then accepts jobs for that connection. list in `known_jobs`, and only then accepts jobs for that connection.
@@ -76,8 +81,16 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
## Gotchas ## Gotchas
- **Registry access is HEAD only** (`internal/master/registry`). Docker Hub counts a manifest - **Registry manifest access is HEAD only** (`internal/master/registry`). Docker Hub counts a
GET as a pull; the client never falls back to GET, and a test asserts this. manifest GET as a pull; the client never GETs `/manifests/`, and a test asserts this. The
only GETs are the `/v2/` ping and `tags/list` (newer-tag detection, once per repo per cycle).
- **`services.newer_ref` is a suggestion only**: no auto jobs, not even for auto services. The
poller writes it only on change; a failed tag list keeps the old value and never sets
`last_poll_error`. The UI re-checks it at render time with `tagver.Newer`, so a stale value
is hidden until the next cycle; it is also hidden when its repo is not the one the service
follows. The suggestion is computed from the service list read at the start of the cycle, so
if the service moved on meanwhile, the render-time check hides it. Tags with one numeric component or no digits (`7`, `pg16`,
`latest`) never get suggestions.
- **`image_versions` uses `''` instead of NULL** for a missing tag or digest, so - **`image_versions` uses `''` instead of NULL** for a missing tag or digest, so
`UNIQUE (repo, tag, digest)` deduplicates. The webhook records `(repo, tag, '')`; that `UNIQUE (repo, tag, digest)` deduplicates. The webhook records `(repo, tag, '')`; that
row is hidden from `RecentVersions` when a row with the same tag and a digest was first row is hidden from `RecentVersions` when a row with the same tag and a digest was first
@@ -124,6 +137,10 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
own Docker daemon at execution time. own Docker daemon at execution time.
- **`protocol.Service.ConfigFiles` is the raw comma-joined label string**, so `Service` - **`protocol.Service.ConfigFiles` is the raw comma-joined label string**, so `Service`
stays comparable with `==` (`inventory.Diff` depends on it). stays comparable with `==` (`inventory.Diff` depends on it).
- **Helper containers carry `se.shcizo.package-updater.self-update.*` labels** and are
not inventoried (the collector skips any container with the `.job` label). The label
strings are constants in `internal/agent/updater/selfupdate.go`, and the `.job` one is
repeated in `internal/agent/inventory`.
- **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged. - **Swarm "succeeded" means the daemon accepted the new spec**, not that tasks converged.
- **Swarm updates need manager-node API access** for the agent doing the update. - **Swarm updates need manager-node API access** for the agent doing the update.
- **The image's default entrypoint is `updater-agent`.** The master overrides it with - **The image's default entrypoint is `updater-agent`.** The master overrides it with
@@ -136,7 +153,7 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
knows about; only the waiting stops. The master never fails or cancels a timed-out job. knows about; only the waiting stops. The master never fails or cancels a timed-out job.
- **A webhook job's target is `<image_repo>:<request tag>`**, built from the service's own - **A webhook job's target is `<image_repo>:<request tag>`**, built from the service's own
repo, never from the request's image string. With an explicit target the "follows no tag" repo, never from the request's image string. With an explicit target the "follows no tag"
refusal does not apply; removed, self, built and container still do. refusal does not apply; removed, built, container and an agent's own Swarm service still do.
- **`available_*` is cleared by `FinishJob`** on success only when the job's `to_ref` - **`available_*` is cleared by `FinishJob`** on success only when the job's `to_ref`
(as created, without `@digest`) equals `available_ref`, `available_at <= job.created_at` (as created, without `@digest`) equals `available_ref`, `available_at <= job.created_at`
(that version is now running) and, when the job's `to_ref` carries a digest, that digest (that version is now running) and, when the job's `to_ref` carries a digest, that digest
+54 -9
View File
@@ -18,7 +18,8 @@ Clicking *Update* on a service re-pulls the service's **current tag** (`repo:tag
agent asks its Docker daemon for the tag's digest and deploys `repo:tag@digest`, so what agent asks its Docker daemon for the tag's digest and deploys `repo:tag@digest`, so what
runs is exactly what was pulled. The log streams to the browser while it runs; job runs is exactly what was pulled. The log streams to the browser while it runs; job
history is kept in Postgres. A service without a tag (`repo@digest`), a plain container, history is kept in Postgres. A service without a tag (`repo@digest`), a plain container,
and the updater's own services cannot be updated, and the UI says why. A bare image and a Swarm agent's own service cannot be updated, and the UI says why (see
[Self-update](#self-update) for the master and Compose agents). A bare image
(`image: postgres`, no tag and no digest) counts as `postgres:latest`; only a reference (`image: postgres`, no tag and no digest) counts as `postgres:latest`; only a reference
with a digest and no tag (`repo@sha256:...`) "follows no tag". Compose services with a with a digest and no tag (`repo@sha256:...`) "follows no tag". Compose services with a
`build:` section are shown but never updated ("built from source (build:)"). One update at `build:` section are shown but never updated ("built from source (build:)"). One update at
@@ -61,7 +62,7 @@ Content-Type: application/json
in the request are ignored for matching; matching is case-sensitive). `tag` is optional. in the request are ignored for matching; matching is case-sensitive). `tag` is optional.
For every match the master records the version as *available* (the overview shows an For every match the master records the version as *available* (the overview shows an
"update available" badge), including services that cannot be updated automatically "update available" badge), including services that cannot be updated automatically
(updater's own, `build:` and plain containers), so the badge shows for them too. Services (`build:` services, plain containers and an agent's own Swarm service), so the badge shows for them too. Services
in **manual** mode stop there. Services in **auto** in **manual** mode stop there. Services in **auto**
mode also get an update job, and the request waits for those jobs to finish, at most mode also get an update job, and the request waits for those jobs to finish, at most
`UPDATE_TIMEOUT`. `UPDATE_TIMEOUT`.
@@ -73,8 +74,8 @@ one that has been pushed. Compose services get pinned to it through
already follows (a bare image counts as `:latest`); a service that follows no tag already follows (a bare image counts as `:latest`); a service that follows no tag
(`repo@digest`) is then not touched and is reported as `refused` ("no tag given and the (`repo@digest`) is then not touched and is reported as `refused` ("no tag given and the
service follows no tag"), manual or auto. service follows no tag"), manual or auto.
With a `tag`, the "follows no tag" refusal does not apply; the other refusals (the With a `tag`, the "follows no tag" refusal does not apply; the other refusals (`build:`
updater's own services, `build:` services, plain containers) still do. services, plain containers, a Swarm agent's own service) still do.
Response (`200` or `500`): Response (`200` or `500`):
@@ -111,8 +112,8 @@ is refused with its reason, in auto mode too, until someone restores or accepts
The webhook never creates services: the first deployment of a stack is done by hand. A The webhook never creates services: the first deployment of a stack is done by hand. A
new service always starts in **manual** mode; switch it with the *Make auto* button on the new service always starts in **manual** mode; switch it with the *Make auto* button on the
overview or the service page. Auto is refused for the updater's own services, `build:` overview or the service page. Auto is refused for `build:` services, plain containers and
services and plain containers. a Swarm agent's own service.
> **Warning:** auto mode deploys whatever CI pushes, without anyone looking at it. Using > **Warning:** auto mode deploys whatever CI pushes, without anyone looking at it. Using
> it on critical services is your own risk. > it on critical services is your own risk.
@@ -143,10 +144,21 @@ multi-arch image the digest is the index digest, the same one `docker inspect` s
it) and starts no job for that digest again, whether the job it) and starts no job for that digest again, whether the job
failed or the user rolled back afterwards. If the service already had a job when the failed or the user rolled back afterwards. If the service already had a job when the
digest was found, the next cycle tries again. digest was found, the next cycle tries again.
- **Newer tags.** For a service that follows a version-like tag (`1.27.3`, `v1.2.3`,
`11.0-latest`; at least two numeric components), the master also lists the repo's tags
once per cycle (`GET /v2/<repo>/tags/list`, which Docker Hub does not count as a pull) and
suggests the highest tag with the same prefix, suffix and number of components and the
same major number, for example `gitea:1.27.3` gives `1.28.0` but never `2.0.0`. The
overview shows a "Newer tag" badge and the service page lists the tag under Versions, from
where you deploy it. It is a suggestion only: nothing is deployed automatically, not even
for **auto** services. Tags such as `latest`, `7` or `pg16` get no suggestion. A failed tag
list keeps the old suggestion and is only logged (`poll_tags_failed`).
- **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and - **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and
services without a registry digest (locally built images). The service page says which. services without a registry digest (locally built images). The service page says which.
The updater's own services and plain containers are polled, so their badge works, but Plain containers are polled, so their badge works, but jobs for them are refused.
jobs for them are still refused. - **Check now.** The service page's *Check now* button runs the same check immediately for that
service only (digest, auto job, newer tag); other services that follow the same tag are
updated at the next cycle.
- **Errors.** A failed lookup is stored for the services that follow that tag and shown as a - **Errors.** A failed lookup is stored for the services that follow that tag and shown as a
"poll failed" badge on the overview and in the "Polling" row of the service page; other "poll failed" badge on the overview and in the "Polling" row of the service page; other
tags continue. A later success clears it. tags continue. A later success clears it.
@@ -224,7 +236,7 @@ version, auto mode or jobs) while the new one has none, the service page suggest
moved. **Merge** moves the old row, with its history, mode and desired version, to the new moved. **Merge** moves the old row, with its history, mode and desired version, to the new
location and deletes the duplicate. Merging is never automatic. It can also be done by location and deletes the duplicate. Merging is never automatic. It can also be done by
hand for any removed service of the same kind (Compose or Swarm, names may differ), but hand for any removed service of the same kind (Compose or Swarm, names may differ), but
not for plain containers, the updater itself, or a service with a queued or running job. not for plain containers, an agent's own service, or a service with a queued or running job.
A merge keeps the old row's mode and desired version, so a manually merged target with its A merge keeps the old row's mode and desired version, so a manually merged target with its
own history may show drift right after the merge and can be accepted. own history may show drift right after the merge and can be accepted.
@@ -276,6 +288,39 @@ Read-only mounts and mounts whose source differs from the destination are refuse
clear message. On Docker Desktop and OrbStack bind sources are translated host paths, so clear message. On Docker Desktop and OrbStack bind sources are translated host paths, so
Compose jobs fail there; that is a development-only limitation. Compose jobs fail there; that is a development-only limitation.
## Self-update
The master's own Compose service and each Compose agent's own service are updated like
any other service; the overview marks the agent's row "This agent".
- **Master.** The agent on the master's host runs the job. The `up -d` recreates the
master, which ends the connection and any webhook request that waits for the job. The
agent keeps the result and sends it again when the new master is up, so the job is
recorded as finished. A client that waited gets an error and has to look at the job
afterwards.
- **Agent.** `up -d` would stop the agent that runs it, so the agent starts a short-lived
helper container from its own image (same mounts) and the helper runs `docker compose
up -d <service>`. The helper carries `se.shcizo.package-updater.self-update.*` labels
with the job ID, the pinned ref and the output so far. When the new agent starts it
finds the helper by those labels, waits for it, adds its logs to the job output, removes
it and reports the result under the old job ID. The old agent reports nothing after it
is stopped.
- **Requirements.** The agent's own stack directory has to be mounted read-write at the
same path (see below), like any stack it updates, and the agent must speak protocol 4
(`agent too old to update itself` otherwise). The master and the agent need to be
upgraded to a version with protocol 4 by hand once.
- **Order.** A job for an agent's own service runs after every other queued job on that
agent. A master and an agent on the same host share an image, so one push starts both
jobs, and the master's job has to finish before the agent replaces itself. The master
also starts a job for an agent's own service after the other jobs of the same poll
cycle or webhook request. Any other job that reaches an agent while it replaces itself
is marked lost when the new agent connects without it.
- **Docker socket.** The helper runs on the default network, so `DOCKER_HOST` only works
with a `docker.sock` bind mount. A TCP socket proxy that is reachable only on a Compose
network does not work for self-update.
- **Swarm.** An agent's own Swarm service is refused: the agent cannot update the service
it runs in.
## How Compose updates work, and their limits ## How Compose updates work, and their limits
The agent never edits your files. It writes one file next to the stack's base file, The agent never edits your files. It writes one file next to the stack's base file,
+5
View File
@@ -64,6 +64,11 @@ func run() error {
defer stop() defer stop()
logMounts(ctx, dockerCli, cfg.SelfID, logger) logMounts(ctx, dockerCli, cfg.SelfID, logger)
// Before the runner and the connection start: the first hello must already
// list the jobs an earlier agent process began.
if err := updater.ResumeSelfUpdates(ctx, dockerCli, cfg.SelfID, runner, logger); err != nil {
logger.Warn("self_update_resume_failed", "error", err)
}
go runner.Run(ctx) go runner.Run(ctx)
go watchLoop(ctx, dockerCli, agent, logger) go watchLoop(ctx, dockerCli, agent, logger)
logger.Info("agent_starting", "version", version, "master_url", cfg.MasterURL, logger.Info("agent_starting", "version", version, "master_url", cfg.MasterURL,
+11 -10
View File
@@ -115,7 +115,17 @@ func run() error {
} }
auth := webauth.New(provider, st, cfg.AllowedGroup, strings.HasPrefix(cfg.PublicURL, "https://"), logger) auth := webauth.New(provider, st, cfg.AllowedGroup, strings.HasPrefix(cfg.PublicURL, "https://"), logger)
ui, err := web.New(st, agentHub, jobService, broker, web.Config{ credDir, credFound, credReadErr := registryCredentialsDir()
logger.Info("registry_credentials", "docker_config", credDir, "config_json_found", credFound,
"config_json_readable", credFound && credReadErr == nil)
if credReadErr != nil {
logger.Warn("registry_credentials_unreadable", "docker_config", credDir, "error", credReadErr)
}
registryPoller := poller.New(st, jobService, registry.New("package-updater-master/"+version, logger), broker, poller.Config{
Interval: cfg.PollInterval, FirstDelay: pollFirstDelay, Workers: pollWorkers,
}, logger)
ui, err := web.New(st, agentHub, jobService, broker, registryPoller, web.Config{
AgentURL: cfg.AgentURL, MasterKey: fingerprint, AgentImage: cfg.AgentImage, AgentURL: cfg.AgentURL, MasterKey: fingerprint, AgentImage: cfg.AgentImage,
PollInterval: cfg.PollInterval, Version: version, PollInterval: cfg.PollInterval, Version: version,
}, logger) }, logger)
@@ -164,17 +174,8 @@ func run() error {
BaseContext: baseContext, BaseContext: baseContext,
} }
credDir, credFound, credReadErr := registryCredentialsDir()
logger.Info("registry_credentials", "docker_config", credDir, "config_json_found", credFound,
"config_json_readable", credFound && credReadErr == nil)
if credReadErr != nil {
logger.Warn("registry_credentials_unreadable", "docker_config", credDir, "error", credReadErr)
}
pollCtx, stopPolling := context.WithCancel(ctx) pollCtx, stopPolling := context.WithCancel(ctx)
defer stopPolling() defer stopPolling()
registryPoller := poller.New(st, jobService, registry.New("package-updater-master/"+version, logger), broker, poller.Config{
Interval: cfg.PollInterval, FirstDelay: pollFirstDelay, Workers: pollWorkers,
}, logger)
pollDone := make(chan struct{}) pollDone := make(chan struct{})
go func() { go func() {
defer close(pollDone) defer close(pollDone)
+1
View File
@@ -16,4 +16,5 @@ services:
volumes: volumes:
- /var/run/docker.sock:/var/run/docker.sock - /var/run/docker.sock:/var/run/docker.sock
# - /path/to/stacks:/path/to/stacks # mount each stacks directory at the same path, read-write # - /path/to/stacks:/path/to/stacks # mount each stacks directory at the same path, read-write
# (also this agent's own stack directory, or it cannot update itself)
# - /root/.docker/config.json:/docker-config/config.json:ro # - /root/.docker/config.json:/docker-config/config.json:ro
@@ -229,13 +229,46 @@ Online/offline is held in memory only.
| Master restarts mid-job | Jobs in `running` are reconciled when the agent reconnects; jobs the agent does not know are marked `failed` ("lost"). | | Master restarts mid-job | Jobs in `running` are reconciled when the agent reconnects; jobs the agent does not know are marked `failed` ("lost"). |
| New trigger while a job for the same service is queued or running | No new job; only `available_*` is updated. Two jobs never race on one service. | | New trigger while a job for the same service is queued or running | No new job; only `available_*` is updated. Two jobs never race on one service. |
| Job exceeds `UPDATE_TIMEOUT` | The agent cancels it, as today. | | Job exceeds `UPDATE_TIMEOUT` | The agent cancels it, as today. |
| Agent updates itself | The old agent runs `up -d` in a helper container and waits. If the helper exits while the old agent still runs, the old agent reports the outcome. If the helper stops the old agent, the old agent records nothing; the new agent finds the helper by its labels before it connects, so its hello lists the job, and it reports the helper's outcome. If no new agent resumes the job, it is marked lost when the agent reconnects without it. If `UPDATE_TIMEOUT` passes first, the old agent reports a timeout and leaves the helper running; a later success is then not recorded. |
| Polling fails for one image | Error stored in `last_poll_error` and shown; other images continue. | | Polling fails for one image | Error stored in `last_poll_error` and shown; other images continue. |
### Self-protection ### Self-update
The agent reports its own container and the master knows its own service. Both are shown The agent reports its own container with `self`; the overview shows "This agent". The
as "updated manually outside the system", and the master refuses to create jobs for master's own Compose service and every Compose agent's own service are normal update
them. An agent recreating itself would kill the job it is running. targets. A Swarm agent's own service is refused ("an agent's own Swarm service cannot
update itself").
- **Master.** No special execution. The agent on the master's host runs the job; the
recreate drops its connection, and it sends the result again after the new master is up
(unacknowledged results are resent).
- **Agent.** The old agent does the usual checks, writes the override and pulls. Only
`up -d <service>` runs in a helper container made from the agent's own image, with the
agent's own bind and volume mounts and no Compose labels. The helper's labels
(`se.shcizo.package-updater.self-update.` plus `job`, `ref`, `started`, `project`,
`service`, `output`) hold what the new agent needs: the job ID, the pinned ref, the
start time and the output so far (tail, 32 KiB). If the helper exits while the old agent
still runs, the old agent copies its logs into the job output, removes it and reports
the outcome itself.
- **Resume.** At startup, before the runner and the connection start, the agent lists
helpers whose project and service labels match its own container and registers each as
a job at the front of its queue. The job is then in `known_jobs` of the first hello.
Its result is a normal `job.result`: output, helper logs, pinned ref on exit 0,
otherwise an error. A helper with unusable labels fails the job with a message saying so.
- **Stopping.** When the agent is stopped (SIGTERM) while a job runs, the runner drops the
job and records no result, so a "context canceled" result cannot overtake the new
agent's.
- **Order.** `job.start` carries `self`; the agent runs a job for its own service only
when no other job is queued (the master's job on a shared host has to finish first).
The master starts such jobs after the other jobs of the same poll cycle or webhook
request; messages to one agent arrive in order, so the others are queued first. A job
that reaches an agent while it replaces itself is marked lost when the new agent
connects without it. Whether to use a helper is decided by the agent from its own
container ID, not from the flag.
- **Protocol 4.** The master routes a job for an agent's own service, and any job for a
service with the `AGENT_IMAGE` repo (an older agent does not report itself), only to an
agent speaking protocol 4 (`agent too old to update itself`). The helper runs on the
default network, so `DOCKER_HOST` must be a `docker.sock` bind mount.
## Registry polling ## Registry polling
@@ -255,6 +288,11 @@ them. An agent recreating itself would kill the job it is running.
HEAD-only and never downloads a manifest. HEAD-only and never downloads a manifest.
- Credentials: a Docker `config.json` (as written by `docker login`) mounted into the - Credentials: a Docker `config.json` (as written by `docker login`) mounted into the
master. No credential UI, no encryption in Postgres. master. No credential UI, no encryption in Postgres.
- Newer version tags (e.g. `gitea:1.27.3` → `1.27.4`): for a followed tag with at least
two numeric components the poller lists the repo's tags once per cycle
(`GET /v2/<repo>/tags/list`) and stores the highest tag with the same prefix, suffix,
component count and major number in `services.newer_ref`. A suggestion only, never
deployed automatically; deploying it is the existing version pick.
## Webhook ## Webhook
@@ -355,7 +393,7 @@ Each phase ends with something that works on its own.
snapshot and live events, Postgres storage, OIDC login, Overview and Agents pages. snapshot and live events, Postgres storage, OIDC login, Overview and Agents pages.
The overview works; nothing can be updated yet. The overview works; nothing can be updated yet.
2. **Jobs and manual update.** Job protocol, agent queue and executors (Compose override 2. **Jobs and manual update.** Job protocol, agent queue and executors (Compose override
file, Swarm with digest), job history, Service and Jobs pages, self-protection. file, Swarm with digest), job history, Service and Jobs pages.
3. **Webhook and auto mode.** `POST /update` on the master, matching, waiting for 3. **Webhook and auto mode.** `POST /update` on the master, matching, waiting for
auto jobs, mode toggle. auto jobs, mode toggle.
4. **Polling and version history.** `image_versions`, registry polling, version picker 4. **Polling and version history.** `image_versions`, registry polling, version picker
@@ -364,14 +402,9 @@ Each phase ends with something that works on its own.
## Out of scope (later specs) ## Out of scope (later specs)
- Detecting newer version *tags* for images we do not build (e.g. `gitea:1.27.3` →
`1.27.4`, `postgres:17.10`, `loki:3.4.2`). Requires listing tags and ordering them;
major upgrades such as Postgres 17 → 18 must never be automatic. Revisit after
checking the other two servers' stack files.
- Notifications. - Notifications.
- Export/import of a fleet configuration to restore a setup. The `services` table - Export/import of a fleet configuration to restore a setup. The `services` table
already holds what an export needs. already holds what an export needs.
- Writing version bumps back to git. - Writing version bumps back to git.
- Self-update of master and agents (`internal/selfupdate` is the starting point).
- Updating plain (non-Compose, non-Swarm) containers. - Updating plain (non-Compose, non-Swarm) containers.
- Credential management UI for registries. - Credential management UI for registries.
+1 -1
View File
@@ -10,6 +10,7 @@ require (
github.com/google/go-containerregistry v0.22.1 github.com/google/go-containerregistry v0.22.1
github.com/jackc/pgx/v5 v5.10.0 github.com/jackc/pgx/v5 v5.10.0
github.com/opencontainers/go-digest v1.0.0 github.com/opencontainers/go-digest v1.0.0
github.com/opencontainers/image-spec v1.1.1
github.com/stretchr/testify v1.11.1 github.com/stretchr/testify v1.11.1
golang.org/x/oauth2 v0.36.0 golang.org/x/oauth2 v0.36.0
gopkg.in/yaml.v3 v3.0.1 gopkg.in/yaml.v3 v3.0.1
@@ -38,7 +39,6 @@ require (
github.com/moby/sys/atomicwriter v0.1.0 // indirect github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect github.com/moby/term v0.5.2 // indirect
github.com/morikuni/aec v1.1.0 // indirect github.com/morikuni/aec v1.1.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pkg/errors v0.9.1 // indirect github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect github.com/sirupsen/logrus v1.9.4 // indirect
+52 -5
View File
@@ -28,6 +28,8 @@ const (
labelConfigFiles = "com.docker.compose.project.config_files" labelConfigFiles = "com.docker.compose.project.config_files"
labelEnvFile = "com.docker.compose.project.environment_file" labelEnvFile = "com.docker.compose.project.environment_file"
labelSwarmTask = "com.docker.swarm.service.id" labelSwarmTask = "com.docker.swarm.service.id"
labelSwarmName = "com.docker.swarm.service.name"
labelSelfUpdateJob = "se.shcizo.package-updater.self-update.job"
) )
// DockerClient is the subset of the Docker SDK the collector needs. // DockerClient is the subset of the Docker SDK the collector needs.
@@ -61,6 +63,7 @@ type Collector struct {
now func() time.Time now func() time.Time
mu sync.Mutex mu sync.Mutex
self *selfInfo // the agent's own container, once inspected
cache map[string]buildEntry // by Compose project cache map[string]buildEntry // by Compose project
warned map[string]bool // logged warnings: "<project>" (check failed) and "<project>/<service>" (service not in config) warned map[string]bool // logged warnings: "<project>" (check failed) and "<project>/<service>" (service not in config)
} }
@@ -77,8 +80,15 @@ type buildEntry struct {
// stackLabels are the Compose labels a build check needs. // stackLabels are the Compose labels a build check needs.
type stackLabels struct{ workingDir, configFiles, envFile string } type stackLabels struct{ workingDir, configFiles, envFile string }
// New returns a Collector. selfID is a prefix of the agent's own container // selfInfo is what the inspect of the agent's own container tells.
// ID (inside a container, the hostname); that container is marked Self. type selfInfo struct {
id string // full container ID
swarmService string // name of the Swarm service the container is a task of, if any
}
// New returns a Collector. selfID is the agent's own container: a prefix of
// its ID (inside a container, the hostname) or its name. That container is
// marked Self, and so is the Swarm service it is a task of.
func New(cli DockerClient, selfID string) *Collector { func New(cli DockerClient, selfID string) *Collector {
return &Collector{cli: cli, selfID: selfID, now: time.Now, cache: map[string]buildEntry{}, warned: map[string]bool{}} return &Collector{cli: cli, selfID: selfID, now: time.Now, cache: map[string]buildEntry{}, warned: map[string]bool{}}
} }
@@ -92,6 +102,29 @@ func (c *Collector) WithBuildCheck(check BuildCheck, log *slog.Logger) *Collecto
return c return c
} }
// resolveSelf inspects the agent's own container once and remembers the
// answer. It returns nil while that fails (or selfID is empty), and the
// caller then compares container IDs with the selfID prefix.
func (c *Collector) resolveSelf(ctx context.Context) *selfInfo {
if c.selfID == "" {
return nil
}
c.mu.Lock()
defer c.mu.Unlock()
if c.self != nil {
return c.self
}
insp, err := c.cli.ContainerInspect(ctx, c.selfID)
if err != nil || insp.ContainerJSONBase == nil || insp.ID == "" {
return nil
}
c.self = &selfInfo{id: insp.ID}
if insp.Config != nil {
c.self.swarmService = insp.Config.Labels[labelSwarmName]
}
return c.self
}
// Hello returns the machine facts sent on connect. AgentVersion is left for // Hello returns the machine facts sent on connect. AgentVersion is left for
// the caller to fill in. // the caller to fill in.
func (c *Collector) Hello(ctx context.Context) (protocol.Hello, error) { func (c *Collector) Hello(ctx context.Context) (protocol.Hello, error) {
@@ -125,17 +158,21 @@ func (c *Collector) Snapshot(ctx context.Context) ([]protocol.Service, error) {
repoDigests[img.ID] = img.RepoDigests repoDigests[img.ID] = img.RepoDigests
} }
self := c.resolveSelf(ctx)
byKey := make(map[string]protocol.Service) byKey := make(map[string]protocol.Service)
stacks := make(map[string]stackLabels) stacks := make(map[string]stackLabels)
for _, ct := range containers { for _, ct := range containers {
if _, isTask := ct.Labels[labelSwarmTask]; isTask { if _, isTask := ct.Labels[labelSwarmTask]; isTask {
continue continue
} }
if _, isHelper := ct.Labels[labelSelfUpdateJob]; isHelper {
continue
}
ref, err := c.imageRef(ctx, ct) ref, err := c.imageRef(ctx, ct)
if err != nil { if err != nil {
return nil, err return nil, err
} }
svc := c.fromContainer(ct, ref, repoDigests[ct.ImageID]) svc := c.fromContainer(ct, ref, repoDigests[ct.ImageID], self)
if svc.Kind == protocol.KindCompose { if svc.Kind == protocol.KindCompose {
if _, ok := stacks[svc.ComposeProject]; !ok { if _, ok := stacks[svc.ComposeProject]; !ok {
stacks[svc.ComposeProject] = stackLabels{svc.WorkingDir, svc.ConfigFiles, ct.Labels[labelEnvFile]} stacks[svc.ComposeProject] = stackLabels{svc.WorkingDir, svc.ConfigFiles, ct.Labels[labelEnvFile]}
@@ -159,6 +196,7 @@ func (c *Collector) Snapshot(ctx context.Context) ([]protocol.Service, error) {
continue continue
} }
svc := fromSwarmService(s) svc := fromSwarmService(s)
svc.Self = self != nil && self.swarmService != "" && svc.SwarmService == self.swarmService
byKey[svc.Key()] = svc byKey[svc.Key()] = svc
} }
} }
@@ -189,12 +227,12 @@ func (c *Collector) imageRef(ctx context.Context, ct container.Summary) (string,
return insp.Config.Image, nil return insp.Config.Image, nil
} }
func (c *Collector) fromContainer(ct container.Summary, ref string, repoDigests []string) protocol.Service { func (c *Collector) fromContainer(ct container.Summary, ref string, repoDigests []string, self *selfInfo) protocol.Service {
svc := protocol.Service{ svc := protocol.Service{
Image: ref, Image: ref,
Digest: digestFor(ref, repoDigests), Digest: digestFor(ref, repoDigests),
Status: containerStatus(ct.State), Status: containerStatus(ct.State),
Self: c.selfID != "" && strings.HasPrefix(ct.ID, c.selfID), Self: c.isSelf(ct.ID, self),
} }
project, service := ct.Labels[labelProject], ct.Labels[labelService] project, service := ct.Labels[labelProject], ct.Labels[labelService]
if project != "" && service != "" { if project != "" && service != "" {
@@ -213,6 +251,15 @@ func (c *Collector) fromContainer(ct container.Summary, ref string, repoDigests
return svc return svc
} }
// isSelf reports whether id is the agent's own container: by the inspected
// ID when there is one, else by the selfID prefix.
func (c *Collector) isSelf(id string, self *selfInfo) bool {
if self != nil {
return id == self.id
}
return c.selfID != "" && strings.HasPrefix(id, c.selfID)
}
func fromSwarmService(s swarm.Service) protocol.Service { func fromSwarmService(s swarm.Service) protocol.Service {
ref := s.Spec.TaskTemplate.ContainerSpec.Image ref := s.Spec.TaskTemplate.ContainerSpec.Image
return protocol.Service{ return protocol.Service{
@@ -135,6 +135,19 @@ func TestSnapshot_PlainContainerAndSwarmTaskSkipped(t *testing.T) {
}}, got) }}, got)
} }
func TestSnapshot_SelfUpdateHelperSkipped(t *testing.T) {
cli := &fakeDocker{containers: []container.Summary{
{ID: "h1", Names: []string{"/quirky_helper"}, Image: "sha256:agentimage", State: "running",
Labels: map[string]string{labelSelfUpdateJob: "7"}},
{ID: "p1", Names: []string{"/adhoc"}, Image: "redis:alpine", State: "exited"},
}}
got, err := New(cli, "").Snapshot(context.Background())
require.NoError(t, err)
assert.Equal(t, []protocol.Service{{
Kind: protocol.KindContainer, ContainerName: "adhoc", Image: "redis:alpine", Status: "stopped",
}}, got)
}
func TestSnapshot_SelfFlag(t *testing.T) { func TestSnapshot_SelfFlag(t *testing.T) {
cli := &fakeDocker{containers: []container.Summary{ cli := &fakeDocker{containers: []container.Summary{
composeContainer("0123456789abcdef", "updater-agent", "agent", "pu:1", "sha256:i", "running"), composeContainer("0123456789abcdef", "updater-agent", "agent", "pu:1", "sha256:i", "running"),
@@ -145,6 +158,63 @@ func TestSnapshot_SelfFlag(t *testing.T) {
assert.True(t, got[0].Self) assert.True(t, got[0].Self)
} }
func TestSnapshot_SelfFlagByContainerName(t *testing.T) {
cli := &fakeDocker{
containers: []container.Summary{
composeContainer("0123456789abcdef", "updater-agent", "agent", "pu:1", "sha256:i", "running"),
composeContainer("fedcba9876543210", "slusk", "web", "slusk:1", "sha256:j", "running"),
},
inspect: map[string]container.InspectResponse{
"updater-agent-agent-1": {ContainerJSONBase: &container.ContainerJSONBase{ID: "0123456789abcdef"}},
},
}
got, err := New(cli, "updater-agent-agent-1").Snapshot(context.Background())
require.NoError(t, err)
require.Len(t, got, 2)
assert.False(t, got[0].Self, "slusk")
assert.True(t, got[1].Self, "updater-agent")
}
func TestSnapshot_SelfFlagFallsBackToIDPrefixWhenInspectFails(t *testing.T) {
cli := &fakeDocker{
containers: []container.Summary{composeContainer("0123456789abcdef", "updater-agent", "agent", "pu:1", "sha256:i", "running")},
inspectErr: errors.New("boom"),
}
got, err := New(cli, "0123456789ab").Snapshot(context.Background())
require.NoError(t, err)
require.Len(t, got, 1)
assert.True(t, got[0].Self)
}
func TestSnapshot_AgentsOwnSwarmServiceIsSelf(t *testing.T) {
swarmSvc := func(name string) swarm.Service {
return swarm.Service{
Spec: swarm.ServiceSpec{
Annotations: swarm.Annotations{Name: name},
TaskTemplate: swarm.TaskSpec{ContainerSpec: &swarm.ContainerSpec{Image: "pu:1"}},
},
ServiceStatus: &swarm.ServiceStatus{RunningTasks: 1, DesiredTasks: 1},
}
}
cli := &fakeDocker{
info: managerInfo(),
services: []swarm.Service{swarmSvc("pu_agent"), swarmSvc("pu_master")},
inspect: map[string]container.InspectResponse{
"abc123": {
ContainerJSONBase: &container.ContainerJSONBase{ID: "abc123full"},
Config: &container.Config{Labels: map[string]string{labelSwarmName: "pu_agent"}},
},
},
}
got, err := New(cli, "abc123").Snapshot(context.Background())
require.NoError(t, err)
require.Len(t, got, 2)
assert.Equal(t, "pu_agent", got[0].SwarmService)
assert.True(t, got[0].Self)
assert.Equal(t, "pu_master", got[1].SwarmService)
assert.False(t, got[1].Self)
}
func TestSnapshot_SwarmServicesOnlyOnManager(t *testing.T) { func TestSnapshot_SwarmServicesOnlyOnManager(t *testing.T) {
svc := swarm.Service{ svc := swarm.Service{
Spec: swarm.ServiceSpec{ Spec: swarm.ServiceSpec{
+39 -10
View File
@@ -6,13 +6,12 @@ import (
"io" "io"
"os" "os"
"os/exec" "os/exec"
"slices"
"strings" "strings"
"time" "time"
"github.com/docker/docker/api/types/container" "github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/filters" "github.com/docker/docker/api/types/filters"
"github.com/shcizo/package-updater/internal/protocol"
) )
const ( const (
@@ -71,6 +70,7 @@ func dockerEnv() []string {
// ComposeDocker is the subset of the Docker SDK ComposeExecutor needs. // ComposeDocker is the subset of the Docker SDK ComposeExecutor needs.
type ComposeDocker interface { type ComposeDocker interface {
SelfInspector SelfInspector
HelperDocker
ContainerList(ctx context.Context, opts container.ListOptions) ([]container.Summary, error) ContainerList(ctx context.Context, opts container.ListOptions) ([]container.Summary, error)
} }
@@ -95,7 +95,10 @@ func NewComposeExecutor(cli ComposeDocker, resolve *Resolver, selfID string, run
// is), writes it to the stack's override file and recreates the service. // is), writes it to the stack's override file and recreates the service.
// Working directory and config files come from the service's container // Working directory and config files come from the service's container
// labels on this machine, never from the job. It returns the pinned ref. // labels on this machine, never from the job. It returns the pinned ref.
func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) { //
// When the service is the agent's own container, the recreate runs in a
// helper container instead (see selfUpdate), because it stops this process.
func (e *ComposeExecutor) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
project, service := job.ComposeProject, job.ComposeService project, service := job.ComposeProject, job.ComposeService
cts, err := e.cli.ContainerList(ctx, container.ListOptions{ cts, err := e.cli.ContainerList(ctx, container.ListOptions{
All: true, All: true,
@@ -110,14 +113,24 @@ func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, ou
if len(cts) == 0 { if len(cts) == 0 {
return "", fmt.Errorf("no container for compose service %s/%s on this machine", project, service) return "", fmt.Errorf("no container for compose service %s/%s on this machine", project, service)
} }
self, err := inspectSelf(ctx, e.cli, e.selfID)
if err != nil {
return "", err
}
isSelf := self.ContainerJSONBase != nil && self.ID != "" &&
slices.ContainsFunc(cts, func(ct container.Summary) bool { return ct.ID == self.ID })
// What an update of the agent's own service has printed so far goes into
// the helper's labels.
var soFar *tailBuffer
if isSelf {
soFar = &tailBuffer{max: selfUpdateOutputMax}
out = io.MultiWriter(out, tailWriter{soFar})
}
workingDir := cts[0].Labels[labelWorkingDir] workingDir := cts[0].Labels[labelWorkingDir]
configFiles := cts[0].Labels[labelConfigFiles] configFiles := cts[0].Labels[labelConfigFiles]
envFileLabel := cts[0].Labels[labelEnvFile] envFileLabel := cts[0].Labels[labelEnvFile]
roots, err := MountRoots(ctx, e.cli, e.selfID) roots := rootsOf(self)
if err != nil {
return "", err
}
files, dir, err := composeFiles(roots, configFiles) files, dir, err := composeFiles(roots, configFiles)
if err != nil { if err != nil {
return "", err return "", err
@@ -163,13 +176,29 @@ func (e *ComposeExecutor) Execute(ctx context.Context, job protocol.JobStart, ou
} }
base := composeBase(project, workingDir, files, envs) base := composeBase(project, workingDir, files, envs)
for _, step := range [][]string{{"pull", service}, {"up", "-d", service}} { run := func(args []string) error {
args := append(append([]string(nil), base...), step...)
fmt.Fprintf(out, "$ docker %s\n", strings.Join(args, " ")) fmt.Fprintf(out, "$ docker %s\n", strings.Join(args, " "))
tail := &tailBuffer{max: errorOutputTail} tail := &tailBuffer{max: errorOutputTail}
if err := e.run(ctx, workingDir, io.MultiWriter(out, tailWriter{tail}), args...); err != nil { if err := e.run(ctx, workingDir, io.MultiWriter(out, tailWriter{tail}), args...); err != nil {
return "", fmt.Errorf("docker compose %s: %w (output: %s)", step[0], err, strings.TrimSpace(tail.String())) return composeError(args[len(base)], err, tail)
} }
return nil
}
if err := run(append(append([]string(nil), base...), "pull", service)); err != nil {
return "", err
}
up := append(append([]string(nil), base...), "up", "-d", service)
if isSelf {
return e.selfUpdate(ctx, job, self, workingDir, up, pinned, out, soFar)
}
if err := run(up); err != nil {
return "", err
} }
return pinned, nil return pinned, nil
} }
// composeError describes a failed `docker compose <step>` call with the tail
// of its output.
func composeError(step string, err error, tail *tailBuffer) error {
return fmt.Errorf("docker compose %s: %w (output: %s)", step, err, strings.TrimSpace(tail.String()))
}
+260 -1
View File
@@ -1,6 +1,7 @@
package updater package updater
import ( import (
"bytes"
"context" "context"
"errors" "errors"
"io" "io"
@@ -8,9 +9,14 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"time"
"unicode/utf8"
"github.com/docker/docker/api/types/container" "github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/mount" "github.com/docker/docker/api/types/mount"
"github.com/docker/docker/api/types/network"
"github.com/docker/docker/pkg/stdcopy"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -26,6 +32,123 @@ func (f fakeCompose) ContainerList(context.Context, container.ListOptions) ([]co
return f.containers, nil return f.containers, nil
} }
// fakeCompose is not meant to start helper containers; these fail the test
// run if an executor tries.
func (fakeCompose) ContainerCreate(context.Context, *container.Config, *container.HostConfig, *network.NetworkingConfig, *ocispec.Platform, string) (container.CreateResponse, error) {
panic("unexpected ContainerCreate")
}
func (fakeCompose) ContainerStart(context.Context, string, container.StartOptions) error {
panic("unexpected ContainerStart")
}
func (fakeCompose) ContainerWait(context.Context, string, container.WaitCondition) (<-chan container.WaitResponse, <-chan error) {
panic("unexpected ContainerWait")
}
func (fakeCompose) ContainerLogs(context.Context, string, container.LogsOptions) (io.ReadCloser, error) {
panic("unexpected ContainerLogs")
}
func (fakeCompose) ContainerRemove(context.Context, string, container.RemoveOptions) error {
panic("unexpected ContainerRemove")
}
// createdHelper is what a fakeHelper was asked to create.
type createdHelper struct {
cfg *container.Config
host *container.HostConfig
}
// fakeHelper is a helper container the test controls. Its logs are framed
// like Docker's multiplexed stream.
type fakeHelper struct {
createErr error
exit int64
hold bool // ContainerWait never reports an exit
logs string // what the helper printed
onStart func()
state container.ContainerState // what an inspect reports; exited when empty
created []createdHelper
started []string
removed []string
waitCond container.WaitCondition
}
func (h *fakeHelper) ContainerCreate(_ context.Context, cfg *container.Config, host *container.HostConfig, _ *network.NetworkingConfig, _ *ocispec.Platform, _ string) (container.CreateResponse, error) {
if h.createErr != nil {
return container.CreateResponse{}, h.createErr
}
h.created = append(h.created, createdHelper{cfg, host})
return container.CreateResponse{ID: "helper1"}, nil
}
func (h *fakeHelper) ContainerInspect(context.Context, string) (container.InspectResponse, error) {
state := h.state
if state == "" {
state = container.StateExited
}
return container.InspectResponse{ContainerJSONBase: &container.ContainerJSONBase{State: &container.State{Status: state}}}, nil
}
func (h *fakeHelper) ContainerStart(_ context.Context, id string, _ container.StartOptions) error {
h.started = append(h.started, id)
if h.onStart != nil {
h.onStart()
}
return nil
}
func (h *fakeHelper) ContainerWait(_ context.Context, _ string, cond container.WaitCondition) (<-chan container.WaitResponse, <-chan error) {
h.waitCond = cond
res, errs := make(chan container.WaitResponse, 1), make(chan error, 1)
if !h.hold {
res <- container.WaitResponse{StatusCode: h.exit}
}
return res, errs
}
func (h *fakeHelper) ContainerLogs(context.Context, string, container.LogsOptions) (io.ReadCloser, error) {
var buf bytes.Buffer
stdcopy.NewStdWriter(&buf, stdcopy.Stdout).Write([]byte(h.logs))
return io.NopCloser(&buf), nil
}
func (h *fakeHelper) ContainerRemove(_ context.Context, id string, opts container.RemoveOptions) error {
if !opts.Force {
return errors.New("remove without force")
}
h.removed = append(h.removed, id)
return nil
}
// fakeHelperCompose is a fakeCompose whose helper containers are h.
type fakeHelperCompose struct {
fakeCompose
h *fakeHelper
}
// ContainerInspect answers for the agent's own container ("me") like fakeCompose
// and for any other ID like the helper.
func (f fakeHelperCompose) ContainerInspect(ctx context.Context, id string) (container.InspectResponse, error) {
if id == "me" {
return f.fakeCompose.ContainerInspect(ctx, id)
}
return f.h.ContainerInspect(ctx, id)
}
func (f fakeHelperCompose) ContainerCreate(ctx context.Context, cfg *container.Config, host *container.HostConfig, n *network.NetworkingConfig, p *ocispec.Platform, name string) (container.CreateResponse, error) {
return f.h.ContainerCreate(ctx, cfg, host, n, p, name)
}
func (f fakeHelperCompose) ContainerStart(ctx context.Context, id string, o container.StartOptions) error {
return f.h.ContainerStart(ctx, id, o)
}
func (f fakeHelperCompose) ContainerWait(ctx context.Context, id string, c container.WaitCondition) (<-chan container.WaitResponse, <-chan error) {
return f.h.ContainerWait(ctx, id, c)
}
func (f fakeHelperCompose) ContainerLogs(ctx context.Context, id string, o container.LogsOptions) (io.ReadCloser, error) {
return f.h.ContainerLogs(ctx, id, o)
}
func (f fakeHelperCompose) ContainerRemove(ctx context.Context, id string, o container.RemoveOptions) error {
return f.h.ContainerRemove(ctx, id, o)
}
// outRunner is a fake OutputRunner for `docker compose config`. // outRunner is a fake OutputRunner for `docker compose config`.
type outRunner struct { type outRunner struct {
json string json string
@@ -70,7 +193,7 @@ func TestComposeExecutor(t *testing.T) {
stack := filepath.Join(root, "slusk") stack := filepath.Join(root, "slusk")
require.NoError(t, os.MkdirAll(stack, 0o755)) require.NoError(t, os.MkdirAll(stack, 0o755))
composeFile := stackFile(t, stack) composeFile := stackFile(t, stack)
job := protocol.JobStart{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", Ref: "gitea.shcizo.se/shcizo/slusk:1.4"} job := Job{ID: "7", Started: time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC), JobStart: protocol.JobStart{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", Ref: "gitea.shcizo.se/shcizo/slusk:1.4"}}
pinned := job.Ref + "@" + d pinned := job.Ref + "@" + d
ct := container.Summary{Labels: map[string]string{ ct := container.Summary{Labels: map[string]string{
@@ -319,3 +442,139 @@ func TestExecDocker_UsesMinimalEnv(t *testing.T) {
assert.NotContains(t, string(raw), "AGENT_TOKEN") assert.NotContains(t, string(raw), "AGENT_TOKEN")
assert.Contains(t, string(raw), "DOCKER_CONFIG=/docker-config") assert.Contains(t, string(raw), "DOCKER_CONFIG=/docker-config")
} }
func TestComposeSelfUpdate(t *testing.T) {
const d = "sha256:d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1d1"
root, _ := filepath.EvalSymlinks(t.TempDir())
stack := filepath.Join(root, "updater")
require.NoError(t, os.MkdirAll(stack, 0o755))
stackFile(t, stack)
job := Job{ID: "7", Started: time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC), JobStart: protocol.JobStart{
Kind: protocol.KindCompose, ComposeProject: "updater", ComposeService: "agent", Ref: "gitea.shcizo.se/shcizo/package-updater:1.4", Self: true}}
pinned := job.Ref + "@" + d
overridePath := filepath.Join(stack, OverrideName)
t.Cleanup(func() { os.Remove(overridePath) })
selfCt := container.Summary{ID: "selfid", Labels: map[string]string{
labelWorkingDir: stack,
labelConfigFiles: filepath.Join(stack, "compose.yaml"),
}}
mounts := []container.MountPoint{
{Type: mount.TypeBind, Source: root, Destination: root, RW: true},
{Type: mount.TypeBind, Source: "/var/run/docker.sock", Destination: "/var/run/docker.sock", RW: true},
{Type: mount.TypeBind, Source: "/root/.docker/config.json", Destination: "/docker-config/config.json", RW: false},
{Type: mount.TypeVolume, Name: "agent-data", Source: "/var/lib/docker/volumes/agent-data/_data", Destination: "/data", RW: true},
{Type: mount.TypeTmpfs, Destination: "/tmp", RW: true},
}
self := fakeSelf{id: "selfid", image: "sha256:agentimage", mounts: mounts}
wantMounts := []mount.Mount{
{Type: mount.TypeBind, Source: root, Target: root},
{Type: mount.TypeBind, Source: "/var/run/docker.sock", Target: "/var/run/docker.sock"},
{Type: mount.TypeBind, Source: "/root/.docker/config.json", Target: "/docker-config/config.json", ReadOnly: true},
{Type: mount.TypeVolume, Source: "agent-data", Target: "/data"},
}
base := []string{"compose", "-p", "updater", "--project-directory", stack,
"-f", filepath.Join(stack, "compose.yaml"), "-f", overridePath}
withStep := func(step ...string) []string { return append(append([]string(nil), base...), step...) }
t.Setenv("PATH", "/usr/bin")
t.Setenv("DOCKER_CONFIG", "/docker-config")
t.Setenv("AGENT_TOKEN", "secret")
cases := []struct {
name string
helper *fakeHelper
cancelled bool // ctx is cancelled when the helper starts
wantErr string
wantCancel bool
wantRemoved []string
wantOut []string
}{
{name: "helper succeeds", helper: &fakeHelper{logs: "Container updater-agent-1 Recreated\n"},
wantRemoved: []string{"helper1"}, wantOut: []string{"Container updater-agent-1 Recreated"}},
{name: "helper fails", helper: &fakeHelper{exit: 1, logs: "network not found\n"},
wantErr: "docker compose up: exit status 1 (output: network not found)", wantRemoved: []string{"helper1"},
wantOut: []string{"network not found"}},
{name: "ctx ends while waiting", helper: &fakeHelper{hold: true}, cancelled: true, wantCancel: true},
{name: "create fails", helper: &fakeHelper{createErr: errors.New("no such image")}, wantErr: "create update helper: no such image"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
t.Cleanup(func() { os.Remove(overridePath) })
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
if c.cancelled {
c.helper.onStart = cancel
}
rec := &recorder{}
cli := fakeHelperCompose{fakeCompose{self, []container.Summary{selfCt}}, c.helper}
e := NewComposeExecutor(cli, NewResolver(&fakeDist{digest: d}, nil), "me", rec.run, (&outRunner{json: `{"services":{"agent":{"image":"x"}}}`}).run)
var out strings.Builder
got, err := e.Execute(ctx, job, &out)
require.Len(t, rec.cmds, 1, "pull ran, up did not")
assert.Equal(t, ranCmd{stack, withStep("pull", "agent")}, rec.cmds[0])
assert.Equal(t, c.wantRemoved, c.helper.removed)
for _, s := range c.wantOut {
assert.Contains(t, out.String(), s)
}
switch {
case c.wantCancel:
assert.ErrorIs(t, err, context.Canceled)
case c.wantErr != "":
assert.EqualError(t, err, c.wantErr)
assert.Empty(t, got)
default:
require.NoError(t, err)
assert.Equal(t, pinned, got)
}
if len(c.helper.created) == 0 {
return
}
cfg, host := c.helper.created[0].cfg, c.helper.created[0].host
assert.Equal(t, "sha256:agentimage", cfg.Image)
assert.Equal(t, []string{"docker"}, []string(cfg.Entrypoint))
assert.Equal(t, withStep("up", "-d", "agent"), []string(cfg.Cmd))
assert.Equal(t, stack, cfg.WorkingDir)
assert.Equal(t, dockerEnv(), cfg.Env)
assert.Equal(t, wantMounts, host.Mounts)
assert.False(t, host.AutoRemove)
assert.Equal(t, container.RestartPolicy{}, host.RestartPolicy)
assert.Equal(t, []string{"helper1"}, c.helper.started)
assert.Equal(t, container.WaitConditionNextExit, c.helper.waitCond)
labels := cfg.Labels
assert.Len(t, labels, 6, "no compose labels")
assert.Equal(t, "7", labels[labelSelfUpdateJob])
assert.Equal(t, pinned, labels[labelSelfUpdateRef])
assert.Equal(t, "2026-10-13T09:00:00Z", labels[labelSelfUpdateStarted])
assert.Equal(t, "updater", labels[labelSelfUpdateProject])
assert.Equal(t, "agent", labels[labelSelfUpdateService])
assert.Contains(t, labels[labelSelfUpdateOutput], "resolved "+pinned)
assert.Contains(t, labels[labelSelfUpdateOutput], "$ docker "+strings.Join(withStep("pull", "agent"), " "))
})
}
t.Run("a service that is not the agent's own never uses a helper", func(t *testing.T) {
t.Cleanup(func() { os.Remove(overridePath) })
other := self
other.id = "otherid" // fakeCompose panics on ContainerCreate
rec := &recorder{}
e := NewComposeExecutor(fakeCompose{other, []container.Summary{selfCt}}, NewResolver(&fakeDist{digest: d}, nil), "me", rec.run,
(&outRunner{json: `{"services":{"agent":{"image":"x"}}}`}).run)
got, err := e.Execute(context.Background(), job, io.Discard)
require.NoError(t, err)
assert.Equal(t, pinned, got)
require.Len(t, rec.cmds, 2)
assert.Equal(t, withStep("up", "-d", "agent"), rec.cmds[1].args)
})
t.Run("output label keeps the tail and says it was cut", func(t *testing.T) {
b := &tailBuffer{max: 4}
b.Write([]byte("abcdef"))
assert.Equal(t, truncatedMarker+"cdef", outputLabel(b))
b2 := &tailBuffer{max: 3}
b2.Write([]byte("aéé")) // the cut falls inside a character
assert.True(t, utf8.ValidString(outputLabel(b2)))
})
}
+1 -1
View File
@@ -16,7 +16,7 @@ func NewDispatch(compose, swarm Executor) Executor {
return dispatch{compose: compose, swarm: swarm} return dispatch{compose: compose, swarm: swarm}
} }
func (d dispatch) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) { func (d dispatch) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
switch job.Kind { switch job.Kind {
case protocol.KindCompose: case protocol.KindCompose:
return d.compose.Execute(ctx, job, out) return d.compose.Execute(ctx, job, out)
+6 -6
View File
@@ -12,20 +12,20 @@ import (
) )
func TestDispatch(t *testing.T) { func TestDispatch(t *testing.T) {
compose := execFunc(func(context.Context, protocol.JobStart, io.Writer) (string, error) { return "compose", nil }) compose := execFunc(func(context.Context, Job, io.Writer) (string, error) { return "compose", nil })
swarm := execFunc(func(context.Context, protocol.JobStart, io.Writer) (string, error) { return "swarm", nil }) swarm := execFunc(func(context.Context, Job, io.Writer) (string, error) { return "swarm", nil })
d := NewDispatch(compose, swarm) d := NewDispatch(compose, swarm)
got, err := d.Execute(context.Background(), protocol.JobStart{Kind: protocol.KindCompose}, io.Discard) got, err := d.Execute(context.Background(), Job{JobStart: protocol.JobStart{Kind: protocol.KindCompose}}, io.Discard)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "compose", got) assert.Equal(t, "compose", got)
got, err = d.Execute(context.Background(), protocol.JobStart{Kind: protocol.KindSwarm}, io.Discard) got, err = d.Execute(context.Background(), Job{JobStart: protocol.JobStart{Kind: protocol.KindSwarm}}, io.Discard)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "swarm", got) assert.Equal(t, "swarm", got)
_, err = d.Execute(context.Background(), protocol.JobStart{Kind: protocol.KindContainer}, io.Discard) _, err = d.Execute(context.Background(), Job{JobStart: protocol.JobStart{Kind: protocol.KindContainer}}, io.Discard)
assert.ErrorContains(t, err, "cannot update") assert.ErrorContains(t, err, "cannot update")
_, err = d.Execute(context.Background(), protocol.JobStart{Kind: "weird"}, io.Discard) _, err = d.Execute(context.Background(), Job{JobStart: protocol.JobStart{Kind: "weird"}}, io.Discard)
assert.ErrorContains(t, err, "cannot update") assert.ErrorContains(t, err, "cannot update")
} }
+18 -3
View File
@@ -177,20 +177,35 @@ var errNoRoots = errors.New("refused: no stack directory is mounted into the age
// path inside the container. Compose labels hold host paths, so only those // path inside the container. Compose labels hold host paths, so only those
// mounts make a label path usable as is. // mounts make a label path usable as is.
func MountRoots(ctx context.Context, cli SelfInspector, selfID string) ([]string, error) { func MountRoots(ctx context.Context, cli SelfInspector, selfID string) ([]string, error) {
insp, err := inspectSelf(ctx, cli, selfID)
if err != nil {
return nil, err
}
return rootsOf(insp), nil
}
// inspectSelf inspects the agent's own container.
func inspectSelf(ctx context.Context, cli SelfInspector, selfID string) (container.InspectResponse, error) {
if selfID == "" { if selfID == "" {
return nil, errNoRoots return container.InspectResponse{}, errNoRoots
} }
insp, err := cli.ContainerInspect(ctx, selfID) insp, err := cli.ContainerInspect(ctx, selfID)
if err != nil { if err != nil {
return nil, fmt.Errorf("inspect own container %s: %w", selfID, err) return container.InspectResponse{}, fmt.Errorf("inspect own container %s: %w", selfID, err)
} }
return insp, nil
}
// rootsOf returns the directories MountRoots describes, from an inspected
// container.
func rootsOf(insp container.InspectResponse) []string {
var roots []string var roots []string
for _, m := range insp.Mounts { for _, m := range insp.Mounts {
if m.Type == mount.TypeBind && m.RW && filepath.Clean(m.Source) == filepath.Clean(m.Destination) { if m.Type == mount.TypeBind && m.RW && filepath.Clean(m.Source) == filepath.Clean(m.Destination) {
roots = append(roots, filepath.Clean(m.Destination)) roots = append(roots, filepath.Clean(m.Destination))
} }
} }
return roots, nil return roots
} }
// composeFiles returns the -f arguments for a job: the stack's config files // composeFiles returns the -f arguments for a job: the stack's config files
+11 -1
View File
@@ -175,10 +175,20 @@ func TestWriteOverride(t *testing.T) {
type fakeSelf struct { type fakeSelf struct {
mounts []container.MountPoint mounts []container.MountPoint
err error err error
id string // the container's ID, when set
image string // the container's image ID, when set
labels map[string]string // the container's labels, when set
} }
func (f fakeSelf) ContainerInspect(context.Context, string) (container.InspectResponse, error) { func (f fakeSelf) ContainerInspect(context.Context, string) (container.InspectResponse, error) {
return container.InspectResponse{Mounts: f.mounts}, f.err insp := container.InspectResponse{Mounts: f.mounts}
if f.id != "" || f.image != "" {
insp.ContainerJSONBase = &container.ContainerJSONBase{ID: f.id, Image: f.image}
}
if f.labels != nil {
insp.Config = &container.Config{Labels: f.labels}
}
return insp, f.err
} }
func TestMountRoots(t *testing.T) { func TestMountRoots(t *testing.T) {
+84 -16
View File
@@ -16,12 +16,25 @@ import (
// chatty job cannot grow a single job.progress message without bound. // chatty job cannot grow a single job.progress message without bound.
const maxProgress = 32 << 10 const maxProgress = 32 << 10
// Job is a job as an Executor sees it: what the master asked for, plus the
// ID and start time the runner gave it. A self-update records both in the
// helper container, so the new agent can finish the job under the same ID.
type Job struct {
protocol.JobStart
ID string
Started time.Time
}
// Executor deploys one job and returns the "repo:tag@digest" it deployed. // Executor deploys one job and returns the "repo:tag@digest" it deployed.
// Everything it writes to out is streamed to the master as job output. // Everything it writes to out is streamed to the master as job output.
type Executor interface { type Executor interface {
Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) Execute(ctx context.Context, job Job, out io.Writer) (string, error)
} }
// ResumeFunc finishes a job that an earlier agent process began. It returns
// the "repo:tag@digest" that was deployed, like Executor.Execute.
type ResumeFunc func(ctx context.Context, out io.Writer) (string, error)
// Progress is output a running job produced since the previous TakeProgress. // Progress is output a running job produced since the previous TakeProgress.
type Progress struct{ ID, Output string } type Progress struct{ ID, Output string }
@@ -52,6 +65,8 @@ func (b *tailBuffer) Reset() { b.buf, b.truncated = nil, false }
type jobState struct { type jobState struct {
start protocol.JobStart start protocol.JobStart
resume ResumeFunc // set for a job resumed after a restart; start is then empty
started time.Time // set with resume
running bool running bool
result *protocol.JobResult result *protocol.JobResult
output tailBuffer // everything, capped at protocol.MaxJobOutput output tailBuffer // everything, capped at protocol.MaxJobOutput
@@ -117,6 +132,28 @@ func (r *Runner) Submit(id string, start protocol.JobStart) {
signal(r.wake) signal(r.wake)
} }
// Resume registers a job that an earlier agent process began and runs fn for
// it, before every job that is already queued. The job is in Known at once,
// so the agent's next hello does not report it lost. The result carries
// started as its start time. A job whose ID the runner already holds is
// ignored.
func (r *Runner) Resume(id string, started time.Time, fn ResumeFunc) {
r.mu.Lock()
if _, ok := r.jobs[id]; ok {
r.mu.Unlock()
return
}
r.jobs[id] = &jobState{
resume: fn,
started: started,
output: tailBuffer{max: protocol.MaxJobOutput},
unsent: tailBuffer{max: maxProgress},
}
r.queue = append([]string{id}, r.queue...)
r.mu.Unlock()
signal(r.wake)
}
// Ack forgets a finished job. Acks for unknown, queued or running jobs are // Ack forgets a finished job. Acks for unknown, queued or running jobs are
// ignored. // ignored.
func (r *Runner) Ack(id string) { func (r *Runner) Ack(id string) {
@@ -196,10 +233,12 @@ func (r *Runner) Pending() []Finished {
func (r *Runner) Changed() <-chan struct{} { return r.changed } func (r *Runner) Changed() <-chan struct{} { return r.changed }
// Run works through the queue until ctx ends. Jobs still queued then are // Run works through the queue until ctx ends. Jobs still queued then are
// dropped; the master marks them lost on the agent's next hello. // dropped; the master marks them lost on the agent's next hello. A job that
// ctx interrupts is dropped too, without a result: a self-updating agent is
// stopped that way, and its successor reports the outcome.
func (r *Runner) Run(ctx context.Context) { func (r *Runner) Run(ctx context.Context) {
for { for {
id, start, ok := r.next() id, ok := r.next()
if !ok { if !ok {
select { select {
case <-ctx.Done(): case <-ctx.Done():
@@ -211,32 +250,62 @@ func (r *Runner) Run(ctx context.Context) {
if ctx.Err() != nil { if ctx.Err() != nil {
return return
} }
r.runOne(ctx, id, start) r.runOne(ctx, id)
} }
} }
// next pops the next queued job and marks it running. // next pops the next job to run and marks it running. Jobs for the agent's
func (r *Runner) next() (string, protocol.JobStart, bool) { // own service run last: a master and an agent on the same host share an
// image, and the master's job must finish before the agent replaces itself.
func (r *Runner) next() (string, bool) {
r.mu.Lock() r.mu.Lock()
defer r.mu.Unlock() defer r.mu.Unlock()
if len(r.queue) == 0 { if len(r.queue) == 0 {
return "", protocol.JobStart{}, false return "", false
} }
id := r.queue[0] pick := 0
r.queue = r.queue[1:] for i, id := range r.queue {
j := r.jobs[id] if !r.jobs[id].start.Self {
j.running = true pick = i
break
}
}
id := r.queue[pick]
r.queue = append(r.queue[:pick:pick], r.queue[pick+1:]...)
r.jobs[id].running = true
signal(r.changed) signal(r.changed)
return id, j.start, true return id, true
} }
func (r *Runner) runOne(ctx context.Context, id string, start protocol.JobStart) { func (r *Runner) runOne(ctx context.Context, id string) {
started := r.now() r.mu.Lock()
j := r.jobs[id]
start, resume, started := j.start, j.resume, j.started
r.mu.Unlock()
if resume == nil {
started = r.now()
}
jobCtx, cancel := context.WithTimeout(ctx, r.timeout) jobCtx, cancel := context.WithTimeout(ctx, r.timeout)
ref, err := r.exec.Execute(jobCtx, start, &jobWriter{r: r, id: id}) var ref string
var err error
if resume != nil {
ref, err = resume(jobCtx, &jobWriter{r: r, id: id})
} else {
ref, err = r.exec.Execute(jobCtx, Job{JobStart: start, ID: id, Started: started}, &jobWriter{r: r, id: id})
}
timedOut := errors.Is(jobCtx.Err(), context.DeadlineExceeded) timedOut := errors.Is(jobCtx.Err(), context.DeadlineExceeded)
cancel() cancel()
if err != nil && ctx.Err() != nil && !timedOut {
// The agent is stopping, not the job failing. Nothing is recorded, so
// no result can pre-empt the one a successor agent may report.
r.mu.Lock()
delete(r.jobs, id)
r.mu.Unlock()
return
}
res := protocol.JobResult{Status: protocol.ResultSucceeded, Ref: ref, StartedAt: started, FinishedAt: r.now()} res := protocol.JobResult{Status: protocol.ResultSucceeded, Ref: ref, StartedAt: started, FinishedAt: r.now()}
switch { switch {
case timedOut: case timedOut:
@@ -246,7 +315,6 @@ func (r *Runner) runOne(ctx context.Context, id string, start protocol.JobStart)
} }
r.mu.Lock() r.mu.Lock()
j := r.jobs[id]
res.Output, res.OutputTruncated = j.output.String(), j.output.Truncated() res.Output, res.OutputTruncated = j.output.String(), j.output.Truncated()
j.running, j.result, j.finished = false, &res, res.FinishedAt j.running, j.result, j.finished = false, &res, res.FinishedAt
j.unsent.Reset() j.unsent.Reset()
+109 -7
View File
@@ -29,7 +29,7 @@ func newFakeExec() *fakeExec {
return &fakeExec{gate: make(chan struct{}), started: make(chan string, 10)} return &fakeExec{gate: make(chan struct{}), started: make(chan string, 10)}
} }
func (f *fakeExec) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) { func (f *fakeExec) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
n := f.cur.Add(1) n := f.cur.Add(1)
defer f.cur.Add(-1) defer f.cur.Add(-1)
for { for {
@@ -148,7 +148,7 @@ func TestRunner_TakeProgress(t *testing.T) {
func TestRunner_TakeProgressReportsStartedJobOnce(t *testing.T) { func TestRunner_TakeProgressReportsStartedJobOnce(t *testing.T) {
gate := make(chan struct{}) gate := make(chan struct{})
started := make(chan struct{}) started := make(chan struct{})
r := startRunner(t, execFunc(func(ctx context.Context, _ protocol.JobStart, _ io.Writer) (string, error) { r := startRunner(t, execFunc(func(ctx context.Context, _ Job, _ io.Writer) (string, error) {
close(started) close(started)
<-gate <-gate
return "x", nil return "x", nil
@@ -160,14 +160,14 @@ func TestRunner_TakeProgressReportsStartedJobOnce(t *testing.T) {
close(gate) close(gate)
} }
type execFunc func(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) type execFunc func(ctx context.Context, job Job, out io.Writer) (string, error)
func (f execFunc) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) { func (f execFunc) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
return f(ctx, job, out) return f(ctx, job, out)
} }
func TestRunner_Timeout(t *testing.T) { func TestRunner_Timeout(t *testing.T) {
r := startRunner(t, execFunc(func(ctx context.Context, _ protocol.JobStart, _ io.Writer) (string, error) { r := startRunner(t, execFunc(func(ctx context.Context, _ Job, _ io.Writer) (string, error) {
<-ctx.Done() <-ctx.Done()
return "", ctx.Err() return "", ctx.Err()
}), 20*time.Millisecond) }), 20*time.Millisecond)
@@ -178,7 +178,7 @@ func TestRunner_Timeout(t *testing.T) {
} }
func TestRunner_OutputCapped(t *testing.T) { func TestRunner_OutputCapped(t *testing.T) {
r := startRunner(t, execFunc(func(_ context.Context, _ protocol.JobStart, out io.Writer) (string, error) { r := startRunner(t, execFunc(func(_ context.Context, _ Job, out io.Writer) (string, error) {
io.WriteString(out, strings.Repeat("a", 10)) io.WriteString(out, strings.Repeat("a", 10))
io.WriteString(out, strings.Repeat("b", protocol.MaxJobOutput)) io.WriteString(out, strings.Repeat("b", protocol.MaxJobOutput))
return "x", nil return "x", nil
@@ -190,7 +190,7 @@ func TestRunner_OutputCapped(t *testing.T) {
} }
func TestRunner_ChangedSignals(t *testing.T) { func TestRunner_ChangedSignals(t *testing.T) {
r := startRunner(t, execFunc(func(context.Context, protocol.JobStart, io.Writer) (string, error) { return "x", nil }), time.Minute) r := startRunner(t, execFunc(func(context.Context, Job, io.Writer) (string, error) { return "x", nil }), time.Minute)
r.Submit("1", protocol.JobStart{Ref: "a:1"}) r.Submit("1", protocol.JobStart{Ref: "a:1"})
select { select {
case <-r.Changed(): case <-r.Changed():
@@ -198,3 +198,105 @@ func TestRunner_ChangedSignals(t *testing.T) {
t.Fatal("no change signal") t.Fatal("no change signal")
} }
} }
func TestRunner_ResumeJumpsTheQueue(t *testing.T) {
fe := newFakeExec()
r := NewRunner(fe, time.Minute)
r.Submit("1", protocol.JobStart{Ref: "a:1"})
started := time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC)
r.Resume("9", started, func(_ context.Context, out io.Writer) (string, error) {
io.WriteString(out, "from the helper\n")
return "resumed@sha256:bb", nil
})
assert.Equal(t, []string{"1", "9"}, r.Known(), "known before the worker starts")
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
wg.Add(1)
go func() { defer wg.Done(); r.Run(ctx) }()
t.Cleanup(func() { cancel(); wg.Wait() })
got := waitPending(t, r, 1)
require.Len(t, got, 1, "the resumed job runs first; job 1 waits on the gate")
assert.Equal(t, "9", got[0].ID)
assert.Equal(t, protocol.ResultSucceeded, got[0].Result.Status)
assert.Equal(t, "resumed@sha256:bb", got[0].Result.Ref)
assert.Equal(t, "from the helper\n", got[0].Result.Output)
assert.True(t, started.Equal(got[0].Result.StartedAt))
r.Resume("9", started, nil) // already held: ignored
assert.Equal(t, "a:1", <-fe.started)
fe.gate <- struct{}{}
waitPending(t, r, 2)
}
func TestRunner_ResumedJobFailure(t *testing.T) {
r := startRunner(t, newFakeExec(), time.Minute)
r.Resume("9", time.Now(), func(context.Context, io.Writer) (string, error) { return "", errors.New("helper failed") })
got := waitPending(t, r, 1)
assert.Equal(t, protocol.ResultFailed, got[0].Result.Status)
assert.Equal(t, "helper failed", got[0].Result.Error)
}
func TestRunner_SelfJobsRunLast(t *testing.T) {
fe := newFakeExec()
r := NewRunner(fe, time.Minute)
r.Submit("1", protocol.JobStart{Ref: "self:1", Self: true})
r.Submit("2", protocol.JobStart{Ref: "a:2"})
r.Submit("3", protocol.JobStart{Ref: "self:3", Self: true})
r.Submit("4", protocol.JobStart{Ref: "a:4"})
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
wg.Add(1)
go func() { defer wg.Done(); r.Run(ctx) }()
t.Cleanup(func() { cancel(); wg.Wait() })
var order []string
for i := 0; i < 4; i++ {
order = append(order, <-fe.started)
fe.gate <- struct{}{}
}
assert.Equal(t, []string{"a:2", "a:4", "self:1", "self:3"}, order)
}
func TestRunner_SelfJobRunsWhenAlone(t *testing.T) {
fe := newFakeExec()
r := startRunner(t, fe, time.Minute)
r.Submit("1", protocol.JobStart{Ref: "self:1", Self: true})
assert.Equal(t, "self:1", <-fe.started)
fe.gate <- struct{}{}
waitPending(t, r, 1)
}
func TestRunner_StoppingTheAgentRecordsNoResult(t *testing.T) {
r := NewRunner(execFunc(func(ctx context.Context, _ Job, _ io.Writer) (string, error) {
<-ctx.Done()
return "", ctx.Err()
}), time.Minute)
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { r.Run(ctx); close(done) }()
r.Submit("1", protocol.JobStart{Ref: "a:1"})
require.Eventually(t, func() bool { return len(r.TakeProgress()) > 0 }, time.Second, 5*time.Millisecond)
cancel()
<-done
assert.Empty(t, r.Pending(), "no \"context canceled\" result")
assert.Empty(t, r.Known(), "the job is dropped, so the master marks it lost unless a successor resumes it")
}
func TestRunner_ExecutorGetsIDAndStart(t *testing.T) {
var got Job
r := startRunner(t, execFunc(func(_ context.Context, job Job, _ io.Writer) (string, error) {
got = job
return "x", nil
}), time.Minute)
now := time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC)
r.now = func() time.Time { return now }
r.Submit("5", protocol.JobStart{Ref: "a:1"})
waitPending(t, r, 1)
assert.Equal(t, "5", got.ID)
assert.Equal(t, "a:1", got.Ref)
assert.True(t, now.Equal(got.Started))
}
+279
View File
@@ -0,0 +1,279 @@
package updater
import (
"context"
"errors"
"fmt"
"io"
"log/slog"
"strings"
"time"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/filters"
"github.com/docker/docker/api/types/mount"
"github.com/docker/docker/api/types/network"
"github.com/docker/docker/pkg/stdcopy"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
// Labels on a self-update helper container. The new agent finds the helper
// by them, and the inventory skips containers that carry the job label.
const (
labelSelfUpdateJob = "se.shcizo.package-updater.self-update.job"
labelSelfUpdateRef = "se.shcizo.package-updater.self-update.ref"
labelSelfUpdateStarted = "se.shcizo.package-updater.self-update.started"
labelSelfUpdateProject = "se.shcizo.package-updater.self-update.project"
labelSelfUpdateService = "se.shcizo.package-updater.self-update.service"
labelSelfUpdateOutput = "se.shcizo.package-updater.self-update.output"
)
// selfUpdateOutputMax caps the job output stored in the helper container's
// output label. The old agent's output so far is all the new agent has of it.
const selfUpdateOutputMax = 32 << 10
// truncatedMarker starts the stored output when its head was cut off.
const truncatedMarker = "…truncated\n"
// resumeSeparator follows the old agent's output in a resumed job's output.
const resumeSeparator = "--- agent restarted; output of the update helper ---\n"
// HelperDocker is the subset of the Docker SDK the self-update helper
// container needs. *client.Client satisfies it.
type HelperDocker interface {
ContainerInspect(ctx context.Context, containerID string) (container.InspectResponse, error)
ContainerCreate(ctx context.Context, config *container.Config, hostConfig *container.HostConfig, networkingConfig *network.NetworkingConfig, platform *ocispec.Platform, containerName string) (container.CreateResponse, error)
ContainerStart(ctx context.Context, containerID string, options container.StartOptions) error
ContainerWait(ctx context.Context, containerID string, condition container.WaitCondition) (<-chan container.WaitResponse, <-chan error)
ContainerLogs(ctx context.Context, containerID string, options container.LogsOptions) (io.ReadCloser, error)
ContainerRemove(ctx context.Context, containerID string, options container.RemoveOptions) error
}
// selfUpdate runs `up -d` for the agent's own service in a short-lived helper
// container. Recreating the service stops this process, so the helper, made
// from the agent's own image with the agent's own mounts, does it and
// outlives the agent. args are the complete `docker` arguments of the call.
//
// If the helper exits while this agent still runs, selfUpdate copies its logs
// into the job output, removes it and returns the outcome. If ctx ends first
// (the agent is being replaced, or the job timed out) it returns ctx's error
// and leaves the helper to the new agent, see ResumeSelfUpdates.
func (e *ComposeExecutor) selfUpdate(ctx context.Context, job Job, self container.InspectResponse, workingDir string, args []string, pinned string, out io.Writer, soFar *tailBuffer) (string, error) {
fmt.Fprintf(out, "$ docker %s (in an update helper container)\n", strings.Join(args, " "))
cfg := &container.Config{
Image: self.Image,
Entrypoint: []string{"docker"},
Cmd: args,
WorkingDir: workingDir,
Env: dockerEnv(),
Labels: map[string]string{
labelSelfUpdateJob: job.ID,
labelSelfUpdateRef: pinned,
labelSelfUpdateStarted: job.Started.UTC().Format(time.RFC3339Nano),
labelSelfUpdateProject: job.ComposeProject,
labelSelfUpdateService: job.ComposeService,
labelSelfUpdateOutput: outputLabel(soFar),
},
}
created, err := e.cli.ContainerCreate(ctx, cfg, &container.HostConfig{Mounts: helperMounts(self)}, nil, nil, "")
if err != nil {
return "", fmt.Errorf("create update helper: %w", err)
}
// Wait is registered before the start so a fast exit is not missed.
waitCh, errCh := e.cli.ContainerWait(ctx, created.ID, container.WaitConditionNextExit)
if err := e.cli.ContainerStart(ctx, created.ID, container.StartOptions{}); err != nil {
if rmErr := e.cli.ContainerRemove(context.WithoutCancel(ctx), created.ID, container.RemoveOptions{Force: true}); rmErr != nil {
fmt.Fprintf(out, "warning: remove the update helper: %v\n", rmErr)
}
return "", fmt.Errorf("start update helper: %w", err)
}
code, err := awaitExit(ctx, waitCh, errCh)
if err != nil {
return "", err
}
tail := &tailBuffer{max: errorOutputTail}
if err := finishHelper(ctx, e.cli, created.ID, io.MultiWriter(out, tailWriter{tail})); err != nil {
fmt.Fprintf(out, "warning: %v\n", err)
}
if code != 0 {
return "", exitError(code, tail)
}
return pinned, nil
}
// outputLabel returns the job output so far as a label value.
func outputLabel(b *tailBuffer) string {
s := strings.ToValidUTF8(b.String(), "")
if b.Truncated() {
s = truncatedMarker + s
}
return s
}
// helperMounts copies the agent's own bind and volume mounts: the Docker
// socket, the stack directories and the registry credentials the compose call
// needs are all among them.
func helperMounts(self container.InspectResponse) []mount.Mount {
var mounts []mount.Mount
for _, m := range self.Mounts {
switch m.Type {
case mount.TypeBind:
mounts = append(mounts, mount.Mount{Type: mount.TypeBind, Source: m.Source, Target: m.Destination, ReadOnly: !m.RW})
case mount.TypeVolume:
mounts = append(mounts, mount.Mount{Type: mount.TypeVolume, Source: m.Name, Target: m.Destination, ReadOnly: !m.RW})
}
}
return mounts
}
// awaitExit waits for a ContainerWait result and returns the exit code. When
// ctx ends it returns ctx's error.
func awaitExit(ctx context.Context, waitCh <-chan container.WaitResponse, errCh <-chan error) (int64, error) {
select {
case r := <-waitCh:
if r.Error != nil {
return 0, fmt.Errorf("wait for update helper: %s", r.Error.Message)
}
return r.StatusCode, nil
case err := <-errCh:
if ctx.Err() != nil {
return 0, ctx.Err()
}
return 0, fmt.Errorf("wait for update helper: %w", err)
case <-ctx.Done():
return 0, ctx.Err()
}
}
// finishHelper copies the exited helper's logs to w and removes the helper.
// Both are attempted; the first failure is returned.
func finishHelper(ctx context.Context, cli HelperDocker, id string, w io.Writer) error {
var first error
rc, err := cli.ContainerLogs(ctx, id, container.LogsOptions{ShowStdout: true, ShowStderr: true})
if err == nil {
_, err = stdcopy.StdCopy(w, w, rc)
rc.Close()
}
if err != nil {
first = fmt.Errorf("read the update helper's logs: %w", err)
}
if err := cli.ContainerRemove(ctx, id, container.RemoveOptions{Force: true}); err != nil && first == nil {
first = fmt.Errorf("remove the update helper: %w", err)
}
return first
}
// exitError describes a failed helper like a failed `docker compose` call.
func exitError(code int64, tail *tailBuffer) error {
return composeError("up", fmt.Errorf("exit status %d", code), tail)
}
// ResumeDocker is the subset of the Docker SDK ResumeSelfUpdates needs.
// *client.Client satisfies it.
type ResumeDocker interface {
SelfInspector
HelperDocker
ContainerList(ctx context.Context, opts container.ListOptions) ([]container.Summary, error)
}
// ResumeSelfUpdates finishes the jobs a previous agent process started with
// selfUpdate. It finds the helper containers made for this agent's own
// Compose service and registers each with runner.Resume, which reports the
// job under its old ID once the helper has exited. Call it before the runner
// and the connection start, so the first hello already lists these jobs. A
// helper with unusable labels is resumed too; its job fails with an error
// that says so.
func ResumeSelfUpdates(ctx context.Context, cli ResumeDocker, selfID string, runner *Runner, log *slog.Logger) error {
self, err := inspectSelf(ctx, cli, selfID)
if err != nil {
return err
}
if self.Config == nil {
return nil
}
project, service := self.Config.Labels[labelComposeProject], self.Config.Labels[labelComposeService]
if project == "" || service == "" {
return nil
}
cts, err := cli.ContainerList(ctx, container.ListOptions{
All: true,
Filters: filters.NewArgs(filters.Arg("label", labelSelfUpdateJob)),
})
if err != nil {
return fmt.Errorf("list update helpers: %w", err)
}
for _, ct := range cts {
if ct.Labels[labelSelfUpdateProject] != project || ct.Labels[labelSelfUpdateService] != service {
continue
}
jobID := ct.Labels[labelSelfUpdateJob]
if jobID == "" {
log.Warn("self_update_helper_without_job", "helper", ct.ID)
continue
}
ref, started, labelErr := parseHelperLabels(ct.Labels)
if started.IsZero() {
started = time.Now()
}
log.Info("self_update_resumed", "job_id", jobID, "helper", ct.ID)
runner.Resume(jobID, started, resumeFunc(cli, ct.ID, ct.Labels[labelSelfUpdateOutput], ref, labelErr))
}
return nil
}
// parseHelperLabels reads the pinned ref and start time from a helper's
// labels. started is zero when it is missing or unreadable, and err then says
// what is wrong.
func parseHelperLabels(labels map[string]string) (ref string, started time.Time, err error) {
ref = labels[labelSelfUpdateRef]
if ref == "" {
err = errors.New("update helper has no ref label")
}
raw := labels[labelSelfUpdateStarted]
if t, perr := time.Parse(time.RFC3339Nano, raw); perr == nil {
started = t
} else if err == nil {
err = fmt.Errorf("update helper has no usable started label %q", raw)
}
return ref, started, err
}
// resumeFunc waits for the helper to stop, then reports the old agent's
// output, a separator line and the helper's own logs as the job output,
// removes the helper and returns the pinned ref, or an error when the helper
// failed or its labels were unusable.
func resumeFunc(cli HelperDocker, helperID, earlier, ref string, labelErr error) ResumeFunc {
return func(ctx context.Context, out io.Writer) (string, error) {
// A helper that was created but never started would read as exited
// with code 0, so it must not be mistaken for a finished update.
if insp, err := cli.ContainerInspect(ctx, helperID); err == nil && insp.ContainerJSONBase != nil &&
insp.State != nil && insp.State.Status == container.StateCreated {
if err := cli.ContainerRemove(context.WithoutCancel(ctx), helperID, container.RemoveOptions{Force: true}); err != nil {
fmt.Fprintf(out, "warning: remove the update helper: %v\n", err)
}
return "", errors.New("update helper was never started")
}
waitCh, errCh := cli.ContainerWait(ctx, helperID, container.WaitConditionNotRunning)
code, err := awaitExit(ctx, waitCh, errCh)
if err != nil {
return "", err
}
io.WriteString(out, earlier)
if earlier != "" && !strings.HasSuffix(earlier, "\n") {
io.WriteString(out, "\n")
}
io.WriteString(out, resumeSeparator)
tail := &tailBuffer{max: errorOutputTail}
if err := finishHelper(ctx, cli, helperID, io.MultiWriter(out, tailWriter{tail})); err != nil {
fmt.Fprintf(out, "warning: %v\n", err)
}
switch {
case labelErr != nil:
return "", labelErr
case code != 0:
return "", exitError(code, tail)
}
return ref, nil
}
}
+116
View File
@@ -0,0 +1,116 @@
package updater
import (
"context"
"io"
"log/slog"
"sync"
"testing"
"time"
"github.com/docker/docker/api/types/container"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/protocol"
)
func TestResumeSelfUpdates(t *testing.T) {
const pinned = "gitea.shcizo.se/shcizo/package-updater:1.4@sha256:aa"
started := time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC)
self := fakeSelf{id: "selfid", labels: map[string]string{labelComposeProject: "updater", labelComposeService: "agent"}}
helper := func(id, project, service string, extra map[string]string) container.Summary {
labels := map[string]string{
labelSelfUpdateJob: "j-" + id,
labelSelfUpdateRef: pinned,
labelSelfUpdateStarted: started.Format(time.RFC3339Nano),
labelSelfUpdateProject: project,
labelSelfUpdateService: service,
labelSelfUpdateOutput: "resolved " + pinned + "\n$ docker compose pull agent\n",
}
for k, v := range extra {
if v == "" {
delete(labels, k)
} else {
labels[k] = v
}
}
return container.Summary{ID: id, Labels: labels}
}
separator := "--- agent restarted; output of the update helper ---\n"
cases := []struct {
name string
helper container.Summary
exit int64
state container.ContainerState
noStarted bool // the old agent's start time is lost with the label
wantStatus string
wantErr string
wantOutput string
}{
{name: "helper succeeded", helper: helper("h1", "updater", "agent", nil), wantStatus: protocol.ResultSucceeded,
wantOutput: "resolved " + pinned + "\n$ docker compose pull agent\n" + separator + "Recreated\n"},
{name: "helper failed", helper: helper("h1", "updater", "agent", nil), exit: 1, wantStatus: protocol.ResultFailed,
wantErr: "docker compose up: exit status 1 (output: Recreated)"},
{name: "no ref label", helper: helper("h1", "updater", "agent", map[string]string{labelSelfUpdateRef: ""}),
wantStatus: protocol.ResultFailed, wantErr: "update helper has no ref label"},
{name: "no started label", helper: helper("h1", "updater", "agent", map[string]string{labelSelfUpdateStarted: ""}),
noStarted: true,
wantStatus: protocol.ResultFailed, wantErr: `update helper has no usable started label ""`},
{name: "helper never started", helper: helper("h1", "updater", "agent", nil), state: container.StateCreated,
wantStatus: protocol.ResultFailed, wantErr: "update helper was never started"},
{name: "no output label", helper: helper("h1", "updater", "agent", map[string]string{labelSelfUpdateOutput: ""}),
wantStatus: protocol.ResultSucceeded, wantOutput: separator + "Recreated\n"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
h := &fakeHelper{exit: c.exit, state: c.state, logs: "Recreated\n"}
unrelated := []container.Summary{
helper("other-project", "other", "agent", nil),
helper("other-service", "updater", "web", nil),
}
cli := fakeHelperCompose{fakeCompose{self, append(unrelated, c.helper)}, h}
r := NewRunner(execFunc(func(context.Context, Job, io.Writer) (string, error) {
t.Error("a resumed job must not reach the executor")
return "", nil
}), time.Minute)
require.NoError(t, ResumeSelfUpdates(context.Background(), cli, "me", r, slog.New(slog.NewTextHandler(io.Discard, nil))))
assert.Equal(t, []string{"j-h1"}, r.Known(), "known before the runner starts, so the first hello lists it")
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
wg.Add(1)
go func() { defer wg.Done(); r.Run(ctx) }()
t.Cleanup(func() { cancel(); wg.Wait() })
got := waitPending(t, r, 1)
require.Len(t, got, 1)
assert.Equal(t, "j-h1", got[0].ID)
res := got[0].Result
assert.Equal(t, c.wantStatus, res.Status)
if c.state != container.StateCreated {
assert.Equal(t, container.WaitConditionNotRunning, h.waitCond)
}
assert.Equal(t, []string{"h1"}, h.removed, "only the matching helper is touched")
if c.wantStatus == protocol.ResultSucceeded {
assert.Equal(t, pinned, res.Ref)
assert.Equal(t, c.wantOutput, res.Output)
} else {
assert.Equal(t, c.wantErr, res.Error)
}
if !c.noStarted {
assert.True(t, started.Equal(res.StartedAt), "the old agent's start time is kept")
}
})
}
t.Run("an agent outside Compose resumes nothing", func(t *testing.T) {
cli := fakeHelperCompose{fakeCompose{fakeSelf{id: "selfid"}, []container.Summary{helper("h1", "updater", "agent", nil)}}, &fakeHelper{}}
r := NewRunner(nil, time.Minute)
require.NoError(t, ResumeSelfUpdates(context.Background(), cli, "me", r, slog.New(slog.NewTextHandler(io.Discard, nil))))
assert.Empty(t, r.Known())
})
}
+1 -2
View File
@@ -9,7 +9,6 @@ import (
"github.com/docker/docker/api/types/system" "github.com/docker/docker/api/types/system"
"github.com/shcizo/package-updater/internal/imageref" "github.com/shcizo/package-updater/internal/imageref"
"github.com/shcizo/package-updater/internal/protocol"
) )
// SwarmClient is the subset of the Docker SDK SwarmExecutor depends on. // SwarmClient is the subset of the Docker SDK SwarmExecutor depends on.
@@ -40,7 +39,7 @@ func NewSwarmExecutor(cli SwarmClient, resolve *Resolver, auth *RegistryAuth) *S
// the service that was inspected, so it changes exactly that object. Success // the service that was inspected, so it changes exactly that object. Success
// means the daemon accepted the new spec, not that the tasks have converged. // means the daemon accepted the new spec, not that the tasks have converged.
// It returns the pinned ref. // It returns the pinned ref.
func (e *SwarmExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) { func (e *SwarmExecutor) Execute(ctx context.Context, job Job, out io.Writer) (string, error) {
info, err := e.cli.Info(ctx) info, err := e.cli.Info(ctx)
if err != nil { if err != nil {
return "", fmt.Errorf("docker info: %w", err) return "", fmt.Errorf("docker info: %w", err)
+2 -2
View File
@@ -79,7 +79,7 @@ func swarmSvc(name string) swarm.Service {
func TestSwarmUsesPinnedRef(t *testing.T) { func TestSwarmUsesPinnedRef(t *testing.T) {
const ref = "gitea/gitea:1.26@sha256:abababababababababababababababababababababababababababababababab" const ref = "gitea/gitea:1.26@sha256:abababababababababababababababababababababababababababababababab"
job := protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: ref} job := Job{JobStart: protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: ref}}
fake := &fakeSwarm{info: managerOf("c1"), svc: swarmSvc("gitea_gitea")} fake := &fakeSwarm{info: managerOf("c1"), svc: swarmSvc("gitea_gitea")}
dist := &fakeDist{digest: "sha256:other"} dist := &fakeDist{digest: "sha256:other"}
var out bytes.Buffer var out bytes.Buffer
@@ -94,7 +94,7 @@ func TestSwarmUsesPinnedRef(t *testing.T) {
} }
func TestSwarmExecutor(t *testing.T) { func TestSwarmExecutor(t *testing.T) {
job := protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: "gitea/gitea:1.27"} job := Job{JobStart: protocol.JobStart{Kind: protocol.KindSwarm, SwarmClusterID: "c1", SwarmService: "gitea_gitea", Ref: "gitea/gitea:1.27"}}
workerInfo := managerOf("c1") workerInfo := managerOf("c1")
workerInfo.Swarm.ControlAvailable = false workerInfo.Swarm.ControlAvailable = false
noContainer := swarmSvc("gitea_gitea") noContainer := swarmSvc("gitea_gitea")
+13
View File
@@ -104,6 +104,19 @@ func (m *memStore) SetLastTagDigest(_ context.Context, id int64, digest string)
return nil return nil
} }
func (m *memStore) SetNewer(_ context.Context, id int64, ref string, at time.Time) error {
m.mu.Lock()
defer m.mu.Unlock()
svc, ok := m.svcRows[id]
if !ok {
return model.ErrNotFound
}
svc.NewerRef = ref
svc.NewerAt = &at
m.svcRows[id] = svc
return nil
}
// seedService adds a service row that jobs can be triggered for. // seedService adds a service row that jobs can be triggered for.
func (m *memStore) seedService(svc model.Service) { func (m *memStore) seedService(svc model.Service) {
m.mu.Lock() m.mu.Lock()
+57 -2
View File
@@ -7,6 +7,7 @@ import (
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
"strconv"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -46,7 +47,7 @@ func newGateExecutor() *gateExecutor {
return &gateExecutor{started: make(chan string, 8), release: make(chan struct{})} return &gateExecutor{started: make(chan string, 8), release: make(chan struct{})}
} }
func (g *gateExecutor) Execute(ctx context.Context, job protocol.JobStart, out io.Writer) (string, error) { func (g *gateExecutor) Execute(ctx context.Context, job updater.Job, out io.Writer) (string, error) {
_, _ = io.WriteString(out, "pulling "+job.Ref+"\n") _, _ = io.WriteString(out, "pulling "+job.Ref+"\n")
g.started <- job.Ref g.started <- job.Ref
select { select {
@@ -69,11 +70,17 @@ type jobAgent struct {
stopRun func() stopRun func()
} }
func startJobAgent(t *testing.T, m testMaster, token string, exec updater.Executor) *jobAgent { // startJobAgent starts an agent. before, if given, runs on the new runner
// before the worker and the connection start, like ResumeSelfUpdates in the
// real agent.
func startJobAgent(t *testing.T, m testMaster, token string, exec updater.Executor, before ...func(*updater.Runner)) *jobAgent {
t.Helper() t.Helper()
src := &staticSource{} src := &staticSource{}
src.set(web) src.set(web)
runner := updater.NewRunner(exec, time.Minute) runner := updater.NewRunner(exec, time.Minute)
for _, f := range before {
f(runner)
}
runCtx, stopRun := context.WithCancel(context.Background()) runCtx, stopRun := context.WithCancel(context.Background())
runDone := make(chan struct{}) runDone := make(chan struct{})
go func() { runner.Run(runCtx); close(runDone) }() go func() { runner.Run(runCtx); close(runDone) }()
@@ -221,6 +228,54 @@ func TestAgentRestartMarksJobLost(t *testing.T) {
assert.Equal(t, lostMessage, lost.Error) assert.Equal(t, lostMessage, lost.Error)
} }
// TestAgentSelfUpdateResumedByNewAgent follows a self-update: the old agent
// dies while its job runs, and the new agent finishes the job under the same
// ID before it says hello, so the master neither loses nor re-runs it.
func TestAgentSelfUpdateResumedByNewAgent(t *testing.T) {
pinned := slusk + ":1.4.2@" + e2eDigest
tests := []struct {
name string
resume updater.ResumeFunc
wantError string
}{
{"helper succeeded", func(_ context.Context, out io.Writer) (string, error) {
_, _ = io.WriteString(out, "helper output\n")
return pinned, nil
}, ""},
{"helper failed", func(context.Context, io.Writer) (string, error) {
return "", errors.New("docker compose up: exit status 1 (output: boom)")
}, "docker compose up: exit status 1 (output: boom)"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
e := newJobEnv(t)
j := e.trigger(t)
e.waitStarted(t)
started := time.Now().Add(-time.Second)
e.agent.stopConn()
e.agent.stopRun()
startJobAgent(t, e.m, "pua_ok", newGateExecutor(), func(r *updater.Runner) {
r.Resume(strconv.FormatInt(j.ID, 10), started, tt.resume)
})
if tt.wantError != "" {
done := e.waitStatus(t, j.ID, model.JobFailed)
assert.Equal(t, tt.wantError, done.Error)
assert.NotEqual(t, lostMessage, done.Error)
return
}
done := e.waitStatus(t, j.ID, model.JobSucceeded)
assert.Empty(t, done.Error)
assert.Equal(t, pinned, done.ToRef)
assert.Contains(t, done.Output, "helper output")
svc, err := e.store.GetService(context.Background(), 1)
require.NoError(t, err)
assert.Equal(t, pinned, svc.DesiredRef)
})
}
}
func TestPhaseOneAgentGetsNoJobs(t *testing.T) { func TestPhaseOneAgentGetsNoJobs(t *testing.T) {
store := &memStore{agents: map[string]model.Agent{agenttoken.Hash("pua_old"): {ID: agentID, Name: "node-2"}}} store := &memStore{agents: map[string]model.Agent{agenttoken.Hash("pua_old"): {ID: agentID, Name: "node-2"}}}
svcRow := model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", svcRow := model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",
+3
View File
@@ -34,6 +34,9 @@ func (f *fixedRegistry) Digest(_ context.Context, ref string) (string, error) {
return f.digests[ref], nil return f.digests[ref], nil
} }
// Tags lists no tags, so no newer tag is ever suggested.
func (f *fixedRegistry) Tags(context.Context, string) ([]string, error) { return nil, nil }
func TestPickPinnedVersion(t *testing.T) { func TestPickPinnedVersion(t *testing.T) {
env := newJobEnv(t) env := newJobEnv(t)
env.store.seedVersion(model.ImageVersion{ID: 5, Repo: slusk, Tag: "1.3.0", Digest: "sha256:old130", Source: model.SourceRunning}) env.store.seedVersion(model.ImageVersion{ID: 5, Repo: slusk, Tag: "1.3.0", Digest: "sha256:old130", Source: model.SourceRunning})
+6 -2
View File
@@ -16,8 +16,9 @@ type candidate struct {
// Route picks the connection a job for svc goes to, among agents that speak // Route picks the connection a job for svc goes to, among agents that speak
// at least protocol version minProtocol. It returns model.ErrAgentOffline or // at least protocol version minProtocol. It returns model.ErrAgentOffline or
// model.ErrAgentTooOld (model.ErrAgentTooOldForPinned when minProtocol asks // model.ErrAgentTooOld (model.ErrAgentTooOldForSelfUpdate when minProtocol
// for a pinned ref) when there is none. // asks for self-update, model.ErrAgentTooOldForPinned when it asks for a
// pinned ref) when there is none.
func (h *Hub) Route(svc model.Service, minProtocol int) (model.Route, error) { func (h *Hub) Route(svc model.Service, minProtocol int) (model.Route, error) {
h.mu.Lock() h.mu.Lock()
conns := make([]candidate, 0, len(h.conns)) conns := make([]candidate, 0, len(h.conns))
@@ -65,6 +66,9 @@ func route(svc model.Service, conns []candidate, minProtocol int) (model.Route,
} }
} }
if tooOld { if tooOld {
if minProtocol >= protocol.MinSelfUpdateVersion {
return model.Route{}, model.ErrAgentTooOldForSelfUpdate
}
if minProtocol >= protocol.MinPinnedRefVersion { if minProtocol >= protocol.MinPinnedRefVersion {
return model.Route{}, model.ErrAgentTooOldForPinned return model.Route{}, model.ErrAgentTooOldForPinned
} }
+12 -1
View File
@@ -14,9 +14,10 @@ func TestRoute(t *testing.T) {
swarmSvc := model.Service{Kind: protocol.KindSwarm, AgentID: ptr(int64(1)), SwarmClusterID: "c1"} swarmSvc := model.Service{Kind: protocol.KindSwarm, AgentID: ptr(int64(1)), SwarmClusterID: "c1"}
v2 := protocol.Hello{ProtocolVersion: 2} v2 := protocol.Hello{ProtocolVersion: 2}
v3 := protocol.Hello{ProtocolVersion: 3} v3 := protocol.Hello{ProtocolVersion: 3}
v4 := protocol.Hello{ProtocolVersion: 4}
mgrV2 := protocol.Hello{ProtocolVersion: 2, SwarmManager: true, SwarmClusterID: "c1"} mgrV2 := protocol.Hello{ProtocolVersion: 2, SwarmManager: true, SwarmClusterID: "c1"}
mgrV3 := protocol.Hello{ProtocolVersion: 3, SwarmManager: true, SwarmClusterID: "c1"} mgrV3 := protocol.Hello{ProtocolVersion: 3, SwarmManager: true, SwarmClusterID: "c1"}
jobs, pinned := protocol.MinJobVersion, protocol.MinPinnedRefVersion jobs, pinned, selfUpdate := protocol.MinJobVersion, protocol.MinPinnedRefVersion, protocol.MinSelfUpdateVersion
tests := []struct { tests := []struct {
name string name string
@@ -47,6 +48,8 @@ func TestRoute(t *testing.T) {
[]candidate{{1, 10, protocol.Hello{ProtocolVersion: 2, SwarmManager: true}}}, jobs, model.Route{}, model.ErrAgentOffline}, []candidate{{1, 10, protocol.Hello{ProtocolVersion: 2, SwarmManager: true}}}, jobs, model.Route{}, model.ErrAgentOffline},
{"swarm, pinned ref falls back to a v3 manager when the reporting one is v2", swarmSvc, {"swarm, pinned ref falls back to a v3 manager when the reporting one is v2", swarmSvc,
[]candidate{{1, 10, mgrV2}, {3, 30, mgrV3}}, pinned, model.Route{AgentID: 3, ConnID: 30}, nil}, []candidate{{1, 10, mgrV2}, {3, 30, mgrV3}}, pinned, model.Route{AgentID: 3, ConnID: 30}, nil},
{"compose, own agent is v3 and the job updates the agent itself", composeSvc, []candidate{{2, 20, v3}}, selfUpdate, model.Route{}, model.ErrAgentTooOldForSelfUpdate},
{"compose, v4 agent and a self-update", composeSvc, []candidate{{2, 20, v4}}, selfUpdate, model.Route{AgentID: 2, ConnID: 20}, nil},
{"swarm, pinned ref and only v2 managers", swarmSvc, []candidate{{1, 10, mgrV2}}, pinned, model.Route{}, model.ErrAgentTooOldForPinned}, {"swarm, pinned ref and only v2 managers", swarmSvc, []candidate{{1, 10, mgrV2}}, pinned, model.Route{}, model.ErrAgentTooOldForPinned},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -57,3 +60,11 @@ func TestRoute(t *testing.T) {
}) })
} }
} }
func TestRoute_TooOldForSelfUpdateIsAlsoTooOld(t *testing.T) {
svc := model.Service{Kind: protocol.KindCompose, AgentID: ptr(int64(2))}
_, err := route(svc, []candidate{{2, 20, protocol.Hello{ProtocolVersion: 3}}}, protocol.MinSelfUpdateVersion)
assert.ErrorIs(t, err, model.ErrAgentTooOldForSelfUpdate)
assert.ErrorIs(t, err, model.ErrAgentTooOld)
assert.EqualError(t, err, "agent too old to update itself")
}
+5 -4
View File
@@ -51,12 +51,13 @@ func TestRestorable(t *testing.T) {
{"never updated", func(s *model.Service) { s.DesiredRef = "" }, model.ReasonNotDrifted}, {"never updated", func(s *model.Service) { s.DesiredRef = "" }, model.ReasonNotDrifted},
{"awaiting inventory", func(s *model.Service) { s.AwaitingInventory = true }, model.ReasonNotDrifted}, {"awaiting inventory", func(s *model.Service) { s.AwaitingInventory = true }, model.ReasonNotDrifted},
{"removed", func(s *model.Service) { s.RemovedAt = ptr(time.Now()) }, model.ReasonRemoved}, {"removed", func(s *model.Service) { s.RemovedAt = ptr(time.Now()) }, model.ReasonRemoved},
{"flagged self", func(s *model.Service) { s.Self = true }, model.ReasonSelf}, {"agent's own Compose service", func(s *model.Service) { s.Self = true }, ""},
{"agent's own Swarm service", func(s *model.Service) { s.Self = true; s.Kind = protocol.KindSwarm }, model.ReasonSelfSwarm},
{"built", func(s *model.Service) { s.Built = true }, model.ReasonBuilt}, {"built", func(s *model.Service) { s.Built = true }, model.ReasonBuilt},
{"container", func(s *model.Service) { s.Kind = protocol.KindContainer }, model.ReasonContainer}, {"container", func(s *model.Service) { s.Kind = protocol.KindContainer }, model.ReasonContainer},
{"desired ref is the agent image", func(s *model.Service) { {"desired ref is the agent image", func(s *model.Service) {
s.DesiredRef = selfImage + "@" + driftD1 s.DesiredRef = "gitea.shcizo.se/shcizo/package-updater:latest@" + driftD1
}, model.ReasonSelf}, }, ""},
} }
s, _ := newService(&fakeStore{}, &fakeHub{}) s, _ := newService(&fakeStore{}, &fakeHub{})
for _, tt := range tests { for _, tt := range tests {
@@ -100,7 +101,7 @@ func TestRestoreDesired_Refusals(t *testing.T) {
reason string reason string
}{ }{
{"not drifted", func(s *model.Service) { s.ActualDigest = driftD1 }, model.ReasonNotDrifted}, {"not drifted", func(s *model.Service) { s.ActualDigest = driftD1 }, model.ReasonNotDrifted},
{"self", func(s *model.Service) { s.Self = true }, model.ReasonSelf}, {"own Swarm service", func(s *model.Service) { s.Self = true; s.Kind = protocol.KindSwarm }, model.ReasonSelfSwarm},
{"built", func(s *model.Service) { s.Built = true }, model.ReasonBuilt}, {"built", func(s *model.Service) { s.Built = true }, model.ReasonBuilt},
{"container", func(s *model.Service) { s.Kind = protocol.KindContainer }, model.ReasonContainer}, {"container", func(s *model.Service) { s.Kind = protocol.KindContainer }, model.ReasonContainer},
} }
+15 -14
View File
@@ -48,8 +48,11 @@ type Service struct {
now func() time.Time now func() time.Time
} }
// New returns a Service. agentImage is the master's AGENT_IMAGE; services // New returns a Service. agentImage is the master's AGENT_IMAGE. A service
// whose image repo equals its repo are the updater itself and are refused. // whose image repo equals its repo may be an agent that reports itself as
// another service (an agent older than protocol version 4 does not tell), so
// such a job is only sent to an agent that speaks protocol version
// protocol.MinSelfUpdateVersion.
func New(store Store, hub Hub, pub Publisher, agentImage string, log *slog.Logger) *Service { func New(store Store, hub Hub, pub Publisher, agentImage string, log *slog.Logger) *Service {
return &Service{ return &Service{
store: store, hub: hub, pub: pub, log: log, now: time.Now, store: store, hub: hub, pub: pub, log: log, now: time.Now,
@@ -102,8 +105,8 @@ func (s *Service) updatable(svc model.Service, explicitRef bool) error {
switch { switch {
case svc.RemovedAt != nil: case svc.RemovedAt != nil:
return &model.NotUpdatableError{Reason: model.ReasonRemoved} return &model.NotUpdatableError{Reason: model.ReasonRemoved}
case svc.Self || imageref.Parse(svc.ActualRef).Repo == s.selfRepo: case svc.Self && svc.Kind == protocol.KindSwarm:
return &model.NotUpdatableError{Reason: model.ReasonSelf} return &model.NotUpdatableError{Reason: model.ReasonSelfSwarm}
case svc.Built: case svc.Built:
return &model.NotUpdatableError{Reason: model.ReasonBuilt} return &model.NotUpdatableError{Reason: model.ReasonBuilt}
case svc.Kind == protocol.KindContainer: case svc.Kind == protocol.KindContainer:
@@ -126,13 +129,7 @@ func (s *Service) Restorable(svc model.Service) error {
if !svc.Drifted() { if !svc.Drifted() {
return &model.NotUpdatableError{Reason: model.ReasonNotDrifted} return &model.NotUpdatableError{Reason: model.ReasonNotDrifted}
} }
if err := s.updatable(svc, true); err != nil { return s.updatable(svc, true)
return err
}
if imageref.Parse(svc.DesiredRef).Repo == s.selfRepo {
return &model.NotUpdatableError{Reason: model.ReasonSelf}
}
return nil
} }
// RestoreDesired starts a manual job that deploys the drifted service's // RestoreDesired starts a manual job that deploys the drifted service's
@@ -232,9 +229,10 @@ func (s *Service) TriggerVersion(ctx context.Context, serviceID, versionID int64
} }
// Start starts an update of svc. It returns the job, which may already be // Start starts an update of svc. It returns the job, which may already be
// failed ("agent offline", "agent too old for updates", or "agent too old to // failed ("agent offline", "agent too old for updates", "agent too old to
// deploy a pinned version" when the ref has a digest or no tag and the agent // deploy a pinned version" when the ref has a digest or no tag and the agent
// speaks protocol version 2); a // speaks protocol version 2, or "agent too old to update itself" for an
// agent's own service before protocol version 4); a
// *model.NotUpdatableError when the service cannot be updated (no job // *model.NotUpdatableError when the service cannot be updated (no job
// created); the active job and model.ErrJobActive when one is queued or // created); the active job and model.ErrJobActive when one is queued or
// running. Poll and webhook jobs are refused for a drifted service; only a // running. Poll and webhook jobs are refused for a drifted service; only a
@@ -255,6 +253,9 @@ func (s *Service) Start(ctx context.Context, svc model.Service, req StartRequest
if p := imageref.Parse(ref); p.Digest != "" || p.Tag == "" { if p := imageref.Parse(ref); p.Digest != "" || p.Tag == "" {
minProtocol = protocol.MinPinnedRefVersion minProtocol = protocol.MinPinnedRefVersion
} }
if svc.Self || imageref.Parse(svc.ActualRef).Repo == s.selfRepo {
minProtocol = protocol.MinSelfUpdateVersion
}
r, routeErr := s.hub.Route(svc, minProtocol) r, routeErr := s.hub.Route(svc, minProtocol)
nj := model.NewJob{ nj := model.NewJob{
ServiceID: svc.ID, FromRef: svc.ActualRef, ToRef: ref, ServiceID: svc.ID, FromRef: svc.ActualRef, ToRef: ref,
@@ -276,7 +277,7 @@ func (s *Service) Start(ctx context.Context, svc model.Service, req StartRequest
failure := routeErr failure := routeErr
if failure == nil { if failure == nil {
start := protocol.JobStart{Kind: svc.Kind, Ref: ref} start := protocol.JobStart{Kind: svc.Kind, Ref: ref, Self: svc.Self}
if svc.Kind == protocol.KindSwarm { if svc.Kind == protocol.KindSwarm {
start.SwarmClusterID, start.SwarmService = svc.SwarmClusterID, svc.SwarmService start.SwarmClusterID, start.SwarmService = svc.SwarmClusterID, svc.SwarmService
} else { } else {
+69 -7
View File
@@ -16,6 +16,8 @@ import (
"github.com/shcizo/package-updater/internal/protocol" "github.com/shcizo/package-updater/internal/protocol"
) )
const selfImage = "gitea.shcizo.se/shcizo/package-updater:latest"
type fakeStore struct { type fakeStore struct {
services map[int64]model.Service services map[int64]model.Service
jobs map[int64]model.Job jobs map[int64]model.Job
@@ -92,6 +94,9 @@ type fakeHub struct {
func (h *fakeHub) Route(_ model.Service, minProtocol int) (model.Route, error) { func (h *fakeHub) Route(_ model.Service, minProtocol int) (model.Route, error) {
h.mins = append(h.mins, minProtocol) h.mins = append(h.mins, minProtocol)
if h.agentProtocol != 0 && minProtocol > h.agentProtocol { if h.agentProtocol != 0 && minProtocol > h.agentProtocol {
if minProtocol >= protocol.MinSelfUpdateVersion {
return model.Route{}, model.ErrAgentTooOldForSelfUpdate
}
if minProtocol >= protocol.MinPinnedRefVersion { if minProtocol >= protocol.MinPinnedRefVersion {
return model.Route{}, model.ErrAgentTooOldForPinned return model.Route{}, model.ErrAgentTooOldForPinned
} }
@@ -109,8 +114,6 @@ type fakePub struct{ topics []events.Topic }
func (p *fakePub) Publish(t events.Topic) { p.topics = append(p.topics, t) } func (p *fakePub) Publish(t events.Topic) { p.topics = append(p.topics, t) }
const selfImage = "gitea.shcizo.se/shcizo/package-updater:latest"
var ( var (
composeSvc = model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", composeSvc = model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",
AgentID: ptr(int64(7)), ActualRef: "gitea.shcizo.se/shcizo/slusk:1.4.2@sha256:d1"} AgentID: ptr(int64(7)), ActualRef: "gitea.shcizo.se/shcizo/slusk:1.4.2@sha256:d1"}
@@ -134,9 +137,10 @@ func TestUpdatable(t *testing.T) {
reason string reason string
}{ }{
{"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved}, {"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved},
{"flagged self", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, model.ReasonSelf}, {"agent's own Compose service", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, ""},
{"agent's own Swarm service", model.Service{Kind: protocol.KindSwarm, ActualRef: "a:1", Self: true}, model.ReasonSelfSwarm},
{"same repo as the agent image", model.Service{Kind: protocol.KindCompose, {"same repo as the agent image", model.Service{Kind: protocol.KindCompose,
ActualRef: "gitea.shcizo.se/shcizo/package-updater:1.2.0@sha256:abc"}, model.ReasonSelf}, ActualRef: "gitea.shcizo.se/shcizo/package-updater:1.2.0@sha256:abc"}, ""},
{"built from source", model.Service{Kind: protocol.KindCompose, ActualRef: "slusk:1", Built: true}, model.ReasonBuilt}, {"built from source", model.Service{Kind: protocol.KindCompose, ActualRef: "slusk:1", Built: true}, model.ReasonBuilt},
{"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer}, {"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer},
{"digest without tag", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, model.ReasonNoTag}, {"digest without tag", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, model.ReasonNoTag},
@@ -298,7 +302,8 @@ func TestStart_ExplicitRefBypassesOnlyTheNoTagRefusal(t *testing.T) {
}{ }{
{"digest only", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, ""}, {"digest only", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, ""},
{"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved}, {"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved},
{"self", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, model.ReasonSelf}, {"own Swarm service", model.Service{Kind: protocol.KindSwarm, ActualRef: "a:1", Self: true}, model.ReasonSelfSwarm},
{"own Compose service", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, ""},
{"built", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Built: true}, model.ReasonBuilt}, {"built", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Built: true}, model.ReasonBuilt},
{"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer}, {"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer},
} }
@@ -342,8 +347,9 @@ func TestAutoAllowed(t *testing.T) {
reason string reason string
}{ }{
{"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved}, {"removed", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", RemovedAt: ptr(time.Now())}, model.ReasonRemoved},
{"self", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, model.ReasonSelf}, {"own Swarm service", model.Service{Kind: protocol.KindSwarm, ActualRef: "a:1", Self: true}, model.ReasonSelfSwarm},
{"agent image repo", model.Service{Kind: protocol.KindCompose, ActualRef: selfImage}, model.ReasonSelf}, {"own Compose service", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Self: true}, ""},
{"agent image repo", model.Service{Kind: protocol.KindCompose, ActualRef: "gitea.shcizo.se/shcizo/package-updater:latest"}, ""},
{"built", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Built: true}, model.ReasonBuilt}, {"built", model.Service{Kind: protocol.KindCompose, ActualRef: "a:1", Built: true}, model.ReasonBuilt},
{"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer}, {"container", model.Service{Kind: protocol.KindContainer, ActualRef: "redis:7"}, model.ReasonContainer},
{"digest without tag is allowed", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, ""}, {"digest without tag is allowed", model.Service{Kind: protocol.KindCompose, ActualRef: "redis@sha256:abc"}, ""},
@@ -396,6 +402,61 @@ func TestStart_RoutesWithTheMinimumProtocolOfTheRef(t *testing.T) {
} }
} }
func TestStart_SelfServiceNeedsASelfUpdatingAgent(t *testing.T) {
self := composeSvc
self.Self = true
ok := model.Route{AgentID: 7, ConnID: 70}
t.Run("routes with the self-update protocol and marks the job", func(t *testing.T) {
st := &fakeStore{services: map[int64]model.Service{1: self}, jobs: map[int64]model.Job{}}
h := &fakeHub{route: ok}
s, _ := newService(st, h)
_, err := s.Start(context.Background(), self, StartRequest{Trigger: model.TriggerManual, Ref: "x/y:2@sha256:a"})
require.NoError(t, err)
assert.Equal(t, []int{protocol.MinSelfUpdateVersion}, h.mins, "also above the pinned-ref minimum")
require.Len(t, h.starts, 1)
assert.True(t, h.starts[0].start.Self)
})
t.Run("a service with the agent image repo needs a self-updating agent but is not marked", func(t *testing.T) {
svc := composeSvc
svc.ActualRef = selfImage
st := &fakeStore{services: map[int64]model.Service{1: svc}, jobs: map[int64]model.Job{}}
h := &fakeHub{route: ok}
s, _ := newService(st, h)
_, err := s.Start(context.Background(), svc, StartRequest{Trigger: model.TriggerManual})
require.NoError(t, err)
assert.Equal(t, []int{protocol.MinSelfUpdateVersion}, h.mins)
require.Len(t, h.starts, 1)
assert.False(t, h.starts[0].start.Self)
})
t.Run("a service that is not the agent's own is not marked", func(t *testing.T) {
st := &fakeStore{services: map[int64]model.Service{1: composeSvc}, jobs: map[int64]model.Job{}}
h := &fakeHub{route: ok}
s, _ := newService(st, h)
_, err := s.Start(context.Background(), composeSvc, StartRequest{Trigger: model.TriggerManual})
require.NoError(t, err)
assert.False(t, h.starts[0].start.Self)
})
t.Run("an older agent fails the job with a clear error", func(t *testing.T) {
st := &fakeStore{services: map[int64]model.Service{1: self}, jobs: map[int64]model.Job{}}
h := &fakeHub{route: ok, agentProtocol: 3}
s, _ := newService(st, h)
job, err := s.Start(context.Background(), self, StartRequest{Trigger: model.TriggerManual})
require.NoError(t, err)
assert.Equal(t, model.JobFailed, job.Status)
assert.Equal(t, "agent too old to update itself", job.Error)
assert.Empty(t, h.starts)
})
}
func TestTriggerVersion_Ref(t *testing.T) { func TestTriggerVersion_Ref(t *testing.T) {
const repo = "gitea.shcizo.se/shcizo/slusk" const repo = "gitea.shcizo.se/shcizo/slusk"
noTag := model.Service{ID: 4, Kind: protocol.KindCompose, ComposeProject: "n", ComposeService: "n", noTag := model.Service{ID: 4, Kind: protocol.KindCompose, ComposeProject: "n", ComposeService: "n",
@@ -411,6 +472,7 @@ func TestTriggerVersion_Ref(t *testing.T) {
}{ }{
{"tag and digest", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"}, repo + ":1.3.0@sha256:old", protocol.MinPinnedRefVersion}, {"tag and digest", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"}, repo + ":1.3.0@sha256:old", protocol.MinPinnedRefVersion},
{"tag only", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "main"}, repo + ":main", protocol.MinJobVersion}, {"tag only", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "main"}, repo + ":main", protocol.MinJobVersion},
{"digest-less version from the tag list", svc, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.28.0", Source: model.SourceTags}, repo + ":1.28.0", protocol.MinJobVersion},
{"digest only", svc, model.ImageVersion{ID: 10, Repo: repo, Digest: "sha256:old"}, repo + "@sha256:old", protocol.MinPinnedRefVersion}, {"digest only", svc, model.ImageVersion{ID: 10, Repo: repo, Digest: "sha256:old"}, repo + "@sha256:old", protocol.MinPinnedRefVersion},
{"service that follows no tag", noTag, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"}, repo + ":1.3.0@sha256:old", protocol.MinPinnedRefVersion}, {"service that follows no tag", noTag, model.ImageVersion{ID: 10, Repo: repo, Tag: "1.3.0", Digest: "sha256:old"}, repo + ":1.3.0@sha256:old", protocol.MinPinnedRefVersion},
} }
+15 -3
View File
@@ -19,6 +19,10 @@ var (
// speaks protocol version 2, which cannot deploy a digest-pinned or // speaks protocol version 2, which cannot deploy a digest-pinned or
// tagless ref. errors.Is(err, ErrAgentTooOld) is true for it as well. // tagless ref. errors.Is(err, ErrAgentTooOld) is true for it as well.
ErrAgentTooOldForPinned error = agentTooOldForPinned{} ErrAgentTooOldForPinned error = agentTooOldForPinned{}
// ErrAgentTooOldForSelfUpdate means the agent that runs the service speaks
// a protocol version before 4 and cannot update itself.
// errors.Is(err, ErrAgentTooOld) is true for it as well.
ErrAgentTooOldForSelfUpdate error = agentTooOldForSelfUpdate{}
) )
type agentTooOldForPinned struct{} type agentTooOldForPinned struct{}
@@ -29,6 +33,12 @@ func (agentTooOldForPinned) Error() string {
func (agentTooOldForPinned) Is(target error) bool { return target == ErrAgentTooOld } func (agentTooOldForPinned) Is(target error) bool { return target == ErrAgentTooOld }
type agentTooOldForSelfUpdate struct{}
func (agentTooOldForSelfUpdate) Error() string { return "agent too old to update itself" }
func (agentTooOldForSelfUpdate) Is(target error) bool { return target == ErrAgentTooOld }
// NotUpdatableError says why a service cannot be updated. Reason is shown // NotUpdatableError says why a service cannot be updated. Reason is shown
// in the UI as is. // in the UI as is.
type NotUpdatableError struct { type NotUpdatableError struct {
@@ -39,11 +49,13 @@ func (e *NotUpdatableError) Error() string { return "not updatable: " + e.Reason
// Reasons a service cannot be updated. // Reasons a service cannot be updated.
const ( const (
ReasonSelf = "updated manually outside the system"
ReasonContainer = "plain containers cannot be updated" ReasonContainer = "plain containers cannot be updated"
ReasonNoTag = "follows no tag" ReasonNoTag = "follows no tag"
ReasonRemoved = "service no longer exists" ReasonRemoved = "service no longer exists"
ReasonBuilt = "built from source (build:)" ReasonBuilt = "built from source (build:)"
// ReasonSelfSwarm is for an agent's own Swarm service, which cannot be
// updated by the agent that runs in it.
ReasonSelfSwarm = "an agent's own Swarm service cannot update itself"
ReasonDrifted = "drifted: restore or accept the running version first" ReasonDrifted = "drifted: restore or accept the running version first"
ReasonNotDrifted = "runs its desired version" ReasonNotDrifted = "runs its desired version"
@@ -57,14 +69,14 @@ type MergeRefusedError struct {
func (e *MergeRefusedError) Error() string { return "cannot merge: " + e.Reason } func (e *MergeRefusedError) Error() string { return "cannot merge: " + e.Reason }
// Reasons two services cannot be merged. ReasonSelf is reused for the // Reasons two services cannot be merged.
// updater itself.
const ( const (
ReasonMergeSame = "a service cannot be merged into itself" ReasonMergeSame = "a service cannot be merged into itself"
ReasonMergeLiveOld = "the old service still exists" ReasonMergeLiveOld = "the old service still exists"
ReasonMergeGoneNew = "the new service no longer exists" ReasonMergeGoneNew = "the new service no longer exists"
ReasonMergeKind = "only services of the same kind can be merged" ReasonMergeKind = "only services of the same kind can be merged"
ReasonMergeContainer = "plain containers cannot be merged" ReasonMergeContainer = "plain containers cannot be merged"
ReasonMergeSelf = "an agent's own service cannot be merged"
) )
// JobStatus is the state of an update job. // JobStatus is the state of an update job.
+9 -1
View File
@@ -66,6 +66,11 @@ type Service struct {
AvailableSource string // SourceWebhook or SourcePoll; "" when AvailableRef is empty AvailableSource string // SourceWebhook or SourcePoll; "" when AvailableRef is empty
AvailableAt *time.Time AvailableAt *time.Time
LastPollError string // last registry error for the followed ref; "" when fine LastPollError string // last registry error for the followed ref; "" when fine
// NewerRef is the newest repo:tag from the registry's tag list that
// tagver.Newer finds for the followed tag; "" if none. A suggestion only:
// it is never deployed automatically, not even for auto services.
NewerRef string
NewerAt *time.Time // when NewerRef was first suggested; nil when NewerRef is empty
// LastTagDigest is the newest digest of the followed tag that was deployed // LastTagDigest is the newest digest of the followed tag that was deployed
// or started: set when the poller starts an auto job, when it sees the service // or started: set when the poller starts an auto job, when it sees the service
// already running that digest, and when any plain tag job succeeds (the // already running that digest, and when any plain tag job succeeds (the
@@ -95,6 +100,9 @@ const (
// SourceRunning is used by image_versions only: an agent reported the // SourceRunning is used by image_versions only: an agent reported the
// version as running. // version as running.
SourceRunning = "running" SourceRunning = "running"
// SourceTags is used by image_versions only: the tag appeared in the
// registry's tag list as a newer tag.
SourceTags = "tags"
) )
// Available is a newer ref reported for a service. // Available is a newer ref reported for a service.
@@ -113,7 +121,7 @@ type ImageVersion struct {
Repo string Repo string
Tag string Tag string
Digest string Digest string
Source string // SourceWebhook, SourcePoll or SourceRunning: the first source that saw it Source string // SourceWebhook, SourcePoll, SourceRunning or SourceTags: the first source that saw it
FirstSeenAt time.Time FirstSeenAt time.Time
} }
+225 -22
View File
@@ -1,24 +1,32 @@
// Package poller checks the registry for new digests of the tags the // Package poller checks the registry for new digests of the tags the
// services follow. A new digest becomes the service's available version, and // services follow. A new digest becomes the service's available version, and
// a service in auto mode gets an update job. // a service in auto mode gets an update job. It also lists the tags of the
// repos whose followed tag looks like a version and suggests a newer one;
// that suggestion never starts a job.
package poller package poller
import ( import (
"context" "context"
"errors" "errors"
"log/slog" "log/slog"
"slices"
"strings" "strings"
"sync" "sync"
"time" "time"
"github.com/shcizo/package-updater/internal/imageref"
"github.com/shcizo/package-updater/internal/master/events" "github.com/shcizo/package-updater/internal/master/events"
"github.com/shcizo/package-updater/internal/master/jobs" "github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model" "github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/master/tagver"
) )
// lookupTimeout bounds one registry lookup. // lookupTimeout bounds one registry lookup.
const lookupTimeout = 30 * time.Second const lookupTimeout = 30 * time.Second
// listTimeout bounds one tag listing, which may take several requests.
const listTimeout = 60 * time.Second
// maxErrorBytes caps the registry error stored in services.last_poll_error. // maxErrorBytes caps the registry error stored in services.last_poll_error.
const maxErrorBytes = 500 const maxErrorBytes = 500
@@ -28,6 +36,7 @@ type Store interface {
GetService(ctx context.Context, id int64) (model.Service, error) GetService(ctx context.Context, id int64) (model.Service, error)
SetAvailable(ctx context.Context, id int64, a model.Available) error SetAvailable(ctx context.Context, id int64, a model.Available) error
SetLastTagDigest(ctx context.Context, id int64, digest string) error SetLastTagDigest(ctx context.Context, id int64, digest string) error
SetNewer(ctx context.Context, id int64, ref string, at time.Time) error
SetPollError(ctx context.Context, ids []int64, msg string) error SetPollError(ctx context.Context, ids []int64, msg string) error
} }
@@ -36,10 +45,11 @@ type Jobs interface {
Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error) Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error)
} }
// Registry looks up the digest a "repo:tag" ref points at. *registry.Client // Registry looks up the digest a "repo:tag" ref points at and lists the tags
// satisfies it. // of a repo. *registry.Client satisfies it.
type Registry interface { type Registry interface {
Digest(ctx context.Context, ref string) (string, error) Digest(ctx context.Context, ref string) (string, error)
Tags(ctx context.Context, repo string) ([]string, error)
} }
// Publisher receives change signals. *events.Broker satisfies it. // Publisher receives change signals. *events.Broker satisfies it.
@@ -49,7 +59,7 @@ type Publisher interface{ Publish(t events.Topic) }
type Config struct { type Config struct {
Interval time.Duration // time between the end of a cycle and the start of the next (POLL_INTERVAL) Interval time.Duration // time between the end of a cycle and the start of the next (POLL_INTERVAL)
FirstDelay time.Duration // wait before the first cycle; 1 minute in production FirstDelay time.Duration // wait before the first cycle; 1 minute in production
Workers int // registry lookups in flight at once; 4 in production Workers int // registry requests in flight at once; 4 in production
} }
// Poller runs polling cycles. // Poller runs polling cycles.
@@ -112,13 +122,16 @@ type lookup struct {
// cycleStats is what one cycle did, for the poll_cycle log line. // cycleStats is what one cycle did, for the poll_cycle log line.
type cycleStats struct { type cycleStats struct {
refs, ok, failed, suggested, jobs, skipped int refs, ok, failed, suggested, jobs, skipped int
reposListed, tagsFailed, newer int
wrote bool wrote bool
} }
// Cycle runs one poll over all services. It looks up every distinct followed // Cycle runs one poll over all services. It looks up every distinct followed
// ref once, with at most Config.Workers lookups at a time, then records the // ref once, with at most Config.Workers lookups at a time, then records the
// results. Writing results and starting jobs is not cut short by ctx ending // results. Then it lists the tags of each repo that has a service following a
// during the lookups, so a job that was marked is always started. // version-like tag, once per repo, and records newer tags. Writing results and
// starting jobs is not cut short by ctx ending during the lookups, so a job
// that was marked is always started.
func (p *Poller) Cycle(ctx context.Context) { func (p *Poller) Cycle(ctx context.Context) {
began := p.now() began := p.now()
svcs, err := p.store.ListServices(ctx) svcs, err := p.store.ListServices(ctx)
@@ -126,6 +139,17 @@ func (p *Poller) Cycle(ctx context.Context) {
p.log.Error("poll_list_failed", "error", err) p.log.Error("poll_list_failed", "error", err)
return return
} }
stats := p.pollServices(ctx, svcs)
p.log.Info("poll_cycle", "refs", stats.refs, "ok", stats.ok, "failed", stats.failed,
"suggested", stats.suggested, "jobs", stats.jobs, "skipped", stats.skipped,
"repos_listed", stats.reposListed, "tags_failed", stats.tagsFailed, "newer", stats.newer,
"duration_ms", p.now().Sub(began).Milliseconds())
}
// pollServices does the work of a cycle for svcs: it looks up every distinct
// followed ref once, records the results and starts auto jobs, then syncs the
// newer-tag suggestions of svcs.
func (p *Poller) pollServices(ctx context.Context, svcs []model.Service) cycleStats {
var stats cycleStats var stats cycleStats
byRef := map[string][]model.Service{} byRef := map[string][]model.Service{}
var refs []string var refs []string
@@ -152,19 +176,61 @@ func (p *Poller) Cycle(ctx context.Context) {
} }
p.record(write, l, byRef[l.ref], &stats) p.record(write, l, byRef[l.ref], &stats)
} }
if stats.wrote { // Publish after the digest phase too: listing tags can take a while and
p.pub.Publish(events.Services) // an "Update available" should not wait for it.
} p.publishIfWrote(&stats)
p.log.Info("poll_cycle", "refs", stats.refs, "ok", stats.ok, "failed", stats.failed, p.syncNewer(ctx, write, svcs, &stats)
"suggested", stats.suggested, "jobs", stats.jobs, "skipped", stats.skipped, p.publishIfWrote(&stats)
"duration_ms", p.now().Sub(began).Milliseconds()) return stats
} }
// lookupAll asks the registry for every ref, Config.Workers at a time. It // NotPolledError says a service cannot be checked against the registry.
// returns the answers in ref order and stops starting new lookups once ctx type NotPolledError struct{ Reason string }
// is done; refs never looked up are missing from the result.
func (p *Poller) lookupAll(ctx context.Context, refs []string) []lookup { func (e *NotPolledError) Error() string { return "not polled: " + e.Reason }
results := make([]*lookup, len(refs))
// PollService does what Cycle does, for the service id only: it looks up the
// followed ref, records the digest, starts an auto job if one is wanted and
// syncs the newer-tag suggestion. Other services that follow the same ref are
// updated at the next cycle. It is safe to run while Run is running: the
// unique index update_jobs_one_active prevents duplicate jobs. A registry
// error is not returned; it is stored as last_poll_error, like in a cycle. It
// returns the store's error when the service cannot be read, model.ErrNotFound
// for a removed service, *NotPolledError when the service is not polled, and
// ctx.Err() when ctx ended.
func (p *Poller) PollService(ctx context.Context, id int64) error {
began := p.now()
svc, err := p.store.GetService(ctx, id)
if err != nil {
return err
}
if svc.RemovedAt != nil {
return model.ErrNotFound
}
if r := SkipReason(svc); r != "" {
return &NotPolledError{Reason: r}
}
stats := p.pollServices(ctx, []model.Service{svc})
p.log.Info("poll_service", "service_id", id, "ref", jobs.FollowedRef(svc.ActualRef),
"ok", stats.ok, "failed", stats.failed, "suggested", stats.suggested, "jobs", stats.jobs,
"tags_failed", stats.tagsFailed, "newer", stats.newer,
"duration_ms", p.now().Sub(began).Milliseconds())
return ctx.Err()
}
// publishIfWrote tells the browsers to refresh if a phase of the cycle wrote
// anything, and starts the next phase with a clean slate.
func (p *Poller) publishIfWrote(stats *cycleStats) {
if stats.wrote {
p.pub.Publish(events.Services)
stats.wrote = false
}
}
// forEach calls fn(i) for i in [0, n), Config.Workers at a time, and stops
// starting new calls once ctx is done. It returns when all started calls
// have returned.
func (p *Poller) forEach(ctx context.Context, n int, fn func(i int)) {
next := make(chan int) next := make(chan int)
var wg sync.WaitGroup var wg sync.WaitGroup
for w := 0; w < p.cfg.Workers; w++ { for w := 0; w < p.cfg.Workers; w++ {
@@ -172,15 +238,12 @@ func (p *Poller) lookupAll(ctx context.Context, refs []string) []lookup {
go func() { go func() {
defer wg.Done() defer wg.Done()
for i := range next { for i := range next {
lctx, cancel := context.WithTimeout(ctx, lookupTimeout) fn(i)
d, err := p.reg.Digest(lctx, refs[i])
cancel()
results[i] = &lookup{ref: refs[i], digest: d, err: err}
} }
}() }()
} }
feed: feed:
for i := range refs { for i := 0; i < n; i++ {
select { select {
case next <- i: case next <- i:
case <-ctx.Done(): case <-ctx.Done():
@@ -189,6 +252,19 @@ feed:
} }
close(next) close(next)
wg.Wait() wg.Wait()
}
// lookupAll asks the registry for every ref, Config.Workers at a time. It
// returns the answers in ref order and stops starting new lookups once ctx
// is done; refs never looked up are missing from the result.
func (p *Poller) lookupAll(ctx context.Context, refs []string) []lookup {
results := make([]*lookup, len(refs))
p.forEach(ctx, len(refs), func(i int) {
lctx, cancel := context.WithTimeout(ctx, lookupTimeout)
d, err := p.reg.Digest(lctx, refs[i])
cancel()
results[i] = &lookup{ref: refs[i], digest: d, err: err}
})
var out []lookup var out []lookup
for _, r := range results { for _, r := range results {
if r != nil { if r != nil {
@@ -198,6 +274,118 @@ feed:
return out return out
} }
// tagList is the registry's answer for the tags of one repo.
type tagList struct {
repo string
tags []string
err error
}
// listAll lists the tags of every repo like lookupAll looks up refs.
func (p *Poller) listAll(ctx context.Context, repos []string) []tagList {
results := make([]*tagList, len(repos))
p.forEach(ctx, len(repos), func(i int) {
lctx, cancel := context.WithTimeout(ctx, listTimeout)
tags, err := p.reg.Tags(lctx, repos[i])
cancel()
results[i] = &tagList{repo: repos[i], tags: tags, err: err}
})
var out []tagList
for _, r := range results {
if r != nil {
out = append(out, *r)
}
}
return out
}
// Followed returns the repo and tag svc follows: a bare image follows
// "latest", and a digest-only reference follows no tag.
func Followed(svc model.Service) (repo, tag string) {
ref := imageref.Parse(jobs.FollowedRef(svc.ActualRef))
return ref.Repo, ref.Tag
}
// followedComparable returns the repo and tag svc follows. ok is false unless
// svc is polled and the tag can have a newer tag.
func followedComparable(svc model.Service) (repo, tag string, ok bool) {
if SkipReason(svc) != "" {
return "", "", false
}
repo, tag = Followed(svc)
return repo, tag, tagver.Comparable(tag)
}
// syncNewer lists the tags of every repo that has a service following a
// version-like tag, once per repo, and stores the newer tag of each service.
// A service whose tag cannot have a newer tag any more gets its suggestion
// cleared; one whose repo could not be listed keeps it, unless it names
// another repo than the service follows now. Nothing is written
// when the value is unchanged. It never starts a job, not even for an auto
// service: a newer tag may be a major step the user has not chosen.
func (p *Poller) syncNewer(ctx, write context.Context, svcs []model.Service, stats *cycleStats) {
seen := map[string]bool{}
var repos []string
for _, svc := range svcs {
if repo, _, ok := followedComparable(svc); ok && svc.RemovedAt == nil && !seen[repo] {
seen[repo] = true
repos = append(repos, repo)
}
}
lists := map[string][]string{}
for _, l := range p.listAll(ctx, repos) {
if l.err != nil {
p.log.Warn("poll_tags_failed", "repo", l.repo, "error", truncate(l.err.Error()))
if ctx.Err() == nil { // cancelled mid-listing: not the registry's fault
stats.tagsFailed++
}
continue
}
stats.reposListed++
lists[l.repo] = l.tags
}
type repoTag struct{ repo, tag string }
newest := map[repoTag]string{}
for _, svc := range svcs {
if svc.RemovedAt != nil {
continue
}
want := ""
if repo, tag, ok := followedComparable(svc); ok {
tags, listed := lists[repo]
if !listed && imageref.Parse(svc.NewerRef).Repo == repo {
if svc.NewerRef != "" {
stats.newer++
}
continue // not listed this cycle: keep what is stored
}
if listed {
key := repoTag{repo, tag}
n, done := newest[key]
if !done {
n, _ = tagver.Newer(tag, tags)
newest[key] = n
}
if n != "" {
want = repo + ":" + n
}
}
}
if want != "" {
stats.newer++
}
if want == svc.NewerRef {
continue
}
if err := p.store.SetNewer(write, svc.ID, want, p.now()); err != nil {
p.log.Error("poll_set_newer_failed", "service_id", svc.ID, "ref", want, "error", err)
continue
}
stats.wrote = true
}
}
// record stores one lookup result for the services that follow l.ref. // record stores one lookup result for the services that follow l.ref.
func (p *Poller) record(ctx context.Context, l lookup, svcs []model.Service, stats *cycleStats) { func (p *Poller) record(ctx context.Context, l lookup, svcs []model.Service, stats *cycleStats) {
if l.err != nil { if l.err != nil {
@@ -209,14 +397,29 @@ func (p *Poller) record(ctx context.Context, l lookup, svcs []model.Service, sta
} }
stats.ok++ stats.ok++
p.setPollError(ctx, svcs, "", stats, func(s model.Service) bool { return s.LastPollError != "" }) p.setPollError(ctx, svcs, "", stats, func(s model.Service) bool { return s.LastPollError != "" })
var auto []model.Service
for _, svc := range svcs { for _, svc := range svcs {
if !p.suggest(ctx, svc, l, stats) { if !p.suggest(ctx, svc, l, stats) {
continue continue
} }
if autoWanted(svc, l.ref, l.digest) { if autoWanted(svc, l.ref, l.digest) {
auto = append(auto, svc)
}
}
// An agent's own service goes last: jobs that reach an agent while it
// replaces itself are lost, and messages to one agent arrive in order.
slices.SortStableFunc(auto, func(a, b model.Service) int { return selfRank(a) - selfRank(b) })
for _, svc := range auto {
p.startAuto(ctx, svc.ID, l.ref, l.digest, stats) p.startAuto(ctx, svc.ID, l.ref, l.digest, stats)
} }
}
// selfRank sorts an agent's own service after all others.
func selfRank(svc model.Service) int {
if svc.Self {
return 1
} }
return 0
} }
// suggest marks l.digest as the available version of svc unless the service // suggest marks l.digest as the available version of svc unless the service
+447 -1
View File
@@ -30,6 +30,12 @@ type pollErrorCall struct {
msg string msg string
} }
type newerCall struct {
id int64
ref string
at time.Time
}
type lastTagCall struct { type lastTagCall struct {
id int64 id int64
digest string digest string
@@ -40,6 +46,7 @@ type fakeStore struct {
services []model.Service services []model.Service
fresh map[int64]model.Service // what GetService returns instead of the listed row fresh map[int64]model.Service // what GetService returns instead of the listed row
lastTag []lastTagCall lastTag []lastTagCall
newer []newerCall
listErr error listErr error
available []availableCall available []availableCall
pollErrors []pollErrorCall pollErrors []pollErrorCall
@@ -86,6 +93,21 @@ func (f *fakeStore) SetAvailable(ctx context.Context, id int64, a model.Availabl
return nil return nil
} }
// SetNewer records the call and, like the database, changes the row the
// next cycle lists.
func (f *fakeStore) SetNewer(ctx context.Context, id int64, ref string, at time.Time) error {
f.mu.Lock()
defer f.mu.Unlock()
f.ctxErrs = append(f.ctxErrs, ctx.Err())
f.newer = append(f.newer, newerCall{id, ref, at})
for i := range f.services {
if f.services[i].ID == id {
f.services[i].NewerRef = ref
}
}
return nil
}
func (f *fakeStore) SetPollError(ctx context.Context, ids []int64, msg string) error { func (f *fakeStore) SetPollError(ctx context.Context, ids []int64, msg string) error {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
@@ -125,6 +147,36 @@ type fakeRegistry struct {
started chan struct{} // receives a value whenever a lookup starts, if set started chan struct{} // receives a value whenever a lookup starts, if set
release chan struct{} // lookups wait for it to be closed, if set release chan struct{} // lookups wait for it to be closed, if set
onResult func() // called after a lookup computed its answer onResult func() // called after a lookup computed its answer
tags map[string][]string // repo -> tags
tagErrs map[string]error // repo -> error
tagCalls []string
tagStarted chan struct{} // receives a value whenever a tag listing starts, if set
tagRelease chan struct{} // tag listings wait for it to be closed, if set
onTags func() // called after a tag listing computed its answer
}
func (f *fakeRegistry) Tags(ctx context.Context, repo string) ([]string, error) {
f.mu.Lock()
f.tagCalls = append(f.tagCalls, repo)
f.mu.Unlock()
if f.tagStarted != nil {
f.tagStarted <- struct{}{}
}
if f.tagRelease != nil {
select {
case <-f.tagRelease:
case <-ctx.Done():
return nil, ctx.Err()
}
}
if f.onTags != nil {
f.onTags()
}
if err := f.tagErrs[repo]; err != nil {
return nil, err
}
return f.tags[repo], nil
} }
func (f *fakeRegistry) Digest(ctx context.Context, ref string) (string, error) { func (f *fakeRegistry) Digest(ctx context.Context, ref string) (string, error) {
@@ -297,6 +349,24 @@ func TestNewDigest(t *testing.T) {
} }
} }
func TestAgentsOwnServiceIsStartedLast(t *testing.T) {
ref := repoA + ":1.4"
own := svc(1, ref+"@sha256:old")
own.UpdateMode, own.Self = model.ModeAuto, true
other := svc(2, ref+"@sha256:old")
other.UpdateMode = model.ModeAuto
st := &fakeStore{services: []model.Service{own, other}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
require.Len(t, j.starts, 2)
assert.Equal(t, int64(2), j.starts[0].svcID)
assert.Equal(t, int64(1), j.starts[1].svcID)
}
func TestSameSuggestionIsIdempotent(t *testing.T) { func TestSameSuggestionIsIdempotent(t *testing.T) {
ref := repoA + ":1.4" ref := repoA + ":1.4"
running := svc(1, ref+"@sha256:old") running := svc(1, ref+"@sha256:old")
@@ -332,7 +402,7 @@ func TestStartErrorsDoNotAbortCycle(t *testing.T) {
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}} reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
j := &fakeJobs{errFor: map[int64]error{ j := &fakeJobs{errFor: map[int64]error{
1: model.ErrJobActive, 1: model.ErrJobActive,
2: &model.NotUpdatableError{Reason: model.ReasonSelf}, 2: &model.NotUpdatableError{Reason: model.ReasonContainer},
3: errors.New("database down"), 3: errors.New("database down"),
}} }}
p, _ := newPoller(st, j, reg, Config{}) p, _ := newPoller(st, j, reg, Config{})
@@ -649,3 +719,379 @@ func TestSeedsTheTagDigestOfARunningService(t *testing.T) {
assert.Empty(t, j.starts, "the rollback stays") assert.Empty(t, j.starts, "the rollback stays")
assert.Len(t, st.lastTag, 1, "nothing more to seed") assert.Len(t, st.lastTag, 1, "nothing more to seed")
} }
// newTagRegistry returns a fake registry that lists tags by repo.
func newTagRegistry(tags map[string][]string) *fakeRegistry {
return &fakeRegistry{digests: map[string]string{}, tags: tags}
}
// newerSvc is a polled service running tag of repo that already runs the
// registry's digest, so the digest phase writes nothing.
func newerSvc(id int64, repo, tag string) model.Service {
s := svc(id, repo+":"+tag)
s.ActualDigest = "sha256:new"
return s
}
// newerRegistry is newTagRegistry that also answers sha256:new for refs.
func newerRegistry(tags map[string][]string, refs ...string) *fakeRegistry {
reg := newTagRegistry(tags)
for _, r := range refs {
reg.digests[r] = "sha256:new"
}
return reg
}
func TestNewerTagSuggested(t *testing.T) {
ref := repoB + ":1.27.3"
st := &fakeStore{services: []model.Service{newerSvc(1, repoB, "1.27.3")}}
reg := newerRegistry(map[string][]string{repoB: {"1.27.4", "1.28.0", "2.0.0", "latest", "1.28.1-rootless"}}, ref)
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Equal(t, []newerCall{{1, repoB + ":1.28.0", time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}}, st.newer)
assert.Equal(t, []events.Topic{events.Services}, pub.topics)
}
func TestOneTagListPerRepo(t *testing.T) {
st := &fakeStore{services: []model.Service{
newerSvc(1, repoB, "1.27.3"),
newerSvc(2, repoB, "2.1.0"),
newerSvc(3, repoB, "1.27.3"),
}}
reg := newerRegistry(map[string][]string{repoB: {"2.1.1", "1.27.4"}},
repoB+":1.27.3", repoB+":2.1.0")
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Equal(t, []string{repoB}, reg.tagCalls)
got := map[int64]string{}
for _, c := range st.newer {
got[c.id] = c.ref
}
assert.Equal(t, map[int64]string{1: repoB + ":1.27.4", 2: repoB + ":2.1.1", 3: repoB + ":1.27.4"}, got)
}
func TestNoTagListWithoutComparableTag(t *testing.T) {
built := newerSvc(5, repoB, "1.2.3")
built.Built = true
noDigest := newerSvc(6, repoB, "1.2.3")
noDigest.ActualDigest = ""
st := &fakeStore{services: []model.Service{
newerSvc(1, repoB, "latest"),
newerSvc(2, repoB, "7"),
newerSvc(3, repoB, "pg16"),
newerSvc(4, repoB+"@sha256:abc", ""),
built,
noDigest,
}}
st.services[3].ActualRef = repoB + "@sha256:abc"
reg := newTagRegistry(map[string][]string{repoB: {"1.2.4"}})
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Empty(t, reg.tagCalls)
assert.Empty(t, st.newer)
}
func TestNewerClearedWhenNoCandidate(t *testing.T) {
s := newerSvc(1, repoB, "1.27.3")
s.NewerRef = repoB + ":1.27.4"
st := &fakeStore{services: []model.Service{s}}
reg := newerRegistry(map[string][]string{repoB: {"1.27.3", "2.0.0"}}, repoB+":1.27.3")
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Equal(t, []newerCall{{1, "", time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}}, st.newer)
}
func TestNewerClearedWhenTagNoLongerComparable(t *testing.T) {
moved := newerSvc(1, repoB, "latest")
moved.NewerRef = repoB + ":1.27.4"
built := newerSvc(2, repoB, "1.27.3")
built.Built = true
built.NewerRef = repoB + ":1.27.4"
st := &fakeStore{services: []model.Service{moved, built}}
reg := newerRegistry(map[string][]string{repoB: {"1.27.4"}}, repoB+":latest")
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
require.Len(t, st.newer, 2)
assert.Equal(t, "", st.newer[0].ref)
assert.Equal(t, "", st.newer[1].ref)
assert.Empty(t, reg.tagCalls)
assert.Equal(t, []events.Topic{events.Services}, pub.topics)
}
func TestTagListErrorKeepsNewerAndPollError(t *testing.T) {
s := newerSvc(1, repoB, "1.27.3")
s.NewerRef = repoB + ":1.27.4"
st := &fakeStore{services: []model.Service{s}}
reg := newerRegistry(nil, repoB+":1.27.3")
reg.tagErrs = map[string]error{repoB: errors.New("list x: registry answered 500 Internal Server Error")}
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Empty(t, st.newer, "the stored suggestion stays")
assert.Empty(t, st.pollErrors, "a failed tag list is not a poll error")
assert.Empty(t, pub.topics)
}
func TestNewerIdempotent(t *testing.T) {
s := newerSvc(1, repoB, "1.27.3")
s.NewerRef = repoB + ":1.28.0"
st := &fakeStore{services: []model.Service{s}}
reg := newerRegistry(map[string][]string{repoB: {"1.28.0"}}, repoB+":1.27.3")
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Empty(t, st.newer)
assert.Empty(t, pub.topics)
}
func TestNewerNeverStartsJobs(t *testing.T) {
s := newerSvc(1, repoB, "1.27.3")
s.UpdateMode = model.ModeAuto
st := &fakeStore{services: []model.Service{s}}
reg := newerRegistry(map[string][]string{repoB: {"1.27.4"}}, repoB+":1.27.3")
j := &fakeJobs{}
p, _ := newPoller(st, j, reg, Config{})
p.Cycle(context.Background())
require.Len(t, st.newer, 1)
assert.Empty(t, j.starts)
}
func TestNewerWritesSurviveCancel(t *testing.T) {
st := &fakeStore{services: []model.Service{newerSvc(1, repoB, "1.27.3")}}
ctx, cancel := context.WithCancel(context.Background())
reg := newerRegistry(map[string][]string{repoB: {"1.27.4"}}, repoB+":1.27.3")
reg.onTags = cancel
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(ctx)
require.Len(t, st.newer, 1)
assert.NoError(t, st.ctxErrs[0], "the write does not see the cancelled context")
}
func TestCancelledListKeepsNewer(t *testing.T) {
s := newerSvc(1, repoB, "1.27.3")
s.NewerRef = repoB + ":1.27.4"
st := &fakeStore{services: []model.Service{s}}
reg := newerRegistry(map[string][]string{repoB: {"1.28.0"}}, repoB+":1.27.3")
reg.tagRelease = make(chan struct{})
reg.tagStarted = make(chan struct{}, 1)
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { p.Cycle(ctx); close(done) }()
<-reg.tagStarted
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Cycle did not return after cancel")
}
assert.Empty(t, st.newer)
assert.Empty(t, pub.topics)
}
func TestFailedListClearsNewerFromAnotherRepo(t *testing.T) {
s := newerSvc(1, repoB, "1.27.3")
s.NewerRef = repoA + ":1.27.4"
st := &fakeStore{services: []model.Service{s}}
reg := newerRegistry(nil, repoB+":1.27.3")
reg.tagErrs = map[string]error{repoB: errors.New("list x: registry answered 500 Internal Server Error")}
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
p.Cycle(context.Background())
assert.Equal(t, []newerCall{{1, "", time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}}, st.newer)
}
func TestDigestWritesArePublishedBeforeTagsAreListed(t *testing.T) {
st := &fakeStore{services: []model.Service{svc(1, repoB+":1.27.3")}}
reg := newerRegistry(map[string][]string{repoB: {"1.27.3"}}, repoB+":1.27.3")
reg.tagRelease = make(chan struct{})
reg.tagStarted = make(chan struct{}, 1)
p, pub := newPoller(st, &fakeJobs{}, reg, Config{})
done := make(chan struct{})
go func() { p.Cycle(context.Background()); close(done) }()
<-reg.tagStarted
pub.mu.Lock()
published := append([]events.Topic(nil), pub.topics...)
pub.mu.Unlock()
assert.Equal(t, []events.Topic{events.Services}, published, "published while the listing is still blocked")
close(reg.tagRelease)
<-done
}
func TestPollService(t *testing.T) {
ref := repoA + ":latest" // not version-like: no tag listing
newDigest := map[string]string{ref: "sha256:new"}
auto := func(edit func(*model.Service)) model.Service {
s := svc(1, ref)
s.UpdateMode = model.ModeAuto
if edit != nil {
edit(&s)
}
return s
}
tests := []struct {
name string
svc model.Service
reg *fakeRegistry
startErr error
wantErr error // matched with ErrorIs
wantNotPoll string
wantHeads []string
wantTags []string
check func(t *testing.T, st *fakeStore, j *fakeJobs)
}{
{name: "manual with a new digest is only marked",
svc: svc(1, ref), reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
require.Len(t, st.available, 1)
assert.Equal(t, model.SourcePoll, st.available[0].a.Source)
assert.Empty(t, j.starts)
}},
{name: "auto with a new digest starts a poll job",
svc: auto(nil), reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
assert.Equal(t, []startCall{{1, jobs.StartRequest{Trigger: model.TriggerPoll, Ref: ref}}}, j.starts)
assert.Equal(t, []lastTagCall{{1, "sha256:new"}}, st.lastTag)
}},
{name: "auto that cannot be updated is marked but last_tag_digest stays",
svc: auto(nil), reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
startErr: &model.NotUpdatableError{Reason: model.ReasonDrifted},
check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
assert.Len(t, st.available, 1)
assert.Len(t, j.starts, 1)
assert.Empty(t, st.lastTag)
}},
{name: "auto that already got the digest is not started",
svc: auto(func(s *model.Service) { s.LastTagDigest = "sha256:new" }),
reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
check: func(t *testing.T, st *fakeStore, j *fakeJobs) { assert.Empty(t, j.starts) }},
{name: "auto whose desired_ref names the digest is not started",
svc: auto(func(s *model.Service) { s.DesiredRef = ref + "@sha256:new" }),
reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
check: func(t *testing.T, st *fakeStore, j *fakeJobs) { assert.Empty(t, j.starts) }},
{name: "registry error is stored, not returned",
svc: svc(1, ref), reg: &fakeRegistry{errs: map[string]error{ref: errors.New("head: boom")}}, wantHeads: []string{ref},
check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
assert.Equal(t, []pollErrorCall{{[]int64{1}, "head: boom"}}, st.pollErrors)
}},
{name: "a previous poll error is cleared on success",
svc: auto(func(s *model.Service) {
s.UpdateMode, s.ActualDigest, s.LastPollError = model.ModeManual, "sha256:new", "head: boom"
}),
reg: &fakeRegistry{digests: newDigest}, wantHeads: []string{ref},
check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
assert.Equal(t, []pollErrorCall{{[]int64{1}, ""}}, st.pollErrors)
}},
{name: "no tag is not polled",
svc: svc(1, repoA+"@sha256:old"), reg: &fakeRegistry{}, wantNotPoll: model.ReasonNoTag},
{name: "built is not polled",
svc: auto(func(s *model.Service) { s.Built = true }), reg: &fakeRegistry{}, wantNotPoll: "built from source"},
{name: "no digest is not polled",
svc: auto(func(s *model.Service) { s.ActualDigest = "" }), reg: &fakeRegistry{}, wantNotPoll: "no registry digest"},
{name: "removed is not found",
svc: auto(func(s *model.Service) { s.RemovedAt = ptr(time.Now()) }), reg: &fakeRegistry{}, wantErr: model.ErrNotFound},
{name: "a comparable tag lists the repo once and suggests the newer tag",
svc: newerSvc(1, repoB, "1.27.3"),
reg: newerRegistry(map[string][]string{repoB: {"1.27.4", "1.28.0"}}, repoB+":1.27.3"),
wantHeads: []string{repoB + ":1.27.3"}, wantTags: []string{repoB},
check: func(t *testing.T, st *fakeStore, j *fakeJobs) {
require.Len(t, st.newer, 1)
assert.Equal(t, newerCall{1, repoB + ":1.28.0", time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)}, st.newer[0])
}},
{name: "a tag that cannot have a newer tag lists nothing",
svc: newerSvc(1, repoB, "latest"), reg: newerRegistry(nil, repoB+":latest"), wantHeads: []string{repoB + ":latest"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
st := &fakeStore{services: []model.Service{tt.svc}}
j := &fakeJobs{errFor: map[int64]error{1: tt.startErr}}
p, _ := newPoller(st, j, tt.reg, Config{})
err := p.PollService(context.Background(), 1)
var np *NotPolledError
switch {
case tt.wantNotPoll != "":
require.ErrorAs(t, err, &np)
assert.Equal(t, tt.wantNotPoll, np.Reason)
case tt.wantErr != nil:
assert.ErrorIs(t, err, tt.wantErr)
default:
require.NoError(t, err)
}
assert.Equal(t, tt.wantHeads, tt.reg.sortedCalls())
assert.Equal(t, tt.wantTags, tt.reg.tagCalls)
if tt.check != nil {
tt.check(t, st, j)
}
})
}
}
func TestPollServiceUnknownID(t *testing.T) {
reg := &fakeRegistry{}
p, _ := newPoller(&fakeStore{}, &fakeJobs{}, reg, Config{})
err := p.PollService(context.Background(), 9)
assert.ErrorIs(t, err, model.ErrNotFound)
assert.Empty(t, reg.calls)
}
func TestPollServiceOnlyThisService(t *testing.T) {
ref := repoA + ":1.4"
st := &fakeStore{services: []model.Service{svc(1, ref), svc(2, ref)}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
require.NoError(t, p.PollService(context.Background(), 1))
require.Len(t, st.available, 1)
assert.Equal(t, int64(1), st.available[0].id)
assert.Equal(t, []string{ref}, reg.sortedCalls())
for _, c := range st.pollErrors {
assert.NotContains(t, c.ids, int64(2))
}
for _, c := range st.newer {
assert.Equal(t, int64(1), c.id)
}
for _, c := range st.lastTag {
assert.Equal(t, int64(1), c.id)
}
}
func TestPollServiceCancelled(t *testing.T) {
ref := repoA + ":1.4"
st := &fakeStore{services: []model.Service{svc(1, ref)}}
reg := &fakeRegistry{release: make(chan struct{})} // lookups only end by cancel
p, _ := newPoller(st, &fakeJobs{}, reg, Config{})
ctx, cancel := context.WithCancel(context.Background())
cancel()
err := p.PollService(ctx, 1)
assert.ErrorIs(t, err, context.Canceled)
assert.Empty(t, st.pollErrors)
assert.Empty(t, st.available)
}
+42 -10
View File
@@ -1,5 +1,6 @@
// Package registry looks up the digest a tag points at in a container // Package registry looks up the digest a tag points at in a container
// registry, using HEAD requests only. // registry and lists the tags of a repository. Manifests are only ever
// requested with HEAD.
package registry package registry
import ( import (
@@ -21,9 +22,10 @@ import (
"github.com/shcizo/package-updater/internal/imageref" "github.com/shcizo/package-updater/internal/imageref"
) )
// Client looks up tag digests with HEAD requests only. Docker Hub counts a // Client looks up tag digests and lists tags. Docker Hub counts a GET of a
// GET of a manifest as a pull against the rate limit and a HEAD as nothing, // manifest as a pull against the rate limit and a HEAD as nothing, so Client
// so Client never issues a manifest GET. // never issues a manifest GET. The only GETs it sends are the /v2/ ping and
// tags/list, which Docker Hub does not count as a pull.
type Client struct { type Client struct {
keychain authn.Keychain keychain authn.Keychain
userAgent string userAgent string
@@ -65,16 +67,46 @@ func (c *Client) Digest(ctx context.Context, ref string) (string, error) {
remote.WithAuth(auth), remote.WithAuth(auth),
remote.WithUserAgent(c.userAgent)) remote.WithUserAgent(c.userAgent))
if err != nil { if err != nil {
var terr *transport.Error return "", c.lookupError("lookup", r.String(), ref, err)
if errors.As(err, &terr) {
return "", fmt.Errorf("lookup %s: %s", r, describe(terr))
}
c.log.Warn("registry_lookup_failed", "ref", ref, "error", err)
return "", fmt.Errorf("lookup %s: %s", r, classify(err))
} }
return desc.Digest.String(), nil return desc.Digest.String(), nil
} }
// Tags returns every tag of repo ("host/path", or a Docker Hub name), in the
// order the registry lists them. It sends GET /v2/<repo>/tags/list, following
// the registry's pagination links. Errors are sanitised like Digest's.
func (c *Client) Tags(ctx context.Context, repo string) ([]string, error) {
r, err := name.NewRepository(repo)
if err != nil {
return nil, fmt.Errorf("parse %s: %w", repo, err)
}
auth, err := c.keychain.Resolve(r)
if err != nil {
return nil, fmt.Errorf("list %s: %w", r, err)
}
tags, err := remote.List(r,
remote.WithContext(ctx),
remote.WithAuth(auth),
remote.WithUserAgent(c.userAgent))
if err != nil {
return nil, c.lookupError("list", r.String(), repo, err)
}
return tags, nil
}
// lookupError turns a failed registry request into an error that is safe to
// show: a registry answer is described by its status and codes, anything
// else by a fixed text. The full error goes to the log. verb and target
// start the message; ref is what the log line names.
func (c *Client) lookupError(verb, target, ref string, err error) error {
var terr *transport.Error
if errors.As(err, &terr) {
return fmt.Errorf("%s %s: %s", verb, target, describe(terr))
}
c.log.Warn("registry_request_failed", "ref", ref, "error", err)
return fmt.Errorf("%s %s: %s", verb, target, classify(err))
}
// classify returns a fixed text for an error that is not a registry answer. // classify returns a fixed text for an error that is not a registry answer.
// ggcr copies registry-controlled text (token responses, authentication // ggcr copies registry-controlled text (token responses, authentication
// challenges, redirect URLs, certificate names) into such errors, so none of // challenges, redirect URLs, certificate names) into such errors, so none of
+101 -1
View File
@@ -134,6 +134,8 @@ func TestDigestIgnoresDigestSuffix(t *testing.T) {
assert.Equal(t, want, got) assert.Equal(t, want, got)
} }
// Digest asks the manifest endpoint with HEAD only; the one other request is
// the GET /v2/ ping that every client sends first.
func TestDigestUsesHeadOnly(t *testing.T) { func TestDigestUsesHeadOnly(t *testing.T) {
dockerConfig(t, "") dockerConfig(t, "")
reg := newTestRegistry(t, "", "") reg := newTestRegistry(t, "", "")
@@ -236,7 +238,7 @@ func TestCredsStoreWithoutHelperFailsTheLookup(t *testing.T) {
} }
// A registry that leaves out Docker-Content-Digest must not make the client // A registry that leaves out Docker-Content-Digest must not make the client
// retry with a manifest download, which Docker Hub counts as a pull. // retry with a manifest GET, which Docker Hub counts as a pull.
func TestMissingDigestHeaderDoesNotFallBackToDownload(t *testing.T) { func TestMissingDigestHeaderDoesNotFallBackToDownload(t *testing.T) {
dockerConfig(t, "") dockerConfig(t, "")
var mu sync.Mutex var mu sync.Mutex
@@ -334,3 +336,101 @@ func TestClassify(t *testing.T) {
}) })
} }
} }
// pushTags stores a random index under each tag of repo.
func (tr *testRegistry) pushTags(t *testing.T, repo string, tags ...string) {
t.Helper()
for _, tag := range tags {
tr.pushIndex(t, repo, tag)
}
}
// Digest and Tags together never GET a manifest: every manifest request is a
// HEAD, and the only other requests are the /v2/ ping and tags/list.
func TestNoManifestGet(t *testing.T) {
dockerConfig(t, "")
reg := newTestRegistry(t, "", "")
reg.pushTags(t, "team/app", "1.2", "1.3")
reg.reset()
_, err := newClient().Digest(context.Background(), reg.host+"/team/app:1.2")
require.NoError(t, err)
_, err = newClient().Tags(context.Background(), reg.host+"/team/app")
require.NoError(t, err)
reg.mu.Lock()
defer reg.mu.Unlock()
for _, r := range reg.requests {
switch {
case strings.Contains(r.path, "/manifests/"):
assert.Equal(t, http.MethodHead, r.method, r.path)
case r.path == "/v2/", r.path == "/v2/team/app/tags/list":
assert.Equal(t, http.MethodGet, r.method, r.path)
default:
assert.Failf(t, "unexpected request", "%s %s", r.method, r.path)
}
}
}
func TestTagsListsAllTags(t *testing.T) {
dockerConfig(t, "")
reg := newTestRegistry(t, "", "")
reg.pushTags(t, "team/app", "1.2", "1.3", "latest")
got, err := newClient().Tags(context.Background(), reg.host+"/team/app")
require.NoError(t, err)
assert.ElementsMatch(t, []string{"1.2", "1.3", "latest"}, got)
}
// ggcr's in-memory registry sends no Link header, so pagination is tested
// against a handler that does.
func TestTagsFollowsLinkPagination(t *testing.T) {
dockerConfig(t, "")
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path == "/v2/":
case r.URL.Path == "/v2/a/b/tags/list" && r.URL.Query().Get("last") == "":
w.Header().Set("Link", `</v2/a/b/tags/list?last=1.3&n=2>; rel="next"`)
_, _ = w.Write([]byte(`{"name":"a/b","tags":["1.2","1.3"]}`))
case r.URL.Path == "/v2/a/b/tags/list":
_, _ = w.Write([]byte(`{"name":"a/b","tags":["1.4"]}`))
default:
w.WriteHeader(http.StatusNotFound)
}
}))
t.Cleanup(srv.Close)
got, err := newClient().Tags(context.Background(), strings.TrimPrefix(srv.URL, "http://")+"/a/b")
require.NoError(t, err)
assert.Equal(t, []string{"1.2", "1.3", "1.4"}, got)
}
func TestTagsUsesDockerConfigCredentials(t *testing.T) {
reg := newTestRegistry(t, "bot", "s3cret")
reg.pushIndex(t, "team/app", "1.2", remote.WithAuth(&authn.Basic{Username: "bot", Password: "s3cret"}))
auth := base64.StdEncoding.EncodeToString([]byte("bot:s3cret"))
dockerConfig(t, `{"auths":{"`+reg.host+`":{"auth":"`+auth+`"}}}`)
got, err := newClient().Tags(context.Background(), reg.host+"/team/app")
require.NoError(t, err)
assert.Equal(t, []string{"1.2"}, got)
}
func TestTagsErrorBodyDoesNotLeak(t *testing.T) {
dockerConfig(t, "")
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusBadRequest)
_, _ = w.Write([]byte(`{"errors":[{"code":"NAME_UNKNOWN","message":"internal-secret-body"}]}`))
}))
t.Cleanup(srv.Close)
host := strings.TrimPrefix(srv.URL, "http://")
_, err := newClient().Tags(context.Background(), host+"/a/b")
require.Error(t, err)
assert.NotContains(t, err.Error(), "internal-secret-body")
assert.ErrorContains(t, err, "list "+host+"/a/b: registry answered 400 Bad Request")
}
@@ -275,6 +275,7 @@ func TestMergeService_MovesHistoryModeAndDesiredToTheNewLocation(t *testing.T) {
require.NoError(t, s.SetLastTagDigest(ctx, f.to, "sha256:fromto")) require.NoError(t, s.SetLastTagDigest(ctx, f.to, "sha256:fromto"))
require.NoError(t, s.SetAvailable(ctx, f.to, model.Available{Ref: sluskDrift + ":3", Digest: digNew, Source: model.SourcePoll, At: t0})) require.NoError(t, s.SetAvailable(ctx, f.to, model.Available{Ref: sluskDrift + ":3", Digest: digNew, Source: model.SourcePoll, At: t0}))
require.NoError(t, s.SetPollError(ctx, []int64{f.to}, "registry down")) require.NoError(t, s.SetPollError(ctx, []int64{f.to}, "registry down"))
require.NoError(t, s.SetNewer(ctx, f.to, sluskDrift+":3.1", t0))
// A job on the old row is finished, and awaiting_inventory is set on both. // A job on the old row is finished, and awaiting_inventory is set on both.
before, err := s.GetService(ctx, f.from) before, err := s.GetService(ctx, f.from)
require.NoError(t, err) require.NoError(t, err)
@@ -297,6 +298,9 @@ func TestMergeService_MovesHistoryModeAndDesiredToTheNewLocation(t *testing.T) {
assert.Equal(t, sluskDrift+":3", got.AvailableRef) assert.Equal(t, sluskDrift+":3", got.AvailableRef)
assert.Equal(t, digNew, got.AvailableDigest) assert.Equal(t, digNew, got.AvailableDigest)
assert.Equal(t, "registry down", got.LastPollError) assert.Equal(t, "registry down", got.LastPollError)
assert.Equal(t, sluskDrift+":3.1", got.NewerRef, "newer tag from the duplicate")
require.NotNil(t, got.NewerAt)
assert.True(t, t0.Equal(*got.NewerAt))
assert.Equal(t, before.LastTagDigest, got.LastTagDigest, "an existing last_tag_digest wins") assert.Equal(t, before.LastTagDigest, got.LastTagDigest, "an existing last_tag_digest wins")
_, err = s.GetService(ctx, f.to) _, err = s.GetService(ctx, f.to)
@@ -418,11 +422,11 @@ func TestMergeService_Refusals(t *testing.T) {
{"the updater itself (new)", func(t *testing.T, s *Store, f moveFixture) error { {"the updater itself (new)", func(t *testing.T, s *Store, f moveFixture) error {
mustExec(t, s, `UPDATE services SET self = true WHERE id = $1`, f.to) mustExec(t, s, `UPDATE services SET self = true WHERE id = $1`, f.to)
return s.MergeService(context.Background(), f.from, f.to) return s.MergeService(context.Background(), f.from, f.to)
}, refused(model.ReasonSelf)}, }, refused(model.ReasonMergeSelf)},
{"the updater itself (old)", func(t *testing.T, s *Store, f moveFixture) error { {"the updater itself (old)", func(t *testing.T, s *Store, f moveFixture) error {
mustExec(t, s, `UPDATE services SET self = true WHERE id = $1`, f.from) mustExec(t, s, `UPDATE services SET self = true WHERE id = $1`, f.from)
return s.MergeService(context.Background(), f.from, f.to) return s.MergeService(context.Background(), f.from, f.to)
}, refused(model.ReasonSelf)}, }, refused(model.ReasonMergeSelf)},
{"active job on the old service", func(t *testing.T, s *Store, f moveFixture) error { {"active job on the old service", func(t *testing.T, s *Store, f moveFixture) error {
_, err := s.CreateJob(context.Background(), newJob(f.from, f.a)) _, err := s.CreateJob(context.Background(), newJob(f.from, f.a))
require.NoError(t, err) require.NoError(t, err)
+8 -7
View File
@@ -106,9 +106,9 @@ type mergeRow struct {
// MergeService moves the live service toID onto the removed service fromID: // MergeService moves the live service toID onto the removed service fromID:
// the old row keeps its id, first_seen_at, update mode and desired ref, takes // the old row keeps its id, first_seen_at, update mode and desired ref, takes
// the location and inventory of the live row (and its available suggestion), // the location and inventory of the live row (and its available and newer-tag
// becomes live, and the live row is deleted. The jobs of the live row move to // suggestions), becomes live, and the live row is deleted. The jobs of the live
// the old one. This is the only place a services row is deleted. // row move to the old one. This is the only place a services row is deleted.
// //
// It returns *model.MergeRefusedError when the pair cannot be merged, // It returns *model.MergeRefusedError when the pair cannot be merged,
// model.ErrJobActive when either row has a queued or running job, // model.ErrJobActive when either row has a queued or running job,
@@ -195,12 +195,12 @@ func (s *Store) MergeService(ctx context.Context, fromID, toID int64) error {
SELECT agent_id, location_key, compose_project, compose_service, working_dir, config_files, built, SELECT agent_id, location_key, compose_project, compose_service, working_dir, config_files, built,
swarm_cluster_id, swarm_service, container_name, image_repo, actual_ref, actual_digest, status, self, swarm_cluster_id, swarm_service, container_name, image_repo, actual_ref, actual_digest, status, self,
updated_at, available_ref, available_digest, available_source, available_at, last_poll_error, updated_at, available_ref, available_digest, available_source, available_at, last_poll_error,
last_tag_digest last_tag_digest, newer_ref, newer_at
FROM services WHERE id = $1`, toID).Scan( FROM services WHERE id = $1`, toID).Scan(
&t.AgentID, &t.LocationKey, &t.ComposeProject, &t.ComposeService, &t.WorkingDir, &t.ConfigFiles, &t.Built, &t.AgentID, &t.LocationKey, &t.ComposeProject, &t.ComposeService, &t.WorkingDir, &t.ConfigFiles, &t.Built,
&t.SwarmClusterID, &t.SwarmService, &t.ContainerName, &t.ImageRepo, &t.ActualRef, &t.ActualDigest, &t.Status, &t.Self, &t.SwarmClusterID, &t.SwarmService, &t.ContainerName, &t.ImageRepo, &t.ActualRef, &t.ActualDigest, &t.Status, &t.Self,
&t.UpdatedAt, &t.AvailableRef, &t.AvailableDigest, &t.AvailableSource, &t.AvailableAt, &t.LastPollError, &t.UpdatedAt, &t.AvailableRef, &t.AvailableDigest, &t.AvailableSource, &t.AvailableAt, &t.LastPollError,
&lastTagDigest) &lastTagDigest, &t.NewerRef, &t.NewerAt)
if err != nil { if err != nil {
return fmt.Errorf("merge: read new service: %w", err) return fmt.Errorf("merge: read new service: %w", err)
} }
@@ -215,13 +215,14 @@ func (s *Store) MergeService(ctx context.Context, fromID, toID int64) error {
available_ref = $18, available_digest = $19, available_source = $20, available_at = $21, available_ref = $18, available_digest = $19, available_source = $20, available_at = $21,
last_poll_error = $22, last_poll_error = $22,
last_tag_digest = CASE WHEN last_tag_digest <> '' THEN last_tag_digest ELSE $23 END, last_tag_digest = CASE WHEN last_tag_digest <> '' THEN last_tag_digest ELSE $23 END,
newer_ref = $24, newer_at = $25,
removed_at = NULL, awaiting_inventory = false removed_at = NULL, awaiting_inventory = false
WHERE id = $1`, WHERE id = $1`,
fromID, t.AgentID, t.LocationKey, t.ComposeProject, t.ComposeService, t.WorkingDir, fromID, t.AgentID, t.LocationKey, t.ComposeProject, t.ComposeService, t.WorkingDir,
t.ConfigFiles, t.Built, t.SwarmClusterID, t.SwarmService, t.ContainerName, t.ConfigFiles, t.Built, t.SwarmClusterID, t.SwarmService, t.ContainerName,
t.ImageRepo, t.ActualRef, t.ActualDigest, t.Status, t.Self, t.UpdatedAt, t.ImageRepo, t.ActualRef, t.ActualDigest, t.Status, t.Self, t.UpdatedAt,
t.AvailableRef, t.AvailableDigest, t.AvailableSource, t.AvailableAt, t.AvailableRef, t.AvailableDigest, t.AvailableSource, t.AvailableAt,
t.LastPollError, lastTagDigest); err != nil { t.LastPollError, lastTagDigest, t.NewerRef, t.NewerAt); err != nil {
return fmt.Errorf("merge: update old service: %w", err) return fmt.Errorf("merge: update old service: %w", err)
} }
if err := tx.Commit(ctx); err != nil { if err := tx.Commit(ctx); err != nil {
@@ -243,7 +244,7 @@ func mergeRefusal(from, to mergeRow) string {
case from.kind == string(protocol.KindContainer): case from.kind == string(protocol.KindContainer):
return model.ReasonMergeContainer return model.ReasonMergeContainer
case from.self || to.self: case from.self || to.self:
return model.ReasonSelf return model.ReasonMergeSelf
} }
return "" return ""
} }
@@ -0,0 +1,9 @@
-- Newer version tag of the followed tag, found in the registry's tag list.
-- A suggestion only: nothing deploys it automatically.
ALTER TABLE services ADD COLUMN newer_ref text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN newer_at timestamptz;
-- A suggested tag is recorded in image_versions with source 'tags'.
ALTER TABLE image_versions DROP CONSTRAINT image_versions_source_check;
ALTER TABLE image_versions ADD CONSTRAINT image_versions_source_check
CHECK (source IN ('webhook', 'poll', 'running', 'tags'));
+46 -2
View File
@@ -175,7 +175,7 @@ const serviceColumns = `s.id, s.kind, s.location_key, s.agent_id, COALESCE(a.nam
s.compose_project, s.compose_service, s.working_dir, s.config_files, s.built, s.swarm_cluster_id, s.compose_project, s.compose_service, s.working_dir, s.config_files, s.built, s.swarm_cluster_id,
s.swarm_service, s.container_name, s.image_repo, s.desired_ref, s.actual_ref, s.actual_digest, s.swarm_service, s.container_name, s.image_repo, s.desired_ref, s.actual_ref, s.actual_digest,
s.status, s.self, s.update_mode, s.available_ref, s.available_digest, s.available_source, s.available_at, s.status, s.self, s.update_mode, s.available_ref, s.available_digest, s.available_source, s.available_at,
s.last_poll_error, s.last_tag_digest, s.awaiting_inventory, s.first_seen_at, s.updated_at, s.removed_at` s.last_poll_error, s.newer_ref, s.newer_at, s.last_tag_digest, s.awaiting_inventory, s.first_seen_at, s.updated_at, s.removed_at`
func scanService(row pgx.Row) (model.Service, error) { func scanService(row pgx.Row) (model.Service, error) {
var m model.Service var m model.Service
@@ -184,7 +184,7 @@ func scanService(row pgx.Row) (model.Service, error) {
&m.ComposeProject, &m.ComposeService, &m.WorkingDir, &m.ConfigFiles, &m.Built, &m.SwarmClusterID, &m.ComposeProject, &m.ComposeService, &m.WorkingDir, &m.ConfigFiles, &m.Built, &m.SwarmClusterID,
&m.SwarmService, &m.ContainerName, &m.ImageRepo, &m.DesiredRef, &m.ActualRef, &m.ActualDigest, &m.SwarmService, &m.ContainerName, &m.ImageRepo, &m.DesiredRef, &m.ActualRef, &m.ActualDigest,
&m.Status, &m.Self, &m.UpdateMode, &m.AvailableRef, &m.AvailableDigest, &m.AvailableSource, &m.AvailableAt, &m.Status, &m.Self, &m.UpdateMode, &m.AvailableRef, &m.AvailableDigest, &m.AvailableSource, &m.AvailableAt,
&m.LastPollError, &m.LastTagDigest, &m.AwaitingInventory, &m.FirstSeenAt, &m.UpdatedAt, &m.RemovedAt) &m.LastPollError, &m.NewerRef, &m.NewerAt, &m.LastTagDigest, &m.AwaitingInventory, &m.FirstSeenAt, &m.UpdatedAt, &m.RemovedAt)
m.Kind = protocol.Kind(kind) m.Kind = protocol.Kind(kind)
return m, err return m, err
} }
@@ -246,6 +246,33 @@ func (s *Store) SetAvailable(ctx context.Context, id int64, a model.Available) e
return nil return nil
} }
// SetNewer sets the service's newer-tag suggestion (model.Service.NewerRef);
// ref "" clears it. A non-empty ref is also recorded in image_versions with
// source "tags", in the same transaction, unless that (repo, tag) is already
// there. The digest is unknown, so the row has none.
func (s *Store) SetNewer(ctx context.Context, id int64, ref string, at time.Time) error {
tx, err := s.pool.Begin(ctx)
if err != nil {
return fmt.Errorf("set newer: begin: %w", err)
}
defer func() { _ = tx.Rollback(ctx) }()
if _, err := tx.Exec(ctx, `
UPDATE services SET newer_ref = $2, newer_at = CASE WHEN $2 = '' THEN NULL ELSE $3::timestamptz END
WHERE id = $1`, id, ref, at); err != nil {
return fmt.Errorf("set newer: %w", err)
}
if ref != "" {
p := imageref.Parse(ref)
if err := insertVersion(ctx, tx, p.Repo, p.Tag, "", model.SourceTags, at); err != nil {
return err
}
}
if err := tx.Commit(ctx); err != nil {
return fmt.Errorf("set newer: commit: %w", err)
}
return nil
}
// SetPollError sets last_poll_error on the given services; msg "" clears it. // SetPollError sets last_poll_error on the given services; msg "" clears it.
func (s *Store) SetPollError(ctx context.Context, ids []int64, msg string) error { func (s *Store) SetPollError(ctx context.Context, ids []int64, msg string) error {
if len(ids) == 0 { if len(ids) == 0 {
@@ -313,6 +340,23 @@ func (s *Store) GetImageVersion(ctx context.Context, id int64) (model.ImageVersi
return v, nil return v, nil
} }
// GetTagVersion returns the version row of repo and tag with an empty digest, which
// SetNewer and the webhook create. Returns model.ErrNotFound if there is none.
func (s *Store) GetTagVersion(ctx context.Context, repo, tag string) (model.ImageVersion, error) {
var v model.ImageVersion
err := s.pool.QueryRow(ctx, `
SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions
WHERE repo = $1 AND tag = $2 AND digest = ''`, repo, tag,
).Scan(&v.ID, &v.Repo, &v.Tag, &v.Digest, &v.Source, &v.FirstSeenAt)
if errors.Is(err, pgx.ErrNoRows) {
return model.ImageVersion{}, model.ErrNotFound
}
if err != nil {
return model.ImageVersion{}, fmt.Errorf("get tag version: %w", err)
}
return v, nil
}
// SetUpdateMode sets the service's update mode (model.ModeManual or // SetUpdateMode sets the service's update mode (model.ModeManual or
// model.ModeAuto). Returns model.ErrNotFound if the service does not exist // model.ModeAuto). Returns model.ErrNotFound if the service does not exist
// or is marked removed. // or is marked removed.
@@ -214,3 +214,79 @@ func TestSetLastTagDigest(t *testing.T) {
require.Len(t, list, 1) require.Len(t, list, 1)
assert.Equal(t, "sha256:n1", list[0].LastTagDigest) assert.Equal(t, "sha256:n1", list[0].LastTagDigest)
} }
func TestSetNewer_SetsAndClears(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
_, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetNewer(ctx, svc, sluskRepo+":1.5.0", t0))
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, sluskRepo+":1.5.0", got.NewerRef)
require.NotNil(t, got.NewerAt)
assert.True(t, t0.Equal(*got.NewerAt))
v, err := s.GetTagVersion(ctx, sluskRepo, "1.5.0")
require.NoError(t, err)
assert.Equal(t, model.SourceTags, v.Source)
assert.Empty(t, v.Digest)
assert.True(t, t0.Equal(v.FirstSeenAt))
require.NoError(t, s.SetNewer(ctx, svc, "", t0.Add(time.Hour)))
got, err = s.GetService(ctx, svc)
require.NoError(t, err)
assert.Empty(t, got.NewerRef)
assert.Nil(t, got.NewerAt)
_, err = s.GetTagVersion(ctx, sluskRepo, "1.5.0")
assert.NoError(t, err, "clearing the suggestion keeps the version")
}
func TestSetNewer_KeepsEarlierVersionSource(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
_, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":1.5.0", Source: model.SourceWebhook, At: t0}))
require.NoError(t, s.SetNewer(ctx, svc, sluskRepo+":1.5.0", t0.Add(time.Hour)))
v, err := s.GetTagVersion(ctx, sluskRepo, "1.5.0")
require.NoError(t, err)
assert.Equal(t, model.SourceWebhook, v.Source)
assert.True(t, t0.Equal(v.FirstSeenAt))
}
func TestRecentVersions_TagsRowHiddenOnceDigestKnown(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
_, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetNewer(ctx, svc, sluskRepo+":1.5.0", t0.Add(time.Minute)))
require.Len(t, versionsOf(t, s, sluskRepo), 2, "running 1.4.2 and the tags row")
require.NoError(t, s.SetAvailable(ctx, svc, model.Available{Ref: sluskRepo + ":1.5.0", Digest: "sha256:n1", Source: model.SourcePoll, At: t0.Add(2 * time.Minute)}))
var refs []string
for _, v := range versionsOf(t, s, sluskRepo) {
refs = append(refs, v.Ref())
}
assert.Equal(t, []string{sluskRepo + ":1.5.0@sha256:n1", sluskRepo + ":1.4.2@" + digD1}, refs)
}
func TestGetTagVersion(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
_, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetNewer(ctx, svc, sluskRepo+":1.5.0", t0))
got, err := s.GetTagVersion(ctx, sluskRepo, "1.5.0")
require.NoError(t, err)
assert.Equal(t, sluskRepo+":1.5.0", got.Ref())
_, err = s.GetTagVersion(ctx, sluskRepo, "9.9.9")
assert.ErrorIs(t, err, model.ErrNotFound)
_, err = s.GetTagVersion(ctx, "other/repo", "1.5.0")
assert.ErrorIs(t, err, model.ErrNotFound)
_, err = s.GetTagVersion(ctx, sluskRepo, "1.4.2")
assert.ErrorIs(t, err, model.ErrNotFound, "a row with a digest is not a tag version")
}
+101
View File
@@ -0,0 +1,101 @@
// Package tagver decides which tag of a repository is a newer version of the
// tag a service follows, using only the shape of the tag names.
package tagver
import (
"regexp"
"strconv"
"strings"
)
// tagPattern splits a tag into a prefix without digits, a dotted numeric
// part and whatever follows it. "v1.2.3-alpine" gives "v", "1.2.3", "-alpine".
var tagPattern = regexp.MustCompile(`^([^0-9]*)([0-9]+(?:\.[0-9]+)*)(.*)$`)
// cosignPattern matches the tags cosign pushes next to an image to store its
// signature, attestation or SBOM. They look like versions to a naive check.
// The pattern is taken from Diun (crazy-max/diun, MIT, pkg/registry/tags.go).
var cosignPattern = regexp.MustCompile(`^sha256-[0-9a-f]{64}(\.att|\.sbom|\.sig)?$`)
// parsed is a tag split into its parts.
type parsed struct {
prefix, suffix string
nums []int
}
// parse splits tag. It returns false for a cosign tag, a tag without digits
// and a numeric component that does not fit an int.
func parse(tag string) (parsed, bool) {
if cosignPattern.MatchString(tag) {
return parsed{}, false
}
m := tagPattern.FindStringSubmatch(tag)
if m == nil {
return parsed{}, false
}
var nums []int
for _, part := range strings.Split(m[2], ".") {
n, err := strconv.Atoi(part)
if err != nil {
return parsed{}, false
}
nums = append(nums, n)
}
return parsed{prefix: m[1], suffix: m[3], nums: nums}, true
}
// Comparable reports whether tag can ever get a newer tag: it splits into
// prefix, numeric part and suffix, has at least two numeric components, and
// is not a cosign artifact tag. A tag with one component ("7", "pg16",
// "8-bookworm") is a moving tag, not a version.
func Comparable(tag string) bool {
p, ok := parse(tag)
return ok && len(p.nums) >= 2
}
// Newer returns the highest candidate that is newer than current, and false
// when there is none. A candidate qualifies when it has the same prefix and
// suffix and the same number of components as current, the same first
// component, and a greater numeric part. Of two candidates with the same
// numeric value the lexically smaller string wins, so the result does not
// depend on the order of candidates. A tag such as "name2-v1.2.3", where a
// digit comes before the version, is not comparable.
func Newer(current string, candidates []string) (string, bool) {
cur, ok := parse(current)
if !ok || len(cur.nums) < 2 {
return "", false
}
best, found := "", false
var bestNums []int
for _, c := range candidates {
p, ok := parse(c)
if !ok || p.prefix != cur.prefix || p.suffix != cur.suffix ||
len(p.nums) != len(cur.nums) || p.nums[0] != cur.nums[0] {
continue
}
if compare(p.nums, cur.nums) <= 0 {
continue
}
if !found {
best, bestNums, found = c, p.nums, true
continue
}
if cmp := compare(p.nums, bestNums); cmp > 0 || (cmp == 0 && c < best) {
best, bestNums = c, p.nums
}
}
return best, found
}
// compare compares two numeric parts of equal length.
func compare(a, b []int) int {
for i := range a {
switch {
case a[i] < b[i]:
return -1
case a[i] > b[i]:
return 1
}
}
return 0
}
+71
View File
@@ -0,0 +1,71 @@
package tagver
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
)
func TestComparable(t *testing.T) {
sig := "sha256-" + strings.Repeat("a", 64) + ".sig"
tests := []struct {
tag string
want bool
}{
{"latest", false},
{"release", false},
{"stable", false},
{"edge", false},
{"alpine", false},
{"7", false},
{"7-alpine", false},
{"8-bookworm", false},
{"pg16", false},
{"14-vectorchord0.4.3-pgvectors0.2.0", false},
{"name2-v1.2.3", false},
{sig, false},
{"99999999999999999999.1", false},
{"1.2", true},
{"1.2.3", true},
{"v1.2.3", true},
{"11.0-latest", true},
}
for _, tt := range tests {
t.Run(tt.tag, func(t *testing.T) {
assert.Equal(t, tt.want, Comparable(tt.tag))
})
}
}
func TestNewer(t *testing.T) {
tests := []struct {
name string
current string
candidates []string
want string
ok bool
}{
{"highest within major", "1.2.3", []string{"1.2.4", "1.3.0", "2.0.0", "1.2.4-rc1", "1.3", "v1.2.4"}, "1.3.0", true},
{"v prefix kept", "v1.2.3", []string{"1.2.4", "v1.2.4"}, "v1.2.4", true},
{"name prefix kept", "name-v1.2.3", []string{"v1.2.4", "name-v1.2.4"}, "name-v1.2.4", true},
{"suffix kept", "11.0-latest", []string{"11.1", "11.1-latest"}, "11.1-latest", true},
{"two components", "7.0", []string{"8.0", "7.0.1", "7.2", "7.1"}, "7.2", true},
{"equal is not newer", "1.2.3", []string{"1.2.3"}, "", false},
{"older is not newer", "1.2.3", []string{"1.2.2", "1.1.9"}, "", false},
{"numeric not lexical", "1.9.0", []string{"1.10.0"}, "1.10.0", true},
{"empty candidates", "1.2.3", nil, "", false},
{"moving tag", "latest", []string{"1.2.3"}, "", false},
{"single component", "7", []string{"8"}, "", false},
{"digit before version", "name2-v1.2.3", []string{"name2-v1.2.4"}, "", false},
{"cosign tag ignored", "1.2.3", []string{"sha256-" + strings.Repeat("0", 64) + ".sig"}, "", false},
{"equal value picks smaller string", "1.2.3", []string{"1.02.4", "1.2.4"}, "1.02.4", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, ok := Newer(tt.current, tt.candidates)
assert.Equal(t, tt.ok, ok)
assert.Equal(t, tt.want, got)
})
}
}
+3 -3
View File
@@ -273,15 +273,15 @@ func TestServiceDetail_DriftAndMoveData(t *testing.T) {
{name: "swarm service can merge, no suggestion", id: 3, wantSources: sources}, {name: "swarm service can merge, no suggestion", id: 3, wantSources: sources},
{name: "container cannot merge", id: 4}, {name: "container cannot merge", id: 4},
{name: "self cannot merge", id: 5}, {name: "self cannot merge", id: 5},
{name: "restore refusal is passed on", id: 1, restorable: &model.NotUpdatableError{Reason: model.ReasonSelf}, {name: "restore refusal is passed on", id: 1, restorable: &model.NotUpdatableError{Reason: model.ReasonContainer},
wantBlocked: model.ReasonSelf, wantFrom: &move, wantSources: sources}, wantBlocked: model.ReasonContainer, wantFrom: &move, wantSources: sources},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
store := &fakeStore{services: []model.Service{live, gone, swarm, container, self}, store := &fakeStore{services: []model.Service{live, gone, swarm, container, self},
moves: []model.Move{move}, sources: sources} moves: []model.Move{move}, sources: sources}
jobs := &fakeJobs{restorable: tt.restorable} jobs := &fakeJobs{restorable: tt.restorable}
s, err := New(store, &fakeHub{}, jobs, events.NewBroker(), Config{}, slog.New(slog.NewTextHandler(io.Discard, nil))) s, err := New(store, &fakeHub{}, jobs, events.NewBroker(), &fakePoller{}, Config{}, slog.New(slog.NewTextHandler(io.Discard, nil)))
require.NoError(t, err) require.NoError(t, err)
req := httptest.NewRequest(http.MethodGet, "/services/"+strconv.FormatInt(tt.id, 10), nil) req := httptest.NewRequest(http.MethodGet, "/services/"+strconv.FormatInt(tt.id, 10), nil)
+86
View File
@@ -0,0 +1,86 @@
package web
import (
"context"
"io"
"net/http"
"net/url"
"sync"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/master/poller"
)
// fakePoller records the services it was asked to check and answers err.
type fakePoller struct {
mu sync.Mutex
err error
calls []int64
}
func (f *fakePoller) PollService(_ context.Context, id int64) error {
f.mu.Lock()
defer f.mu.Unlock()
f.calls = append(f.calls, id)
return f.err
}
func (f *fakePoller) called() []int64 {
f.mu.Lock()
defer f.mu.Unlock()
return append([]int64(nil), f.calls...)
}
func TestPollService(t *testing.T) {
tests := []struct {
name string
id string
err error
wantStatus int
wantLocation string
wantBody string
wantCalls []int64
}{
{name: "redirects to the service page", id: "1", wantStatus: http.StatusSeeOther, wantLocation: "/services/1", wantCalls: []int64{1}},
{name: "unknown service is 404", id: "1", err: model.ErrNotFound, wantStatus: http.StatusNotFound, wantCalls: []int64{1}},
{name: "invalid id is 400", id: "abc", wantStatus: http.StatusBadRequest},
{name: "not polled is 409 with the reason", id: "1", err: &poller.NotPolledError{Reason: model.ReasonNoTag},
wantStatus: http.StatusConflict, wantBody: "Cannot check: follows no tag.", wantCalls: []int64{1}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
fp := &fakePoller{err: tt.err}
srv, _, _, _, _ := newTestEnvFull(t, func(*Config) {}, fp)
resp, err := noRedirectClient().PostForm(srv.URL+"/services/"+tt.id+"/poll", url.Values{"csrf_token": {"csrf"}})
require.NoError(t, err)
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
assert.Equal(t, tt.wantStatus, resp.StatusCode)
assert.Equal(t, tt.wantLocation, resp.Header.Get("Location"))
assert.Contains(t, string(body), tt.wantBody)
assert.Equal(t, tt.wantCalls, fp.called())
})
}
}
func TestServicePage_CheckNowForm(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].ActualDigest = "sha256:" + "a"
_, body := get(t, srv, "/services/1")
assert.Contains(t, body, `action="/services/1/poll"`)
assert.Contains(t, body, `<button type="submit">Check now</button>`)
assert.NotContains(t, body, "Check unavailable")
}
func TestServicePage_CheckNowDisabledWhenNotPolled(t *testing.T) {
srv, _, _, _, _ := newTestEnv(t)
_, body := get(t, srv, "/services/2") // follows no tag
assert.Contains(t, body, `<button type="submit" disabled>Check now</button>`)
assert.Contains(t, body, "Check unavailable: follows no tag.")
}
+72 -2
View File
@@ -18,6 +18,7 @@ import (
"github.com/shcizo/package-updater/internal/agenttoken" "github.com/shcizo/package-updater/internal/agenttoken"
"github.com/shcizo/package-updater/internal/master/events" "github.com/shcizo/package-updater/internal/master/events"
"github.com/shcizo/package-updater/internal/master/model" "github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/master/poller"
"github.com/shcizo/package-updater/internal/master/web/icons" "github.com/shcizo/package-updater/internal/master/web/icons"
"github.com/shcizo/package-updater/internal/master/webauth" "github.com/shcizo/package-updater/internal/master/webauth"
"github.com/shcizo/package-updater/internal/protocol" "github.com/shcizo/package-updater/internal/protocol"
@@ -38,6 +39,9 @@ var agentNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`)
type Store interface { type Store interface {
ListServices(ctx context.Context) ([]model.Service, error) ListServices(ctx context.Context) ([]model.Service, error)
GetService(ctx context.Context, id int64) (model.Service, error) GetService(ctx context.Context, id int64) (model.Service, error)
// GetTagVersion returns the version of repo with this tag and no digest;
// model.ErrNotFound if there is none.
GetTagVersion(ctx context.Context, repo, tag string) (model.ImageVersion, error)
// RecentVersions returns up to limit versions of repo the master has // RecentVersions returns up to limit versions of repo the master has
// seen, most recently first seen first. // seen, most recently first seen first.
RecentVersions(ctx context.Context, repo string, limit int) ([]model.ImageVersion, error) RecentVersions(ctx context.Context, repo string, limit int) ([]model.ImageVersion, error)
@@ -82,6 +86,13 @@ type Jobs interface {
RestoreDesired(ctx context.Context, serviceID int64, requestedBy string) (model.Job, error) RestoreDesired(ctx context.Context, serviceID int64, requestedBy string) (model.Job, error)
} }
// Poller checks the registry for one service. *poller.Poller satisfies it.
type Poller interface {
// PollService returns *poller.NotPolledError when the service is not
// polled and model.ErrNotFound for an unknown or removed one.
PollService(ctx context.Context, id int64) error
}
// Broker delivers change signals to open streams and accepts signals from // Broker delivers change signals to open streams and accepts signals from
// UI actions. *events.Broker satisfies it. // UI actions. *events.Broker satisfies it.
type Broker interface { type Broker interface {
@@ -105,6 +116,7 @@ type Server struct {
hub Hub hub Hub
jobs Jobs jobs Jobs
broker Broker broker Broker
poller Poller
cfg Config cfg Config
log *slog.Logger log *slog.Logger
now func() time.Time now func() time.Time
@@ -113,7 +125,7 @@ type Server struct {
// New parses the templates. Each page is its own template set so every page // New parses the templates. Each page is its own template set so every page
// can define its own "content" block. // can define its own "content" block.
func New(store Store, hub Hub, jobs Jobs, broker Broker, cfg Config, log *slog.Logger) (*Server, error) { func New(store Store, hub Hub, jobs Jobs, broker Broker, poller Poller, cfg Config, log *slog.Logger) (*Server, error) {
funcs := template.FuncMap{"shortRef": shortRef, "duration": jobDuration, "followsNoTag": followsNoTag, "middle": middle} funcs := template.FuncMap{"shortRef": shortRef, "duration": jobDuration, "followsNoTag": followsNoTag, "middle": middle}
pages := map[string]*template.Template{} pages := map[string]*template.Template{}
for _, page := range []string{"overview.html", "agents.html", "service.html", "jobs.html", "job.html"} { for _, page := range []string{"overview.html", "agents.html", "service.html", "jobs.html", "job.html"} {
@@ -123,7 +135,7 @@ func New(store Store, hub Hub, jobs Jobs, broker Broker, cfg Config, log *slog.L
} }
pages[page] = t pages[page] = t
} }
return &Server{store: store, hub: hub, jobs: jobs, broker: broker, cfg: cfg, log: log, now: time.Now, pages: pages}, nil return &Server{store: store, hub: hub, jobs: jobs, broker: broker, poller: poller, cfg: cfg, log: log, now: time.Now, pages: pages}, nil
} }
// Register adds the UI routes to mux. protect wraps every route except // Register adds the UI routes to mux. protect wraps every route except
@@ -137,6 +149,7 @@ func (s *Server) Register(mux *http.ServeMux, protect func(http.Handler) http.Ha
route("GET /fragments/services/{id}/jobs", s.serviceJobsFragment) route("GET /fragments/services/{id}/jobs", s.serviceJobsFragment)
route("POST /services/{id}/update", s.updateService) route("POST /services/{id}/update", s.updateService)
route("POST /services/{id}/mode", s.setMode) route("POST /services/{id}/mode", s.setMode)
route("POST /services/{id}/poll", s.pollService)
route("POST /services/{id}/restore", s.restoreDesired) route("POST /services/{id}/restore", s.restoreDesired)
route("POST /services/{id}/accept", s.acceptRunning) route("POST /services/{id}/accept", s.acceptRunning)
route("POST /services/{id}/merge", s.mergeService) route("POST /services/{id}/merge", s.mergeService)
@@ -201,7 +214,9 @@ type pageData struct {
ConfigFiles []string ConfigFiles []string
Reason string // why Service cannot be updated; "" when it can Reason string // why Service cannot be updated; "" when it can
AutoBlocked string // why Service cannot be set to auto; "" when it can AutoBlocked string // why Service cannot be set to auto; "" when it can
PollBlocked string // why Service cannot be checked against the registry; "" when it can
Versions []versionRow Versions []versionRow
NewerTag string // Service's newer tag while it still holds; "" otherwise
Polling string // what the service page says about registry polling Polling string // what the service page says about registry polling
PollFailed bool // Polling is the last registry error PollFailed bool // Polling is the last registry error
// RestoreBlocked is why a drifted Service cannot be restored; "" when it // RestoreBlocked is why a drifted Service cannot be restored; "" when it
@@ -376,6 +391,7 @@ func (s *Server) serviceDetail(r *http.Request, id int64) (pageData, error) {
return data, err return data, err
} }
data.AutoBlocked = s.autoBlocked(svc) data.AutoBlocked = s.autoBlocked(svc)
data.PollBlocked = pollBlocked(svc)
if err := s.jobs.Restorable(svc); errors.As(err, &nu) { if err := s.jobs.Restorable(svc); errors.As(err, &nu) {
data.RestoreBlocked = nu.Reason data.RestoreBlocked = nu.Reason
} else if err != nil { } else if err != nil {
@@ -388,6 +404,17 @@ func (s *Server) serviceDetail(r *http.Request, id int64) (pageData, error) {
if err != nil { if err != nil {
return data, err return data, err
} }
data.NewerTag = newerTag(svc)
if data.NewerTag != "" && !slices.ContainsFunc(versions, func(v model.ImageVersion) bool { return v.Tag == data.NewerTag && v.Digest == "" }) {
// The suggested tag can be older than the listed versions; it must
// still be pickable.
v, err := s.store.GetTagVersion(r.Context(), svc.ImageRepo, data.NewerTag)
if err == nil {
versions = append(versions, v)
} else if !errors.Is(err, model.ErrNotFound) {
return data, err
}
}
data.Versions = versionRows(versions, svc.ActualDigest) data.Versions = versionRows(versions, svc.ActualDigest)
data.Polling, data.PollFailed = pollingStatus(svc, s.cfg.PollInterval) data.Polling, data.PollFailed = pollingStatus(svc, s.cfg.PollInterval)
data.Jobs, err = s.store.ListJobs(r.Context(), model.JobFilter{ServiceID: id, Limit: serviceJobsLimit}) data.Jobs, err = s.store.ListJobs(r.Context(), model.JobFilter{ServiceID: id, Limit: serviceJobsLimit})
@@ -590,6 +617,34 @@ func (s *Server) renderServiceError(w http.ResponseWriter, r *http.Request, id i
s.page(w, r, status, "service.html", data) s.page(w, r, status, "service.html", data)
} }
// pollService checks the registry for one service now and waits for the
// answer. A registry error is not a failure of the request: it shows on the
// service page as the last check failing. The poller publishes the change
// itself when it writes.
func (s *Server) pollService(w http.ResponseWriter, r *http.Request) {
id, ok := pathID(w, r)
if !ok {
return
}
err := s.poller.PollService(r.Context(), id)
var np *poller.NotPolledError
switch {
case errors.As(err, &np):
s.renderServiceError(w, r, id, http.StatusConflict, "Cannot check: "+np.Reason+".")
return
case r.Context().Err() != nil:
// The client is gone or the master is shutting down. Redirect anyway:
// a browser still waiting gets the page, not an empty 200.
case err != nil:
s.failLookup(w, err)
return
default:
sess, _ := webauth.SessionFrom(r.Context())
s.log.Info("poll_requested", "service_id", id, "subject", sess.Subject)
}
http.Redirect(w, r, "/services/"+strconv.FormatInt(id, 10), http.StatusSeeOther)
}
// setMode switches a service between manual and auto. Manual is always // setMode switches a service between manual and auto. Manual is always
// allowed; auto only when jobs.AutoAllowed says so. // allowed; auto only when jobs.AutoAllowed says so.
func (s *Server) setMode(w http.ResponseWriter, r *http.Request) { func (s *Server) setMode(w http.ResponseWriter, r *http.Request) {
@@ -613,6 +668,10 @@ func (s *Server) setMode(w http.ResponseWriter, r *http.Request) {
} }
var nu *model.NotUpdatableError var nu *model.NotUpdatableError
if err := s.jobs.AutoAllowed(svc); errors.As(err, &nu) { if err := s.jobs.AutoAllowed(svc); errors.As(err, &nu) {
if isHTMX(r) {
// htmx does not swap a 409; the service page shows the reason.
w.Header().Set("HX-Redirect", "/services/"+strconv.FormatInt(id, 10))
}
s.renderServiceError(w, r, id, http.StatusConflict, "Cannot enable auto: "+nu.Reason+".") s.renderServiceError(w, r, id, http.StatusConflict, "Cannot enable auto: "+nu.Reason+".")
return return
} else if err != nil { } else if err != nil {
@@ -626,6 +685,11 @@ func (s *Server) setMode(w http.ResponseWriter, r *http.Request) {
} }
s.broker.Publish(events.Services) s.broker.Publish(events.Services)
if r.PostFormValue("return") == "overview" { if r.PostFormValue("return") == "overview" {
if isHTMX(r) {
// Swap the list in place instead of redirecting, so the page keeps its scroll position.
s.servicesFragment(w, r)
return
}
http.Redirect(w, r, "/", http.StatusSeeOther) http.Redirect(w, r, "/", http.StatusSeeOther)
return return
} }
@@ -866,6 +930,11 @@ func (s *Server) fail(w http.ResponseWriter, err error) {
http.Error(w, "internal error", http.StatusInternalServerError) http.Error(w, "internal error", http.StatusInternalServerError)
} }
// isHTMX reports whether r was sent by htmx rather than by a plain form submit.
func isHTMX(r *http.Request) bool {
return r.Header.Get("HX-Request") == "true"
}
// agentSnippet is the compose file shown once when an agent is added. // agentSnippet is the compose file shown once when an agent is added.
func agentSnippet(cfg Config, token string) string { func agentSnippet(cfg Config, token string) string {
return fmt.Sprintf(`services: return fmt.Sprintf(`services:
@@ -883,6 +952,7 @@ func agentSnippet(cfg Config, token string) string {
volumes: volumes:
- /var/run/docker.sock:/var/run/docker.sock - /var/run/docker.sock:/var/run/docker.sock
# - /path/to/stacks:/path/to/stacks # mount each stacks directory at the same path, read-write # - /path/to/stacks:/path/to/stacks # mount each stacks directory at the same path, read-write
# (also this agent's own stack directory, or it cannot update itself)
# - /root/.docker/config.json:/docker-config/config.json:ro # - /root/.docker/config.json:/docker-config/config.json:ro
`, cfg.AgentImage, cfg.AgentURL, cfg.MasterKey, token) `, cfg.AgentImage, cfg.AgentURL, cfg.MasterKey, token)
} }
+85 -4
View File
@@ -232,6 +232,12 @@ func newTestEnvVersion(t *testing.T, version string) (*httptest.Server, *fakeSto
// newTestEnvConfig is newTestEnv with edit applied to the UI's Config. // newTestEnvConfig is newTestEnv with edit applied to the UI's Config.
func newTestEnvConfig(t *testing.T, edit func(*Config)) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) { func newTestEnvConfig(t *testing.T, edit func(*Config)) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) {
t.Helper()
return newTestEnvFull(t, edit, &fakePoller{})
}
// newTestEnvFull is newTestEnvConfig with p as the poller.
func newTestEnvFull(t *testing.T, edit func(*Config), p Poller) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker, *fakeJobs) {
t.Helper() t.Helper()
agentID, offlineAgentID := int64(2), int64(3) agentID, offlineAgentID := int64(2), int64(3)
store := &fakeStore{ store := &fakeStore{
@@ -260,7 +266,7 @@ func newTestEnvConfig(t *testing.T, edit func(*Config)) (*httptest.Server, *fake
AgentURL: "wss://10.0.0.5:8090/agent", MasterKey: "sha256:abc", AgentImage: "gitea.shcizo.se/shcizo/package-updater:latest", AgentURL: "wss://10.0.0.5:8090/agent", MasterKey: "sha256:abc", AgentImage: "gitea.shcizo.se/shcizo/package-updater:latest",
} }
edit(&cfg) edit(&cfg)
s, err := New(store, hub, jobs, broker, cfg, slog.New(slog.NewTextHandler(io.Discard, nil))) s, err := New(store, hub, jobs, broker, p, cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
require.NoError(t, err) require.NoError(t, err)
mux := http.NewServeMux() mux := http.NewServeMux()
s.Register(mux, func(h http.Handler) http.Handler { s.Register(mux, func(h http.Handler) http.Handler {
@@ -303,7 +309,8 @@ func TestOverview_GroupsServicesPerMachineAndSwarm(t *testing.T) {
assert.Contains(t, body, `<span class="stk">slusk</span>`) assert.Contains(t, body, `<span class="stk">slusk</span>`)
assert.Contains(t, body, "jellyfin_jellyfin", "Swarm service names are shown as they are") assert.Contains(t, body, "jellyfin_jellyfin", "Swarm service names are shown as they are")
assert.Contains(t, body, "sha256:0f42497a69fa…", "digests are shortened") assert.Contains(t, body, "sha256:0f42497a69fa…", "digests are shortened")
assert.Contains(t, body, "Managed outside") assert.Contains(t, body, `<span class="tag f-out">This agent</span>`)
assert.NotContains(t, body, "Managed outside")
assert.Less(t, strings.Index(body, ">Swarm</h2>"), strings.Index(body, ">node-2</h2>"), "a cluster is followed by its managers' machines") assert.Less(t, strings.Index(body, ">Swarm</h2>"), strings.Index(body, ">node-2</h2>"), "a cluster is followed by its managers' machines")
assert.Contains(t, body, `class="sub" data-machine="agent-2"`, "the machine filter lists the manager under the cluster") assert.Contains(t, body, `class="sub" data-machine="agent-2"`, "the machine filter lists the manager under the cluster")
} }
@@ -513,6 +520,7 @@ func TestAgentSnippet(t *testing.T) {
assert.Contains(t, snippet, "UPDATE_TIMEOUT: 5m") assert.Contains(t, snippet, "UPDATE_TIMEOUT: 5m")
assert.Contains(t, snippet, " # DOCKER_CONFIG: /docker-config\n") assert.Contains(t, snippet, " # DOCKER_CONFIG: /docker-config\n")
assert.Contains(t, snippet, " # - /path/to/stacks:/path/to/stacks # mount each stacks directory at the same path, read-write\n") assert.Contains(t, snippet, " # - /path/to/stacks:/path/to/stacks # mount each stacks directory at the same path, read-write\n")
assert.Contains(t, snippet, "also this agent's own stack directory")
assert.Contains(t, snippet, "AGENT_TOKEN: tok") assert.Contains(t, snippet, "AGENT_TOKEN: tok")
} }
@@ -537,7 +545,7 @@ func TestServicePage_ShowsUpdateButton(t *testing.T) {
} }
func TestServicePage_DisabledReason(t *testing.T) { func TestServicePage_DisabledReason(t *testing.T) {
for _, reason := range []string{model.ReasonSelf, model.ReasonContainer, model.ReasonNoTag, model.ReasonRemoved, model.ReasonBuilt} { for _, reason := range []string{model.ReasonContainer, model.ReasonNoTag, model.ReasonRemoved, model.ReasonBuilt} {
t.Run(reason, func(t *testing.T) { t.Run(reason, func(t *testing.T) {
srv, _, _, _, jobs := newTestEnv(t) srv, _, _, _, jobs := newTestEnv(t)
jobs.updatable = &model.NotUpdatableError{Reason: reason} jobs.updatable = &model.NotUpdatableError{Reason: reason}
@@ -705,6 +713,47 @@ func TestSetMode_ReturnToOverview(t *testing.T) {
assert.Equal(t, "auto", store.modeOf(1)) assert.Equal(t, "auto", store.modeOf(1))
} }
func postModeHTMX(t *testing.T, srv *httptest.Server, id string, form url.Values) (*http.Response, string) {
t.Helper()
form.Set("csrf_token", "csrf")
req, err := http.NewRequest(http.MethodPost, srv.URL+"/services/"+id+"/mode", strings.NewReader(form.Encode()))
require.NoError(t, err)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("HX-Request", "true")
resp, err := noRedirectClient().Do(req)
require.NoError(t, err)
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
return resp, string(body)
}
func TestSetMode_HTMXFromOverviewReturnsServicesFragment(t *testing.T) {
srv, store, _, broker, _ := newTestEnv(t)
sub, cancel := broker.Subscribe()
defer cancel()
resp, body := postModeHTMX(t, srv, "1", url.Values{"mode": {"auto"}, "return": {"overview"}})
assert.Equal(t, http.StatusOK, resp.StatusCode, "no redirect, so the page keeps its scroll position")
assert.Empty(t, resp.Header.Get("Location"))
assert.NotContains(t, body, "<html")
assert.Contains(t, body, `id="machines" hx-swap-oob`)
assert.Contains(t, body, `action="/services/1/mode"`)
assert.Equal(t, "auto", store.modeOf(1))
assert.Equal(t, events.Services, <-sub)
}
func TestSetMode_HTMXRefusedRedirectsToService(t *testing.T) {
srv, store, _, _, jobs := newTestEnv(t)
jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonBuilt}
resp, _ := postModeHTMX(t, srv, "1", url.Values{"mode": {"auto"}, "return": {"overview"}})
assert.Equal(t, http.StatusConflict, resp.StatusCode)
assert.Equal(t, "/services/1", resp.Header.Get("HX-Redirect"), "htmx does not swap a 409, so send the user to the page that shows why")
assert.Equal(t, "manual", store.modeOf(1))
}
func TestSetMode_AutoRefusedShows409(t *testing.T) { func TestSetMode_AutoRefusedShows409(t *testing.T) {
srv, store, _, _, jobs := newTestEnv(t) srv, store, _, _, jobs := newTestEnv(t)
jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonBuilt} jobs.autoErr = &model.NotUpdatableError{Reason: model.ReasonBuilt}
@@ -762,7 +811,8 @@ func TestOverview_ShowsModeToggleAndAvailableBadge(t *testing.T) {
assert.Contains(t, body, `<input type="hidden" name="mode" value="auto">`, "manual services toggle to auto: "+path) assert.Contains(t, body, `<input type="hidden" name="mode" value="auto">`, "manual services toggle to auto: "+path)
assert.Contains(t, body, `aria-checked="false" disabled title="`+model.ReasonBuilt+`">`, "blocked manual services get a disabled switch: "+path) assert.Contains(t, body, `aria-checked="false" disabled title="`+model.ReasonBuilt+`">`, "blocked manual services get a disabled switch: "+path)
assert.Contains(t, body, `<span class="na">`+model.ReasonBuilt+`</span>`, "and say why: "+path) assert.Contains(t, body, `<span class="na">`+model.ReasonBuilt+`</span>`, "and say why: "+path)
assert.Contains(t, body, `<span class="na">n/a</span>`, "services managed outside have no switch: "+path) assert.Contains(t, body, `action="/services/3/mode"`, "the agent's own service has a switch like any other: "+path)
assert.NotContains(t, body, `<span class="na">n/a</span>`, path)
} }
} }
@@ -787,3 +837,34 @@ func TestServicePage_AutoBlockedDisablesToggle(t *testing.T) {
_, body := get(t, srv, "/services/1") _, body := get(t, srv, "/services/1")
assert.Contains(t, body, "Auto unavailable: "+model.ReasonContainer+".") assert.Contains(t, body, "Auto unavailable: "+model.ReasonContainer+".")
} }
// GetTagVersion returns the digest-less version of repo with this tag.
func (f *fakeStore) GetTagVersion(_ context.Context, repo, tag string) (model.ImageVersion, error) {
for _, v := range f.versions {
if v.Repo == repo && v.Tag == tag && v.Digest == "" {
return v, nil
}
}
return model.ImageVersion{}, model.ErrNotFound
}
func TestOverview_ShowsNewerTagChipBesideUpdateChip(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].AvailableRef = "gitea.shcizo.se/shcizo/slusk:1.4.2"
store.services[0].NewerRef = "gitea.shcizo.se/shcizo/slusk:1.5.0"
for _, path := range []string{"/", "/fragments/services"} {
_, body := get(t, srv, path)
assert.Contains(t, body, `class="tag f-newer" title="gitea.shcizo.se/shcizo/slusk:1.5.0">Newer tag<em>1.5.0</em>`, path)
assert.Contains(t, body, `class="tag f-up"`, path)
}
}
func TestOverview_HidesStaleNewerTag(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].NewerRef = "gitea.shcizo.se/shcizo/slusk:1.4.1" // not newer than 1.4.2
_, body := get(t, srv, "/")
assert.NotContains(t, body, "f-newer")
}
+5 -1
View File
@@ -66,6 +66,9 @@ summary.btn::-webkit-details-marker{display:none}
.seg button{display:inline-flex;align-items:center;gap:6px;height:28px;padding:0 10px;border:0;border-radius:5px;background:transparent;color:var(--mut);font:500 12.5px var(--sans);cursor:pointer} .seg button{display:inline-flex;align-items:center;gap:6px;height:28px;padding:0 10px;border:0;border-radius:5px;background:transparent;color:var(--mut);font:500 12.5px var(--sans);cursor:pointer}
.seg button:hover{color:var(--fg)} .seg button:hover{color:var(--fg)}
.seg button[aria-pressed=true]{background:var(--hover);color:var(--fg);box-shadow:inset 0 0 0 1px var(--line2)} .seg button[aria-pressed=true]{background:var(--hover);color:var(--fg);box-shadow:inset 0 0 0 1px var(--line2)}
/* htmx inserts the new list before removing the old one; scroll anchoring on the
focused switch would scroll by the list height and then land at the bottom. */
#services{overflow-anchor:none}
.grp{margin-top:28px} .grp{margin-top:28px}
.gh{display:flex;flex-wrap:wrap;align-items:baseline;gap:4px 12px;padding-bottom:8px;border-bottom:2px solid var(--fg)} .gh{display:flex;flex-wrap:wrap;align-items:baseline;gap:4px 12px;padding-bottom:8px;border-bottom:2px solid var(--fg)}
.gh h2{margin:0;font:700 20px var(--sans);letter-spacing:-.015em} .gh h2{margin:0;font:700 20px var(--sans);letter-spacing:-.015em}
@@ -106,8 +109,9 @@ summary.btn::-webkit-details-marker{display:none}
.tag em{font:500 11.5px var(--mono);font-style:normal;margin-left:6px} .tag em{font:500 11.5px var(--mono);font-style:normal;margin-left:6px}
.tag em:empty{display:none} .tag em:empty{display:none}
.f-up{color:var(--orange);background:var(--orangebg)} .f-up{color:var(--orange);background:var(--orangebg)}
.f-newer em{color:var(--orange)}
.f-err{color:var(--red);background:var(--redbg)} .f-err{color:var(--red);background:var(--redbg)}
.f-notag{color:var(--mut);box-shadow:inset 0 0 0 1px var(--line2)} .f-newer,.f-notag{color:var(--mut);box-shadow:inset 0 0 0 1px var(--line2)}
.f-out{color:var(--dim);box-shadow:inset 0 0 0 1px var(--line)} .f-out{color:var(--dim);box-shadow:inset 0 0 0 1px var(--line)}
.mode{display:flex;flex-direction:column;align-items:flex-start;gap:2px} .mode{display:flex;flex-direction:column;align-items:flex-start;gap:2px}
.mode form{margin:0;display:flex;flex-direction:column;align-items:flex-start;gap:2px} .mode form{margin:0;display:flex;flex-direction:column;align-items:flex-start;gap:2px}
+4 -4
View File
@@ -77,13 +77,13 @@
</div> </div>
<div class="flags"> <div class="flags">
{{if .AvailableRef}}<span class="tag f-up" title="{{.AvailableRef}}">Update available<em>{{.AvailableTag}}</em></span>{{end}} {{if .AvailableRef}}<span class="tag f-up" title="{{.AvailableRef}}">Update available<em>{{.AvailableTag}}</em></span>{{end}}
{{if .NewerTag}}<span class="tag f-newer" title="{{.NewerRef}}">Newer tag<em>{{.NewerTag}}</em></span>{{end}}
{{if followsNoTag .Service}}<span class="tag f-notag">Follows no tag</span>{{end}} {{if followsNoTag .Service}}<span class="tag f-notag">Follows no tag</span>{{end}}
{{if .LastPollError}}<span class="tag f-err" title="{{.LastPollError}}">Poll failed</span>{{end}} {{if .LastPollError}}<span class="tag f-err" title="{{.LastPollError}}">Poll failed</span>{{end}}
{{if .Self}}<span class="tag f-out">Managed outside</span>{{end}} {{if .Self}}<span class="tag f-out">This agent</span>{{end}}
</div> </div>
<div class="mode"> <div class="mode">
{{if .Self}}<span class="na">n/a</span> <form method="post" action="/services/{{.ID}}/mode" hx-post="/services/{{.ID}}/mode" hx-target="#services" hx-swap="innerHTML">
{{else}}<form method="post" action="/services/{{.ID}}/mode">
<input type="hidden" name="csrf_token" value="{{$.CSRF}}"> <input type="hidden" name="csrf_token" value="{{$.CSRF}}">
<input type="hidden" name="return" value="overview"> <input type="hidden" name="return" value="overview">
{{if eq .UpdateMode "auto"}}<input type="hidden" name="mode" value="manual"> {{if eq .UpdateMode "auto"}}<input type="hidden" name="mode" value="manual">
@@ -91,7 +91,7 @@
{{else}}<input type="hidden" name="mode" value="auto"> {{else}}<input type="hidden" name="mode" value="auto">
<button type="submit" class="sw" role="switch" aria-checked="false"{{if .AutoBlocked}} disabled title="{{.AutoBlocked}}"{{end}}><i></i>Manual</button> <button type="submit" class="sw" role="switch" aria-checked="false"{{if .AutoBlocked}} disabled title="{{.AutoBlocked}}"{{end}}><i></i>Manual</button>
{{if .AutoBlocked}}<span class="na">{{.AutoBlocked}}</span>{{end}}{{end}} {{if .AutoBlocked}}<span class="na">{{.AutoBlocked}}</span>{{end}}{{end}}
</form>{{end}} </form>
</div> </div>
</div> </div>
{{end}} {{end}}
@@ -9,6 +9,7 @@
<tr><th>Status</th><td class="status status-{{.Service.Status}}">{{.Service.Status}}</td></tr> <tr><th>Status</th><td class="status status-{{.Service.Status}}">{{.Service.Status}}</td></tr>
{{if .Service.DesiredRef}}<tr><th>Desired image</th><td><code>{{shortRef .Service.DesiredRef}}</code></td></tr>{{end}} {{if .Service.DesiredRef}}<tr><th>Desired image</th><td><code>{{shortRef .Service.DesiredRef}}</code></td></tr>{{end}}
{{if .Service.AvailableRef}}<tr><th>Available</th><td><code>{{.Service.AvailableRef}}</code> <span class="muted">via {{.Service.AvailableSource}}{{with .Service.AvailableAt}}, {{.Format "2006-01-02 15:04"}}{{end}}</span></td></tr>{{end}} {{if .Service.AvailableRef}}<tr><th>Available</th><td><code>{{.Service.AvailableRef}}</code> <span class="muted">via {{.Service.AvailableSource}}{{with .Service.AvailableAt}}, {{.Format "2006-01-02 15:04"}}{{end}}</span></td></tr>{{end}}
{{if .NewerTag}}<tr><th>Newer tag</th><td><code>{{.Service.NewerRef}}</code> <span class="muted">from the registry's tag list{{with .Service.NewerAt}}, {{.Format "2006-01-02 15:04"}}{{end}}; deploy it from Versions</span></td></tr>{{end}}
<tr><th>Mode</th><td>{{.Service.UpdateMode}}</td></tr> <tr><th>Mode</th><td>{{.Service.UpdateMode}}</td></tr>
<tr><th>Polling</th><td{{if .PollFailed}} class="error"{{end}}>{{.Polling}}</td></tr> <tr><th>Polling</th><td{{if .PollFailed}} class="error"{{end}}>{{.Polling}}</td></tr>
{{if .ConfigFiles}}<tr><th>Config files</th><td>{{range .ConfigFiles}}<code>{{.}}</code><br>{{end}}</td></tr>{{end}} {{if .ConfigFiles}}<tr><th>Config files</th><td>{{range .ConfigFiles}}<code>{{.}}</code><br>{{end}}</td></tr>{{end}}
@@ -22,6 +23,11 @@
</form> </form>
<p class="muted">Pulls the current tag again and redeploys the service.</p> <p class="muted">Pulls the current tag again and redeploys the service.</p>
{{end}} {{end}}
<form method="post" action="/services/{{.Service.ID}}/poll">
<input type="hidden" name="csrf_token" value="{{.CSRF}}">
<button type="submit"{{if .PollBlocked}} disabled{{end}}>Check now</button>
<p class="muted">{{if .PollBlocked}}Check unavailable: {{.PollBlocked}}.{{else}}Asks the registry now instead of waiting for the next poll.{{end}}</p>
</form>
<form method="post" action="/services/{{.Service.ID}}/mode"> <form method="post" action="/services/{{.Service.ID}}/mode">
<input type="hidden" name="csrf_token" value="{{.CSRF}}"> <input type="hidden" name="csrf_token" value="{{.CSRF}}">
{{if eq .Service.UpdateMode "auto"}}<input type="hidden" name="mode" value="manual"> {{if eq .Service.UpdateMode "auto"}}<input type="hidden" name="mode" value="manual">
+51 -1
View File
@@ -68,7 +68,7 @@ func TestServicePage_VersionPickerWarnsForAutoServices(t *testing.T) {
} }
func TestServicePage_VersionPickerHiddenWhenServiceCannotBeUpdated(t *testing.T) { func TestServicePage_VersionPickerHiddenWhenServiceCannotBeUpdated(t *testing.T) {
for _, reason := range []string{model.ReasonSelf, model.ReasonContainer, model.ReasonRemoved, model.ReasonBuilt} { for _, reason := range []string{model.ReasonContainer, model.ReasonRemoved, model.ReasonBuilt} {
t.Run(reason, func(t *testing.T) { t.Run(reason, func(t *testing.T) {
srv, store, _, _, jobs := newTestEnv(t) srv, store, _, _, jobs := newTestEnv(t)
store.services[0].ImageRepo = sluskRepo store.services[0].ImageRepo = sluskRepo
@@ -219,3 +219,53 @@ func TestOverview_ShowsFollowsNoTagAndPollFailedBadges(t *testing.T) {
assert.Equal(t, 1, strings.Count(body, "Poll failed"), path) assert.Equal(t, 1, strings.Count(body, "Poll failed"), path)
} }
} }
func TestServicePage_VersionPickerAppendsNewerTagBeyondLimit(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].ImageRepo = sluskRepo
store.services[0].NewerRef = sluskRepo + ":1.5.0"
for i := int64(1); i <= 12; i++ {
store.versions = append(store.versions, model.ImageVersion{ID: i, Repo: sluskRepo, Tag: "t", Digest: "sha256:d", Source: "poll"})
}
store.versions = append(store.versions, model.ImageVersion{ID: 99, Repo: sluskRepo, Tag: "1.5.0", Source: "tags"})
_, body := get(t, srv, "/services/1")
assert.Equal(t, 11, strings.Count(body, `name="version"`), "the ten recent versions and the newer tag")
assert.Contains(t, body, `<input type="hidden" name="version" value="99">`)
assert.Contains(t, body, "<th>Newer tag</th>")
}
func TestServicePage_NewerTagAlreadyListedIsNotAppendedTwice(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].ImageRepo = sluskRepo
store.services[0].NewerRef = sluskRepo + ":1.5.0"
store.versions = []model.ImageVersion{{ID: 31, Repo: sluskRepo, Tag: "1.5.0", Source: "tags"}}
_, body := get(t, srv, "/services/1")
assert.Equal(t, 1, strings.Count(body, `name="version"`))
}
func TestServicePage_PinnedRowOfTheNewerTagDoesNotHideThePlainRow(t *testing.T) {
srv, store, _, _, _ := newTestEnv(t)
store.services[0].ImageRepo = sluskRepo
store.services[0].NewerRef = sluskRepo + ":1.5.0"
store.versions = []model.ImageVersion{
{ID: 31, Repo: sluskRepo, Tag: "1.5.0", Digest: "sha256:d", Source: "poll"},
{ID: 32, Repo: sluskRepo, Tag: "1.5.0", Source: "tags"},
}
_, body := get(t, srv, "/services/1")
assert.Equal(t, 2, strings.Count(body, `name="version"`))
assert.Contains(t, body, `<input type="hidden" name="version" value="32">`)
}
func TestServicePage_NoNewerTagRowWithoutSuggestion(t *testing.T) {
srv, _, _, _, _ := newTestEnv(t)
_, body := get(t, srv, "/services/1")
assert.NotContains(t, body, "Newer tag")
}
+32
View File
@@ -12,6 +12,7 @@ import (
"github.com/shcizo/package-updater/internal/master/jobs" "github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model" "github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/master/poller" "github.com/shcizo/package-updater/internal/master/poller"
"github.com/shcizo/package-updater/internal/master/tagver"
"github.com/shcizo/package-updater/internal/master/web/icons" "github.com/shcizo/package-updater/internal/master/web/icons"
"github.com/shcizo/package-updater/internal/protocol" "github.com/shcizo/package-updater/internal/protocol"
) )
@@ -45,6 +46,7 @@ type serviceRow struct {
Tag string // ":tag", or "" Tag string // ":tag", or ""
Digest string // "@sha256:<12 hex>…" when the image has no tag, else "" Digest string // "@sha256:<12 hex>…" when the image has no tag, else ""
AvailableTag string // the tag of AvailableRef, or "" AvailableTag string // the tag of AvailableRef, or ""
NewerTag string // the tag of NewerRef while it is still newer than the followed tag, or ""
Icon string // Simple Icons slug; "" when there is no icon Icon string // Simple Icons slug; "" when there is no icon
Letter string // shown instead of an icon Letter string // shown instead of an icon
LetterClass string // lc-0 .. lc-3 LetterClass string // lc-0 .. lc-3
@@ -66,6 +68,7 @@ func newServiceRow(s model.Service, autoBlocked string) serviceRow {
} }
r.Repo, r.Tag, r.Digest = imageParts(s.ActualRef) r.Repo, r.Tag, r.Digest = imageParts(s.ActualRef)
r.AvailableTag = imageref.Parse(s.AvailableRef).Tag r.AvailableTag = imageref.Parse(s.AvailableRef).Tag
r.NewerTag = newerTag(s)
r.Icon = icons.ForImage(s.ActualRef) r.Icon = icons.ForImage(s.ActualRef)
r.Letter = "?" r.Letter = "?"
if first, _ := utf8.DecodeRuneInString(r.Name); first != utf8.RuneError { if first, _ := utf8.DecodeRuneInString(r.Name); first != utf8.RuneError {
@@ -243,6 +246,26 @@ func followsNoTag(svc model.Service) bool {
return jobs.FollowedRef(svc.ActualRef) == "" return jobs.FollowedRef(svc.ActualRef) == ""
} }
// newerTag returns the tag of the service's newer-tag suggestion, or "" when
// there is none or it no longer holds. The poller writes the suggestion once
// per cycle, so after the service moved to another tag the stored value can
// be stale until the next cycle; checking again here hides it meanwhile. A
// suggestion from another repo than the service follows now is hidden too.
func newerTag(s model.Service) string {
if s.RemovedAt != nil || s.NewerRef == "" {
return ""
}
newer := imageref.Parse(s.NewerRef)
repo, followed := poller.Followed(s)
if newer.Repo != repo {
return ""
}
if _, ok := tagver.Newer(followed, []string{newer.Tag}); !ok {
return ""
}
return newer.Tag
}
// versionRow is an image version as shown on the service page. // versionRow is an image version as shown on the service page.
type versionRow struct { type versionRow struct {
model.ImageVersion model.ImageVersion
@@ -258,6 +281,15 @@ func versionRows(versions []model.ImageVersion, actualDigest string) []versionRo
return rows return rows
} }
// pollBlocked returns why the registry cannot be checked for svc, or "" when
// it can.
func pollBlocked(svc model.Service) string {
if svc.RemovedAt != nil {
return model.ReasonRemoved
}
return poller.SkipReason(svc)
}
// pollingStatus is the text of the "Polling" row on the service page and // pollingStatus is the text of the "Polling" row on the service page and
// whether it is a registry error. // whether it is a registry error.
func pollingStatus(svc model.Service, interval time.Duration) (text string, failed bool) { func pollingStatus(svc model.Service, interval time.Duration) (text string, failed bool) {
+22
View File
@@ -211,3 +211,25 @@ func TestGroupServices(t *testing.T) {
assert.Equal(t, "because", groups[0].Services[0].AutoBlocked) assert.Equal(t, "because", groups[0].Services[0].AutoBlocked)
}) })
} }
func TestNewerTag(t *testing.T) {
removed := time.Now()
tests := []struct {
name string
svc model.Service
want string
}{
{"newer", model.Service{ActualRef: "gitea/gitea:1.27.3", NewerRef: "gitea/gitea:1.28.0"}, "1.28.0"},
{"bare image follows latest", model.Service{ActualRef: "gitea/gitea", NewerRef: "gitea/gitea:1.28.0"}, ""},
{"stale after the tag changed", model.Service{ActualRef: "gitea/gitea:1.28.0", NewerRef: "gitea/gitea:1.28.0"}, ""},
{"stale after the tag moved past it", model.Service{ActualRef: "gitea/gitea:1.29.0", NewerRef: "gitea/gitea:1.28.0"}, ""},
{"other repo", model.Service{ActualRef: "redis:7.2.4", NewerRef: "bitnami/redis:7.2.5"}, ""},
{"removed", model.Service{ActualRef: "gitea/gitea:1.27.3", NewerRef: "gitea/gitea:1.28.0", RemovedAt: &removed}, ""},
{"empty", model.Service{ActualRef: "gitea/gitea:1.27.3"}, ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, newerTag(tt.svc))
})
}
}
+12
View File
@@ -15,6 +15,7 @@ import (
"log/slog" "log/slog"
"net/http" "net/http"
"regexp" "regexp"
"slices"
"strconv" "strconv"
"strings" "strings"
"time" "time"
@@ -181,6 +182,9 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
resp := response{RequestID: requestID, Image: req.Image, Tag: req.Tag, Matched: len(services)} resp := response{RequestID: requestID, Image: req.Image, Tag: req.Tag, Matched: len(services)}
pending := map[int]int64{} // index into resp.Results -> job ID pending := map[int]int64{} // index into resp.Results -> job ID
// An agent's own service goes last: jobs that reach an agent while it
// replaces itself are lost, and messages to one agent arrive in order.
slices.SortStableFunc(services, func(a, b model.Service) int { return selfRank(a) - selfRank(b) })
for _, svc := range services { for _, svc := range services {
res, job := h.handle(work, svc, req.Tag, started, log) res, job := h.handle(work, svc, req.Tag, started, log)
resp.Results = append(resp.Results, res) resp.Results = append(resp.Results, res)
@@ -252,6 +256,14 @@ func (h *Handler) handle(ctx context.Context, svc model.Service, tag string, sta
return res, nil return res, nil
} }
// selfRank sorts an agent's own service after all others.
func selfRank(svc model.Service) int {
if svc.Self {
return 1
}
return 0
}
func (h *Handler) setJob(res *result, job model.Job) { func (h *Handler) setJob(res *result, job model.Job) {
res.JobID = job.ID res.JobID = job.ID
res.JobURL = h.cfg.PublicURL + "/jobs/" + strconv.FormatInt(job.ID, 10) res.JobURL = h.cfg.PublicURL + "/jobs/" + strconv.FormatInt(job.ID, 10)
+17 -2
View File
@@ -470,14 +470,14 @@ func TestBusyIsNotAFailure(t *testing.T) {
func TestRefusedIsNotAFailure(t *testing.T) { func TestRefusedIsNotAFailure(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2")) e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.jobs.errFor[1] = &model.NotUpdatableError{Reason: model.ReasonSelf} e.jobs.errFor[1] = &model.NotUpdatableError{Reason: model.ReasonContainer}
rec := e.post(`{"image":"` + slusk + `","tag":"2"}`) rec := e.post(`{"image":"` + slusk + `","tag":"2"}`)
assert.Equal(t, 200, rec.Code) assert.Equal(t, 200, rec.Code)
r := decode(t, rec).Results[0] r := decode(t, rec).Results[0]
assert.Equal(t, statusRefused, r.Status) assert.Equal(t, statusRefused, r.Status)
assert.Equal(t, model.ReasonSelf, r.Error) assert.Equal(t, model.ReasonContainer, r.Error)
assert.Zero(t, r.JobID) assert.Zero(t, r.JobID)
assert.NotEmpty(t, e.store.avail) assert.NotEmpty(t, e.store.avail)
} }
@@ -500,6 +500,21 @@ func TestMixedAutoAndManual(t *testing.T) {
assert.Len(t, e.store.avail, 3) assert.Len(t, e.store.avail, 3)
} }
func TestAgentsOwnServiceIsStartedLast(t *testing.T) {
own := compose(1, model.ModeAuto, slusk+":1")
own.Self = true
e := newEnv(t, own, compose(2, model.ModeAuto, slusk+":1"))
e.jobs.jobFor[1] = model.Job{Status: model.JobSucceeded}
e.jobs.jobFor[2] = model.Job{Status: model.JobSucceeded}
e.post(`{"image":"` + slusk + `","tag":"2"}`)
calls := e.jobs.startCalls()
require.Len(t, calls, 2)
assert.Equal(t, int64(2), calls[0].svc.ID)
assert.Equal(t, int64(1), calls[1].svc.ID)
}
func TestClientCancelStopsWaitingButNotStarting(t *testing.T) { func TestClientCancelStopsWaitingButNotStarting(t *testing.T) {
e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2")) e := newEnv(t, compose(1, model.ModeAuto, slusk+":1.4.2"))
e.h.cfg.Timeout = time.Minute e.h.cfg.Timeout = time.Minute
+10 -1
View File
@@ -16,7 +16,7 @@ import (
// Version is the protocol version this build speaks. Agents send it in // Version is the protocol version this build speaks. Agents send it in
// Hello. Phase-1 agents do not send the field, which reads as 0. // Hello. Phase-1 agents do not send the field, which reads as 0.
const Version = 3 const Version = 4
// MinJobVersion is the lowest agent protocol version the master sends jobs to. // MinJobVersion is the lowest agent protocol version the master sends jobs to.
const MinJobVersion = 2 const MinJobVersion = 2
@@ -26,6 +26,11 @@ const MinJobVersion = 2
// with a digest. // with a digest.
const MinPinnedRefVersion = 3 const MinPinnedRefVersion = 3
// MinSelfUpdateVersion is the lowest agent protocol version the master sends a
// job for the agent's own Compose service to. Older agents neither run the
// update in a helper container nor resume it after restarting.
const MinSelfUpdateVersion = 4
// MaxJobOutput caps the job output an agent keeps and the master stores, in // MaxJobOutput caps the job output an agent keeps and the master stores, in
// bytes. Only the tail is kept. With JSON escaping (worst case 6 bytes per // bytes. Only the tail is kept. With JSON escaping (worst case 6 bytes per
// byte) a JobResult carrying this much output still fits the 1 MiB read limit. // byte) a JobResult carrying this much output still fits the 1 MiB read limit.
@@ -141,6 +146,10 @@ type JobStart struct {
SwarmClusterID string `json:"swarm_cluster_id,omitempty"` SwarmClusterID string `json:"swarm_cluster_id,omitempty"`
SwarmService string `json:"swarm_service,omitempty"` SwarmService string `json:"swarm_service,omitempty"`
Ref string `json:"ref"` Ref string `json:"ref"`
// Self says the service is the agent's own. The agent runs such a job
// after every other queued job; whether it needs a helper container is
// decided by the agent itself.
Self bool `json:"self,omitempty"`
} }
// JobProgress carries output produced since the previous JobProgress for the // JobProgress carries output produced since the previous JobProgress for the