docs: auto rollbacks, bare image as latest and registry request scope (#9)
This commit is contained in:
@@ -74,12 +74,21 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
|
||||
GET as a pull; the client never falls back to GET, and a test asserts this.
|
||||
- **`image_versions` uses `''` instead of NULL** for a missing tag or digest, so
|
||||
`UNIQUE (repo, tag, digest)` deduplicates. The webhook records `(repo, tag, '')`; that
|
||||
row is hidden from `RecentVersions` once a row with the same tag and a digest exists.
|
||||
row is hidden from `RecentVersions` when a row with the same tag and a digest was first
|
||||
seen at or after it.
|
||||
- **The UI picks a version by `image_versions.id`, never by a ref from the form**, and the row
|
||||
must belong to the service's `image_repo`. A digest-pinned or tagless pick needs agent
|
||||
protocol 3 (`protocol.MinPinnedRefVersion`); `Hub.Route` takes the minimum version.
|
||||
- **Rolling back a service that follows no tag afterwards:** a pick of `repo@digest` makes the
|
||||
service follow no tag, so it is no longer polled.
|
||||
- **Picking a tagless version (`repo@digest`) leaves the service following no tag:** it is no
|
||||
longer polled and the plain Update button is refused.
|
||||
- **The poller starts one auto job per digest** (`services.last_auto_digest`, set only when a
|
||||
job row was created; `ErrJobActive` and `NotUpdatableError` leave it so the next cycle
|
||||
retries). It also skips a digest that `desired_ref` (`ref@digest`) already names, and
|
||||
re-reads the service before `Start`. That is why a rollback of an auto service holds until
|
||||
a new digest appears. The webhook does not set it, so a webhook-started update can be
|
||||
followed by a poll job after a rollback.
|
||||
- **A bare image's running version is recorded with tag `latest`** (like `jobs.FollowedRef`);
|
||||
`repo@digest` is recorded with tag `''`. Only `sha256:<64 hex>` digests are recorded.
|
||||
- **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where
|
||||
the first colon is a port, not a tag. See `internal/imageref`.
|
||||
- **Path safety uses `filepath.Rel` + ".." prefix check**, NOT `strings.HasPrefix`.
|
||||
|
||||
@@ -129,9 +129,12 @@ multi-arch image the digest is the index digest, the same one `docker inspect` s
|
||||
|
||||
- **What counts as new.** A digest different from the one the service runs becomes the
|
||||
service's *available* version (the overview badge, source `poll`). A digest the master
|
||||
already suggested is not suggested again. A service in **auto** mode also gets an update
|
||||
job to `repo:tag` (the agent resolves the digest itself, so agents on protocol version 2 keep
|
||||
working). A failed auto job is not retried every interval.
|
||||
already suggested is not suggested again. A service in **auto** mode also gets one update
|
||||
job to `repo:tag` per digest (the agent resolves the digest itself, so agents on protocol
|
||||
version 2 keep working); the master remembers that digest in `services.last_auto_digest`
|
||||
and does not start another job for it, whether the job failed or the user rolled back
|
||||
afterwards. If the service already had a job when the digest was found, the next cycle
|
||||
tries again.
|
||||
- **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and
|
||||
services without a registry digest (locally built images). The service page says which.
|
||||
The updater's own services and plain containers are polled, so their badge works, but
|
||||
@@ -146,25 +149,35 @@ multi-arch image the digest is the index digest, the same one `docker inspect` s
|
||||
in $PATH`, because the container has no credential helper binaries. The master logs
|
||||
`registry_credentials` at startup with the directory and whether `config.json` exists,
|
||||
never its contents.
|
||||
- **Requests go wherever the image name points.** The master sends the lookup to the
|
||||
registry host in the image name, including a `GET /v2/` ping that follows redirects.
|
||||
Anyone who can start containers on a managed host can therefore make the master send
|
||||
blind requests to internal hosts. The registry's answer is not shown: an error keeps only
|
||||
the HTTP status and the registry's error codes.
|
||||
|
||||
### Version history and rollback
|
||||
|
||||
The master records every version it sees (webhook, polling, and what agents report as
|
||||
running) in `image_versions`. The service page lists the 10 most recently first seen
|
||||
versions of the service's image repo, marks the one that is running, and has a *Deploy*
|
||||
button on each row. A webhook announces a tag but not its digest, so such a row has no digest until polling or an
|
||||
agent reports one; it is hidden once a row with the same tag and a digest exists.
|
||||
button on each row. A webhook announces a tag but not its digest, so such a row has no
|
||||
digest until polling or an agent reports one; it is hidden when a row with the same tag
|
||||
and a digest was first seen at or after it. A bare image (`postgres`) is recorded as tag
|
||||
`latest`.
|
||||
|
||||
Picking a version starts a manual job to exactly that reference. Picking an older one is a
|
||||
rollback, including to a digest-pinned reference. A version without a tag deploys `repo@digest` and the
|
||||
service then follows no tag, so it is no longer polled or updatable with the plain *Update* button until
|
||||
it is moved to a tagged version again. Picks of digest-pinned or tagless versions need an agent speaking protocol
|
||||
version 3 (this release); older agents get a failed job "agent too old for updates".
|
||||
rollback, including to a digest-pinned reference. A version without a tag deploys
|
||||
`repo@digest` and the service then follows no tag, so it is no longer polled or updatable
|
||||
with the plain *Update* button until it is moved to a tagged version again. Picks of
|
||||
digest-pinned or tagless versions need an agent speaking protocol version 3 (this
|
||||
release); on an older agent the job fails with "agent too old to deploy a pinned version
|
||||
(needs protocol 3)".
|
||||
|
||||
On a service in **auto** mode a rollback is only a warning in the UI: the service stays auto and
|
||||
is moved forward again at the next new version, announced by the webhook or found by polling.
|
||||
Rolling back to an older digest of the tag the service follows keeps the newer version marked as
|
||||
available.
|
||||
A service in **auto** mode stays auto after a rollback. Polling does not undo the rollback:
|
||||
it started at most one job per digest, so the service stays on the older version until a
|
||||
new digest of the followed tag appears. The newer version it was rolled back from stays
|
||||
marked as available. A webhook that names a tag still starts a job for an auto service
|
||||
every time, so a CI push moves it forward again.
|
||||
|
||||
## Adding agents
|
||||
|
||||
|
||||
@@ -28,11 +28,11 @@
|
||||
|
||||
## Deliberate deviations from the spec in this phase
|
||||
|
||||
1. **Webhook records `(repo, tag, '')`, no digest.** The master does not resolve the digest at webhook time (no registry round-trip/credentials in the CI path). The digest arrives later from polling or running inventory. The picker hides a digest-less row once a row with the same `(repo, tag)` and a digest exists.
|
||||
1. **Webhook records `(repo, tag, '')`, no digest.** The master does not resolve the digest at webhook time (no registry round-trip/credentials in the CI path). The digest arrives later from polling or running inventory. The picker hides a digest-less row when a row with the same `(repo, tag)` and a digest was first seen at or after it.
|
||||
2. **Empty string instead of NULL** for missing tag or digest in `image_versions`, so `UNIQUE (repo, tag, digest)` works. A CHECK forbids both being empty.
|
||||
3. **`first_seen_at` is never bumped.** Re-observation is `ON CONFLICT DO NOTHING`; `source` is the first source that saw it. "10 most recent" = "10 most recently first seen". A new digest of a tag is a new row at the top; old digest stays below (enables rollback of moving tags).
|
||||
4. **Not polled:** services that follow no tag, `build:` services, services with empty `actual_digest`. UI shows the reason. Self and container services ARE polled (badge informative; jobs still refused by `jobs`).
|
||||
5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, nothing happens for that service (no `available_at` bump, no new auto job). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears.
|
||||
5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, the badge is not written again (no `available_at` bump). The auto job is decided separately: it starts only if the service is auto, `actual_digest != D`, `D != last_auto_digest` and `desired_ref != ref@D`, on a fresh read of the row. `last_auto_digest` is set when a job row was created (not on `ErrJobActive`/`NotUpdatableError`, which retry next cycle). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears. The webhook does not set `last_auto_digest`.
|
||||
6. **FinishJob clearing rule gets a digest condition** (C3). Rollback to an older digest of the same tag no longer clears the newer-version badge.
|
||||
7. **Poll auto jobs use `repo:tag`, not `repo:tag@D`.** v2 agents keep working for poll and webhook jobs. Only UI picks of digest-pinned or tagless versions need a v3 agent.
|
||||
8. **First cycle runs 1 minute after start**, then every `POLL_INTERVAL` measured from the end of the previous cycle.
|
||||
@@ -68,7 +68,7 @@
|
||||
| `ErrJobActive` / `NotUpdatableError` from `Start` do not abort the cycle. | poller `TestStartErrorsDoNotAbortCycle` |
|
||||
| Concurrency within limit; cancellation stops the cycle; marking/starting on `WithoutCancel`. | poller `TestConcurrencyLimit`, `TestRunStopsOnCancel` |
|
||||
| Form cannot deploy an arbitrary ref: unknown version ID → 404; version of another repo → 404, no job. | jobs `TestTriggerVersion_*`, web `TestUpdateService_Version*` |
|
||||
| Pinned/tagless refs routed only to protocol ≥ 3 agents; old agents → failed "agent too old" job. | jobs `TestTriggerVersion_RequiresPinnedProtocol`, hub `TestRoute` |
|
||||
| Pinned/tagless refs routed only to protocol ≥ 3 agents; old agents → failed "agent too old to deploy a pinned version (needs protocol 3)" job (`ErrAgentTooOldForPinned`, which also satisfies `errors.Is(err, ErrAgentTooOld)`). | jobs `TestTriggerVersion_RequiresPinnedProtocol`, hub `TestRoute` |
|
||||
| Agent deploys a pinned ref verbatim, no registry resolve. | updater `TestResolve`, `TestComposeWritesPinnedRef`, `TestSwarmUsesPinnedRef` |
|
||||
| Rollback to older digest of same tag keeps newer suggestion; pick of suggested digest clears it. | store `TestFinishJob_ClearsAvailableOnlyWhenThatVersionWasDeployed` (extended) |
|
||||
| Running versions recorded only with a digest; re-snapshot does not duplicate or reorder. | store `TestSnapshotRecordsRunningVersions` |
|
||||
@@ -86,6 +86,7 @@ ALTER TABLE services ADD CONSTRAINT services_available_source_check
|
||||
CHECK (available_source IN ('', 'webhook', 'poll'));
|
||||
ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT '';
|
||||
ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT '';
|
||||
ALTER TABLE services ADD COLUMN last_auto_digest text NOT NULL DEFAULT '';
|
||||
|
||||
-- Every version the master has seen. '' means "no tag" / "digest unknown";
|
||||
-- never NULL, so the unique constraint deduplicates.
|
||||
@@ -136,7 +137,7 @@ SetAvailable(ctx, id int64, a model.Available) error
|
||||
|
||||
// RecentVersions returns up to limit versions of repo, newest first_seen_at
|
||||
// first (ties: id DESC). It leaves out a digest-less row when a row with the
|
||||
// same repo and tag and a digest exists.
|
||||
// same repo and tag and a digest was first seen at or after it.
|
||||
RecentVersions(ctx, repo string, limit int) ([]model.ImageVersion, error)
|
||||
|
||||
// GetImageVersion returns one version or model.ErrNotFound.
|
||||
@@ -153,21 +154,22 @@ SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions v
|
||||
WHERE v.repo = $1
|
||||
AND NOT (v.digest = '' AND EXISTS (
|
||||
SELECT 1 FROM image_versions d
|
||||
WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> ''))
|
||||
WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> ''
|
||||
AND d.first_seen_at >= v.first_seen_at))
|
||||
ORDER BY v.first_seen_at DESC, v.id DESC
|
||||
LIMIT $2
|
||||
```
|
||||
|
||||
`upsertService` (running inventory): after the service upsert, on the same `execer`, only when `svc.Digest != ""`:
|
||||
`upsertService` (running inventory): after the service upsert, on the same `execer`, only when `svc.Digest` matches `^sha256:[0-9a-f]{64}$`:
|
||||
|
||||
```sql
|
||||
INSERT INTO image_versions (repo, tag, digest, source, first_seen_at)
|
||||
VALUES ($1, $2, $3, 'running', $4) ON CONFLICT DO NOTHING
|
||||
```
|
||||
|
||||
`repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image is recorded with `tag = ''`, not `latest`. Covers `ApplySnapshot` and `UpsertService` with no hub change.
|
||||
`repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image (no tag, no digest in the name) is recorded with `tag = 'latest'`, like `jobs.FollowedRef`, so it dedupes with the poll row; `repo@digest` is recorded with `tag = ''`. Covers `ApplySnapshot` and `UpsertService` with no hub change.
|
||||
|
||||
`serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error`.
|
||||
`serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error, s.last_auto_digest`. `SetLastAutoDigest(ctx, id, digest)` sets the last column; inventory writes never touch it.
|
||||
|
||||
`FinishJob` clearing rule (replaces phase 3's). With `requestedRef, requestedDigest, _ := strings.Cut(to_ref_as_created, "@")`:
|
||||
|
||||
@@ -197,7 +199,9 @@ Implementation: `name.ParseReference(repo+":"+tag)`; `remote.Head(r, remote.With
|
||||
```go
|
||||
type Store interface {
|
||||
ListServices(ctx context.Context) ([]model.Service, error)
|
||||
GetService(ctx context.Context, id int64) (model.Service, error)
|
||||
SetAvailable(ctx context.Context, id int64, a model.Available) error
|
||||
SetLastAutoDigest(ctx context.Context, id int64, digest string) error
|
||||
SetPollError(ctx context.Context, ids []int64, msg string) error
|
||||
}
|
||||
type Jobs interface {
|
||||
@@ -225,9 +229,8 @@ Cycle semantics, per distinct `ref = jobs.FollowedRef(svc.ActualRef)` over non-r
|
||||
1. `D, err := reg.Digest(ctx, ref)` with a 30s timeout per call.
|
||||
2. On error: `SetPollError(ids whose LastPollError != msg, msg)`, log `poll_failed` (ref, error).
|
||||
3. On success: `SetPollError(ids with LastPollError != "", "")`. Then per service:
|
||||
- `svc.ActualDigest == D` → nothing.
|
||||
- `svc.AvailableRef == ref && svc.AvailableDigest == D` → nothing.
|
||||
- Otherwise `SetAvailable({ref, D, poll, now})`, and if `UpdateMode == auto`, `Start(TriggerPoll, Ref: ref)`. `ErrJobActive` and `NotUpdatableError` logged at debug and ignored.
|
||||
- Suggestion: `svc.ActualDigest == D` → nothing; `svc.AvailableRef == ref && svc.AvailableDigest == D` → no write; otherwise `SetAvailable({ref, D, poll, now})`.
|
||||
- Auto job, independent of the suggestion: if `UpdateMode == auto && ActualDigest != D && D != LastAutoDigest && DesiredRef != ref+"@"+D`, re-read the service (`GetService`), re-check the same condition and removed/mode on the fresh row, then `Start(TriggerPoll, Ref: ref)`. If a job row was created, `SetLastAutoDigest(D)`. `ErrJobActive` and `NotUpdatableError` are logged at debug and ignored, leaving `last_auto_digest` unset so the next cycle retries.
|
||||
|
||||
Steps 2 and 3 use `context.WithoutCancel(ctx)`. At the end, publish `events.Services` if anything was written; log `poll_cycle` with refs, ok, failed, suggested, jobs, skipped, duration_ms.
|
||||
|
||||
@@ -277,7 +280,7 @@ const MinPinnedRefVersion = 3 // jobs whose Ref has a digest or no tag
|
||||
## Tasks
|
||||
|
||||
### Task 1: Store and model (C1, C2, C3)
|
||||
Migration, model fields, store methods, `upsertService` running versions, `FinishJob` rule. Integration tests: SetAvailable writes digest/source poll and records version, second call keeps first_seen_at; TestSnapshotRecordsRunningVersions (tag+digest, tagless digest, no digest skipped, re-snapshot idempotent); TestRecentVersions (order, limit 10, repo filter, digest-less row hidden only when a digest row for same tag exists); GetImageVersion ErrNotFound; SetPollError sets/clears, upsert does not touch last_poll_error/available_*; FinishJob table extended (tag job clears poll suggestion; pick tag@old keeps suggestion tag@new; pick tag@new clears; webhook suggestion (digest '') cleared by matching tag; tagless pick keeps it). Update the e2e `memStore` in the same commit.
|
||||
Migration, model fields, store methods, `upsertService` running versions, `FinishJob` rule. Integration tests: SetAvailable writes digest/source poll and records version, second call keeps first_seen_at; TestSnapshotRecordsRunningVersions (tag+digest, tagless digest, no digest skipped, re-snapshot idempotent); TestRecentVersions (order, limit 10, repo filter, digest-less row hidden only when a digest row for the same tag was first seen at or after it); GetImageVersion ErrNotFound; SetPollError sets/clears, upsert does not touch last_poll_error/available_*; FinishJob table extended (tag job clears poll suggestion; pick tag@old keeps suggestion tag@new; pick tag@new clears; webhook suggestion (digest '') cleared by matching tag; tagless pick keeps it). Update the e2e `memStore` in the same commit.
|
||||
|
||||
### Task 2: Protocol, agent and hub routing (C7, C6 Route)
|
||||
Bump protocol, MinPinnedRefVersion, docs; Resolver pass-through; compose/swarm output line; `hub.Route(svc, minProtocol)`. Tests: TestResolve table (tag resolves; tag@d and @d pass through with zero inspect calls; bare errors); compose pinned ref verbatim in override; swarm spec image equals pinned ref; route rows v2 agent + min 3 → ErrAgentTooOld, v3 → ok, swarm fallback to v3 manager when reporting manager is v2.
|
||||
|
||||
Reference in New Issue
Block a user