diff --git a/CLAUDE.md b/CLAUDE.md index d00e89d..3c1dbb6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -74,12 +74,21 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/ GET as a pull; the client never falls back to GET, and a test asserts this. - **`image_versions` uses `''` instead of NULL** for a missing tag or digest, so `UNIQUE (repo, tag, digest)` deduplicates. The webhook records `(repo, tag, '')`; that - row is hidden from `RecentVersions` once a row with the same tag and a digest exists. + row is hidden from `RecentVersions` when a row with the same tag and a digest was first + seen at or after it. - **The UI picks a version by `image_versions.id`, never by a ref from the form**, and the row must belong to the service's `image_repo`. A digest-pinned or tagless pick needs agent protocol 3 (`protocol.MinPinnedRefVersion`); `Hub.Route` takes the minimum version. -- **Rolling back a service that follows no tag afterwards:** a pick of `repo@digest` makes the - service follow no tag, so it is no longer polled. +- **Picking a tagless version (`repo@digest`) leaves the service following no tag:** it is no + longer polled and the plain Update button is refused. +- **The poller starts one auto job per digest** (`services.last_auto_digest`, set only when a + job row was created; `ErrJobActive` and `NotUpdatableError` leave it so the next cycle + retries). It also skips a digest that `desired_ref` (`ref@digest`) already names, and + re-reads the service before `Start`. That is why a rollback of an auto service holds until + a new digest appears. The webhook does not set it, so a webhook-started update can be + followed by a poll job after a rollback. +- **A bare image's running version is recorded with tag `latest`** (like `jobs.FollowedRef`); + `repo@digest` is recorded with tag `''`. Only `sha256:<64 hex>` digests are recorded. - **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where the first colon is a port, not a tag. See `internal/imageref`. - **Path safety uses `filepath.Rel` + ".." prefix check**, NOT `strings.HasPrefix`. diff --git a/README.md b/README.md index 1fce50b..b3270e3 100644 --- a/README.md +++ b/README.md @@ -129,9 +129,12 @@ multi-arch image the digest is the index digest, the same one `docker inspect` s - **What counts as new.** A digest different from the one the service runs becomes the service's *available* version (the overview badge, source `poll`). A digest the master - already suggested is not suggested again. A service in **auto** mode also gets an update - job to `repo:tag` (the agent resolves the digest itself, so agents on protocol version 2 keep - working). A failed auto job is not retried every interval. + already suggested is not suggested again. A service in **auto** mode also gets one update + job to `repo:tag` per digest (the agent resolves the digest itself, so agents on protocol + version 2 keep working); the master remembers that digest in `services.last_auto_digest` + and does not start another job for it, whether the job failed or the user rolled back + afterwards. If the service already had a job when the digest was found, the next cycle + tries again. - **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and services without a registry digest (locally built images). The service page says which. The updater's own services and plain containers are polled, so their badge works, but @@ -146,25 +149,35 @@ multi-arch image the digest is the index digest, the same one `docker inspect` s in $PATH`, because the container has no credential helper binaries. The master logs `registry_credentials` at startup with the directory and whether `config.json` exists, never its contents. +- **Requests go wherever the image name points.** The master sends the lookup to the + registry host in the image name, including a `GET /v2/` ping that follows redirects. + Anyone who can start containers on a managed host can therefore make the master send + blind requests to internal hosts. The registry's answer is not shown: an error keeps only + the HTTP status and the registry's error codes. ### Version history and rollback The master records every version it sees (webhook, polling, and what agents report as running) in `image_versions`. The service page lists the 10 most recently first seen versions of the service's image repo, marks the one that is running, and has a *Deploy* -button on each row. A webhook announces a tag but not its digest, so such a row has no digest until polling or an -agent reports one; it is hidden once a row with the same tag and a digest exists. +button on each row. A webhook announces a tag but not its digest, so such a row has no +digest until polling or an agent reports one; it is hidden when a row with the same tag +and a digest was first seen at or after it. A bare image (`postgres`) is recorded as tag +`latest`. Picking a version starts a manual job to exactly that reference. Picking an older one is a -rollback, including to a digest-pinned reference. A version without a tag deploys `repo@digest` and the -service then follows no tag, so it is no longer polled or updatable with the plain *Update* button until -it is moved to a tagged version again. Picks of digest-pinned or tagless versions need an agent speaking protocol -version 3 (this release); older agents get a failed job "agent too old for updates". +rollback, including to a digest-pinned reference. A version without a tag deploys +`repo@digest` and the service then follows no tag, so it is no longer polled or updatable +with the plain *Update* button until it is moved to a tagged version again. Picks of +digest-pinned or tagless versions need an agent speaking protocol version 3 (this +release); on an older agent the job fails with "agent too old to deploy a pinned version +(needs protocol 3)". -On a service in **auto** mode a rollback is only a warning in the UI: the service stays auto and -is moved forward again at the next new version, announced by the webhook or found by polling. -Rolling back to an older digest of the tag the service follows keeps the newer version marked as -available. +A service in **auto** mode stays auto after a rollback. Polling does not undo the rollback: +it started at most one job per digest, so the service stays on the older version until a +new digest of the followed tag appears. The newer version it was rolled back from stays +marked as available. A webhook that names a tag still starts a job for an auto service +every time, so a CI push moves it forward again. ## Adding agents diff --git a/docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md b/docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md index e28dae2..d3dee9d 100644 --- a/docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md +++ b/docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md @@ -28,11 +28,11 @@ ## Deliberate deviations from the spec in this phase -1. **Webhook records `(repo, tag, '')`, no digest.** The master does not resolve the digest at webhook time (no registry round-trip/credentials in the CI path). The digest arrives later from polling or running inventory. The picker hides a digest-less row once a row with the same `(repo, tag)` and a digest exists. +1. **Webhook records `(repo, tag, '')`, no digest.** The master does not resolve the digest at webhook time (no registry round-trip/credentials in the CI path). The digest arrives later from polling or running inventory. The picker hides a digest-less row when a row with the same `(repo, tag)` and a digest was first seen at or after it. 2. **Empty string instead of NULL** for missing tag or digest in `image_versions`, so `UNIQUE (repo, tag, digest)` works. A CHECK forbids both being empty. 3. **`first_seen_at` is never bumped.** Re-observation is `ON CONFLICT DO NOTHING`; `source` is the first source that saw it. "10 most recent" = "10 most recently first seen". A new digest of a tag is a new row at the top; old digest stays below (enables rollback of moving tags). 4. **Not polled:** services that follow no tag, `build:` services, services with empty `actual_digest`. UI shows the reason. Self and container services ARE polled (badge informative; jobs still refused by `jobs`). -5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, nothing happens for that service (no `available_at` bump, no new auto job). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears. +5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, the badge is not written again (no `available_at` bump). The auto job is decided separately: it starts only if the service is auto, `actual_digest != D`, `D != last_auto_digest` and `desired_ref != ref@D`, on a fresh read of the row. `last_auto_digest` is set when a job row was created (not on `ErrJobActive`/`NotUpdatableError`, which retry next cycle). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears. The webhook does not set `last_auto_digest`. 6. **FinishJob clearing rule gets a digest condition** (C3). Rollback to an older digest of the same tag no longer clears the newer-version badge. 7. **Poll auto jobs use `repo:tag`, not `repo:tag@D`.** v2 agents keep working for poll and webhook jobs. Only UI picks of digest-pinned or tagless versions need a v3 agent. 8. **First cycle runs 1 minute after start**, then every `POLL_INTERVAL` measured from the end of the previous cycle. @@ -68,7 +68,7 @@ | `ErrJobActive` / `NotUpdatableError` from `Start` do not abort the cycle. | poller `TestStartErrorsDoNotAbortCycle` | | Concurrency within limit; cancellation stops the cycle; marking/starting on `WithoutCancel`. | poller `TestConcurrencyLimit`, `TestRunStopsOnCancel` | | Form cannot deploy an arbitrary ref: unknown version ID → 404; version of another repo → 404, no job. | jobs `TestTriggerVersion_*`, web `TestUpdateService_Version*` | -| Pinned/tagless refs routed only to protocol ≥ 3 agents; old agents → failed "agent too old" job. | jobs `TestTriggerVersion_RequiresPinnedProtocol`, hub `TestRoute` | +| Pinned/tagless refs routed only to protocol ≥ 3 agents; old agents → failed "agent too old to deploy a pinned version (needs protocol 3)" job (`ErrAgentTooOldForPinned`, which also satisfies `errors.Is(err, ErrAgentTooOld)`). | jobs `TestTriggerVersion_RequiresPinnedProtocol`, hub `TestRoute` | | Agent deploys a pinned ref verbatim, no registry resolve. | updater `TestResolve`, `TestComposeWritesPinnedRef`, `TestSwarmUsesPinnedRef` | | Rollback to older digest of same tag keeps newer suggestion; pick of suggested digest clears it. | store `TestFinishJob_ClearsAvailableOnlyWhenThatVersionWasDeployed` (extended) | | Running versions recorded only with a digest; re-snapshot does not duplicate or reorder. | store `TestSnapshotRecordsRunningVersions` | @@ -86,6 +86,7 @@ ALTER TABLE services ADD CONSTRAINT services_available_source_check CHECK (available_source IN ('', 'webhook', 'poll')); ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT ''; ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT ''; +ALTER TABLE services ADD COLUMN last_auto_digest text NOT NULL DEFAULT ''; -- Every version the master has seen. '' means "no tag" / "digest unknown"; -- never NULL, so the unique constraint deduplicates. @@ -136,7 +137,7 @@ SetAvailable(ctx, id int64, a model.Available) error // RecentVersions returns up to limit versions of repo, newest first_seen_at // first (ties: id DESC). It leaves out a digest-less row when a row with the -// same repo and tag and a digest exists. +// same repo and tag and a digest was first seen at or after it. RecentVersions(ctx, repo string, limit int) ([]model.ImageVersion, error) // GetImageVersion returns one version or model.ErrNotFound. @@ -153,21 +154,22 @@ SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions v WHERE v.repo = $1 AND NOT (v.digest = '' AND EXISTS ( SELECT 1 FROM image_versions d - WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> '')) + WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> '' + AND d.first_seen_at >= v.first_seen_at)) ORDER BY v.first_seen_at DESC, v.id DESC LIMIT $2 ``` -`upsertService` (running inventory): after the service upsert, on the same `execer`, only when `svc.Digest != ""`: +`upsertService` (running inventory): after the service upsert, on the same `execer`, only when `svc.Digest` matches `^sha256:[0-9a-f]{64}$`: ```sql INSERT INTO image_versions (repo, tag, digest, source, first_seen_at) VALUES ($1, $2, $3, 'running', $4) ON CONFLICT DO NOTHING ``` -`repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image is recorded with `tag = ''`, not `latest`. Covers `ApplySnapshot` and `UpsertService` with no hub change. +`repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image (no tag, no digest in the name) is recorded with `tag = 'latest'`, like `jobs.FollowedRef`, so it dedupes with the poll row; `repo@digest` is recorded with `tag = ''`. Covers `ApplySnapshot` and `UpsertService` with no hub change. -`serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error`. +`serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error, s.last_auto_digest`. `SetLastAutoDigest(ctx, id, digest)` sets the last column; inventory writes never touch it. `FinishJob` clearing rule (replaces phase 3's). With `requestedRef, requestedDigest, _ := strings.Cut(to_ref_as_created, "@")`: @@ -197,7 +199,9 @@ Implementation: `name.ParseReference(repo+":"+tag)`; `remote.Head(r, remote.With ```go type Store interface { ListServices(ctx context.Context) ([]model.Service, error) + GetService(ctx context.Context, id int64) (model.Service, error) SetAvailable(ctx context.Context, id int64, a model.Available) error + SetLastAutoDigest(ctx context.Context, id int64, digest string) error SetPollError(ctx context.Context, ids []int64, msg string) error } type Jobs interface { @@ -225,9 +229,8 @@ Cycle semantics, per distinct `ref = jobs.FollowedRef(svc.ActualRef)` over non-r 1. `D, err := reg.Digest(ctx, ref)` with a 30s timeout per call. 2. On error: `SetPollError(ids whose LastPollError != msg, msg)`, log `poll_failed` (ref, error). 3. On success: `SetPollError(ids with LastPollError != "", "")`. Then per service: - - `svc.ActualDigest == D` → nothing. - - `svc.AvailableRef == ref && svc.AvailableDigest == D` → nothing. - - Otherwise `SetAvailable({ref, D, poll, now})`, and if `UpdateMode == auto`, `Start(TriggerPoll, Ref: ref)`. `ErrJobActive` and `NotUpdatableError` logged at debug and ignored. + - Suggestion: `svc.ActualDigest == D` → nothing; `svc.AvailableRef == ref && svc.AvailableDigest == D` → no write; otherwise `SetAvailable({ref, D, poll, now})`. + - Auto job, independent of the suggestion: if `UpdateMode == auto && ActualDigest != D && D != LastAutoDigest && DesiredRef != ref+"@"+D`, re-read the service (`GetService`), re-check the same condition and removed/mode on the fresh row, then `Start(TriggerPoll, Ref: ref)`. If a job row was created, `SetLastAutoDigest(D)`. `ErrJobActive` and `NotUpdatableError` are logged at debug and ignored, leaving `last_auto_digest` unset so the next cycle retries. Steps 2 and 3 use `context.WithoutCancel(ctx)`. At the end, publish `events.Services` if anything was written; log `poll_cycle` with refs, ok, failed, suggested, jobs, skipped, duration_ms. @@ -277,7 +280,7 @@ const MinPinnedRefVersion = 3 // jobs whose Ref has a digest or no tag ## Tasks ### Task 1: Store and model (C1, C2, C3) -Migration, model fields, store methods, `upsertService` running versions, `FinishJob` rule. Integration tests: SetAvailable writes digest/source poll and records version, second call keeps first_seen_at; TestSnapshotRecordsRunningVersions (tag+digest, tagless digest, no digest skipped, re-snapshot idempotent); TestRecentVersions (order, limit 10, repo filter, digest-less row hidden only when a digest row for same tag exists); GetImageVersion ErrNotFound; SetPollError sets/clears, upsert does not touch last_poll_error/available_*; FinishJob table extended (tag job clears poll suggestion; pick tag@old keeps suggestion tag@new; pick tag@new clears; webhook suggestion (digest '') cleared by matching tag; tagless pick keeps it). Update the e2e `memStore` in the same commit. +Migration, model fields, store methods, `upsertService` running versions, `FinishJob` rule. Integration tests: SetAvailable writes digest/source poll and records version, second call keeps first_seen_at; TestSnapshotRecordsRunningVersions (tag+digest, tagless digest, no digest skipped, re-snapshot idempotent); TestRecentVersions (order, limit 10, repo filter, digest-less row hidden only when a digest row for the same tag was first seen at or after it); GetImageVersion ErrNotFound; SetPollError sets/clears, upsert does not touch last_poll_error/available_*; FinishJob table extended (tag job clears poll suggestion; pick tag@old keeps suggestion tag@new; pick tag@new clears; webhook suggestion (digest '') cleared by matching tag; tagless pick keeps it). Update the e2e `memStore` in the same commit. ### Task 2: Protocol, agent and hub routing (C7, C6 Route) Bump protocol, MinPinnedRefVersion, docs; Resolver pass-through; compose/swarm output line; `hub.Route(svc, minProtocol)`. Tests: TestResolve table (tag resolves; tag@d and @d pass through with zero inspect calls; bare errors); compose pinned ref verbatim in override; swarm spec image equals pinned ref; route rows v2 agent + min 3 → ErrAgentTooOld, v3 → ok, swarm fallback to v3 manager when reporting manager is v2.