feat: master/agent phase 4 — registry polling and version history (#9) (#23)
Docker Build / docker (push) Successful in 47s
Docker Build / docker (push) Successful in 47s
This commit was merged in pull request #23.
This commit is contained in:
@@ -0,0 +1,311 @@
|
||||
# Master/Agent Phase 4 — Registry Polling and Version History Implementation Plan
|
||||
|
||||
> This plan documents what to build for issue #9 and where it deviates from the spec. Code in "Frozen contracts" is final; tasks give the contract and the test cases.
|
||||
|
||||
**Goal:** The master records every version it sees (webhook, registry polling, running inventory) in `image_versions`. Every `POLL_INTERVAL` (default 6h) it checks each distinct followed `repo:tag` against its registry with one HEAD request. A new digest becomes the service's available suggestion, and `auto` services get a job. The Service page lists the 10 most recent versions of the service's repo. Picking one starts a manual job to exactly that reference; picking an older one is a rollback, including to a digest-pinned or tagless reference.
|
||||
|
||||
**Architecture:**
|
||||
- New package `internal/master/registry`: thin wrapper around `go-containerregistry`'s `remote.Head` with `authn.DefaultKeychain` (reads `DOCKER_CONFIG`/`config.json`).
|
||||
- New package `internal/master/poller`: owns the cycle (list services, group by followed ref, run HEADs with bounded concurrency, then set available / start auto jobs / store poll error per service). `cmd/master` runs it as a background goroutine on the signal context.
|
||||
- Store: `image_versions` written in `SetAvailable` (webhook and poll, same transaction) and in `upsertService` (running inventory, inside the existing snapshot/upsert transactions).
|
||||
- Jobs gains `TriggerVersion`, which takes a version ID, never a ref from the form. `Hub.Route` gains a minimum protocol version so a pinned or tagless ref only goes to agents that understand it.
|
||||
- Agent: resolver passes digest-pinned refs through unchanged. Protocol version goes to 3.
|
||||
|
||||
**Tech Stack:** adds `github.com/google/go-containerregistry` (`pkg/name`, `pkg/v1/remote`, `pkg/authn`; `pkg/registry` and `pkg/v1/random` in tests only).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-12-master-agent-design.md`, sections "Versions: what the system has seen", "Data model", "Update", "Failure handling", "Registry polling", "UI". Phase 3 plan conventions still apply.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- `go test ./...` passes with no Docker, no Postgres and no network. Registry tests use `httptest` plus `ggcr/pkg/registry`. Postgres tests carry `//go:build integration`.
|
||||
- Interfaces declared at the consumer (`poller.Store`, `poller.Jobs`, `poller.Registry`). Hand-written fakes and testify, table-driven where there is more than one case.
|
||||
- GoDoc on every exported identifier. Conventional Commits, each ending with ` (#9)`.
|
||||
- Registry credentials never appear in logs or in `last_poll_error`. Error text stored truncated to 500 bytes.
|
||||
- Registry traffic is HEAD only: never `GET /v2/<repo>/manifests/...` (Docker Hub counts GETs as pulls). A test asserts this.
|
||||
- A pick from the UI is a POST with the CSRF token and a version ID. The ref is built server-side from the `image_versions` row, and the row must belong to the service's `image_repo`.
|
||||
- The poller never overlaps itself. Next cycle is scheduled after the previous one finishes.
|
||||
- `go.mod` `go` directive must stay `1.26.3` after `go get`.
|
||||
|
||||
## Deliberate deviations from the spec in this phase
|
||||
|
||||
1. **Webhook records `(repo, tag, '')`, no digest.** The master does not resolve the digest at webhook time (no registry round-trip/credentials in the CI path). The digest arrives later from polling or running inventory. The picker hides a digest-less row when a row with the same `(repo, tag)` and a digest was first seen at or after it.
|
||||
2. **Empty string instead of NULL** for missing tag or digest in `image_versions`, so `UNIQUE (repo, tag, digest)` works. A CHECK forbids both being empty.
|
||||
3. **`first_seen_at` is never bumped.** Re-observation is `ON CONFLICT DO NOTHING`; `source` is the first source that saw it. "10 most recent" = "10 most recently first seen". A new digest of a tag is a new row at the top; old digest stays below (enables rollback of moving tags).
|
||||
4. **Not polled:** services that follow no tag, `build:` services, services with empty `actual_digest`. UI shows the reason. Self and container services ARE polled (badge informative; jobs still refused by `jobs`).
|
||||
5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, the badge is not written again (no `available_at` bump). The auto job is decided separately: it starts only if the service is auto, `actual_digest != D`, `D != last_tag_digest` and `desired_ref != ref@D`, on a fresh read of the row. `last_tag_digest` is set when a job row was created (not on `ErrJobActive`/`NotUpdatableError`, which retry next cycle). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears. The poller also sets `last_tag_digest` to D when `actual_digest == D` (seeding services that predate the column). `FinishJob` also sets `last_tag_digest` to the pinned digest after any successful job whose `to_ref` as created has no digest (manual, webhook or poll); pinned and tagless picks leave it, so a rollback holds whoever started the update.
|
||||
6. **FinishJob clearing rule gets a digest condition** (C3). Rollback to an older digest of the same tag no longer clears the newer-version badge.
|
||||
7. **Poll auto jobs use `repo:tag`, not `repo:tag@D`.** v2 agents keep working for poll and webhook jobs. Only UI picks of digest-pinned or tagless versions need a v3 agent.
|
||||
8. **First cycle runs 1 minute after start**, then every `POLL_INTERVAL` measured from the end of the previous cycle.
|
||||
9. **No `last_polled_at` column** (YAGNI).
|
||||
10. **Overview gets two new badges: "follows no tag" and "poll failed"** (error as `title`). Service page gets a "Polling" row. "Drifted" is not in this phase.
|
||||
11. **Credentials from `DOCKER_CONFIG`** (a directory), same name and meaning as on the agent. Only inline `auths` entries supported in the container.
|
||||
|
||||
## Implementation notes (differences found while building)
|
||||
|
||||
- The plan's test names `TestResolve` and `TestComposeWritesPinnedRef` became `TestResolver`
|
||||
(extended), a subtest of `TestComposeExecutor` and `TestSwarmUsesPinnedRef`.
|
||||
- Tasks 2 and 3 are one commit: changing `Hub.Route`'s signature breaks `jobs` and the e2e hub
|
||||
wrapper until both are updated.
|
||||
- `Poller.Cycle` runs the registry lookups concurrently and then records the results
|
||||
sequentially (on `context.WithoutCancel`).
|
||||
- `web.Config` gained `PollInterval` for the "every <interval>" text.
|
||||
- `authn.DefaultKeychain` with a `credsStore` whose helper binary is missing makes the lookup fail
|
||||
with `error getting credentials - err: exec: "docker-credential-...": executable file not found in $PATH`
|
||||
(covered by `TestCredsStoreWithoutHelperFailsTheLookup`); it does not fall back to anonymous access.
|
||||
|
||||
## Review Focus
|
||||
|
||||
| Failure case | Test that covers it |
|
||||
|---|---|
|
||||
| Registry access is HEAD only; no manifest GET ever happens. | registry `TestDigestUsesHeadOnly` |
|
||||
| Multi-arch: digest returned is the index digest (same as `RepoDigests` / Swarm spec). | registry `TestDigestOfIndexIsIndexDigest` |
|
||||
| Credentials from `$DOCKER_CONFIG/config.json` are sent; anonymous works without the file. | registry `TestCredentialsFromDockerConfig`, `TestAnonymousWithoutConfig` |
|
||||
| One HEAD per distinct followed ref (`repo:tag`, `repo:tag@d`, bare → `:latest`). | poller `TestOneHeadPerDistinctRef` |
|
||||
| No tag / built / no digest are skipped, no HEAD. | poller `TestSkipped` |
|
||||
| Error on one ref goes into its services' `last_poll_error`; others continue; later success clears it. | poller `TestErrorIsPerRef`, `TestErrorClearedOnSuccess` |
|
||||
| Already-suggested digest neither re-suggested nor re-started. | poller `TestSameSuggestionIsIdempotent` |
|
||||
| Auto gets exactly one `TriggerPoll` job to `repo:tag`; manual only gets marked. | poller `TestNewDigest` table |
|
||||
| `ErrJobActive` / `NotUpdatableError` from `Start` do not abort the cycle. | poller `TestStartErrorsDoNotAbortCycle` |
|
||||
| Concurrency within limit; cancellation stops the cycle; marking/starting on `WithoutCancel`. | poller `TestConcurrencyLimit`, `TestRunStopsOnCancel` |
|
||||
| Form cannot deploy an arbitrary ref: unknown version ID → 404; version of another repo → 404, no job. | jobs `TestTriggerVersion_*`, web `TestUpdateService_Version*` |
|
||||
| Pinned/tagless refs routed only to protocol ≥ 3 agents; old agents → failed "agent too old to deploy a pinned version (needs protocol 3)" job (`ErrAgentTooOldForPinned`, which also satisfies `errors.Is(err, ErrAgentTooOld)`). | jobs `TestTriggerVersion_RequiresPinnedProtocol`, hub `TestRoute` |
|
||||
| Agent deploys a pinned ref verbatim, no registry resolve. | updater `TestResolve`, `TestComposeWritesPinnedRef`, `TestSwarmUsesPinnedRef` |
|
||||
| Rollback to older digest of same tag keeps newer suggestion; pick of suggested digest clears it. | store `TestFinishJob_ClearsAvailableOnlyWhenThatVersionWasDeployed` (extended) |
|
||||
| Running versions recorded only with a digest; re-snapshot does not duplicate or reorder. | store `TestSnapshotRecordsRunningVersions` |
|
||||
| Digest-less webhook row hidden once the tag's digest is known; order, limit, repo filter. | store `TestRecentVersions` |
|
||||
| `available_source = 'poll'` passes the CHECK. | store `TestSetAvailable_RecordsVersion` |
|
||||
| E2E: pick of a pinned version reaches the agent as that exact ref. | e2e `TestPickPinnedVersion` |
|
||||
|
||||
## Frozen contracts
|
||||
|
||||
### C1. Migration `internal/master/store/migrations/0004_polling.sql`
|
||||
|
||||
```sql
|
||||
ALTER TABLE services DROP CONSTRAINT services_available_source_check;
|
||||
ALTER TABLE services ADD CONSTRAINT services_available_source_check
|
||||
CHECK (available_source IN ('', 'webhook', 'poll'));
|
||||
ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT '';
|
||||
ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT '';
|
||||
ALTER TABLE services ADD COLUMN last_tag_digest text NOT NULL DEFAULT '';
|
||||
|
||||
-- Every version the master has seen. '' means "no tag" / "digest unknown";
|
||||
-- never NULL, so the unique constraint deduplicates.
|
||||
CREATE TABLE image_versions (
|
||||
id bigserial PRIMARY KEY,
|
||||
repo text NOT NULL,
|
||||
tag text NOT NULL DEFAULT '',
|
||||
digest text NOT NULL DEFAULT '',
|
||||
source text NOT NULL CHECK (source IN ('webhook', 'poll', 'running')),
|
||||
first_seen_at timestamptz NOT NULL,
|
||||
CONSTRAINT image_versions_tag_or_digest CHECK (tag <> '' OR digest <> ''),
|
||||
CONSTRAINT image_versions_unique UNIQUE (repo, tag, digest)
|
||||
);
|
||||
CREATE INDEX image_versions_recent ON image_versions (repo, first_seen_at DESC, id DESC);
|
||||
```
|
||||
|
||||
Confirm the existing constraint name in `0003_webhook.sql` before relying on it.
|
||||
|
||||
### C2. `internal/master/model`
|
||||
|
||||
```go
|
||||
// Service gains:
|
||||
AvailableDigest string // digest of AvailableRef when known (poll); "" from the webhook
|
||||
LastPollError string // last registry error for this service's followed ref; "" when fine
|
||||
|
||||
const SourcePoll = "poll" // alongside SourceWebhook
|
||||
const SourceRunning = "running" // image_versions only
|
||||
|
||||
type Available struct { Ref, Digest, Source string; At time.Time } // Digest added
|
||||
|
||||
// ImageVersion is one row of image_versions.
|
||||
type ImageVersion struct {
|
||||
ID int64
|
||||
Repo, Tag, Digest, Source string
|
||||
FirstSeenAt time.Time
|
||||
}
|
||||
// Ref returns "repo:tag@digest", "repo:tag" or "repo@digest".
|
||||
func (v ImageVersion) Ref() string
|
||||
```
|
||||
|
||||
### C3. Store
|
||||
|
||||
```go
|
||||
// SetAvailable sets available_ref/_digest/_source/_at and, in the same
|
||||
// transaction, inserts image_versions(repo, tag parsed from a.Ref, a.Digest,
|
||||
// a.Source, a.At) ON CONFLICT DO NOTHING.
|
||||
SetAvailable(ctx, id int64, a model.Available) error
|
||||
|
||||
// RecentVersions returns up to limit versions of repo, newest first_seen_at
|
||||
// first (ties: id DESC). It leaves out a digest-less row when a row with the
|
||||
// same repo and tag and a digest was first seen at or after it.
|
||||
RecentVersions(ctx, repo string, limit int) ([]model.ImageVersion, error)
|
||||
|
||||
// GetImageVersion returns one version or model.ErrNotFound.
|
||||
GetImageVersion(ctx, id int64) (model.ImageVersion, error)
|
||||
|
||||
// SetPollError sets last_poll_error on the given services ("" clears).
|
||||
SetPollError(ctx, ids []int64, msg string) error
|
||||
```
|
||||
|
||||
`RecentVersions` SQL:
|
||||
|
||||
```sql
|
||||
SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions v
|
||||
WHERE v.repo = $1
|
||||
AND NOT (v.digest = '' AND EXISTS (
|
||||
SELECT 1 FROM image_versions d
|
||||
WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> ''
|
||||
AND d.first_seen_at >= v.first_seen_at))
|
||||
ORDER BY v.first_seen_at DESC, v.id DESC
|
||||
LIMIT $2
|
||||
```
|
||||
|
||||
`upsertService` (running inventory): after the service upsert, on the same `execer`, only when `svc.Digest` matches `^sha256:[0-9a-f]{64}$`:
|
||||
|
||||
```sql
|
||||
INSERT INTO image_versions (repo, tag, digest, source, first_seen_at)
|
||||
VALUES ($1, $2, $3, 'running', $4) ON CONFLICT DO NOTHING
|
||||
```
|
||||
|
||||
`repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image (no tag, no digest in the name) is recorded with `tag = 'latest'`, like `jobs.FollowedRef`, so it dedupes with the poll row; `repo@digest` is recorded with `tag = ''`. Covers `ApplySnapshot` and `UpsertService` with no hub change.
|
||||
|
||||
`serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error, s.last_tag_digest`. `SetLastTagDigest(ctx, id, digest)` sets the last column; inventory writes never touch it.
|
||||
|
||||
`FinishJob` clearing rule (replaces phase 3's). With `requestedRef, requestedDigest, _ := strings.Cut(to_ref_as_created, "@")`:
|
||||
|
||||
```sql
|
||||
available_ref = $4 AND available_at <= $3
|
||||
AND ($5 = '' OR available_digest = $5)
|
||||
```
|
||||
|
||||
`$5` = `requestedDigest`. A digest-pinned job does NOT clear a digest-less (webhook) suggestion: a rollback to `main@old` must not hide that a newer `main` was announced. The badge then stays until a plain tag job succeeds; that is accepted. Clearing sets `available_ref=''`, `available_digest=''`, `available_source=''`, `available_at=NULL`.
|
||||
|
||||
### C4. `internal/master/registry`
|
||||
|
||||
```go
|
||||
// Client looks up tag digests with HEAD requests only.
|
||||
type Client struct{ /* keychain authn.Keychain, userAgent string */ }
|
||||
// New uses authn.DefaultKeychain ($DOCKER_CONFIG/config.json, else ~/.docker/config.json).
|
||||
func New(userAgent string) *Client
|
||||
// Digest returns the digest ref ("repo:tag", any digest suffix ignored) points
|
||||
// at: the index digest for multi-arch images. HEAD /v2/<repo>/manifests/<tag> only.
|
||||
func (c *Client) Digest(ctx context.Context, ref string) (string, error)
|
||||
```
|
||||
|
||||
Implementation: `name.ParseReference(repo+":"+tag)`; `remote.Head(r, remote.WithContext(ctx), remote.WithAuthFromKeychain(c.keychain), remote.WithUserAgent(c.userAgent))`; return `desc.Digest.String()`. Never fall back to GET if `Docker-Content-Digest` is missing.
|
||||
|
||||
### C5. `internal/master/poller`
|
||||
|
||||
```go
|
||||
type Store interface {
|
||||
ListServices(ctx context.Context) ([]model.Service, error)
|
||||
GetService(ctx context.Context, id int64) (model.Service, error)
|
||||
SetAvailable(ctx context.Context, id int64, a model.Available) error
|
||||
SetLastTagDigest(ctx context.Context, id int64, digest string) error
|
||||
SetPollError(ctx context.Context, ids []int64, msg string) error
|
||||
}
|
||||
type Jobs interface {
|
||||
Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error)
|
||||
}
|
||||
type Registry interface { Digest(ctx context.Context, ref string) (string, error) }
|
||||
type Publisher interface { Publish(t events.Topic) }
|
||||
|
||||
type Config struct {
|
||||
Interval time.Duration // POLL_INTERVAL
|
||||
FirstDelay time.Duration // 1m in production; small in tests
|
||||
Workers int // 4
|
||||
}
|
||||
func New(st Store, j Jobs, reg Registry, pub Publisher, cfg Config, log *slog.Logger) *Poller
|
||||
// Run polls until ctx is done. Blocking.
|
||||
func (p *Poller) Run(ctx context.Context)
|
||||
// Cycle runs one poll over all services (exported for tests and e2e).
|
||||
func (p *Poller) Cycle(ctx context.Context)
|
||||
// SkipReason says why svc is not polled: "follows no tag", "built from
|
||||
// source", "no registry digest", or "" when it is polled.
|
||||
func SkipReason(svc model.Service) string
|
||||
```
|
||||
|
||||
Cycle semantics, per distinct `ref = jobs.FollowedRef(svc.ActualRef)` over non-removed, non-skipped services:
|
||||
1. `D, err := reg.Digest(ctx, ref)` with a 30s timeout per call.
|
||||
2. On error: `SetPollError(ids whose LastPollError != msg, msg)`, log `poll_failed` (ref, error).
|
||||
3. On success: `SetPollError(ids with LastPollError != "", "")`. Then per service:
|
||||
- Suggestion: `svc.ActualDigest == D` → nothing; `svc.AvailableRef == ref && svc.AvailableDigest == D` → no write; otherwise `SetAvailable({ref, D, poll, now})`.
|
||||
- Auto job, independent of the suggestion: if `UpdateMode == auto && ActualDigest != D && D != LastTagDigest && DesiredRef != ref+"@"+D`, re-read the service (`GetService`), re-check the same condition and removed/mode on the fresh row, then `Start(TriggerPoll, Ref: ref)`. If a job row was created, `SetLastTagDigest(D)`. `ErrJobActive` and `NotUpdatableError` are logged at debug and ignored, leaving `last_tag_digest` unset so the next cycle retries.
|
||||
|
||||
Steps 2 and 3 use `context.WithoutCancel(ctx)`. At the end, publish `events.Services` if anything was written; log `poll_cycle` with refs, ok, failed, suggested, jobs, skipped, duration_ms.
|
||||
|
||||
### C6. `internal/master/jobs` and hub
|
||||
|
||||
```go
|
||||
// Hub.Route gains the minimum protocol version for this job:
|
||||
Route(svc model.Service, minProtocol int) (model.Route, error)
|
||||
|
||||
// Store gains:
|
||||
GetImageVersion(ctx context.Context, id int64) (model.ImageVersion, error)
|
||||
|
||||
// TriggerVersion starts a manual job to exactly version versionID of the
|
||||
// service's repo. model.ErrNotFound for an unknown service, an unknown
|
||||
// version, or a version of another repo. Otherwise what Start returns.
|
||||
func (s *Service) TriggerVersion(ctx context.Context, serviceID, versionID int64, requestedBy string) (model.Job, error)
|
||||
```
|
||||
|
||||
- Ref = `svc.ImageRepo` + `":"+tag` if tag set + `"@"+digest` if digest set.
|
||||
- Calls `Start` with `TriggerManual` and explicit ref ("follows no tag" refusal does not apply; removed, self, built, container still do).
|
||||
- `Start` passes `protocol.MinPinnedRefVersion` to `Route` when the ref has a digest or no tag, otherwise `protocol.MinJobVersion`.
|
||||
|
||||
### C7. Protocol and agent
|
||||
|
||||
```go
|
||||
const Version = 3
|
||||
const MinJobVersion = 2 // unchanged: repo:tag jobs
|
||||
const MinPinnedRefVersion = 3 // jobs whose Ref has a digest or no tag
|
||||
```
|
||||
|
||||
`JobStart` doc: "Ref is `repo:tag`, `repo:tag@digest` or `repo@digest`. For `repo:tag` the agent resolves the tag's current digest; a ref with a digest is deployed as is."
|
||||
|
||||
`Resolver.Resolve`: digest present → return ref unchanged, no `DistributionInspect`; tag present → resolve as today; neither → error "has no tag to resolve" (unchanged). Compose writes the ref verbatim into the override, Swarm sets it as the image. Executors print `pinned <ref>` instead of `resolved <ref>` when nothing was resolved.
|
||||
|
||||
### C8. Master configuration
|
||||
|
||||
- `POLL_INTERVAL`: default `6h`, positive Go duration, error `"POLL_INTERVAL must be a positive duration such as 6h, got %q"`.
|
||||
- `DOCKER_CONFIG`: not parsed by `config`; `go-containerregistry` reads it. `main` logs `registry_credentials` with the path and whether `config.json` exists, never its contents.
|
||||
|
||||
### C9. Web
|
||||
|
||||
- `POST /services/{id}/update` gains optional form field `version` (an `image_versions.id`): empty → `Trigger` as today; non-numeric → `400 invalid version`; otherwise `TriggerVersion`. Results: `303` to the job (also on `ErrJobActive`), `404` on `ErrNotFound`, `409` re-render "Cannot update: <reason>."
|
||||
- `web.Store` gains `RecentVersions`; `web.Jobs` gains `TriggerVersion`.
|
||||
- Service page: "Versions" table, up to 10 rows: ref (shortRef, full ref as title), source, first seen, "running" marker when digest equals `ActualDigest`, a POST form with `csrf_token` and `version`. Forms shown only when `AutoBlocked == ""`. Rows with empty tag note "the service will follow no tag afterwards". When the service is `auto`, header notes auto mode will move it forward at the next new version. A "Polling" row shows `SkipReason`, or `last_poll_error`, or "every <interval>".
|
||||
- Overview: badge "follows no tag" when `FollowedRef(ActualRef) == ""`; badge "poll failed" (title = error) when `LastPollError != ""`. Template func `followsNoTag`.
|
||||
|
||||
## Tasks
|
||||
|
||||
### Task 1: Store and model (C1, C2, C3)
|
||||
Migration, model fields, store methods, `upsertService` running versions, `FinishJob` rule. Integration tests: SetAvailable writes digest/source poll and records version, second call keeps first_seen_at; TestSnapshotRecordsRunningVersions (tag+digest, tagless digest, no digest skipped, re-snapshot idempotent); TestRecentVersions (order, limit 10, repo filter, digest-less row hidden only when a digest row for the same tag was first seen at or after it); GetImageVersion ErrNotFound; SetPollError sets/clears, upsert does not touch last_poll_error/available_*; FinishJob table extended (tag job clears poll suggestion; pick tag@old keeps suggestion tag@new; pick tag@new clears; webhook suggestion (digest '') cleared by matching tag; tagless pick keeps it). Update the e2e `memStore` in the same commit.
|
||||
|
||||
### Task 2: Protocol, agent and hub routing (C7, C6 Route)
|
||||
Bump protocol, MinPinnedRefVersion, docs; Resolver pass-through; compose/swarm output line; `hub.Route(svc, minProtocol)`. Tests: TestResolve table (tag resolves; tag@d and @d pass through with zero inspect calls; bare errors); compose pinned ref verbatim in override; swarm spec image equals pinned ref; route rows v2 agent + min 3 → ErrAgentTooOld, v3 → ok, swarm fallback to v3 manager when reporting manager is v2.
|
||||
|
||||
### Task 3: Jobs (C6)
|
||||
`Start` picks min version from the ref; `TriggerVersion`. Tests: version of other repo → ErrNotFound, no CreateJob; ref construction table; min version passed to fake hub (plain Trigger → 2, pinned → 3); explicit ref bypasses no-tag but not removed/self/built/container; existing tests green.
|
||||
|
||||
### Task 4: Registry client and poller (C4, C5)
|
||||
Registry tests (local httptest + ggcr/pkg/registry, no network): random.Index digest equals index digest; recording handler shows only HEAD on /manifests/; basic-auth handler succeeds with temp DOCKER_CONFIG (t.Setenv) and fails without; 404 → error. Verify behaviour of DefaultKeychain when config.json has credsStore but no helper binary; document. Poller tests per Review Focus, plus Run test (first cycle after delay, no overlap, returns on cancel).
|
||||
|
||||
### Task 5: Config, wiring, web (C8, C9)
|
||||
Config POLL_INTERVAL with tests; add to setAll list. `cmd/master/main.go`: registry.New, poller, `go p.Run(ctx)` with done channel, wait for it after HTTP shutdown and before st.Close; log registry_credentials. Web handlers/templates per C9 with tests (picker ≤10 rows with CSRF and version IDs; hidden for built/self/container/removed; POST with version → TriggerVersion, without → Trigger; 400/404/409; overview badges; Polling row).
|
||||
|
||||
### Task 6: End-to-end
|
||||
`internal/e2e/versions_test.go`: TestPickPinnedVersion; TestPollStartsAutoJob (poller.Cycle with fake registry → one TriggerPoll job to repo:tag; second Cycle starts nothing). `currentHub.Route` new signature.
|
||||
|
||||
### Task 7: Documentation
|
||||
README (POLL_INTERVAL, DOCKER_CONFIG, "Registry polling" section: polled/skipped, credentials only inline auths, Docker Hub note, version picker & rollback, auto caveat, agent protocol 3 for pinned/tagless picks). Spec: replace "Believed not to count against Docker Hub pull limits for HEAD requests — verify during implementation" with verified statement + source link (docs.docker.com/docker-hub/usage/pulls: "Using GET emulates a real pull and counts towards the limit. Using HEAD won't."; note the separate unquantified abuse rate limit). `deploy/master.stack.yml`: mount config dir read-only, set DOCKER_CONFIG. CLAUDE.md: architecture entries for registry and poller; gotchas (HEAD only; digest-less webhook row hidden once digest known; image_versions uses '' not NULL; picks by version ID and pinned/tagless need protocol 3; FinishJob rule includes digest (replace existing bullet); rollback to repo@digest makes the service follow no tag); phase 4 plan reference. Save this plan as `docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md`.
|
||||
|
||||
## Final verification
|
||||
|
||||
```bash
|
||||
gofmt -l . ; go vet ./...
|
||||
go build ./cmd/master ./cmd/agent && rm -f master agent
|
||||
go test ./...
|
||||
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
|
||||
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
|
||||
```
|
||||
@@ -247,8 +247,12 @@ them. An agent recreating itself would kill the job it is running.
|
||||
- Mechanism: `HEAD /v2/<repo>/manifests/<tag>`; the returned digest is compared with
|
||||
`actual_digest`. Use `go-containerregistry` — it handles Docker Hub's token auth and
|
||||
multi-arch indexes.
|
||||
- Interval `POLL_INTERVAL`, default 6h. Believed not to count against Docker Hub pull
|
||||
limits for HEAD requests — verify during implementation.
|
||||
- Interval `POLL_INTERVAL`, default 6h. Docker Hub documents that HEAD requests do not
|
||||
count against the pull limit ("Using GET emulates a real pull and counts towards the
|
||||
limit. Using HEAD won't."; verified against
|
||||
https://docs.docker.com/docker-hub/usage/pulls/). The same page mentions a separate,
|
||||
unquantified abuse rate limit for many pulls from one IP address, so the poller stays
|
||||
HEAD-only and never downloads a manifest.
|
||||
- Credentials: a Docker `config.json` (as written by `docker login`) mounted into the
|
||||
master. No credential UI, no encryption in Postgres.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user