package-updater/docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md

Master/Agent Phase 4 — Registry Polling and Version History Implementation Plan

This plan documents what to build for issue #9 and where it deviates from the spec. Code in "Frozen contracts" is final; tasks give the contract and the test cases.

Goal: The master records every version it sees (webhook, registry polling, running inventory) in image_versions. Every POLL_INTERVAL (default 6h) it checks each distinct followed repo:tag against its registry with one HEAD request. A new digest becomes the service's available suggestion, and auto services get a job. The Service page lists the 10 most recent versions of the service's repo. Picking one starts a manual job to exactly that reference; picking an older one is a rollback, including to a digest-pinned or tagless reference.

Architecture:

• New package internal/master/registry: thin wrapper around go-containerregistry's remote.Head with authn.DefaultKeychain (reads DOCKER_CONFIG/config.json).
• New package internal/master/poller: owns the cycle (list services, group by followed ref, run HEADs with bounded concurrency, then set available / start auto jobs / store poll error per service). cmd/master runs it as a background goroutine on the signal context.
• Store: image_versions written in SetAvailable (webhook and poll, same transaction) and in upsertService (running inventory, inside the existing snapshot/upsert transactions).
• Jobs gains TriggerVersion, which takes a version ID, never a ref from the form. Hub.Route gains a minimum protocol version so a pinned or tagless ref only goes to agents that understand it.
• Agent: resolver passes digest-pinned refs through unchanged. Protocol version goes to 3.

Tech Stack: adds github.com/google/go-containerregistry (pkg/name, pkg/v1/remote, pkg/authn; pkg/registry and pkg/v1/random in tests only).

Spec: docs/superpowers/specs/2026-10-12-master-agent-design.md, sections "Versions: what the system has seen", "Data model", "Update", "Failure handling", "Registry polling", "UI". Phase 3 plan conventions still apply.

Global Constraints

• go test ./... passes with no Docker, no Postgres and no network. Registry tests use httptest plus ggcr/pkg/registry. Postgres tests carry //go:build integration.
• Interfaces declared at the consumer (poller.Store, poller.Jobs, poller.Registry). Hand-written fakes and testify, table-driven where there is more than one case.
• GoDoc on every exported identifier. Conventional Commits, each ending with (#9).
• Registry credentials never appear in logs or in last_poll_error. Error text stored truncated to 500 bytes.
• Registry traffic is HEAD only: never GET /v2/<repo>/manifests/... (Docker Hub counts GETs as pulls). A test asserts this.
• A pick from the UI is a POST with the CSRF token and a version ID. The ref is built server-side from the image_versions row, and the row must belong to the service's image_repo.
• The poller never overlaps itself. Next cycle is scheduled after the previous one finishes.
• go.mod go directive must stay 1.26.3 after go get.

Deliberate deviations from the spec in this phase

1. Webhook records (repo, tag, ''), no digest. The master does not resolve the digest at webhook time (no registry round-trip/credentials in the CI path). The digest arrives later from polling or running inventory. The picker hides a digest-less row once a row with the same (repo, tag) and a digest exists.
2. Empty string instead of NULL for missing tag or digest in image_versions, so UNIQUE (repo, tag, digest) works. A CHECK forbids both being empty.
3. first_seen_at is never bumped. Re-observation is ON CONFLICT DO NOTHING; source is the first source that saw it. "10 most recent" = "10 most recently first seen". A new digest of a tag is a new row at the top; old digest stays below (enables rollback of moving tags).
4. Not polled: services that follow no tag, build: services, services with empty actual_digest. UI shows the reason. Self and container services ARE polled (badge informative; jobs still refused by jobs).
5. Poller does not re-suggest the same version. If available_ref == ref && available_digest == D, nothing happens for that service (no available_at bump, no new auto job). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears.
6. FinishJob clearing rule gets a digest condition (C3). Rollback to an older digest of the same tag no longer clears the newer-version badge.
7. Poll auto jobs use repo:tag, not repo:tag@D. v2 agents keep working for poll and webhook jobs. Only UI picks of digest-pinned or tagless versions need a v3 agent.
8. First cycle runs 1 minute after start, then every POLL_INTERVAL measured from the end of the previous cycle.
9. No last_polled_at column (YAGNI).
10. Overview gets two new badges: "follows no tag" and "poll failed" (error as title). Service page gets a "Polling" row. "Drifted" is not in this phase.
11. Credentials from DOCKER_CONFIG (a directory), same name and meaning as on the agent. Only inline auths entries supported in the container.

Implementation notes (differences found while building)

• The plan's test names TestResolve and TestComposeWritesPinnedRef became TestResolver (extended), a subtest of TestComposeExecutor and TestSwarmUsesPinnedRef.
• Tasks 2 and 3 are one commit: changing Hub.Route's signature breaks jobs and the e2e hub wrapper until both are updated.
• Poller.Cycle runs the registry lookups concurrently and then records the results sequentially (on context.WithoutCancel).
• web.Config gained PollInterval for the "every " text.
• authn.DefaultKeychain with a credsStore whose helper binary is missing makes the lookup fail with error getting credentials - err: exec: "docker-credential-...": executable file not found in $PATH (covered by TestCredsStoreWithoutHelperFailsTheLookup); it does not fall back to anonymous access.

Review Focus

Failure case	Test that covers it
Registry access is HEAD only; no manifest GET ever happens.	registry TestDigestUsesHeadOnly
Multi-arch: digest returned is the index digest (same as RepoDigests / Swarm spec).	registry TestDigestOfIndexIsIndexDigest
Credentials from $DOCKER_CONFIG/config.json are sent; anonymous works without the file.	registry TestCredentialsFromDockerConfig, TestAnonymousWithoutConfig
One HEAD per distinct followed ref (repo:tag, repo:tag@d, bare → :latest).	poller TestOneHeadPerDistinctRef
No tag / built / no digest are skipped, no HEAD.	poller TestSkipped
Error on one ref goes into its services' last_poll_error; others continue; later success clears it.	poller TestErrorIsPerRef, TestErrorClearedOnSuccess
Already-suggested digest neither re-suggested nor re-started.	poller TestSameSuggestionIsIdempotent
Auto gets exactly one TriggerPoll job to repo:tag; manual only gets marked.	poller TestNewDigest table
ErrJobActive / NotUpdatableError from Start do not abort the cycle.	poller TestStartErrorsDoNotAbortCycle
Concurrency within limit; cancellation stops the cycle; marking/starting on WithoutCancel.	poller TestConcurrencyLimit, TestRunStopsOnCancel
Form cannot deploy an arbitrary ref: unknown version ID → 404; version of another repo → 404, no job.	jobs TestTriggerVersion_*, web TestUpdateService_Version*
Pinned/tagless refs routed only to protocol ≥ 3 agents; old agents → failed "agent too old" job.	jobs TestTriggerVersion_RequiresPinnedProtocol, hub TestRoute
Agent deploys a pinned ref verbatim, no registry resolve.	updater TestResolve, TestComposeWritesPinnedRef, TestSwarmUsesPinnedRef
Rollback to older digest of same tag keeps newer suggestion; pick of suggested digest clears it.	store TestFinishJob_ClearsAvailableOnlyWhenThatVersionWasDeployed (extended)
Running versions recorded only with a digest; re-snapshot does not duplicate or reorder.	store TestSnapshotRecordsRunningVersions
Digest-less webhook row hidden once the tag's digest is known; order, limit, repo filter.	store TestRecentVersions
available_source = 'poll' passes the CHECK.	store TestSetAvailable_RecordsVersion
E2E: pick of a pinned version reaches the agent as that exact ref.	e2e TestPickPinnedVersion

Frozen contracts

C1. Migration internal/master/store/migrations/0004_polling.sql

ALTER TABLE services DROP CONSTRAINT services_available_source_check;
ALTER TABLE services ADD CONSTRAINT services_available_source_check
    CHECK (available_source IN ('', 'webhook', 'poll'));
ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_poll_error  text NOT NULL DEFAULT '';

-- Every version the master has seen. '' means "no tag" / "digest unknown";
-- never NULL, so the unique constraint deduplicates.
CREATE TABLE image_versions (
    id            bigserial   PRIMARY KEY,
    repo          text        NOT NULL,
    tag           text        NOT NULL DEFAULT '',
    digest        text        NOT NULL DEFAULT '',
    source        text        NOT NULL CHECK (source IN ('webhook', 'poll', 'running')),
    first_seen_at timestamptz NOT NULL,
    CONSTRAINT image_versions_tag_or_digest CHECK (tag <> '' OR digest <> ''),
    CONSTRAINT image_versions_unique UNIQUE (repo, tag, digest)
);
CREATE INDEX image_versions_recent ON image_versions (repo, first_seen_at DESC, id DESC);

Confirm the existing constraint name in 0003_webhook.sql before relying on it.

C2. internal/master/model

// Service gains:
AvailableDigest string // digest of AvailableRef when known (poll); "" from the webhook
LastPollError   string // last registry error for this service's followed ref; "" when fine

const SourcePoll = "poll"       // alongside SourceWebhook
const SourceRunning = "running" // image_versions only

type Available struct { Ref, Digest, Source string; At time.Time } // Digest added

// ImageVersion is one row of image_versions.
type ImageVersion struct {
    ID          int64
    Repo, Tag, Digest, Source string
    FirstSeenAt time.Time
}
// Ref returns "repo:tag@digest", "repo:tag" or "repo@digest".
func (v ImageVersion) Ref() string

C3. Store

// SetAvailable sets available_ref/_digest/_source/_at and, in the same
// transaction, inserts image_versions(repo, tag parsed from a.Ref, a.Digest,
// a.Source, a.At) ON CONFLICT DO NOTHING.
SetAvailable(ctx, id int64, a model.Available) error

// RecentVersions returns up to limit versions of repo, newest first_seen_at
// first (ties: id DESC). It leaves out a digest-less row when a row with the
// same repo and tag and a digest exists.
RecentVersions(ctx, repo string, limit int) ([]model.ImageVersion, error)

// GetImageVersion returns one version or model.ErrNotFound.
GetImageVersion(ctx, id int64) (model.ImageVersion, error)

// SetPollError sets last_poll_error on the given services ("" clears).
SetPollError(ctx, ids []int64, msg string) error

RecentVersions SQL:

SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions v
WHERE v.repo = $1
  AND NOT (v.digest = '' AND EXISTS (
        SELECT 1 FROM image_versions d
        WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> ''))
ORDER BY v.first_seen_at DESC, v.id DESC
LIMIT $2

upsertService (running inventory): after the service upsert, on the same execer, only when svc.Digest != "":

INSERT INTO image_versions (repo, tag, digest, source, first_seen_at)
VALUES ($1, $2, $3, 'running', $4) ON CONFLICT DO NOTHING

repo/tag from imageref.Parse(svc.Image), digest = svc.Digest. A bare image is recorded with tag = '', not latest. Covers ApplySnapshot and UpsertService with no hub change.

serviceColumns / scanService gain s.available_digest, s.last_poll_error.

FinishJob clearing rule (replaces phase 3's). With requestedRef, requestedDigest, _ := strings.Cut(to_ref_as_created, "@"):

available_ref = $4 AND available_at <= $3
  AND ($5 = '' OR available_digest = $5)

$5 = requestedDigest. A digest-pinned job does NOT clear a digest-less (webhook) suggestion: a rollback to main@old must not hide that a newer main was announced. The badge then stays until a plain tag job succeeds; that is accepted. Clearing sets available_ref='', available_digest='', available_source='', available_at=NULL.

C4. internal/master/registry

// Client looks up tag digests with HEAD requests only.
type Client struct{ /* keychain authn.Keychain, userAgent string */ }
// New uses authn.DefaultKeychain ($DOCKER_CONFIG/config.json, else ~/.docker/config.json).
func New(userAgent string) *Client
// Digest returns the digest ref ("repo:tag", any digest suffix ignored) points
// at: the index digest for multi-arch images. HEAD /v2/<repo>/manifests/<tag> only.
func (c *Client) Digest(ctx context.Context, ref string) (string, error)

Implementation: name.ParseReference(repo+":"+tag); remote.Head(r, remote.WithContext(ctx), remote.WithAuthFromKeychain(c.keychain), remote.WithUserAgent(c.userAgent)); return desc.Digest.String(). Never fall back to GET if Docker-Content-Digest is missing.

C5. internal/master/poller

type Store interface {
    ListServices(ctx context.Context) ([]model.Service, error)
    SetAvailable(ctx context.Context, id int64, a model.Available) error
    SetPollError(ctx context.Context, ids []int64, msg string) error
}
type Jobs interface {
    Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error)
}
type Registry interface { Digest(ctx context.Context, ref string) (string, error) }
type Publisher interface { Publish(t events.Topic) }

type Config struct {
    Interval   time.Duration // POLL_INTERVAL
    FirstDelay time.Duration // 1m in production; small in tests
    Workers    int           // 4
}
func New(st Store, j Jobs, reg Registry, pub Publisher, cfg Config, log *slog.Logger) *Poller
// Run polls until ctx is done. Blocking.
func (p *Poller) Run(ctx context.Context)
// Cycle runs one poll over all services (exported for tests and e2e).
func (p *Poller) Cycle(ctx context.Context)
// SkipReason says why svc is not polled: "follows no tag", "built from
// source", "no registry digest", or "" when it is polled.
func SkipReason(svc model.Service) string

Cycle semantics, per distinct ref = jobs.FollowedRef(svc.ActualRef) over non-removed, non-skipped services:

1. D, err := reg.Digest(ctx, ref) with a 30s timeout per call.
2. On error: SetPollError(ids whose LastPollError != msg, msg), log poll_failed (ref, error).
3. On success: SetPollError(ids with LastPollError != "", ""). Then per service:
   • svc.ActualDigest == D → nothing.
   • svc.AvailableRef == ref && svc.AvailableDigest == D → nothing.
   • Otherwise SetAvailable({ref, D, poll, now}), and if UpdateMode == auto, Start(TriggerPoll, Ref: ref). ErrJobActive and NotUpdatableError logged at debug and ignored.

Steps 2 and 3 use context.WithoutCancel(ctx). At the end, publish events.Services if anything was written; log poll_cycle with refs, ok, failed, suggested, jobs, skipped, duration_ms.

C6. internal/master/jobs and hub

// Hub.Route gains the minimum protocol version for this job:
Route(svc model.Service, minProtocol int) (model.Route, error)

// Store gains:
GetImageVersion(ctx context.Context, id int64) (model.ImageVersion, error)

// TriggerVersion starts a manual job to exactly version versionID of the
// service's repo. model.ErrNotFound for an unknown service, an unknown
// version, or a version of another repo. Otherwise what Start returns.
func (s *Service) TriggerVersion(ctx context.Context, serviceID, versionID int64, requestedBy string) (model.Job, error)

• Ref = svc.ImageRepo + ":"+tag if tag set + "@"+digest if digest set.
• Calls Start with TriggerManual and explicit ref ("follows no tag" refusal does not apply; removed, self, built, container still do).
• Start passes protocol.MinPinnedRefVersion to Route when the ref has a digest or no tag, otherwise protocol.MinJobVersion.

C7. Protocol and agent

const Version = 3
const MinJobVersion = 2       // unchanged: repo:tag jobs
const MinPinnedRefVersion = 3 // jobs whose Ref has a digest or no tag

JobStart doc: "Ref is repo:tag, repo:tag@digest or repo@digest. For repo:tag the agent resolves the tag's current digest; a ref with a digest is deployed as is."

Resolver.Resolve: digest present → return ref unchanged, no DistributionInspect; tag present → resolve as today; neither → error "has no tag to resolve" (unchanged). Compose writes the ref verbatim into the override, Swarm sets it as the image. Executors print pinned <ref> instead of resolved <ref> when nothing was resolved.

C8. Master configuration

• POLL_INTERVAL: default 6h, positive Go duration, error "POLL_INTERVAL must be a positive duration such as 6h, got %q".
• DOCKER_CONFIG: not parsed by config; go-containerregistry reads it. main logs registry_credentials with the path and whether config.json exists, never its contents.

C9. Web

• POST /services/{id}/update gains optional form field version (an image_versions.id): empty → Trigger as today; non-numeric → 400 invalid version; otherwise TriggerVersion. Results: 303 to the job (also on ErrJobActive), 404 on ErrNotFound, 409 re-render "Cannot update: ."
• web.Store gains RecentVersions; web.Jobs gains TriggerVersion.
• Service page: "Versions" table, up to 10 rows: ref (shortRef, full ref as title), source, first seen, "running" marker when digest equals ActualDigest, a POST form with csrf_token and version. Forms shown only when AutoBlocked == "". Rows with empty tag note "the service will follow no tag afterwards". When the service is auto, header notes auto mode will move it forward at the next new version. A "Polling" row shows SkipReason, or last_poll_error, or "every ".
• Overview: badge "follows no tag" when FollowedRef(ActualRef) == ""; badge "poll failed" (title = error) when LastPollError != "". Template func followsNoTag.

Tasks

Task 1: Store and model (C1, C2, C3)

Migration, model fields, store methods, upsertService running versions, FinishJob rule. Integration tests: SetAvailable writes digest/source poll and records version, second call keeps first_seen_at; TestSnapshotRecordsRunningVersions (tag+digest, tagless digest, no digest skipped, re-snapshot idempotent); TestRecentVersions (order, limit 10, repo filter, digest-less row hidden only when a digest row for same tag exists); GetImageVersion ErrNotFound; SetPollError sets/clears, upsert does not touch last_poll_error/available_*; FinishJob table extended (tag job clears poll suggestion; pick tag@old keeps suggestion tag@new; pick tag@new clears; webhook suggestion (digest '') cleared by matching tag; tagless pick keeps it). Update the e2e memStore in the same commit.

Task 2: Protocol, agent and hub routing (C7, C6 Route)

Bump protocol, MinPinnedRefVersion, docs; Resolver pass-through; compose/swarm output line; hub.Route(svc, minProtocol). Tests: TestResolve table (tag resolves; tag@d and @d pass through with zero inspect calls; bare errors); compose pinned ref verbatim in override; swarm spec image equals pinned ref; route rows v2 agent + min 3 → ErrAgentTooOld, v3 → ok, swarm fallback to v3 manager when reporting manager is v2.

Task 3: Jobs (C6)

Start picks min version from the ref; TriggerVersion. Tests: version of other repo → ErrNotFound, no CreateJob; ref construction table; min version passed to fake hub (plain Trigger → 2, pinned → 3); explicit ref bypasses no-tag but not removed/self/built/container; existing tests green.

Task 4: Registry client and poller (C4, C5)

Registry tests (local httptest + ggcr/pkg/registry, no network): random.Index digest equals index digest; recording handler shows only HEAD on /manifests/; basic-auth handler succeeds with temp DOCKER_CONFIG (t.Setenv) and fails without; 404 → error. Verify behaviour of DefaultKeychain when config.json has credsStore but no helper binary; document. Poller tests per Review Focus, plus Run test (first cycle after delay, no overlap, returns on cancel).

Task 5: Config, wiring, web (C8, C9)

Config POLL_INTERVAL with tests; add to setAll list. cmd/master/main.go: registry.New, poller, go p.Run(ctx) with done channel, wait for it after HTTP shutdown and before st.Close; log registry_credentials. Web handlers/templates per C9 with tests (picker ≤10 rows with CSRF and version IDs; hidden for built/self/container/removed; POST with version → TriggerVersion, without → Trigger; 400/404/409; overview badges; Polling row).

Task 6: End-to-end

internal/e2e/versions_test.go: TestPickPinnedVersion; TestPollStartsAutoJob (poller.Cycle with fake registry → one TriggerPoll job to repo:tag; second Cycle starts nothing). currentHub.Route new signature.

Task 7: Documentation

README (POLL_INTERVAL, DOCKER_CONFIG, "Registry polling" section: polled/skipped, credentials only inline auths, Docker Hub note, version picker & rollback, auto caveat, agent protocol 3 for pinned/tagless picks). Spec: replace "Believed not to count against Docker Hub pull limits for HEAD requests — verify during implementation" with verified statement + source link (docs.docker.com/docker-hub/usage/pulls: "Using GET emulates a real pull and counts towards the limit. Using HEAD won't."; note the separate unquantified abuse rate limit). deploy/master.stack.yml: mount config dir read-only, set DOCKER_CONFIG. CLAUDE.md: architecture entries for registry and poller; gotchas (HEAD only; digest-less webhook row hidden once digest known; image_versions uses '' not NULL; picks by version ID and pinned/tagless need protocol 3; FinishJob rule includes digest (replace existing bullet); rollback to repo@digest makes the service follow no tag); phase 4 plan reference. Save this plan as docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md.

Final verification

gofmt -l . ; go vet ./...
go build ./cmd/master ./cmd/agent && rm -f master agent
go test ./...
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/