# Master/Agent Phase 4 — Registry Polling and Version History Implementation Plan

> This plan documents what to build for issue #9 and where it deviates from the spec. Code in "Frozen contracts" is final; tasks give the contract and the test cases.

**Goal:** The master records every version it sees (webhook, registry polling, running inventory) in `image_versions`. Every `POLL_INTERVAL` (default 6h) it checks each distinct followed `repo:tag` against its registry with one HEAD request. A new digest becomes the service's available suggestion, and `auto` services get a job. The Service page lists the 10 most recent versions of the service's repo. Picking one starts a manual job to exactly that reference; picking an older one is a rollback, including to a digest-pinned or tagless reference.

**Architecture:**
- New package `internal/master/registry`: thin wrapper around `go-containerregistry`'s `remote.Head` with `authn.DefaultKeychain` (reads `DOCKER_CONFIG`/`config.json`).
- New package `internal/master/poller`: owns the cycle (list services, group by followed ref, run HEADs with bounded concurrency, then set available / start auto jobs / store poll error per service). `cmd/master` runs it as a background goroutine on the signal context.
- Store: `image_versions` written in `SetAvailable` (webhook and poll, same transaction) and in `upsertService` (running inventory, inside the existing snapshot/upsert transactions).
- Jobs gains `TriggerVersion`, which takes a version ID, never a ref from the form. `Hub.Route` gains a minimum protocol version so a pinned or tagless ref only goes to agents that understand it.
- Agent: resolver passes digest-pinned refs through unchanged. Protocol version goes to 3.

**Tech Stack:** adds `github.com/google/go-containerregistry` (`pkg/name`, `pkg/v1/remote`, `pkg/authn`; `pkg/registry` and `pkg/v1/random` in tests only).

**Spec:** `docs/superpowers/specs/2026-10-12-master-agent-design.md`, sections "Versions: what the system has seen", "Data model", "Update", "Failure handling", "Registry polling", "UI". Phase 3 plan conventions still apply.

## Global Constraints

- `go test ./...` passes with no Docker, no Postgres and no network. Registry tests use `httptest` plus `ggcr/pkg/registry`. Postgres tests carry `//go:build integration`.
- Interfaces declared at the consumer (`poller.Store`, `poller.Jobs`, `poller.Registry`). Hand-written fakes and testify, table-driven where there is more than one case.
- GoDoc on every exported identifier. Conventional Commits, each ending with ` (#9)`.
- Registry credentials never appear in logs or in `last_poll_error`. Error text stored truncated to 500 bytes.
- Registry traffic is HEAD only: never `GET /v2/<repo>/manifests/...` (Docker Hub counts GETs as pulls). A test asserts this.
- A pick from the UI is a POST with the CSRF token and a version ID. The ref is built server-side from the `image_versions` row, and the row must belong to the service's `image_repo`.
- The poller never overlaps itself. Next cycle is scheduled after the previous one finishes.
- `go.mod` `go` directive must stay `1.26.3` after `go get`.

## Deliberate deviations from the spec in this phase

1. **Webhook records `(repo, tag, '')`, no digest.** The master does not resolve the digest at webhook time (no registry round-trip/credentials in the CI path). The digest arrives later from polling or running inventory. The picker hides a digest-less row when a row with the same `(repo, tag)` and a digest was first seen at or after it.
2. **Empty string instead of NULL** for missing tag or digest in `image_versions`, so `UNIQUE (repo, tag, digest)` works. A CHECK forbids both being empty.
3. **`first_seen_at` is never bumped.** Re-observation is `ON CONFLICT DO NOTHING`; `source` is the first source that saw it. "10 most recent" = "10 most recently first seen". A new digest of a tag is a new row at the top; old digest stays below (enables rollback of moving tags).
4. **Not polled:** services that follow no tag, `build:` services, services with empty `actual_digest`. UI shows the reason. Self and container services ARE polled (badge informative; jobs still refused by `jobs`).
5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, the badge is not written again (no `available_at` bump). The auto job is decided separately: it starts only if the service is auto, `actual_digest != D`, `D != last_tag_digest` and `desired_ref != ref@D`, on a fresh read of the row. `last_tag_digest` is set when a job row was created (not on `ErrJobActive`/`NotUpdatableError`, which retry next cycle). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears. The poller also sets `last_tag_digest` to D when `actual_digest == D` (seeding services that predate the column). `FinishJob` also sets `last_tag_digest` to the pinned digest after any successful job whose `to_ref` as created has no digest (manual, webhook or poll); pinned and tagless picks leave it, so a rollback holds whoever started the update.
6. **FinishJob clearing rule gets a digest condition** (C3). Rollback to an older digest of the same tag no longer clears the newer-version badge.
7. **Poll auto jobs use `repo:tag`, not `repo:tag@D`.** v2 agents keep working for poll and webhook jobs. Only UI picks of digest-pinned or tagless versions need a v3 agent.
8. **First cycle runs 1 minute after start**, then every `POLL_INTERVAL` measured from the end of the previous cycle.
9. **No `last_polled_at` column** (YAGNI).
10. **Overview gets two new badges: "follows no tag" and "poll failed"** (error as `title`). Service page gets a "Polling" row. "Drifted" is not in this phase.
11. **Credentials from `DOCKER_CONFIG`** (a directory), same name and meaning as on the agent. Only inline `auths` entries supported in the container.

## Implementation notes (differences found while building)

- The plan's test names `TestResolve` and `TestComposeWritesPinnedRef` became `TestResolver`
  (extended), a subtest of `TestComposeExecutor` and `TestSwarmUsesPinnedRef`.
- Tasks 2 and 3 are one commit: changing `Hub.Route`'s signature breaks `jobs` and the e2e hub
  wrapper until both are updated.
- `Poller.Cycle` runs the registry lookups concurrently and then records the results
  sequentially (on `context.WithoutCancel`).
- `web.Config` gained `PollInterval` for the "every <interval>" text.
- `authn.DefaultKeychain` with a `credsStore` whose helper binary is missing makes the lookup fail
  with `error getting credentials - err: exec: "docker-credential-...": executable file not found in $PATH`
  (covered by `TestCredsStoreWithoutHelperFailsTheLookup`); it does not fall back to anonymous access.

## Review Focus

| Failure case | Test that covers it |
|---|---|
| Registry access is HEAD only; no manifest GET ever happens. | registry `TestDigestUsesHeadOnly` |
| Multi-arch: digest returned is the index digest (same as `RepoDigests` / Swarm spec). | registry `TestDigestOfIndexIsIndexDigest` |
| Credentials from `$DOCKER_CONFIG/config.json` are sent; anonymous works without the file. | registry `TestCredentialsFromDockerConfig`, `TestAnonymousWithoutConfig` |
| One HEAD per distinct followed ref (`repo:tag`, `repo:tag@d`, bare → `:latest`). | poller `TestOneHeadPerDistinctRef` |
| No tag / built / no digest are skipped, no HEAD. | poller `TestSkipped` |
| Error on one ref goes into its services' `last_poll_error`; others continue; later success clears it. | poller `TestErrorIsPerRef`, `TestErrorClearedOnSuccess` |
| Already-suggested digest neither re-suggested nor re-started. | poller `TestSameSuggestionIsIdempotent` |
| Auto gets exactly one `TriggerPoll` job to `repo:tag`; manual only gets marked. | poller `TestNewDigest` table |
| `ErrJobActive` / `NotUpdatableError` from `Start` do not abort the cycle. | poller `TestStartErrorsDoNotAbortCycle` |
| Concurrency within limit; cancellation stops the cycle; marking/starting on `WithoutCancel`. | poller `TestConcurrencyLimit`, `TestRunStopsOnCancel` |
| Form cannot deploy an arbitrary ref: unknown version ID → 404; version of another repo → 404, no job. | jobs `TestTriggerVersion_*`, web `TestUpdateService_Version*` |
| Pinned/tagless refs routed only to protocol ≥ 3 agents; old agents → failed "agent too old to deploy a pinned version (needs protocol 3)" job (`ErrAgentTooOldForPinned`, which also satisfies `errors.Is(err, ErrAgentTooOld)`). | jobs `TestTriggerVersion_RequiresPinnedProtocol`, hub `TestRoute` |
| Agent deploys a pinned ref verbatim, no registry resolve. | updater `TestResolve`, `TestComposeWritesPinnedRef`, `TestSwarmUsesPinnedRef` |
| Rollback to older digest of same tag keeps newer suggestion; pick of suggested digest clears it. | store `TestFinishJob_ClearsAvailableOnlyWhenThatVersionWasDeployed` (extended) |
| Running versions recorded only with a digest; re-snapshot does not duplicate or reorder. | store `TestSnapshotRecordsRunningVersions` |
| Digest-less webhook row hidden once the tag's digest is known; order, limit, repo filter. | store `TestRecentVersions` |
| `available_source = 'poll'` passes the CHECK. | store `TestSetAvailable_RecordsVersion` |
| E2E: pick of a pinned version reaches the agent as that exact ref. | e2e `TestPickPinnedVersion` |

## Frozen contracts

### C1. Migration `internal/master/store/migrations/0004_polling.sql`

```sql
ALTER TABLE services DROP CONSTRAINT services_available_source_check;
ALTER TABLE services ADD CONSTRAINT services_available_source_check
    CHECK (available_source IN ('', 'webhook', 'poll'));
ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_poll_error  text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_tag_digest text NOT NULL DEFAULT '';

-- Every version the master has seen. '' means "no tag" / "digest unknown";
-- never NULL, so the unique constraint deduplicates.
CREATE TABLE image_versions (
    id            bigserial   PRIMARY KEY,
    repo          text        NOT NULL,
    tag           text        NOT NULL DEFAULT '',
    digest        text        NOT NULL DEFAULT '',
    source        text        NOT NULL CHECK (source IN ('webhook', 'poll', 'running')),
    first_seen_at timestamptz NOT NULL,
    CONSTRAINT image_versions_tag_or_digest CHECK (tag <> '' OR digest <> ''),
    CONSTRAINT image_versions_unique UNIQUE (repo, tag, digest)
);
CREATE INDEX image_versions_recent ON image_versions (repo, first_seen_at DESC, id DESC);
```

Confirm the existing constraint name in `0003_webhook.sql` before relying on it.

### C2. `internal/master/model`

```go
// Service gains:
AvailableDigest string // digest of AvailableRef when known (poll); "" from the webhook
LastPollError   string // last registry error for this service's followed ref; "" when fine

const SourcePoll = "poll"       // alongside SourceWebhook
const SourceRunning = "running" // image_versions only

type Available struct { Ref, Digest, Source string; At time.Time } // Digest added

// ImageVersion is one row of image_versions.
type ImageVersion struct {
    ID          int64
    Repo, Tag, Digest, Source string
    FirstSeenAt time.Time
}
// Ref returns "repo:tag@digest", "repo:tag" or "repo@digest".
func (v ImageVersion) Ref() string
```

### C3. Store

```go
// SetAvailable sets available_ref/_digest/_source/_at and, in the same
// transaction, inserts image_versions(repo, tag parsed from a.Ref, a.Digest,
// a.Source, a.At) ON CONFLICT DO NOTHING.
SetAvailable(ctx, id int64, a model.Available) error

// RecentVersions returns up to limit versions of repo, newest first_seen_at
// first (ties: id DESC). It leaves out a digest-less row when a row with the
// same repo and tag and a digest was first seen at or after it.
RecentVersions(ctx, repo string, limit int) ([]model.ImageVersion, error)

// GetImageVersion returns one version or model.ErrNotFound.
GetImageVersion(ctx, id int64) (model.ImageVersion, error)

// SetPollError sets last_poll_error on the given services ("" clears).
SetPollError(ctx, ids []int64, msg string) error
```

`RecentVersions` SQL:

```sql
SELECT id, repo, tag, digest, source, first_seen_at FROM image_versions v
WHERE v.repo = $1
  AND NOT (v.digest = '' AND EXISTS (
        SELECT 1 FROM image_versions d
        WHERE d.repo = v.repo AND d.tag = v.tag AND d.digest <> ''
          AND d.first_seen_at >= v.first_seen_at))
ORDER BY v.first_seen_at DESC, v.id DESC
LIMIT $2
```

`upsertService` (running inventory): after the service upsert, on the same `execer`, only when `svc.Digest` matches `^sha256:[0-9a-f]{64}$`:

```sql
INSERT INTO image_versions (repo, tag, digest, source, first_seen_at)
VALUES ($1, $2, $3, 'running', $4) ON CONFLICT DO NOTHING
```

`repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image (no tag, no digest in the name) is recorded with `tag = 'latest'`, like `jobs.FollowedRef`, so it dedupes with the poll row; `repo@digest` is recorded with `tag = ''`. Covers `ApplySnapshot` and `UpsertService` with no hub change.

`serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error, s.last_tag_digest`. `SetLastTagDigest(ctx, id, digest)` sets the last column; inventory writes never touch it.

`FinishJob` clearing rule (replaces phase 3's). With `requestedRef, requestedDigest, _ := strings.Cut(to_ref_as_created, "@")`:

```sql
available_ref = $4 AND available_at <= $3
  AND ($5 = '' OR available_digest = $5)
```

`$5` = `requestedDigest`. A digest-pinned job does NOT clear a digest-less (webhook) suggestion: a rollback to `main@old` must not hide that a newer `main` was announced. The badge then stays until a plain tag job succeeds; that is accepted. Clearing sets `available_ref=''`, `available_digest=''`, `available_source=''`, `available_at=NULL`.

### C4. `internal/master/registry`

```go
// Client looks up tag digests with HEAD requests only.
type Client struct{ /* keychain authn.Keychain, userAgent string */ }
// New uses authn.DefaultKeychain ($DOCKER_CONFIG/config.json, else ~/.docker/config.json).
func New(userAgent string) *Client
// Digest returns the digest ref ("repo:tag", any digest suffix ignored) points
// at: the index digest for multi-arch images. HEAD /v2/<repo>/manifests/<tag> only.
func (c *Client) Digest(ctx context.Context, ref string) (string, error)
```

Implementation: `name.ParseReference(repo+":"+tag)`; `remote.Head(r, remote.WithContext(ctx), remote.WithAuthFromKeychain(c.keychain), remote.WithUserAgent(c.userAgent))`; return `desc.Digest.String()`. Never fall back to GET if `Docker-Content-Digest` is missing.

### C5. `internal/master/poller`

```go
type Store interface {
    ListServices(ctx context.Context) ([]model.Service, error)
    GetService(ctx context.Context, id int64) (model.Service, error)
    SetAvailable(ctx context.Context, id int64, a model.Available) error
    SetLastTagDigest(ctx context.Context, id int64, digest string) error
    SetPollError(ctx context.Context, ids []int64, msg string) error
}
type Jobs interface {
    Start(ctx context.Context, svc model.Service, req jobs.StartRequest) (model.Job, error)
}
type Registry interface { Digest(ctx context.Context, ref string) (string, error) }
type Publisher interface { Publish(t events.Topic) }

type Config struct {
    Interval   time.Duration // POLL_INTERVAL
    FirstDelay time.Duration // 1m in production; small in tests
    Workers    int           // 4
}
func New(st Store, j Jobs, reg Registry, pub Publisher, cfg Config, log *slog.Logger) *Poller
// Run polls until ctx is done. Blocking.
func (p *Poller) Run(ctx context.Context)
// Cycle runs one poll over all services (exported for tests and e2e).
func (p *Poller) Cycle(ctx context.Context)
// SkipReason says why svc is not polled: "follows no tag", "built from
// source", "no registry digest", or "" when it is polled.
func SkipReason(svc model.Service) string
```

Cycle semantics, per distinct `ref = jobs.FollowedRef(svc.ActualRef)` over non-removed, non-skipped services:
1. `D, err := reg.Digest(ctx, ref)` with a 30s timeout per call.
2. On error: `SetPollError(ids whose LastPollError != msg, msg)`, log `poll_failed` (ref, error).
3. On success: `SetPollError(ids with LastPollError != "", "")`. Then per service:
   - Suggestion: `svc.ActualDigest == D` → nothing; `svc.AvailableRef == ref && svc.AvailableDigest == D` → no write; otherwise `SetAvailable({ref, D, poll, now})`.
   - Auto job, independent of the suggestion: if `UpdateMode == auto && ActualDigest != D && D != LastTagDigest && DesiredRef != ref+"@"+D`, re-read the service (`GetService`), re-check the same condition and removed/mode on the fresh row, then `Start(TriggerPoll, Ref: ref)`. If a job row was created, `SetLastTagDigest(D)`. `ErrJobActive` and `NotUpdatableError` are logged at debug and ignored, leaving `last_tag_digest` unset so the next cycle retries.

Steps 2 and 3 use `context.WithoutCancel(ctx)`. At the end, publish `events.Services` if anything was written; log `poll_cycle` with refs, ok, failed, suggested, jobs, skipped, duration_ms.

### C6. `internal/master/jobs` and hub

```go
// Hub.Route gains the minimum protocol version for this job:
Route(svc model.Service, minProtocol int) (model.Route, error)

// Store gains:
GetImageVersion(ctx context.Context, id int64) (model.ImageVersion, error)

// TriggerVersion starts a manual job to exactly version versionID of the
// service's repo. model.ErrNotFound for an unknown service, an unknown
// version, or a version of another repo. Otherwise what Start returns.
func (s *Service) TriggerVersion(ctx context.Context, serviceID, versionID int64, requestedBy string) (model.Job, error)
```

- Ref = `svc.ImageRepo` + `":"+tag` if tag set + `"@"+digest` if digest set.
- Calls `Start` with `TriggerManual` and explicit ref ("follows no tag" refusal does not apply; removed, self, built, container still do).
- `Start` passes `protocol.MinPinnedRefVersion` to `Route` when the ref has a digest or no tag, otherwise `protocol.MinJobVersion`.

### C7. Protocol and agent

```go
const Version = 3
const MinJobVersion = 2       // unchanged: repo:tag jobs
const MinPinnedRefVersion = 3 // jobs whose Ref has a digest or no tag
```

`JobStart` doc: "Ref is `repo:tag`, `repo:tag@digest` or `repo@digest`. For `repo:tag` the agent resolves the tag's current digest; a ref with a digest is deployed as is."

`Resolver.Resolve`: digest present → return ref unchanged, no `DistributionInspect`; tag present → resolve as today; neither → error "has no tag to resolve" (unchanged). Compose writes the ref verbatim into the override, Swarm sets it as the image. Executors print `pinned <ref>` instead of `resolved <ref>` when nothing was resolved.

### C8. Master configuration

- `POLL_INTERVAL`: default `6h`, positive Go duration, error `"POLL_INTERVAL must be a positive duration such as 6h, got %q"`.
- `DOCKER_CONFIG`: not parsed by `config`; `go-containerregistry` reads it. `main` logs `registry_credentials` with the path and whether `config.json` exists, never its contents.

### C9. Web

- `POST /services/{id}/update` gains optional form field `version` (an `image_versions.id`): empty → `Trigger` as today; non-numeric → `400 invalid version`; otherwise `TriggerVersion`. Results: `303` to the job (also on `ErrJobActive`), `404` on `ErrNotFound`, `409` re-render "Cannot update: <reason>."
- `web.Store` gains `RecentVersions`; `web.Jobs` gains `TriggerVersion`.
- Service page: "Versions" table, up to 10 rows: ref (shortRef, full ref as title), source, first seen, "running" marker when digest equals `ActualDigest`, a POST form with `csrf_token` and `version`. Forms shown only when `AutoBlocked == ""`. Rows with empty tag note "the service will follow no tag afterwards". When the service is `auto`, header notes auto mode will move it forward at the next new version. A "Polling" row shows `SkipReason`, or `last_poll_error`, or "every <interval>".
- Overview: badge "follows no tag" when `FollowedRef(ActualRef) == ""`; badge "poll failed" (title = error) when `LastPollError != ""`. Template func `followsNoTag`.

## Tasks

### Task 1: Store and model (C1, C2, C3)
Migration, model fields, store methods, `upsertService` running versions, `FinishJob` rule. Integration tests: SetAvailable writes digest/source poll and records version, second call keeps first_seen_at; TestSnapshotRecordsRunningVersions (tag+digest, tagless digest, no digest skipped, re-snapshot idempotent); TestRecentVersions (order, limit 10, repo filter, digest-less row hidden only when a digest row for the same tag was first seen at or after it); GetImageVersion ErrNotFound; SetPollError sets/clears, upsert does not touch last_poll_error/available_*; FinishJob table extended (tag job clears poll suggestion; pick tag@old keeps suggestion tag@new; pick tag@new clears; webhook suggestion (digest '') cleared by matching tag; tagless pick keeps it). Update the e2e `memStore` in the same commit.

### Task 2: Protocol, agent and hub routing (C7, C6 Route)
Bump protocol, MinPinnedRefVersion, docs; Resolver pass-through; compose/swarm output line; `hub.Route(svc, minProtocol)`. Tests: TestResolve table (tag resolves; tag@d and @d pass through with zero inspect calls; bare errors); compose pinned ref verbatim in override; swarm spec image equals pinned ref; route rows v2 agent + min 3 → ErrAgentTooOld, v3 → ok, swarm fallback to v3 manager when reporting manager is v2.

### Task 3: Jobs (C6)
`Start` picks min version from the ref; `TriggerVersion`. Tests: version of other repo → ErrNotFound, no CreateJob; ref construction table; min version passed to fake hub (plain Trigger → 2, pinned → 3); explicit ref bypasses no-tag but not removed/self/built/container; existing tests green.

### Task 4: Registry client and poller (C4, C5)
Registry tests (local httptest + ggcr/pkg/registry, no network): random.Index digest equals index digest; recording handler shows only HEAD on /manifests/; basic-auth handler succeeds with temp DOCKER_CONFIG (t.Setenv) and fails without; 404 → error. Verify behaviour of DefaultKeychain when config.json has credsStore but no helper binary; document. Poller tests per Review Focus, plus Run test (first cycle after delay, no overlap, returns on cancel).

### Task 5: Config, wiring, web (C8, C9)
Config POLL_INTERVAL with tests; add to setAll list. `cmd/master/main.go`: registry.New, poller, `go p.Run(ctx)` with done channel, wait for it after HTTP shutdown and before st.Close; log registry_credentials. Web handlers/templates per C9 with tests (picker ≤10 rows with CSRF and version IDs; hidden for built/self/container/removed; POST with version → TriggerVersion, without → Trigger; 400/404/409; overview badges; Polling row).

### Task 6: End-to-end
`internal/e2e/versions_test.go`: TestPickPinnedVersion; TestPollStartsAutoJob (poller.Cycle with fake registry → one TriggerPoll job to repo:tag; second Cycle starts nothing). `currentHub.Route` new signature.

### Task 7: Documentation
README (POLL_INTERVAL, DOCKER_CONFIG, "Registry polling" section: polled/skipped, credentials only inline auths, Docker Hub note, version picker & rollback, auto caveat, agent protocol 3 for pinned/tagless picks). Spec: replace "Believed not to count against Docker Hub pull limits for HEAD requests — verify during implementation" with verified statement + source link (docs.docker.com/docker-hub/usage/pulls: "Using GET emulates a real pull and counts towards the limit. Using HEAD won't."; note the separate unquantified abuse rate limit). `deploy/master.stack.yml`: mount config dir read-only, set DOCKER_CONFIG. CLAUDE.md: architecture entries for registry and poller; gotchas (HEAD only; digest-less webhook row hidden once digest known; image_versions uses '' not NULL; picks by version ID and pinned/tagless need protocol 3; FinishJob rule includes digest (replace existing bullet); rollback to repo@digest makes the service follow no tag); phase 4 plan reference. Save this plan as `docs/superpowers/plans/2026-10-13-master-agent-phase-4-polling.md`.

## Final verification

```bash
gofmt -l . ; go vet ./...
go build ./cmd/master ./cmd/agent && rm -f master agent
go test ./...
go test -race ./internal/agent/... ./internal/master/... ./internal/e2e/
DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
```