feat: master/agent phase 4 — registry polling and version history (#9) #23

Merged
shcizo merged 14 commits from feat/registry-polling-version-history-9 into main 2026-10-13 07:02:38 +00:00
13 changed files with 149 additions and 56 deletions
Showing only changes of commit 3166874433 - Show all commits
+6 -6
View File
@@ -81,12 +81,12 @@ DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/
protocol 3 (`protocol.MinPinnedRefVersion`); `Hub.Route` takes the minimum version. protocol 3 (`protocol.MinPinnedRefVersion`); `Hub.Route` takes the minimum version.
- **Picking a tagless version (`repo@digest`) leaves the service following no tag:** it is no - **Picking a tagless version (`repo@digest`) leaves the service following no tag:** it is no
longer polled and the plain Update button is refused. longer polled and the plain Update button is refused.
- **The poller starts one auto job per digest** (`services.last_auto_digest`, set only when a - **`services.last_tag_digest`** is set by the poller when it starts an auto job and by
job row was created; `ErrJobActive` and `NotUpdatableError` leave it so the next cycle `FinishJob` after any successful plain tag job (not pinned/tagless picks; the digest is the
retries). It also skips a digest that `desired_ref` (`ref@digest`) already names, and one the agent pinned). The poller starts no job for that digest, nor for one `desired_ref`
re-reads the service before `Start`. That is why a rollback of an auto service holds until (`ref@digest`) already names, and re-reads the service before `Start`; so a rollback of an
a new digest appears. The webhook does not set it, so a webhook-started update can be auto service holds until the registry shows a different digest. `ErrJobActive` and
followed by a poll job after a rollback. `NotUpdatableError` leave it unset so the next cycle retries.
- **A bare image's running version is recorded with tag `latest`** (like `jobs.FollowedRef`); - **A bare image's running version is recorded with tag `latest`** (like `jobs.FollowedRef`);
`repo@digest` is recorded with tag `''`. Only `sha256:<64 hex>` digests are recorded. `repo@digest` is recorded with tag `''`. Only `sha256:<64 hex>` digests are recorded.
- **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where - **Image matching is case-sensitive.** The tricky case is `localhost:5000/foo:v1` where
+12 -11
View File
@@ -129,12 +129,13 @@ multi-arch image the digest is the index digest, the same one `docker inspect` s
- **What counts as new.** A digest different from the one the service runs becomes the - **What counts as new.** A digest different from the one the service runs becomes the
service's *available* version (the overview badge, source `poll`). A digest the master service's *available* version (the overview badge, source `poll`). A digest the master
already suggested is not suggested again. A service in **auto** mode also gets one update already suggested is not suggested again. A service in **auto** mode also gets an update
job to `repo:tag` per digest (the agent resolves the digest itself, so agents on protocol job to `repo:tag` (the agent resolves the digest itself, so agents on protocol version 2
version 2 keep working); the master remembers that digest in `services.last_auto_digest` keep working). The master remembers the newest digest it started or deployed for the tag
and does not start another job for it, whether the job failed or the user rolled back in `services.last_tag_digest` (set when polling starts a job and when any plain tag job
afterwards. If the service already had a job when the digest was found, the next cycle succeeds, whoever started it) and starts no job for that digest again, whether the job
tries again. failed or the user rolled back afterwards. If the service already had a job when the
digest was found, the next cycle tries again.
- **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and - **What is not polled.** Services that follow no tag (`repo@digest`), `build:` services, and
services without a registry digest (locally built images). The service page says which. services without a registry digest (locally built images). The service page says which.
The updater's own services and plain containers are polled, so their badge works, but The updater's own services and plain containers are polled, so their badge works, but
@@ -173,11 +174,11 @@ digest-pinned or tagless versions need an agent speaking protocol version 3 (thi
release); on an older agent the job fails with "agent too old to deploy a pinned version release); on an older agent the job fails with "agent too old to deploy a pinned version
(needs protocol 3)". (needs protocol 3)".
A service in **auto** mode stays auto after a rollback. Polling does not undo the rollback: A service in **auto** mode stays auto after a rollback. Polling does not undo it: the
it started at most one job per digest, so the service stays on the older version until a service stays on the older version until the registry shows a digest of the followed tag
new digest of the followed tag appears. The newer version it was rolled back from stays that differs from the newest one the master deployed. The newer version it was rolled back
marked as available. A webhook that names a tag still starts a job for an auto service from stays marked as available. A webhook that names a tag still starts a job for an auto
every time, so a CI push moves it forward again. service every time, so a CI push moves it forward again.
## Adding agents ## Adding agents
@@ -32,7 +32,7 @@
2. **Empty string instead of NULL** for missing tag or digest in `image_versions`, so `UNIQUE (repo, tag, digest)` works. A CHECK forbids both being empty. 2. **Empty string instead of NULL** for missing tag or digest in `image_versions`, so `UNIQUE (repo, tag, digest)` works. A CHECK forbids both being empty.
3. **`first_seen_at` is never bumped.** Re-observation is `ON CONFLICT DO NOTHING`; `source` is the first source that saw it. "10 most recent" = "10 most recently first seen". A new digest of a tag is a new row at the top; old digest stays below (enables rollback of moving tags). 3. **`first_seen_at` is never bumped.** Re-observation is `ON CONFLICT DO NOTHING`; `source` is the first source that saw it. "10 most recent" = "10 most recently first seen". A new digest of a tag is a new row at the top; old digest stays below (enables rollback of moving tags).
4. **Not polled:** services that follow no tag, `build:` services, services with empty `actual_digest`. UI shows the reason. Self and container services ARE polled (badge informative; jobs still refused by `jobs`). 4. **Not polled:** services that follow no tag, `build:` services, services with empty `actual_digest`. UI shows the reason. Self and container services ARE polled (badge informative; jobs still refused by `jobs`).
5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, the badge is not written again (no `available_at` bump). The auto job is decided separately: it starts only if the service is auto, `actual_digest != D`, `D != last_auto_digest` and `desired_ref != ref@D`, on a fresh read of the row. `last_auto_digest` is set when a job row was created (not on `ErrJobActive`/`NotUpdatableError`, which retry next cycle). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears. The webhook does not set `last_auto_digest`. 5. **Poller does not re-suggest the same version.** If `available_ref == ref && available_digest == D`, the badge is not written again (no `available_at` bump). The auto job is decided separately: it starts only if the service is auto, `actual_digest != D`, `D != last_tag_digest` and `desired_ref != ref@D`, on a fresh read of the row. `last_tag_digest` is set when a job row was created (not on `ErrJobActive`/`NotUpdatableError`, which retry next cycle). A failed auto job is not retried every interval; a rollback of an auto service is not undone by polling until a new digest appears. `FinishJob` also sets `last_tag_digest` to the pinned digest after any successful job whose `to_ref` as created has no digest (manual, webhook or poll); pinned and tagless picks leave it, so a rollback holds whoever started the update.
6. **FinishJob clearing rule gets a digest condition** (C3). Rollback to an older digest of the same tag no longer clears the newer-version badge. 6. **FinishJob clearing rule gets a digest condition** (C3). Rollback to an older digest of the same tag no longer clears the newer-version badge.
7. **Poll auto jobs use `repo:tag`, not `repo:tag@D`.** v2 agents keep working for poll and webhook jobs. Only UI picks of digest-pinned or tagless versions need a v3 agent. 7. **Poll auto jobs use `repo:tag`, not `repo:tag@D`.** v2 agents keep working for poll and webhook jobs. Only UI picks of digest-pinned or tagless versions need a v3 agent.
8. **First cycle runs 1 minute after start**, then every `POLL_INTERVAL` measured from the end of the previous cycle. 8. **First cycle runs 1 minute after start**, then every `POLL_INTERVAL` measured from the end of the previous cycle.
@@ -86,7 +86,7 @@ ALTER TABLE services ADD CONSTRAINT services_available_source_check
CHECK (available_source IN ('', 'webhook', 'poll')); CHECK (available_source IN ('', 'webhook', 'poll'));
ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT ''; ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT ''; ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_auto_digest text NOT NULL DEFAULT ''; ALTER TABLE services ADD COLUMN last_tag_digest text NOT NULL DEFAULT '';
-- Every version the master has seen. '' means "no tag" / "digest unknown"; -- Every version the master has seen. '' means "no tag" / "digest unknown";
-- never NULL, so the unique constraint deduplicates. -- never NULL, so the unique constraint deduplicates.
@@ -169,7 +169,7 @@ VALUES ($1, $2, $3, 'running', $4) ON CONFLICT DO NOTHING
`repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image (no tag, no digest in the name) is recorded with `tag = 'latest'`, like `jobs.FollowedRef`, so it dedupes with the poll row; `repo@digest` is recorded with `tag = ''`. Covers `ApplySnapshot` and `UpsertService` with no hub change. `repo`/`tag` from `imageref.Parse(svc.Image)`, `digest` = `svc.Digest`. A bare image (no tag, no digest in the name) is recorded with `tag = 'latest'`, like `jobs.FollowedRef`, so it dedupes with the poll row; `repo@digest` is recorded with `tag = ''`. Covers `ApplySnapshot` and `UpsertService` with no hub change.
`serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error, s.last_auto_digest`. `SetLastAutoDigest(ctx, id, digest)` sets the last column; inventory writes never touch it. `serviceColumns` / `scanService` gain `s.available_digest, s.last_poll_error, s.last_tag_digest`. `SetLastTagDigest(ctx, id, digest)` sets the last column; inventory writes never touch it.
`FinishJob` clearing rule (replaces phase 3's). With `requestedRef, requestedDigest, _ := strings.Cut(to_ref_as_created, "@")`: `FinishJob` clearing rule (replaces phase 3's). With `requestedRef, requestedDigest, _ := strings.Cut(to_ref_as_created, "@")`:
@@ -201,7 +201,7 @@ type Store interface {
ListServices(ctx context.Context) ([]model.Service, error) ListServices(ctx context.Context) ([]model.Service, error)
GetService(ctx context.Context, id int64) (model.Service, error) GetService(ctx context.Context, id int64) (model.Service, error)
SetAvailable(ctx context.Context, id int64, a model.Available) error SetAvailable(ctx context.Context, id int64, a model.Available) error
SetLastAutoDigest(ctx context.Context, id int64, digest string) error SetLastTagDigest(ctx context.Context, id int64, digest string) error
SetPollError(ctx context.Context, ids []int64, msg string) error SetPollError(ctx context.Context, ids []int64, msg string) error
} }
type Jobs interface { type Jobs interface {
@@ -230,7 +230,7 @@ Cycle semantics, per distinct `ref = jobs.FollowedRef(svc.ActualRef)` over non-r
2. On error: `SetPollError(ids whose LastPollError != msg, msg)`, log `poll_failed` (ref, error). 2. On error: `SetPollError(ids whose LastPollError != msg, msg)`, log `poll_failed` (ref, error).
3. On success: `SetPollError(ids with LastPollError != "", "")`. Then per service: 3. On success: `SetPollError(ids with LastPollError != "", "")`. Then per service:
- Suggestion: `svc.ActualDigest == D` → nothing; `svc.AvailableRef == ref && svc.AvailableDigest == D` → no write; otherwise `SetAvailable({ref, D, poll, now})`. - Suggestion: `svc.ActualDigest == D` → nothing; `svc.AvailableRef == ref && svc.AvailableDigest == D` → no write; otherwise `SetAvailable({ref, D, poll, now})`.
- Auto job, independent of the suggestion: if `UpdateMode == auto && ActualDigest != D && D != LastAutoDigest && DesiredRef != ref+"@"+D`, re-read the service (`GetService`), re-check the same condition and removed/mode on the fresh row, then `Start(TriggerPoll, Ref: ref)`. If a job row was created, `SetLastAutoDigest(D)`. `ErrJobActive` and `NotUpdatableError` are logged at debug and ignored, leaving `last_auto_digest` unset so the next cycle retries. - Auto job, independent of the suggestion: if `UpdateMode == auto && ActualDigest != D && D != LastTagDigest && DesiredRef != ref+"@"+D`, re-read the service (`GetService`), re-check the same condition and removed/mode on the fresh row, then `Start(TriggerPoll, Ref: ref)`. If a job row was created, `SetLastTagDigest(D)`. `ErrJobActive` and `NotUpdatableError` are logged at debug and ignored, leaving `last_tag_digest` unset so the next cycle retries.
Steps 2 and 3 use `context.WithoutCancel(ctx)`. At the end, publish `events.Services` if anything was written; log `poll_cycle` with refs, ok, failed, suggested, jobs, skipped, duration_ms. Steps 2 and 3 use `context.WithoutCancel(ctx)`. At the end, publish `events.Services` if anything was written; log `poll_cycle` with refs, ok, failed, suggested, jobs, skipped, duration_ms.
+5 -2
View File
@@ -91,14 +91,14 @@ func (m *memStore) SetPollError(_ context.Context, ids []int64, msg string) erro
return nil return nil
} }
func (m *memStore) SetLastAutoDigest(_ context.Context, id int64, digest string) error { func (m *memStore) SetLastTagDigest(_ context.Context, id int64, digest string) error {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
svc, ok := m.svcRows[id] svc, ok := m.svcRows[id]
if !ok { if !ok {
return model.ErrNotFound return model.ErrNotFound
} }
svc.LastAutoDigest = digest svc.LastTagDigest = digest
m.svcRows[id] = svc m.svcRows[id] = svc
return nil return nil
} }
@@ -241,6 +241,9 @@ func (m *memStore) FinishJob(_ context.Context, agentID, jobID int64, f model.Jo
if f.Status == model.JobSucceeded { if f.Status == model.JobSucceeded {
svc := m.svcRows[j.ServiceID] svc := m.svcRows[j.ServiceID]
svc.DesiredRef = f.Ref svc.DesiredRef = f.Ref
if _, d, _ := strings.Cut(f.Ref, "@"); !strings.Contains(requestedRef, "@") && d != "" {
svc.LastTagDigest = d
}
// Same rule as the SQL: the job was created for the suggested ref after it was reported. // Same rule as the SQL: the job was created for the suggested ref after it was reported.
requested, digest, _ := strings.Cut(requestedRef, "@") requested, digest, _ := strings.Cut(requestedRef, "@")
if svc.AvailableAt != nil && svc.AvailableRef == requested && !svc.AvailableAt.After(j.CreatedAt) && if svc.AvailableAt != nil && svc.AvailableRef == requested && !svc.AvailableAt.After(j.CreatedAt) &&
+35 -1
View File
@@ -10,6 +10,7 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"github.com/shcizo/package-updater/internal/master/jobs"
"github.com/shcizo/package-updater/internal/master/model" "github.com/shcizo/package-updater/internal/master/model"
"github.com/shcizo/package-updater/internal/master/poller" "github.com/shcizo/package-updater/internal/master/poller"
"github.com/shcizo/package-updater/internal/protocol" "github.com/shcizo/package-updater/internal/protocol"
@@ -88,7 +89,7 @@ func TestPollStartsAutoJob(t *testing.T) {
assert.Equal(t, slusk+":1.4.2", svc.AvailableRef) assert.Equal(t, slusk+":1.4.2", svc.AvailableRef)
assert.Equal(t, "sha256:new", svc.AvailableDigest) assert.Equal(t, "sha256:new", svc.AvailableDigest)
assert.Equal(t, model.SourcePoll, svc.AvailableSource) assert.Equal(t, model.SourcePoll, svc.AvailableSource)
assert.Equal(t, "sha256:new", svc.LastAutoDigest) assert.Equal(t, "sha256:new", svc.LastTagDigest)
p.Cycle(context.Background()) p.Cycle(context.Background())
@@ -97,3 +98,36 @@ func TestPollStartsAutoJob(t *testing.T) {
close(env.exec.release) close(env.exec.release)
env.waitStatus(t, job.ID, model.JobSucceeded) env.waitStatus(t, job.ID, model.JobSucceeded)
} }
// An update started by the webhook, then a rollback by hand: polling must not
// redo the update while the registry still shows the digest that was deployed.
func TestPollKeepsARollbackAfterAWebhookUpdate(t *testing.T) {
env := newJobEnv(t)
ctx := context.Background()
close(env.exec.release)
env.store.seedService(model.Service{ID: 1, Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web",
AgentID: ptr(agentID), ImageRepo: slusk, ActualRef: slusk + ":main@sha256:old", ActualDigest: "sha256:old",
UpdateMode: model.ModeAuto})
env.store.seedVersion(model.ImageVersion{ID: 5, Repo: slusk, Tag: "main", Digest: "sha256:old", Source: model.SourceRunning})
svc, err := env.store.GetService(ctx, 1)
require.NoError(t, err)
hook, err := env.jobs.Start(ctx, svc, jobs.StartRequest{Trigger: model.TriggerWebhook, Ref: slusk + ":main"})
require.NoError(t, err)
env.waitStatus(t, hook.ID, model.JobSucceeded)
svc, err = env.store.GetService(ctx, 1)
require.NoError(t, err)
assert.Equal(t, "sha256:e2e", svc.LastTagDigest, "the deployed digest of the tag is remembered")
pick, err := env.jobs.TriggerVersion(ctx, 1, 5, "alice")
require.NoError(t, err)
env.waitStatus(t, pick.ID, model.JobSucceeded)
jobsBefore := env.store.jobCount()
reg := &fixedRegistry{digests: map[string]string{slusk + ":main": "sha256:e2e"}}
p := poller.New(env.store, env.jobs, reg, env.m.broker, poller.Config{Workers: 1}, slog.New(slog.NewTextHandler(io.Discard, nil)))
p.Cycle(ctx)
assert.Equal(t, 1, reg.calls)
assert.Equal(t, jobsBefore, env.store.jobCount(), "the rollback stays")
}
+9 -7
View File
@@ -61,13 +61,15 @@ type Service struct {
AvailableSource string // SourceWebhook or SourcePoll; "" when AvailableRef is empty AvailableSource string // SourceWebhook or SourcePoll; "" when AvailableRef is empty
AvailableAt *time.Time AvailableAt *time.Time
LastPollError string // last registry error for the followed ref; "" when fine LastPollError string // last registry error for the followed ref; "" when fine
// LastAutoDigest is the registry digest the poller last started an // LastTagDigest is the newest digest of the followed tag that was deployed
// automatic update for; "" if never. The poller does not start another // or started: set when the poller starts an auto job and when any plain tag
// one for the same digest, so a rollback sticks until a new digest appears. // job succeeds (the digest the agent pinned); "" if never. The poller does
LastAutoDigest string // not start another job for the same digest, so a rollback sticks until the
FirstSeenAt time.Time // registry shows a different one.
UpdatedAt time.Time LastTagDigest string
RemovedAt *time.Time FirstSeenAt time.Time
UpdatedAt time.Time
RemovedAt *time.Time
} }
// Update modes of a service. // Update modes of a service.
+6 -6
View File
@@ -27,7 +27,7 @@ type Store interface {
ListServices(ctx context.Context) ([]model.Service, error) ListServices(ctx context.Context) ([]model.Service, error)
GetService(ctx context.Context, id int64) (model.Service, error) GetService(ctx context.Context, id int64) (model.Service, error)
SetAvailable(ctx context.Context, id int64, a model.Available) error SetAvailable(ctx context.Context, id int64, a model.Available) error
SetLastAutoDigest(ctx context.Context, id int64, digest string) error SetLastTagDigest(ctx context.Context, id int64, digest string) error
SetPollError(ctx context.Context, ids []int64, msg string) error SetPollError(ctx context.Context, ids []int64, msg string) error
} }
@@ -246,16 +246,16 @@ func autoWanted(svc model.Service, ref, digest string) bool {
return svc.RemovedAt == nil && return svc.RemovedAt == nil &&
svc.UpdateMode == model.ModeAuto && svc.UpdateMode == model.ModeAuto &&
svc.ActualDigest != digest && svc.ActualDigest != digest &&
svc.LastAutoDigest != digest && svc.LastTagDigest != digest &&
svc.DesiredRef != ref+"@"+digest svc.DesiredRef != ref+"@"+digest
} }
// startAuto starts the update job of an auto service to ref and remembers // startAuto starts the update job of an auto service to ref and remembers
// digest as services.last_auto_digest, so the same digest is not started // digest as services.last_tag_digest, so the same digest is not started
// again, for instance after the user rolled back. It re-reads the service // again, for instance after the user rolled back. It re-reads the service
// first, because a cycle can run for a long time and the user may have // first, because a cycle can run for a long time and the user may have
// switched the service to manual meanwhile. A service that already has a // switched the service to manual meanwhile. A service that already has a
// job or cannot be updated is not an error for the cycle, and last_auto_digest // job or cannot be updated is not an error for the cycle, and last_tag_digest
// is left alone so the next cycle tries again. // is left alone so the next cycle tries again.
func (p *Poller) startAuto(ctx context.Context, id int64, ref, digest string, stats *cycleStats) { func (p *Poller) startAuto(ctx context.Context, id int64, ref, digest string, stats *cycleStats) {
svc, err := p.store.GetService(ctx, id) svc, err := p.store.GetService(ctx, id)
@@ -271,8 +271,8 @@ func (p *Poller) startAuto(ctx context.Context, id int64, ref, digest string, st
switch { switch {
case err == nil: case err == nil:
stats.jobs++ stats.jobs++
if err := p.store.SetLastAutoDigest(ctx, id, digest); err != nil { if err := p.store.SetLastTagDigest(ctx, id, digest); err != nil {
p.log.Error("poll_set_last_auto_digest_failed", "service_id", id, "error", err) p.log.Error("poll_set_last_tag_digest_failed", "service_id", id, "error", err)
} }
case errors.Is(err, model.ErrJobActive), errors.As(err, &notUpdatable): case errors.Is(err, model.ErrJobActive), errors.As(err, &notUpdatable):
p.log.Debug("poll_job_not_started", "service_id", id, "ref", ref, "reason", err.Error()) p.log.Debug("poll_job_not_started", "service_id", id, "ref", ref, "reason", err.Error())
+5 -5
View File
@@ -64,15 +64,15 @@ func (f *fakeStore) GetService(_ context.Context, id int64) (model.Service, erro
return model.Service{}, model.ErrNotFound return model.Service{}, model.ErrNotFound
} }
// SetLastAutoDigest records the call and, like the database, changes the // SetLastTagDigest records the call and, like the database, changes the
// row the next cycle lists. // row the next cycle lists.
func (f *fakeStore) SetLastAutoDigest(_ context.Context, id int64, digest string) error { func (f *fakeStore) SetLastTagDigest(_ context.Context, id int64, digest string) error {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
f.lastAuto = append(f.lastAuto, lastAutoCall{id, digest}) f.lastAuto = append(f.lastAuto, lastAutoCall{id, digest})
for i := range f.services { for i := range f.services {
if f.services[i].ID == id { if f.services[i].ID == id {
f.services[i].LastAutoDigest = digest f.services[i].LastTagDigest = digest
} }
} }
return nil return nil
@@ -305,7 +305,7 @@ func TestSameSuggestionIsIdempotent(t *testing.T) {
suggested := svc(2, ref) suggested := svc(2, ref)
suggested.UpdateMode = model.ModeAuto suggested.UpdateMode = model.ModeAuto
suggested.AvailableRef, suggested.AvailableDigest = ref, "sha256:new" suggested.AvailableRef, suggested.AvailableDigest = ref, "sha256:new"
suggested.LastAutoDigest = "sha256:new" // its job was started in an earlier cycle suggested.LastTagDigest = "sha256:new" // its job was started in an earlier cycle
staleSuggestion := svc(3, ref) staleSuggestion := svc(3, ref)
staleSuggestion.AvailableRef, staleSuggestion.AvailableDigest = ref, "sha256:older" staleSuggestion.AvailableRef, staleSuggestion.AvailableDigest = ref, "sha256:older"
st := &fakeStore{services: []model.Service{running, suggested, staleSuggestion}} st := &fakeStore{services: []model.Service{running, suggested, staleSuggestion}}
@@ -349,7 +349,7 @@ func TestRollbackOfAutoServiceIsNotUndone(t *testing.T) {
// Updated to sha256:new automatically, then rolled back to sha256:old by hand. // Updated to sha256:new automatically, then rolled back to sha256:old by hand.
s := svc(1, ref+"@sha256:old") s := svc(1, ref+"@sha256:old")
s.UpdateMode = model.ModeAuto s.UpdateMode = model.ModeAuto
s.LastAutoDigest = "sha256:new" s.LastTagDigest = "sha256:new"
s.DesiredRef = ref + "@sha256:old" s.DesiredRef = ref + "@sha256:old"
st := &fakeStore{services: []model.Service{s}} st := &fakeStore{services: []model.Service{s}}
reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}} reg := &fakeRegistry{digests: map[string]string{ref: "sha256:new"}}
+8 -1
View File
@@ -229,13 +229,20 @@ func (s *Store) FinishJob(ctx context.Context, agentID, jobID int64, f model.Job
} }
if f.Status == model.JobSucceeded { if f.Status == model.JobSucceeded {
requestedRef, requestedDigest, _ := strings.Cut(requestedRef, "@") requestedRef, requestedDigest, _ := strings.Cut(requestedRef, "@")
// A plain tag job, whoever started it, deployed the tag at the digest
// the agent pinned; remember it. Pinned and tagless picks do not.
tagDigest := ""
if _, d, _ := strings.Cut(f.Ref, "@"); requestedDigest == "" && digestPattern.MatchString(d) {
tagDigest = d
}
if _, err := tx.Exec(ctx, ` if _, err := tx.Exec(ctx, `
UPDATE services SET desired_ref = $2, UPDATE services SET desired_ref = $2,
last_tag_digest = CASE WHEN $6 <> '' THEN $6 ELSE last_tag_digest END,
available_ref = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_ref END, available_ref = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_ref END,
available_digest = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_digest END, available_digest = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_digest END,
available_source = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_source END, available_source = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN '' ELSE available_source END,
available_at = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN NULL ELSE available_at END available_at = CASE WHEN available_ref = $4 AND available_at <= $3 AND ($5 = '' OR available_digest = $5) THEN NULL ELSE available_at END
WHERE id = $1`, serviceID, f.Ref, createdAt, requestedRef, requestedDigest); err != nil { WHERE id = $1`, serviceID, f.Ref, createdAt, requestedRef, requestedDigest, tagDigest); err != nil {
return false, fmt.Errorf("finish job: desired ref: %w", err) return false, fmt.Errorf("finish job: desired ref: %w", err)
} }
} }
@@ -3,7 +3,7 @@ ALTER TABLE services ADD CONSTRAINT services_available_source_check
CHECK (available_source IN ('', 'webhook', 'poll')); CHECK (available_source IN ('', 'webhook', 'poll'));
ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT ''; ALTER TABLE services ADD COLUMN available_digest text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT ''; ALTER TABLE services ADD COLUMN last_poll_error text NOT NULL DEFAULT '';
ALTER TABLE services ADD COLUMN last_auto_digest text NOT NULL DEFAULT ''; ALTER TABLE services ADD COLUMN last_tag_digest text NOT NULL DEFAULT '';
-- Every version the master has seen. '' means "no tag" / "digest unknown"; -- Every version the master has seen. '' means "no tag" / "digest unknown";
-- never NULL, so the unique constraint deduplicates. -- never NULL, so the unique constraint deduplicates.
+6 -6
View File
@@ -178,7 +178,7 @@ const serviceColumns = `s.id, s.kind, s.location_key, s.agent_id, COALESCE(a.nam
s.compose_project, s.compose_service, s.working_dir, s.config_files, s.built, s.swarm_cluster_id, s.compose_project, s.compose_service, s.working_dir, s.config_files, s.built, s.swarm_cluster_id,
s.swarm_service, s.container_name, s.image_repo, s.desired_ref, s.actual_ref, s.actual_digest, s.swarm_service, s.container_name, s.image_repo, s.desired_ref, s.actual_ref, s.actual_digest,
s.status, s.self, s.update_mode, s.available_ref, s.available_digest, s.available_source, s.available_at, s.status, s.self, s.update_mode, s.available_ref, s.available_digest, s.available_source, s.available_at,
s.last_poll_error, s.last_auto_digest, s.first_seen_at, s.updated_at, s.removed_at` s.last_poll_error, s.last_tag_digest, s.first_seen_at, s.updated_at, s.removed_at`
func scanService(row pgx.Row) (model.Service, error) { func scanService(row pgx.Row) (model.Service, error) {
var m model.Service var m model.Service
@@ -187,7 +187,7 @@ func scanService(row pgx.Row) (model.Service, error) {
&m.ComposeProject, &m.ComposeService, &m.WorkingDir, &m.ConfigFiles, &m.Built, &m.SwarmClusterID, &m.ComposeProject, &m.ComposeService, &m.WorkingDir, &m.ConfigFiles, &m.Built, &m.SwarmClusterID,
&m.SwarmService, &m.ContainerName, &m.ImageRepo, &m.DesiredRef, &m.ActualRef, &m.ActualDigest, &m.SwarmService, &m.ContainerName, &m.ImageRepo, &m.DesiredRef, &m.ActualRef, &m.ActualDigest,
&m.Status, &m.Self, &m.UpdateMode, &m.AvailableRef, &m.AvailableDigest, &m.AvailableSource, &m.AvailableAt, &m.Status, &m.Self, &m.UpdateMode, &m.AvailableRef, &m.AvailableDigest, &m.AvailableSource, &m.AvailableAt,
&m.LastPollError, &m.LastAutoDigest, &m.FirstSeenAt, &m.UpdatedAt, &m.RemovedAt) &m.LastPollError, &m.LastTagDigest, &m.FirstSeenAt, &m.UpdatedAt, &m.RemovedAt)
m.Kind = protocol.Kind(kind) m.Kind = protocol.Kind(kind)
return m, err return m, err
} }
@@ -261,11 +261,11 @@ func (s *Store) SetPollError(ctx context.Context, ids []int64, msg string) error
return nil return nil
} }
// SetLastAutoDigest records the digest an automatic update was last started // SetLastTagDigest records the digest of the followed tag that the poller
// for; see model.Service.LastAutoDigest. Inventory writes never change it. // started an update for; see model.Service.LastTagDigest. Inventory writes never change it.
func (s *Store) SetLastAutoDigest(ctx context.Context, id int64, digest string) error { func (s *Store) SetLastTagDigest(ctx context.Context, id int64, digest string) error {
if _, err := s.pool.Exec(ctx, if _, err := s.pool.Exec(ctx,
`UPDATE services SET last_auto_digest = $2 WHERE id = $1`, id, digest); err != nil { `UPDATE services SET last_tag_digest = $2 WHERE id = $1`, id, digest); err != nil {
return fmt.Errorf("set last auto digest: %w", err) return fmt.Errorf("set last auto digest: %w", err)
} }
return nil return nil
@@ -195,22 +195,22 @@ func TestSetPollError(t *testing.T) {
assert.Empty(t, got.LastPollError) assert.Empty(t, got.LastPollError)
} }
func TestSetLastAutoDigest(t *testing.T) { func TestSetLastTagDigest(t *testing.T) {
s := openTestStore(t) s := openTestStore(t)
ctx := context.Background() ctx := context.Background()
agent, svc := jobFixture(t, s, "a") agent, svc := jobFixture(t, s, "a")
got, err := s.GetService(ctx, svc) got, err := s.GetService(ctx, svc)
require.NoError(t, err) require.NoError(t, err)
assert.Empty(t, got.LastAutoDigest) assert.Empty(t, got.LastTagDigest)
require.NoError(t, s.SetLastAutoDigest(ctx, svc, "sha256:n1")) require.NoError(t, s.SetLastTagDigest(ctx, svc, "sha256:n1"))
require.NoError(t, s.UpsertService(ctx, agent, "", slusk, t0.Add(time.Hour))) require.NoError(t, s.UpsertService(ctx, agent, "", slusk, t0.Add(time.Hour)))
got, err = s.GetService(ctx, svc) got, err = s.GetService(ctx, svc)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "sha256:n1", got.LastAutoDigest, "inventory writes do not touch it") assert.Equal(t, "sha256:n1", got.LastTagDigest, "inventory writes do not touch it")
list, err := s.ListServices(ctx) list, err := s.ListServices(ctx)
require.NoError(t, err) require.NoError(t, err)
require.Len(t, list, 1) require.Len(t, list, 1)
assert.Equal(t, "sha256:n1", list[0].LastAutoDigest) assert.Equal(t, "sha256:n1", list[0].LastTagDigest)
} }
@@ -4,6 +4,7 @@ package store
import ( import (
"context" "context"
"strings"
"testing" "testing"
"time" "time"
@@ -160,3 +161,48 @@ func TestFinishJob_WithoutAvailableLeavesItEmpty(t *testing.T) {
assert.Nil(t, got.AvailableAt) assert.Nil(t, got.AvailableAt)
assert.Equal(t, "r:1@sha256:d", got.DesiredRef) assert.Equal(t, "r:1@sha256:d", got.DesiredRef)
} }
func TestFinishJob_RemembersTheDeployedDigestOfATagJob(t *testing.T) {
good := "sha256:" + strings.Repeat("a", 64)
old := "sha256:" + strings.Repeat("b", 64)
ok := func(ref string) model.JobFinish {
return model.JobFinish{Status: model.JobSucceeded, Ref: ref, StartedAt: t0, FinishedAt: t0}
}
tests := []struct {
name string
trigger model.JobTrigger
toRef string
finish model.JobFinish
want string
}{
{"webhook tag job", model.TriggerWebhook, "r:2", ok("r:2@" + good), good},
{"manual tag job", model.TriggerManual, "r:2", ok("r:2@" + good), good},
{"poll tag job", model.TriggerPoll, "r:2", ok("r:2@" + good), good},
{"pinned pick leaves it", model.TriggerManual, "r:2@" + old, ok("r:2@" + old), "sha256:before"},
{"tagless pick leaves it", model.TriggerManual, "r@" + old, ok("r@" + old), "sha256:before"},
{"failed job leaves it", model.TriggerWebhook, "r:2",
model.JobFinish{Status: model.JobFailed, Error: "boom", Ref: "r:2@" + good, StartedAt: t0, FinishedAt: t0}, "sha256:before"},
{"malformed digest leaves it", model.TriggerWebhook, "r:2", ok("r:2@sha256:d"), "sha256:before"},
{"no pinned ref leaves it", model.TriggerWebhook, "r:2", ok(""), "sha256:before"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := openTestStore(t)
ctx := context.Background()
agent, svc := jobFixture(t, s, "a")
require.NoError(t, s.SetLastTagDigest(ctx, svc, "sha256:before"))
nj := newJob(svc, agent)
nj.ToRef, nj.Trigger = tt.toRef, tt.trigger
j, err := s.CreateJob(ctx, nj)
require.NoError(t, err)
done, err := s.FinishJob(ctx, agent, j.ID, tt.finish)
require.NoError(t, err)
require.True(t, done)
got, err := s.GetService(ctx, svc)
require.NoError(t, err)
assert.Equal(t, tt.want, got.LastTagDigest)
})
}
}