diff --git a/docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md b/docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md new file mode 100644 index 0000000..3bed171 --- /dev/null +++ b/docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md @@ -0,0 +1,5620 @@ +# Master/Agent Phase 1 — Inventory Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A master and per-machine agents that show every service on every machine live in a web UI, behind Authentik login. Nothing can be updated yet. + +**Architecture:** Each agent dials the master over WebSocket (TLS with the master's key pinned by fingerprint, per-agent bearer token), sends a hello and a full inventory snapshot, then sends diffs whenever Docker events settle. The master stores inventory in Postgres, tracks which agents are online in memory, and pushes "something changed" signals to browsers over SSE; htmx refetches the affected fragment. + +**Tech Stack:** Go 1.26.3, `github.com/coder/websocket`, `github.com/jackc/pgx/v5`, `github.com/coreos/go-oidc/v3` + `golang.org/x/oauth2`, `html/template` + htmx 2.0.4 + htmx-ext-sse 2.2.2 (vendored), Docker SDK v28 (already in `go.mod`), testify. + +**Spec:** `docs/superpowers/specs/2026-10-12-master-agent-design.md`. This plan implements spec phase 1 only. Phases 2–5 get their own plans once this one has landed. + +## Global Constraints + +- Go `1.26.3` in `go.mod`; Dockerfile stays on `golang:1.26-alpine`. Never bump one without the other. +- `go test ./...` must pass with no Docker daemon and no Postgres. Postgres tests carry `//go:build integration` and need `DATABASE_URL`. +- Do not modify `cmd/server`, `internal/api`, `internal/config`, `internal/discovery`, `internal/updater`, `internal/selfupdate`. Today's updater keeps running in production until phase 3. +- Interfaces are declared in the consuming package. Tests use hand-written fakes and `stretchr/testify`, table-driven where there is more than one case. +- Every exported identifier gets a GoDoc comment (`// Name does X.`). +- Conventional Commits (`feat:`, `fix:`, `docs:`, `build:`, `test:`). +- Agent tokens have the prefix `pua_`. Only `agenttoken.Hash(token)` is stored. Tokens never appear in logs. +- `MASTER_KEY` format: `sha256:` followed by 64 lowercase hex characters (SHA-256 of the PKIX DER public key). +- Every newly discovered service is stored with `update_mode = 'manual'`. +- UI text is English. +- WebSocket read limit is `1 << 20` bytes on both sides (the library default is 32 KiB). + +## Deliberate deviations from the spec in this phase + +- The agent detects changes by re-running the full inventory after Docker events settle and diffing it against the last one sent, rather than re-inspecting individual items. Same messages on the wire, much less code. +- The envelope is `{type, payload}`. The `id` field arrives in phase 2 together with jobs. +- Only the tables phase 1 needs are created. `desired_ref`, `available_*`, `last_poll_error`, `image_versions` and `update_jobs` arrive in later phases' migrations. + +## Review Focus + +- A snapshot from a busy machine (40+ services) exceeds the WebSocket library's default 32 KiB read limit; it must arrive intact, not drop the connection in a reconnect loop. Test: Task 11 (500 services). +- Revoking an agent in the UI while it is connected must cut the live connection at once, not just block the next login. Tests: Task 11 (`Disconnect`) and Task 13 (revoke calls `Disconnect`). +- A service removed while its agent was offline must disappear when the agent reconnects; if it comes back, it keeps its ID and first-seen time. Test: Task 9. +- Two manager agents in the same Swarm must not create duplicate Swarm services, and one agent's snapshot must never mark another agent's Compose services as removed. Test: Task 9. +- An htmx fragment request with an expired session must get `401` + `HX-Redirect`, not the login page's HTML injected into the table. Test: Task 12. + +## File Structure + +``` +internal/imageref/ Parse "repo:tag@digest" (Task 1) +internal/protocol/ Wire messages shared by master and agent (Task 2) +internal/pin/ Master key pair, self-signed cert, fingerprint, pinned TLS client config (Task 3) +internal/agenttoken/ Generate and hash per-agent tokens (Task 3) +internal/agent/inventory/ Docker → []protocol.Service, diff, event watcher (Tasks 4–5) +internal/agent/conn/ Agent's connection loop to the master (Task 6) +internal/agent/config/ Agent env config (Task 7) +cmd/agent/ Agent binary (Task 7) +internal/master/model/ Domain types and errors shared by master packages (Task 8) +internal/master/store/ Postgres storage + embedded migrations (Tasks 8–9) +internal/master/events/ In-process pub/sub for "something changed" (Task 10) +internal/master/hub/ WebSocket endpoint for agents (Task 11) +internal/master/webauth/ OIDC login, sessions, CSRF (Task 12) +internal/master/web/ htmx UI: templates, static assets, handlers, SSE (Task 13) +internal/master/config/ Master env config (Task 14) +cmd/master/ Master binary (Task 14) +internal/e2e/ Agent ↔ master over real TLS, in process (Task 15) +deploy/master.stack.yml Example Swarm stack for the master (Task 16) +``` + +--- + +### Task 1: Image reference parsing + +**Files:** +- Create: `internal/imageref/imageref.go` +- Test: `internal/imageref/imageref_test.go` + +**Interfaces:** +- Consumes: nothing. +- Produces: `type Ref struct { Repo, Tag, Digest string }`, `func Parse(ref string) Ref`, `func (r Ref) String() string`. + +- [ ] **Step 1: Write the failing test** + +```go +package imageref + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestParse(t *testing.T) { + cases := []struct { + in string + want Ref + }{ + {"postgres", Ref{Repo: "postgres"}}, + {"postgres:17.10", Ref{Repo: "postgres", Tag: "17.10"}}, + {"gitea/gitea:1.27.3@sha256:87a6", Ref{Repo: "gitea/gitea", Tag: "1.27.3", Digest: "sha256:87a6"}}, + {"lscr.io/linuxserver/jellyfin@sha256:0f42", Ref{Repo: "lscr.io/linuxserver/jellyfin", Digest: "sha256:0f42"}}, + {"localhost:5000/foo", Ref{Repo: "localhost:5000/foo"}}, + {"localhost:5000/foo:v1", Ref{Repo: "localhost:5000/foo", Tag: "v1"}}, + {"localhost:5000/foo:v1@sha256:ab", Ref{Repo: "localhost:5000/foo", Tag: "v1", Digest: "sha256:ab"}}, + } + for _, c := range cases { + t.Run(c.in, func(t *testing.T) { + got := Parse(c.in) + assert.Equal(t, c.want, got) + assert.Equal(t, c.in, got.String(), "String must round-trip") + }) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/imageref/` +Expected: FAIL — `undefined: Ref` / `undefined: Parse`. + +- [ ] **Step 3: Write minimal implementation** + +```go +// Package imageref splits container image references into repository, tag +// and digest. +package imageref + +import "strings" + +// Ref is an image reference split into its parts. Tag and Digest are empty +// when the reference has none. +type Ref struct { + Repo string + Tag string + Digest string +} + +// Parse splits ref into repository, tag and digest. A colon before the last +// "/" belongs to a registry host and port ("localhost:5000/foo"), not a tag. +func Parse(ref string) Ref { + var r Ref + if at := strings.Index(ref, "@"); at >= 0 { + r.Digest = ref[at+1:] + ref = ref[:at] + } + slash := strings.LastIndex(ref, "/") + last := ref[slash+1:] + if colon := strings.Index(last, ":"); colon >= 0 { + r.Tag = last[colon+1:] + ref = ref[:slash+1+colon] + } + r.Repo = ref + return r +} + +// String reassembles the reference as repo[:tag][@digest]. +func (r Ref) String() string { + s := r.Repo + if r.Tag != "" { + s += ":" + r.Tag + } + if r.Digest != "" { + s += "@" + r.Digest + } + return s +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `go test ./internal/imageref/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/imageref +git commit -m "feat(imageref): parse image references into repo, tag and digest" +``` + +--- + +### Task 2: Wire protocol + +**Files:** +- Create: `internal/protocol/protocol.go` +- Test: `internal/protocol/protocol_test.go` + +**Interfaces:** +- Consumes: nothing. +- Produces: + - Constants `TypeHello`, `TypeSnapshot`, `TypeUpsert`, `TypeRemove` (strings). + - `type Envelope struct { Type string; Payload json.RawMessage }` + - `type Hello struct { AgentVersion, Hostname string; SwarmManager bool; SwarmClusterID string }` + - `type Kind string` with `KindCompose`, `KindSwarm`, `KindContainer`. + - `type Service struct { Kind Kind; ComposeProject, ComposeService, WorkingDir, SwarmService, ContainerName, Image, Digest, Status string; Self bool }` with `func (s Service) Key() string`. + - `type Snapshot struct { Services []Service }`, `type Upsert struct { Service Service }`, `type Remove struct { Key string }`. + - `func Encode(typ string, payload any) (Envelope, error)`, `func Decode[T any](env Envelope) (T, error)`. + +- [ ] **Step 1: Write the failing test** + +```go +package protocol + +import ( + "encoding/json" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestServiceKey(t *testing.T) { + cases := []struct { + name string + svc Service + want string + }{ + {"compose", Service{Kind: KindCompose, ComposeProject: "slusk", ComposeService: "web"}, "compose|slusk|web"}, + {"swarm", Service{Kind: KindSwarm, SwarmService: "gitea_gitea"}, "swarm|gitea_gitea"}, + {"container", Service{Kind: KindContainer, ContainerName: "adhoc"}, "container|adhoc"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + assert.Equal(t, c.want, c.svc.Key()) + }) + } +} + +func TestEncodeDecodeRoundTrip(t *testing.T) { + in := Snapshot{Services: []Service{ + {Kind: KindCompose, ComposeProject: "slusk", ComposeService: "web", Image: "slusk:1.4.2", Status: "running"}, + }} + env, err := Encode(TypeSnapshot, in) + require.NoError(t, err) + + raw, err := json.Marshal(env) + require.NoError(t, err) + var back Envelope + require.NoError(t, json.Unmarshal(raw, &back)) + + assert.Equal(t, TypeSnapshot, back.Type) + out, err := Decode[Snapshot](back) + require.NoError(t, err) + assert.Equal(t, in, out) +} + +func TestDecodeRejectsMalformedPayload(t *testing.T) { + _, err := Decode[Hello](Envelope{Type: TypeHello, Payload: json.RawMessage(`{"hostname": 5}`)}) + assert.ErrorContains(t, err, "decode hello") +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/protocol/` +Expected: FAIL — undefined identifiers. + +- [ ] **Step 3: Write minimal implementation** + +```go +// Package protocol defines the messages agents and the master exchange over +// their WebSocket connection. Every message is an Envelope whose Payload is +// one of the payload types below, selected by Type. +package protocol + +import ( + "encoding/json" + "fmt" +) + +// Message types sent from agent to master. +const ( + TypeHello = "hello" + TypeSnapshot = "snapshot" + TypeUpsert = "upsert" + TypeRemove = "remove" +) + +// Envelope wraps every message on the wire. +type Envelope struct { + Type string `json:"type"` + Payload json.RawMessage `json:"payload,omitempty"` +} + +// Hello is the first message an agent sends on every connection. +type Hello struct { + AgentVersion string `json:"agent_version"` + Hostname string `json:"hostname"` + SwarmManager bool `json:"swarm_manager"` + SwarmClusterID string `json:"swarm_cluster_id,omitempty"` +} + +// Kind says how a service is deployed. +type Kind string + +// Service kinds. +const ( + KindCompose Kind = "compose" + KindSwarm Kind = "swarm" + KindContainer Kind = "container" +) + +// Service is one deployable unit as an agent sees it. Which location fields +// are set depends on Kind. +type Service struct { + Kind Kind `json:"kind"` + ComposeProject string `json:"compose_project,omitempty"` + ComposeService string `json:"compose_service,omitempty"` + WorkingDir string `json:"working_dir,omitempty"` + SwarmService string `json:"swarm_service,omitempty"` + ContainerName string `json:"container_name,omitempty"` + Image string `json:"image"` + Digest string `json:"digest,omitempty"` + Status string `json:"status"` + Self bool `json:"self,omitempty"` +} + +// Key identifies the service within one agent's report. It is stable across +// restarts and is what Remove refers to. +func (s Service) Key() string { + switch s.Kind { + case KindCompose: + return "compose|" + s.ComposeProject + "|" + s.ComposeService + case KindSwarm: + return "swarm|" + s.SwarmService + default: + return "container|" + s.ContainerName + } +} + +// Snapshot is the agent's complete inventory. Sent after every Hello. +type Snapshot struct { + Services []Service `json:"services"` +} + +// Upsert reports a service that appeared or changed. +type Upsert struct { + Service Service `json:"service"` +} + +// Remove reports that the service with Key is gone. +type Remove struct { + Key string `json:"key"` +} + +// Encode wraps payload in an Envelope of the given type. +func Encode(typ string, payload any) (Envelope, error) { + raw, err := json.Marshal(payload) + if err != nil { + return Envelope{}, fmt.Errorf("encode %s: %w", typ, err) + } + return Envelope{Type: typ, Payload: raw}, nil +} + +// Decode unpacks env's payload into T. +func Decode[T any](env Envelope) (T, error) { + var v T + if err := json.Unmarshal(env.Payload, &v); err != nil { + return v, fmt.Errorf("decode %s: %w", env.Type, err) + } + return v, nil +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `go test ./internal/protocol/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/protocol +git commit -m "feat(protocol): define agent-to-master wire messages" +``` + +--- + +### Task 3: Credentials — pinned master key and agent tokens + +**Files:** +- Create: `internal/pin/pin.go`, `internal/agenttoken/agenttoken.go` +- Test: `internal/pin/pin_test.go`, `internal/agenttoken/agenttoken_test.go` + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `pin.GenerateKey() (ed25519.PrivateKey, error)` + - `pin.Certificate(key ed25519.PrivateKey) (tls.Certificate, error)` + - `pin.Fingerprint(pub crypto.PublicKey) (string, error)` → `"sha256:<64 hex>"` + - `pin.ClientConfig(fingerprint string) *tls.Config` + - `agenttoken.Generate() (string, error)` → `"pua_<43 base64url chars>"` + - `agenttoken.Hash(token string) string` → 64 hex chars + +- [ ] **Step 1: Write the failing tests** + +`internal/pin/pin_test.go`: + +```go +package pin + +import ( + "crypto/tls" + "net/http" + "net/http/httptest" + "regexp" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func newPinnedServer(t *testing.T) (*httptest.Server, string) { + t.Helper() + key, err := GenerateKey() + require.NoError(t, err) + cert, err := Certificate(key) + require.NoError(t, err) + fp, err := Fingerprint(key.Public()) + require.NoError(t, err) + + srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte("ok")) + })) + srv.TLS = &tls.Config{Certificates: []tls.Certificate{cert}} + srv.StartTLS() + t.Cleanup(srv.Close) + return srv, fp +} + +func clientFor(fp string) *http.Client { + return &http.Client{Transport: &http.Transport{TLSClientConfig: ClientConfig(fp)}} +} + +func TestFingerprintFormat(t *testing.T) { + key, err := GenerateKey() + require.NoError(t, err) + fp, err := Fingerprint(key.Public()) + require.NoError(t, err) + assert.Regexp(t, regexp.MustCompile(`^sha256:[0-9a-f]{64}$`), fp) +} + +func TestClientConfigAcceptsPinnedKey(t *testing.T) { + srv, fp := newPinnedServer(t) + resp, err := clientFor(fp).Get(srv.URL) + require.NoError(t, err) + _ = resp.Body.Close() + assert.Equal(t, http.StatusOK, resp.StatusCode) +} + +func TestClientConfigRejectsOtherKey(t *testing.T) { + srv, _ := newPinnedServer(t) + other, err := GenerateKey() + require.NoError(t, err) + otherFP, err := Fingerprint(other.Public()) + require.NoError(t, err) + + _, err = clientFor(otherFP).Get(srv.URL) + assert.ErrorContains(t, err, "fingerprint mismatch") +} +``` + +`internal/agenttoken/agenttoken_test.go`: + +```go +package agenttoken + +import ( + "regexp" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestGenerate(t *testing.T) { + a, err := Generate() + require.NoError(t, err) + b, err := Generate() + require.NoError(t, err) + + assert.Regexp(t, regexp.MustCompile(`^pua_[A-Za-z0-9_-]{43}$`), a) + assert.NotEqual(t, a, b) +} + +func TestHash(t *testing.T) { + assert.Equal(t, Hash("pua_x"), Hash("pua_x")) + assert.NotEqual(t, Hash("pua_x"), Hash("pua_y")) + assert.Regexp(t, regexp.MustCompile(`^[0-9a-f]{64}$`), Hash("pua_x")) +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `go test ./internal/pin/ ./internal/agenttoken/` +Expected: FAIL — undefined identifiers. + +- [ ] **Step 3: Write minimal implementation** + +`internal/pin/pin.go`: + +```go +// Package pin lets agents trust the master without a certificate authority. +// The master serves a self-signed certificate for its own key pair; agents +// are configured with the fingerprint of that public key and accept the +// connection only if it matches. The TLS handshake proves the master holds +// the private key, the same way SSH host keys work. +package pin + +import ( + "crypto" + "crypto/ed25519" + "crypto/rand" + "crypto/sha256" + "crypto/subtle" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "encoding/hex" + "errors" + "fmt" + "math/big" + "time" +) + +// GenerateKey creates a new master key pair. +func GenerateKey() (ed25519.PrivateKey, error) { + _, priv, err := ed25519.GenerateKey(rand.Reader) + if err != nil { + return nil, fmt.Errorf("generate key: %w", err) + } + return priv, nil +} + +// Certificate returns a self-signed TLS certificate for key. Its validity is +// effectively unlimited because trust comes from the pinned fingerprint, not +// from the certificate's dates or issuer. +func Certificate(key ed25519.PrivateKey) (tls.Certificate, error) { + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "package-updater-master"}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().AddDate(100, 0, 0), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, key.Public(), key) + if err != nil { + return tls.Certificate{}, fmt.Errorf("create certificate: %w", err) + } + return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}, nil +} + +// Fingerprint returns "sha256:" + hex SHA-256 of pub's PKIX DER encoding. +// This is the MASTER_KEY value agents are configured with. +func Fingerprint(pub crypto.PublicKey) (string, error) { + der, err := x509.MarshalPKIXPublicKey(pub) + if err != nil { + return "", fmt.Errorf("marshal public key: %w", err) + } + sum := sha256.Sum256(der) + return "sha256:" + hex.EncodeToString(sum[:]), nil +} + +// ClientConfig returns a TLS config that accepts only a server whose +// certificate carries the public key with the given fingerprint. +func ClientConfig(fingerprint string) *tls.Config { + return &tls.Config{ + MinVersion: tls.VersionTLS13, + // Chain and hostname verification are replaced by the fingerprint + // check below: the master's certificate is self-signed by design. + // The handshake still verifies the server's signature with the + // certificate's key, so presenting a copied certificate fails. + InsecureSkipVerify: true, //nolint:gosec // see comment above + VerifyPeerCertificate: func(raw [][]byte, _ [][]*x509.Certificate) error { + if len(raw) == 0 { + return errors.New("pin: server sent no certificate") + } + cert, err := x509.ParseCertificate(raw[0]) + if err != nil { + return fmt.Errorf("pin: parse certificate: %w", err) + } + got, err := Fingerprint(cert.PublicKey) + if err != nil { + return err + } + if subtle.ConstantTimeCompare([]byte(got), []byte(fingerprint)) != 1 { + return fmt.Errorf("pin: fingerprint mismatch: server presented %s", got) + } + return nil + }, + } +} +``` + +`internal/agenttoken/agenttoken.go`: + +```go +// Package agenttoken creates and hashes the per-agent tokens agents present +// to the master. +package agenttoken + +import ( + "crypto/rand" + "crypto/sha256" + "encoding/base64" + "encoding/hex" + "fmt" +) + +const prefix = "pua_" + +// Generate returns a new random agent token. Show it to the operator once; +// store only Hash(token). +func Generate() (string, error) { + var b [32]byte + if _, err := rand.Read(b[:]); err != nil { + return "", fmt.Errorf("generate agent token: %w", err) + } + return prefix + base64.RawURLEncoding.EncodeToString(b[:]), nil +} + +// Hash returns the value stored for token. A plain SHA-256 is sufficient +// because tokens carry 256 bits of randomness; salting protects low-entropy +// passwords, which these are not. +func Hash(token string) string { + sum := sha256.Sum256([]byte(token)) + return hex.EncodeToString(sum[:]) +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `go test ./internal/pin/ ./internal/agenttoken/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/pin internal/agenttoken +git commit -m "feat: pinned master key TLS and per-agent tokens" +``` + +--- + +### Task 4: Agent inventory collector + +**Files:** +- Create: `internal/agent/inventory/collector.go` +- Test: `internal/agent/inventory/collector_test.go` + +**Interfaces:** +- Consumes: `imageref.Parse` (Task 1); `protocol.Service`, `protocol.Hello`, kinds (Task 2). +- Produces: + - `type DockerClient interface { ContainerList; ImageList; ServiceList; Info }` (signatures below; `*client.Client` satisfies it). + - `func New(cli DockerClient, selfID string) *Collector` + - `func (c *Collector) Hello(ctx context.Context) (protocol.Hello, error)` — hostname and Swarm facts; `AgentVersion` left empty for the caller. + - `func (c *Collector) Snapshot(ctx context.Context) ([]protocol.Service, error)` — sorted by `Key()`. + +Rules: containers labelled `com.docker.swarm.service.id` are Swarm tasks and skipped (the service is reported instead). Containers with Compose project and service labels become one `compose` service per project+service; replicas merge, `running` wins. Anything else is a `container` named after its first name. Swarm services are listed only when `Info().Swarm.ControlAvailable` is true. Status: containers `running`/`stopped`; Swarm services `running`, `degraded` (fewer running than desired), `stopped` (desired 0) or `unknown` (no status). `Self` is true for containers whose ID starts with `selfID`. + +- [ ] **Step 1: Write the failing test** + +```go +package inventory + +import ( + "context" + "errors" + "testing" + + "github.com/docker/docker/api/types/container" + "github.com/docker/docker/api/types/image" + "github.com/docker/docker/api/types/swarm" + "github.com/docker/docker/api/types/system" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/protocol" +) + +type fakeDocker struct { + containers []container.Summary + images []image.Summary + services []swarm.Service + info system.Info + containerErr error + serviceListCalls int +} + +func (f *fakeDocker) ContainerList(context.Context, container.ListOptions) ([]container.Summary, error) { + return f.containers, f.containerErr +} + +func (f *fakeDocker) ImageList(context.Context, image.ListOptions) ([]image.Summary, error) { + return f.images, nil +} + +func (f *fakeDocker) ServiceList(context.Context, swarm.ServiceListOptions) ([]swarm.Service, error) { + f.serviceListCalls++ + return f.services, nil +} + +func (f *fakeDocker) Info(context.Context) (system.Info, error) { return f.info, nil } + +func composeContainer(id, project, service, img, imageID, state string) container.Summary { + return container.Summary{ + ID: id, Image: img, ImageID: imageID, State: state, Names: []string{"/" + project + "-" + service + "-1"}, + Labels: map[string]string{ + labelProject: project, + labelService: service, + labelWorkingDir: "/home/shcizo/self-hosted/" + project, + }, + } +} + +func managerInfo() system.Info { + var info system.Info + info.Name = "swarm-mgr" + info.Swarm.ControlAvailable = true + info.Swarm.Cluster = &swarm.ClusterInfo{ID: "cluster-1"} + return info +} + +func TestSnapshot_ComposeContainerWithRepoDigest(t *testing.T) { + cli := &fakeDocker{ + containers: []container.Summary{composeContainer("abc123", "slusk", "web", "gitea.shcizo.se/shcizo/slusk:1.4.2", "sha256:img1", "running")}, + images: []image.Summary{{ID: "sha256:img1", RepoDigests: []string{"gitea.shcizo.se/shcizo/slusk@sha256:d1"}}}, + } + got, err := New(cli, "").Snapshot(context.Background()) + require.NoError(t, err) + assert.Equal(t, []protocol.Service{{ + Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", + WorkingDir: "/home/shcizo/self-hosted/slusk", + Image: "gitea.shcizo.se/shcizo/slusk:1.4.2", Digest: "sha256:d1", Status: "running", + }}, got) +} + +func TestSnapshot_ReplicasMergeAndRunningWins(t *testing.T) { + cli := &fakeDocker{containers: []container.Summary{ + composeContainer("a1", "slusk", "worker", "slusk:1", "sha256:i", "exited"), + composeContainer("a2", "slusk", "worker", "slusk:1", "sha256:i", "running"), + }} + got, err := New(cli, "").Snapshot(context.Background()) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, "running", got[0].Status) +} + +func TestSnapshot_PlainContainerAndSwarmTaskSkipped(t *testing.T) { + cli := &fakeDocker{containers: []container.Summary{ + {ID: "p1", Names: []string{"/adhoc"}, Image: "redis:alpine", State: "exited"}, + {ID: "t1", Names: []string{"/gitea_gitea.1.x"}, Image: "gitea/gitea:1.27.3", State: "running", + Labels: map[string]string{labelSwarmTask: "svc1"}}, + }} + got, err := New(cli, "").Snapshot(context.Background()) + require.NoError(t, err) + assert.Equal(t, []protocol.Service{{ + Kind: protocol.KindContainer, ContainerName: "adhoc", Image: "redis:alpine", Status: "stopped", + }}, got) +} + +func TestSnapshot_SelfFlag(t *testing.T) { + cli := &fakeDocker{containers: []container.Summary{ + composeContainer("0123456789abcdef", "updater-agent", "agent", "pu:1", "sha256:i", "running"), + }} + got, err := New(cli, "0123456789ab").Snapshot(context.Background()) + require.NoError(t, err) + require.Len(t, got, 1) + assert.True(t, got[0].Self) +} + +func TestSnapshot_SwarmServicesOnlyOnManager(t *testing.T) { + svc := swarm.Service{ + Spec: swarm.ServiceSpec{ + Annotations: swarm.Annotations{Name: "gitea_gitea"}, + TaskTemplate: swarm.TaskSpec{ContainerSpec: &swarm.ContainerSpec{Image: "gitea/gitea:1.27.3@sha256:87a6"}}, + }, + ServiceStatus: &swarm.ServiceStatus{RunningTasks: 1, DesiredTasks: 1}, + } + + worker := &fakeDocker{services: []swarm.Service{svc}} + got, err := New(worker, "").Snapshot(context.Background()) + require.NoError(t, err) + assert.Empty(t, got) + assert.Equal(t, 0, worker.serviceListCalls, "workers must not call the manager-only API") + + manager := &fakeDocker{services: []swarm.Service{svc}, info: managerInfo()} + got, err = New(manager, "").Snapshot(context.Background()) + require.NoError(t, err) + assert.Equal(t, []protocol.Service{{ + Kind: protocol.KindSwarm, SwarmService: "gitea_gitea", + Image: "gitea/gitea:1.27.3@sha256:87a6", Digest: "sha256:87a6", Status: "running", + }}, got) +} + +func TestSwarmStatus(t *testing.T) { + cases := []struct { + name string + in *swarm.ServiceStatus + want string + }{ + {"no status", nil, "unknown"}, + {"scaled to zero", &swarm.ServiceStatus{DesiredTasks: 0}, "stopped"}, + {"all running", &swarm.ServiceStatus{RunningTasks: 2, DesiredTasks: 2}, "running"}, + {"some running", &swarm.ServiceStatus{RunningTasks: 1, DesiredTasks: 2}, "degraded"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { assert.Equal(t, c.want, swarmStatus(c.in)) }) + } +} + +func TestSnapshot_SortedByKey(t *testing.T) { + cli := &fakeDocker{containers: []container.Summary{ + {ID: "2", Names: []string{"/zeta"}, Image: "z", State: "running"}, + {ID: "1", Names: []string{"/alpha"}, Image: "a", State: "running"}, + }} + got, err := New(cli, "").Snapshot(context.Background()) + require.NoError(t, err) + require.Len(t, got, 2) + assert.Equal(t, "alpha", got[0].ContainerName) +} + +func TestSnapshot_PropagatesDockerErrors(t *testing.T) { + cli := &fakeDocker{containerErr: errors.New("daemon down")} + _, err := New(cli, "").Snapshot(context.Background()) + assert.ErrorContains(t, err, "daemon down") +} + +func TestHello(t *testing.T) { + got, err := New(&fakeDocker{info: managerInfo()}, "").Hello(context.Background()) + require.NoError(t, err) + assert.Equal(t, protocol.Hello{Hostname: "swarm-mgr", SwarmManager: true, SwarmClusterID: "cluster-1"}, got) +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/agent/inventory/` +Expected: FAIL — undefined identifiers. + +- [ ] **Step 3: Write minimal implementation** + +```go +// Package inventory turns what the local Docker daemon runs into the +// services an agent reports to the master. +package inventory + +import ( + "context" + "fmt" + "sort" + "strings" + + "github.com/docker/docker/api/types/container" + "github.com/docker/docker/api/types/image" + "github.com/docker/docker/api/types/swarm" + "github.com/docker/docker/api/types/system" + + "github.com/shcizo/package-updater/internal/imageref" + "github.com/shcizo/package-updater/internal/protocol" +) + +const ( + labelProject = "com.docker.compose.project" + labelService = "com.docker.compose.service" + labelWorkingDir = "com.docker.compose.project.working_dir" + labelSwarmTask = "com.docker.swarm.service.id" +) + +// DockerClient is the subset of the Docker SDK the collector needs. +// *client.Client satisfies it. +type DockerClient interface { + ContainerList(ctx context.Context, opts container.ListOptions) ([]container.Summary, error) + ImageList(ctx context.Context, opts image.ListOptions) ([]image.Summary, error) + ServiceList(ctx context.Context, opts swarm.ServiceListOptions) ([]swarm.Service, error) + Info(ctx context.Context) (system.Info, error) +} + +// Collector reads the local daemon's inventory. +type Collector struct { + cli DockerClient + selfID string +} + +// New returns a Collector. selfID is a prefix of the agent's own container +// ID (inside a container, the hostname); that container is marked Self. +func New(cli DockerClient, selfID string) *Collector { + return &Collector{cli: cli, selfID: selfID} +} + +// Hello returns the machine facts sent on connect. AgentVersion is left for +// the caller to fill in. +func (c *Collector) Hello(ctx context.Context) (protocol.Hello, error) { + info, err := c.cli.Info(ctx) + if err != nil { + return protocol.Hello{}, fmt.Errorf("docker info: %w", err) + } + h := protocol.Hello{Hostname: info.Name, SwarmManager: info.Swarm.ControlAvailable} + if info.Swarm.Cluster != nil { + h.SwarmClusterID = info.Swarm.Cluster.ID + } + return h, nil +} + +// Snapshot returns every service on this machine, sorted by Key. +func (c *Collector) Snapshot(ctx context.Context) ([]protocol.Service, error) { + info, err := c.cli.Info(ctx) + if err != nil { + return nil, fmt.Errorf("docker info: %w", err) + } + containers, err := c.cli.ContainerList(ctx, container.ListOptions{All: true}) + if err != nil { + return nil, fmt.Errorf("docker container list: %w", err) + } + images, err := c.cli.ImageList(ctx, image.ListOptions{}) + if err != nil { + return nil, fmt.Errorf("docker image list: %w", err) + } + repoDigests := make(map[string][]string, len(images)) + for _, img := range images { + repoDigests[img.ID] = img.RepoDigests + } + + byKey := make(map[string]protocol.Service) + for _, ct := range containers { + if _, isTask := ct.Labels[labelSwarmTask]; isTask { + continue + } + svc := c.fromContainer(ct, repoDigests[ct.ImageID]) + if prev, ok := byKey[svc.Key()]; ok { + svc = mergeReplicas(prev, svc) + } + byKey[svc.Key()] = svc + } + + if info.Swarm.ControlAvailable { + services, err := c.cli.ServiceList(ctx, swarm.ServiceListOptions{Status: true}) + if err != nil { + return nil, fmt.Errorf("docker service list: %w", err) + } + for _, s := range services { + if s.Spec.TaskTemplate.ContainerSpec == nil { + continue + } + svc := fromSwarmService(s) + byKey[svc.Key()] = svc + } + } + + out := make([]protocol.Service, 0, len(byKey)) + for _, s := range byKey { + out = append(out, s) + } + sort.Slice(out, func(i, j int) bool { return out[i].Key() < out[j].Key() }) + return out, nil +} + +func (c *Collector) fromContainer(ct container.Summary, repoDigests []string) protocol.Service { + svc := protocol.Service{ + Image: ct.Image, + Digest: digestFor(ct.Image, repoDigests), + Status: containerStatus(ct.State), + Self: c.selfID != "" && strings.HasPrefix(ct.ID, c.selfID), + } + project, service := ct.Labels[labelProject], ct.Labels[labelService] + if project != "" && service != "" { + svc.Kind = protocol.KindCompose + svc.ComposeProject = project + svc.ComposeService = service + svc.WorkingDir = ct.Labels[labelWorkingDir] + return svc + } + svc.Kind = protocol.KindContainer + svc.ContainerName = ct.ID + if len(ct.Names) > 0 { + svc.ContainerName = strings.TrimPrefix(ct.Names[0], "/") + } + return svc +} + +func fromSwarmService(s swarm.Service) protocol.Service { + ref := s.Spec.TaskTemplate.ContainerSpec.Image + return protocol.Service{ + Kind: protocol.KindSwarm, + SwarmService: s.Spec.Name, + Image: ref, + Digest: imageref.Parse(ref).Digest, + Status: swarmStatus(s.ServiceStatus), + } +} + +// mergeReplicas folds a second container of the same Compose service into +// the first: the service counts as running if any replica runs. +func mergeReplicas(a, b protocol.Service) protocol.Service { + if b.Status == "running" { + a.Status = "running" + } + a.Self = a.Self || b.Self + return a +} + +// digestFor returns the registry digest the container's image was pulled +// as. A reference that already pins a digest wins; otherwise the image's +// RepoDigests entry for the same repository is used. +func digestFor(ref string, repoDigests []string) string { + r := imageref.Parse(ref) + if r.Digest != "" { + return r.Digest + } + for _, rd := range repoDigests { + if p := imageref.Parse(rd); p.Repo == r.Repo { + return p.Digest + } + } + return "" +} + +func containerStatus(state string) string { + if state == "running" { + return "running" + } + return "stopped" +} + +func swarmStatus(st *swarm.ServiceStatus) string { + switch { + case st == nil: + return "unknown" + case st.DesiredTasks == 0: + return "stopped" + case st.RunningTasks >= st.DesiredTasks: + return "running" + default: + return "degraded" + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `go test ./internal/agent/inventory/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/agent/inventory +git commit -m "feat(agent): collect compose, swarm and container inventory from docker" +``` + +--- + +### Task 5: Inventory diff and Docker event watcher + +**Files:** +- Create: `internal/agent/inventory/diff.go`, `internal/agent/inventory/watch.go` +- Test: `internal/agent/inventory/diff_test.go`, `internal/agent/inventory/watch_test.go` + +**Interfaces:** +- Consumes: `protocol.Service` (Task 2). +- Produces: + - `func Diff(prev, next []protocol.Service) (upserts []protocol.Service, removes []string)` + - `type EventSource interface { Events(ctx context.Context, opts events.ListOptions) (<-chan events.Message, <-chan error) }` (`*client.Client` satisfies it) + - `func Watch(ctx context.Context, src EventSource, quiet time.Duration, onChange func()) error` — calls `onChange` once per burst of events, after `quiet` without new events. Returns `nil` when `ctx` ends, an error when the event stream fails. + +- [ ] **Step 1: Write the failing tests** + +`diff_test.go`: + +```go +package inventory + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/shcizo/package-updater/internal/protocol" +) + +func TestDiff(t *testing.T) { + a := protocol.Service{Kind: protocol.KindContainer, ContainerName: "a", Image: "a:1", Status: "running"} + b := protocol.Service{Kind: protocol.KindContainer, ContainerName: "b", Image: "b:1", Status: "running"} + bStopped := b + bStopped.Status = "stopped" + c := protocol.Service{Kind: protocol.KindContainer, ContainerName: "c", Image: "c:1", Status: "running"} + + cases := []struct { + name string + prev, next []protocol.Service + wantUpserts []protocol.Service + wantRemoves []string + }{ + {"unchanged", []protocol.Service{a, b}, []protocol.Service{a, b}, nil, nil}, + {"added", []protocol.Service{a}, []protocol.Service{a, c}, []protocol.Service{c}, nil}, + {"changed", []protocol.Service{a, b}, []protocol.Service{a, bStopped}, []protocol.Service{bStopped}, nil}, + {"removed", []protocol.Service{a, b}, []protocol.Service{a}, nil, []string{"container|b"}}, + {"from empty", nil, []protocol.Service{a}, []protocol.Service{a}, nil}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ups, rms := Diff(tc.prev, tc.next) + assert.Equal(t, tc.wantUpserts, ups) + assert.Equal(t, tc.wantRemoves, rms) + }) + } +} +``` + +`watch_test.go`: + +```go +package inventory + +import ( + "context" + "errors" + "sync/atomic" + "testing" + "time" + + "github.com/docker/docker/api/types/events" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +type fakeEvents struct { + msgs chan events.Message + errs chan error +} + +func newFakeEvents() *fakeEvents { + return &fakeEvents{msgs: make(chan events.Message, 10), errs: make(chan error, 1)} +} + +func (f *fakeEvents) Events(context.Context, events.ListOptions) (<-chan events.Message, <-chan error) { + return f.msgs, f.errs +} + +func TestWatch_DebouncesBursts(t *testing.T) { + src := newFakeEvents() + var calls atomic.Int32 + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- Watch(ctx, src, 30*time.Millisecond, func() { calls.Add(1) }) }() + + for range 3 { + src.msgs <- events.Message{Type: events.ContainerEventType, Action: "start"} + } + require.Eventually(t, func() bool { return calls.Load() == 1 }, time.Second, 5*time.Millisecond) + + src.msgs <- events.Message{Type: events.ServiceEventType, Action: "update"} + require.Eventually(t, func() bool { return calls.Load() == 2 }, time.Second, 5*time.Millisecond) + + cancel() + assert.NoError(t, <-done) +} + +func TestWatch_ReturnsStreamError(t *testing.T) { + src := newFakeEvents() + src.errs <- errors.New("daemon restarted") + err := Watch(context.Background(), src, time.Millisecond, func() {}) + assert.ErrorContains(t, err, "daemon restarted") +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `go test ./internal/agent/inventory/` +Expected: FAIL — `undefined: Diff`, `undefined: Watch`. + +- [ ] **Step 3: Write minimal implementation** + +`diff.go`: + +```go +package inventory + +import "github.com/shcizo/package-updater/internal/protocol" + +// Diff compares two snapshots. upserts holds services that are new or +// changed in next; removes holds the keys of services missing from next. +func Diff(prev, next []protocol.Service) (upserts []protocol.Service, removes []string) { + old := make(map[string]protocol.Service, len(prev)) + for _, s := range prev { + old[s.Key()] = s + } + seen := make(map[string]struct{}, len(next)) + for _, s := range next { + seen[s.Key()] = struct{}{} + if o, ok := old[s.Key()]; !ok || o != s { + upserts = append(upserts, s) + } + } + for _, s := range prev { + if _, ok := seen[s.Key()]; !ok { + removes = append(removes, s.Key()) + } + } + return upserts, removes +} +``` + +`watch.go`: + +```go +package inventory + +import ( + "context" + "fmt" + "time" + + "github.com/docker/docker/api/types/events" + "github.com/docker/docker/api/types/filters" +) + +// EventSource is the subset of the Docker SDK Watch needs. +// *client.Client satisfies it. +type EventSource interface { + Events(ctx context.Context, opts events.ListOptions) (<-chan events.Message, <-chan error) +} + +// Watch subscribes to container and service lifecycle events and calls +// onChange once after each burst has been quiet for the given duration. +// A Compose restart emits a dozen events; this turns them into one +// inventory refresh. Returns nil when ctx ends, or an error if the event +// stream breaks (for example when the daemon restarts) — the caller should +// resubscribe and refresh, since events may have been missed. +func Watch(ctx context.Context, src EventSource, quiet time.Duration, onChange func()) error { + args := filters.NewArgs( + filters.Arg("type", string(events.ContainerEventType)), + filters.Arg("type", string(events.ServiceEventType)), + ) + for _, action := range []string{"create", "start", "stop", "die", "destroy", "rename", "update", "remove"} { + args.Add("event", action) + } + msgs, errs := src.Events(ctx, events.ListOptions{Filters: args}) + + timer := time.NewTimer(quiet) + timer.Stop() + defer timer.Stop() + for { + select { + case <-ctx.Done(): + return nil + case err := <-errs: + if ctx.Err() != nil { + return nil + } + return fmt.Errorf("docker events: %w", err) + case <-msgs: + timer.Reset(quiet) + case <-timer.C: + onChange() + } + } +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `go test -race ./internal/agent/inventory/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/agent/inventory +git commit -m "feat(agent): diff inventories and debounce docker events" +``` + +--- + +### Task 6: Agent connection loop + +**Files:** +- Create: `internal/agent/conn/conn.go` +- Test: `internal/agent/conn/conn_test.go` + +**Interfaces:** +- Consumes: `pin.ClientConfig`, `pin.GenerateKey`, `pin.Certificate`, `pin.Fingerprint` (Task 3); `protocol.*` (Task 2); `inventory.Diff` (Task 5). +- Produces: + - `type Source interface { Hello(ctx) (protocol.Hello, error); Snapshot(ctx) ([]protocol.Service, error) }` — `*inventory.Collector` satisfies it. + - `type Config struct { URL, Token, Fingerprint, Version string; MinBackoff, MaxBackoff time.Duration }` + - `func New(cfg Config, src Source, log *slog.Logger) *Agent` + - `func (a *Agent) Notify()` — never blocks; coalesces. + - `func (a *Agent) Run(ctx context.Context) error` — returns `nil` when `ctx` ends. + +Add the dependency first: `go get github.com/coder/websocket@latest`. + +- [ ] **Step 1: Write the failing test** + +```go +package conn + +import ( + "context" + "crypto/tls" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + + "github.com/coder/websocket" + "github.com/coder/websocket/wsjson" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/pin" + "github.com/shcizo/package-updater/internal/protocol" +) + +type fakeMaster struct { + srv *httptest.Server + fingerprint string + envs chan protocol.Envelope + auth chan string + conns chan *websocket.Conn +} + +func newFakeMaster(t *testing.T) *fakeMaster { + t.Helper() + key, err := pin.GenerateKey() + require.NoError(t, err) + cert, err := pin.Certificate(key) + require.NoError(t, err) + fp, err := pin.Fingerprint(key.Public()) + require.NoError(t, err) + + fm := &fakeMaster{ + fingerprint: fp, + envs: make(chan protocol.Envelope, 100), + auth: make(chan string, 10), + conns: make(chan *websocket.Conn, 10), + } + fm.srv = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + fm.auth <- r.Header.Get("Authorization") + c, err := websocket.Accept(w, r, nil) + if err != nil { + return + } + c.SetReadLimit(1 << 20) + fm.conns <- c + for { + var env protocol.Envelope + if err := wsjson.Read(r.Context(), c, &env); err != nil { + return + } + fm.envs <- env + } + })) + fm.srv.TLS = &tls.Config{Certificates: []tls.Certificate{cert}} + fm.srv.StartTLS() + t.Cleanup(fm.srv.Close) + return fm +} + +func (fm *fakeMaster) url() string { return "wss" + strings.TrimPrefix(fm.srv.URL, "https") } + +func (fm *fakeMaster) next(t *testing.T) protocol.Envelope { + t.Helper() + select { + case e := <-fm.envs: + return e + case <-time.After(2 * time.Second): + t.Fatal("timed out waiting for a message from the agent") + return protocol.Envelope{} + } +} + +type fakeSource struct { + mu sync.Mutex + services []protocol.Service +} + +func (f *fakeSource) set(s ...protocol.Service) { + f.mu.Lock() + defer f.mu.Unlock() + f.services = s +} + +func (f *fakeSource) Hello(context.Context) (protocol.Hello, error) { + return protocol.Hello{Hostname: "node-2"}, nil +} + +func (f *fakeSource) Snapshot(context.Context) ([]protocol.Service, error) { + f.mu.Lock() + defer f.mu.Unlock() + return append([]protocol.Service(nil), f.services...), nil +} + +var ( + svcA = protocol.Service{Kind: protocol.KindContainer, ContainerName: "a", Image: "a:1", Status: "running"} + svcB = protocol.Service{Kind: protocol.KindContainer, ContainerName: "b", Image: "b:1", Status: "running"} +) + +func startAgent(t *testing.T, fm *fakeMaster, fingerprint string, src Source) *Agent { + t.Helper() + a := New(Config{ + URL: fm.url(), Token: "pua_test", Fingerprint: fingerprint, Version: "v-test", + MinBackoff: 10 * time.Millisecond, MaxBackoff: 50 * time.Millisecond, + }, src, slog.New(slog.NewTextHandler(io.Discard, nil))) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { _ = a.Run(ctx); close(done) }() + t.Cleanup(func() { cancel(); <-done }) + return a +} + +func TestAgent_SendsHelloThenSnapshotWithToken(t *testing.T) { + fm := newFakeMaster(t) + src := &fakeSource{} + src.set(svcA) + startAgent(t, fm, fm.fingerprint, src) + + assert.Equal(t, "Bearer pua_test", <-fm.auth) + hello := fm.next(t) + require.Equal(t, protocol.TypeHello, hello.Type) + h, err := protocol.Decode[protocol.Hello](hello) + require.NoError(t, err) + assert.Equal(t, protocol.Hello{Hostname: "node-2", AgentVersion: "v-test"}, h) + + snap := fm.next(t) + require.Equal(t, protocol.TypeSnapshot, snap.Type) + s, err := protocol.Decode[protocol.Snapshot](snap) + require.NoError(t, err) + assert.Equal(t, []protocol.Service{svcA}, s.Services) +} + +func TestAgent_NotifySendsDiff(t *testing.T) { + fm := newFakeMaster(t) + src := &fakeSource{} + src.set(svcA) + a := startAgent(t, fm, fm.fingerprint, src) + fm.next(t) // hello + fm.next(t) // snapshot + + src.set(svcB) + a.Notify() + + up := fm.next(t) + require.Equal(t, protocol.TypeUpsert, up.Type) + u, err := protocol.Decode[protocol.Upsert](up) + require.NoError(t, err) + assert.Equal(t, svcB, u.Service) + + rm := fm.next(t) + require.Equal(t, protocol.TypeRemove, rm.Type) + r, err := protocol.Decode[protocol.Remove](rm) + require.NoError(t, err) + assert.Equal(t, "container|a", r.Key) +} + +func TestAgent_ReconnectsAndResendsSnapshot(t *testing.T) { + fm := newFakeMaster(t) + src := &fakeSource{} + src.set(svcA) + startAgent(t, fm, fm.fingerprint, src) + fm.next(t) + fm.next(t) + + first := <-fm.conns + first.CloseNow() + + assert.Equal(t, protocol.TypeHello, fm.next(t).Type) + assert.Equal(t, protocol.TypeSnapshot, fm.next(t).Type) +} + +func TestAgent_RefusesMasterWithWrongKey(t *testing.T) { + fm := newFakeMaster(t) + other, err := pin.GenerateKey() + require.NoError(t, err) + wrong, err := pin.Fingerprint(other.Public()) + require.NoError(t, err) + + startAgent(t, fm, wrong, &fakeSource{}) + time.Sleep(200 * time.Millisecond) + assert.Empty(t, fm.auth, "the token must never be sent to a server with the wrong key") +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/agent/conn/` +Expected: FAIL — undefined identifiers. + +- [ ] **Step 3: Write minimal implementation** + +```go +// Package conn keeps an agent connected to the master: it dials over pinned +// TLS, announces itself, sends its inventory, and streams changes until the +// connection drops, then reconnects with backoff. +package conn + +import ( + "context" + "fmt" + "log/slog" + "net/http" + "time" + + "github.com/coder/websocket" + "github.com/coder/websocket/wsjson" + + "github.com/shcizo/package-updater/internal/agent/inventory" + "github.com/shcizo/package-updater/internal/pin" + "github.com/shcizo/package-updater/internal/protocol" +) + +// readLimit caps a single incoming message. The library default (32 KiB) +// is too small for the master's future job messages; matches the master. +const readLimit = 1 << 20 + +// Source supplies what the agent reports. *inventory.Collector satisfies it. +type Source interface { + Hello(ctx context.Context) (protocol.Hello, error) + Snapshot(ctx context.Context) ([]protocol.Service, error) +} + +// Config holds connection settings. +type Config struct { + URL string // wss://host:port/agent + Token string // AGENT_TOKEN + Fingerprint string // MASTER_KEY + Version string + MinBackoff time.Duration // default 1s + MaxBackoff time.Duration // default 30s +} + +// Agent is the agent's side of the master connection. +type Agent struct { + cfg Config + src Source + log *slog.Logger + changes chan struct{} +} + +// New returns an Agent. Call Run to connect. +func New(cfg Config, src Source, log *slog.Logger) *Agent { + if cfg.MinBackoff == 0 { + cfg.MinBackoff = time.Second + } + if cfg.MaxBackoff == 0 { + cfg.MaxBackoff = 30 * time.Second + } + return &Agent{cfg: cfg, src: src, log: log, changes: make(chan struct{}, 1)} +} + +// Notify tells the agent its inventory may have changed. It never blocks; +// several calls before the agent gets to them collapse into one refresh. +func (a *Agent) Notify() { + select { + case a.changes <- struct{}{}: + default: + } +} + +// Run keeps the agent connected until ctx ends. +func (a *Agent) Run(ctx context.Context) error { + backoff := a.cfg.MinBackoff + for { + connected, err := a.session(ctx) + if ctx.Err() != nil { + return nil + } + if connected { + backoff = a.cfg.MinBackoff + } + a.log.Warn("master_connection_lost", "error", err, "retry_in", backoff.String()) + select { + case <-ctx.Done(): + return nil + case <-time.After(backoff): + } + backoff = min(backoff*2, a.cfg.MaxBackoff) + } +} + +// session runs one connection until it fails. connected reports whether the +// hello was delivered, so Run resets its backoff only after real progress. +func (a *Agent) session(parent context.Context) (connected bool, err error) { + httpClient := &http.Client{Transport: &http.Transport{TLSClientConfig: pin.ClientConfig(a.cfg.Fingerprint)}} + c, _, err := websocket.Dial(parent, a.cfg.URL, &websocket.DialOptions{ + HTTPClient: httpClient, + HTTPHeader: http.Header{"Authorization": []string{"Bearer " + a.cfg.Token}}, + }) + if err != nil { + return false, fmt.Errorf("dial master: %w", err) + } + defer c.CloseNow() + c.SetReadLimit(readLimit) + + ctx, cancel := context.WithCancel(parent) + defer cancel() + // The master sends nothing yet in this phase, but reading is how a + // dropped connection is noticed, so a reader runs for the whole session. + go func() { + defer cancel() + for { + if _, _, err := c.Read(ctx); err != nil { + return + } + } + }() + + hello, err := a.src.Hello(ctx) + if err != nil { + return false, err + } + hello.AgentVersion = a.cfg.Version + if err := send(ctx, c, protocol.TypeHello, hello); err != nil { + return false, err + } + + last, err := a.src.Snapshot(ctx) + if err != nil { + return true, err + } + if err := send(ctx, c, protocol.TypeSnapshot, protocol.Snapshot{Services: last}); err != nil { + return true, err + } + a.log.Info("master_connected", "services", len(last)) + + for { + select { + case <-ctx.Done(): + _ = c.Close(websocket.StatusNormalClosure, "") + return true, ctx.Err() + case <-a.changes: + next, err := a.src.Snapshot(ctx) + if err != nil { + return true, err + } + upserts, removes := inventory.Diff(last, next) + for _, s := range upserts { + if err := send(ctx, c, protocol.TypeUpsert, protocol.Upsert{Service: s}); err != nil { + return true, err + } + } + for _, k := range removes { + if err := send(ctx, c, protocol.TypeRemove, protocol.Remove{Key: k}); err != nil { + return true, err + } + } + last = next + } + } +} + +func send(ctx context.Context, c *websocket.Conn, typ string, payload any) error { + env, err := protocol.Encode(typ, payload) + if err != nil { + return err + } + if err := wsjson.Write(ctx, c, env); err != nil { + return fmt.Errorf("send %s: %w", typ, err) + } + return nil +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `go test -race ./internal/agent/conn/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add go.mod go.sum internal/agent/conn +git commit -m "feat(agent): connect to master over pinned TLS and stream inventory" +``` + +--- + +### Task 7: Agent binary, config and image + +**Files:** +- Create: `internal/agent/config/config.go`, `cmd/agent/main.go` +- Modify: `Dockerfile` (build step and copy) +- Test: `internal/agent/config/config_test.go` + +**Interfaces:** +- Consumes: `inventory.New`, `inventory.Watch` (Tasks 4–5); `conn.New`, `(*Agent).Run`, `(*Agent).Notify` (Task 6); `logging.New` (existing). +- Produces: `config.Load() (*config.Config, error)` with fields `MasterURL, MasterKey, Token, SelfID, LogLevel string`; binary `/usr/local/bin/updater-agent` in the image. + +- [ ] **Step 1: Write the failing test** + +```go +package config + +import ( + "os" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const validKey = "sha256:0000000000000000000000000000000000000000000000000000000000000000" + +func setValid(t *testing.T) { + t.Setenv("MASTER_URL", "wss://10.0.0.5:8090/agent") + t.Setenv("MASTER_KEY", validKey) + t.Setenv("AGENT_TOKEN", "pua_abc") + t.Setenv("SELF_ID", "") + t.Setenv("LOG_LEVEL", "") +} + +func TestLoad_Valid(t *testing.T) { + setValid(t) + cfg, err := Load() + require.NoError(t, err) + host, _ := os.Hostname() + assert.Equal(t, &Config{ + MasterURL: "wss://10.0.0.5:8090/agent", MasterKey: validKey, Token: "pua_abc", + SelfID: host, LogLevel: "info", + }, cfg) +} + +func TestLoad_Invalid(t *testing.T) { + cases := []struct { + name, key, value, wantErr string + }{ + {"missing url", "MASTER_URL", "", "MASTER_URL is required"}, + {"plain ws", "MASTER_URL", "ws://10.0.0.5:8090/agent", "MASTER_URL must start with wss://"}, + {"missing key", "MASTER_KEY", "", "MASTER_KEY is required"}, + {"malformed key", "MASTER_KEY", "abc", "MASTER_KEY must look like sha256:<64 hex>"}, + {"missing token", "AGENT_TOKEN", "", "AGENT_TOKEN is required"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + setValid(t) + t.Setenv(c.key, c.value) + _, err := Load() + assert.ErrorContains(t, err, c.wantErr) + }) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/agent/config/` +Expected: FAIL — `undefined: Load`. + +- [ ] **Step 3: Write the implementation** + +`internal/agent/config/config.go`: + +```go +// Package config loads the agent's configuration from environment variables. +package config + +import ( + "errors" + "os" + "regexp" + "strings" +) + +var masterKeyPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) + +// Config holds the agent's runtime configuration. +type Config struct { + MasterURL string // MASTER_URL, wss://host:port/agent + MasterKey string // MASTER_KEY, fingerprint shown in the master UI + Token string // AGENT_TOKEN, shown once when the agent was added + SelfID string // SELF_ID, defaults to the hostname (= container ID prefix) + LogLevel string // LOG_LEVEL, default info +} + +// Load reads and validates the agent configuration. It fails fast so a +// misconfigured agent never starts dialling. +func Load() (*Config, error) { + cfg := &Config{ + MasterURL: os.Getenv("MASTER_URL"), + MasterKey: os.Getenv("MASTER_KEY"), + Token: os.Getenv("AGENT_TOKEN"), + SelfID: os.Getenv("SELF_ID"), + LogLevel: os.Getenv("LOG_LEVEL"), + } + switch { + case cfg.MasterURL == "": + return nil, errors.New("MASTER_URL is required") + case !strings.HasPrefix(cfg.MasterURL, "wss://"): + return nil, errors.New("MASTER_URL must start with wss://") + case cfg.MasterKey == "": + return nil, errors.New("MASTER_KEY is required") + case !masterKeyPattern.MatchString(cfg.MasterKey): + return nil, errors.New("MASTER_KEY must look like sha256:<64 hex>") + case cfg.Token == "": + return nil, errors.New("AGENT_TOKEN is required") + } + if cfg.SelfID == "" { + cfg.SelfID, _ = os.Hostname() + } + if cfg.LogLevel == "" { + cfg.LogLevel = "info" + } + return cfg, nil +} +``` + +`cmd/agent/main.go`: + +```go +// Command updater-agent reports this machine's Docker services to the +// package-updater master. +package main + +import ( + "context" + "fmt" + "log/slog" + "os" + "os/signal" + "syscall" + "time" + + "github.com/docker/docker/client" + + "github.com/shcizo/package-updater/internal/agent/config" + "github.com/shcizo/package-updater/internal/agent/conn" + "github.com/shcizo/package-updater/internal/agent/inventory" + "github.com/shcizo/package-updater/internal/logging" +) + +// Build info is injected at link time via -ldflags="-X main.version=...". +var version = "dev" + +func main() { + if err := run(); err != nil { + fmt.Fprintln(os.Stderr, "fatal:", err) + os.Exit(1) + } +} + +func run() error { + cfg, err := config.Load() + if err != nil { + return err + } + logger := logging.New(cfg.LogLevel) + + dockerCli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation()) + if err != nil { + return fmt.Errorf("docker client: %w", err) + } + defer dockerCli.Close() + + collector := inventory.New(dockerCli, cfg.SelfID) + agent := conn.New(conn.Config{ + URL: cfg.MasterURL, Token: cfg.Token, Fingerprint: cfg.MasterKey, Version: version, + }, collector, logger) + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + go watchLoop(ctx, dockerCli, agent, logger) + logger.Info("agent_starting", "version", version, "master_url", cfg.MasterURL) + return agent.Run(ctx) +} + +// watchLoop keeps a Docker event subscription alive. When the stream breaks +// (daemon restart), it resubscribes and asks for a refresh, because events +// may have been missed in between. +func watchLoop(ctx context.Context, src inventory.EventSource, agent *conn.Agent, log *slog.Logger) { + for { + err := inventory.Watch(ctx, src, 2*time.Second, agent.Notify) + if ctx.Err() != nil { + return + } + log.Warn("docker_events_lost", "error", err) + select { + case <-ctx.Done(): + return + case <-time.After(5 * time.Second): + } + agent.Notify() + } +} +``` + +`Dockerfile` — replace the single `RUN CGO_ENABLED=0 go build ...` step and the `COPY --from=build` line so the image carries all three binaries: + +```dockerfile +RUN for cmd in server:package-updater master:updater-master agent:updater-agent; do \ + CGO_ENABLED=0 go build \ + -ldflags="-s -w \ + -X main.version=${VERSION} \ + -X main.commit=${COMMIT} \ + -X main.buildTime=${BUILD_TIME}" \ + -o /out/${cmd#*:} ./cmd/${cmd%%:*} || exit 1; \ + done +``` + +```dockerfile +COPY --from=build /out/ /usr/local/bin/ +``` + +`cmd/master` does not exist until Task 14. Until then, temporarily remove `master:updater-master` from the loop, and add it back in Task 14 (Task 14 includes that step). The unused `-X main.commit`/`main.buildTime` flags are ignored by the linker for binaries without those variables. + +- [ ] **Step 4: Run tests and build** + +Run: `go test ./internal/agent/... && go build ./cmd/agent && rm -f agent && docker build -t package-updater:dev .` +Expected: tests PASS, both builds succeed. If Docker is not available locally, skip the `docker build` and say so in the task report. + +- [ ] **Step 5: Commit** + +```bash +git add internal/agent/config cmd/agent Dockerfile +git commit -m "feat(agent): add updater-agent binary and ship it in the image" +``` + +--- + +### Task 8: Master model, store bootstrap, identity and agents + +**Files:** +- Create: `internal/master/model/model.go`, `internal/master/store/store.go`, `internal/master/store/migrate.go`, `internal/master/store/identity.go`, `internal/master/store/agents.go`, `internal/master/store/migrations/0001_init.sql` +- Test: `internal/master/model/model_test.go`, `internal/master/store/helpers_integration_test.go`, `internal/master/store/agents_integration_test.go` + +**Interfaces:** +- Consumes: `protocol.Hello`, `protocol.Kind` (Task 2); `pin.GenerateKey` (Task 3, tests only). +- Produces: + - `model.ErrNotFound`, `model.ErrConflict` + - `model.Agent{ID int64; Name, Hostname, AgentVersion string; SwarmManager bool; LastSeenAt, RevokedAt *time.Time; CreatedAt time.Time}` + - `model.Service{ID int64; Kind protocol.Kind; LocationKey string; AgentID *int64; AgentName, ComposeProject, ComposeService, WorkingDir, SwarmClusterID, SwarmService, ContainerName, ImageRepo, ActualRef, ActualDigest, Status, UpdateMode string; Self bool; FirstSeenAt, UpdatedAt time.Time}` with `func (s Service) DisplayName() string` + - `model.Session{ID, Subject, Name, Email, CSRFToken string; ExpiresAt time.Time}` + - `model.LocationKey(agentID int64, clusterID, agentKey string) string` + - `store.Open(ctx, url string) (*Store, error)`, `(*Store).Close()`, `(*Store).Ping(ctx) error` + - `(*Store).LoadOrCreateIdentity(ctx, generate func() (ed25519.PrivateKey, error)) (ed25519.PrivateKey, error)` + - `(*Store).CreateAgent(ctx, name, tokenHash string) (model.Agent, error)` — `ErrConflict` on duplicate name + - `(*Store).ListAgents(ctx) ([]model.Agent, error)` + - `(*Store).AgentByTokenHash(ctx, hash string) (model.Agent, error)` — `ErrNotFound` + - `(*Store).RevokeAgent(ctx, id int64, at time.Time) error` + - `(*Store).RecordHello(ctx, id int64, h protocol.Hello, at time.Time) error` + - `(*Store).MarkSeen(ctx, id int64, at time.Time) error` + +Add the dependency: `go get github.com/jackc/pgx/v5@latest`. + +To run integration tests locally: + +```bash +docker run --rm -d --name pu-test-pg -p 5432:5432 -e POSTGRES_PASSWORD=pw postgres:17 +export DATABASE_URL='postgres://postgres:pw@localhost:5432/postgres?sslmode=disable' +go test -tags integration ./internal/master/store/ +``` + +- [ ] **Step 1: Write the failing tests** + +`internal/master/model/model_test.go`: + +```go +package model + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/shcizo/package-updater/internal/protocol" +) + +func TestLocationKey(t *testing.T) { + cases := []struct { + name, agentKey, want string + }{ + {"compose is per agent", "compose|slusk|web", "compose|7|slusk|web"}, + {"container is per agent", "container|adhoc", "container|7|adhoc"}, + {"swarm is per cluster", "swarm|gitea_gitea", "swarm|cluster-1|gitea_gitea"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + assert.Equal(t, c.want, LocationKey(7, "cluster-1", c.agentKey)) + }) + } +} + +func TestDisplayName(t *testing.T) { + assert.Equal(t, "slusk/web", Service{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web"}.DisplayName()) + assert.Equal(t, "gitea_gitea", Service{Kind: protocol.KindSwarm, SwarmService: "gitea_gitea"}.DisplayName()) + assert.Equal(t, "adhoc", Service{Kind: protocol.KindContainer, ContainerName: "adhoc"}.DisplayName()) +} +``` + +`internal/master/store/helpers_integration_test.go`: + +```go +//go:build integration + +package store + +import ( + "context" + "fmt" + "os" + "testing" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" + "github.com/stretchr/testify/require" +) + +// openTestStore returns a Store migrated into a fresh schema that is dropped +// when the test ends, so tests never see each other's rows. +func openTestStore(t *testing.T) *Store { + t.Helper() + url := os.Getenv("DATABASE_URL") + if url == "" { + t.Skip("DATABASE_URL not set") + } + ctx := context.Background() + admin, err := pgx.Connect(ctx, url) + require.NoError(t, err) + schema := fmt.Sprintf("test_%d", time.Now().UnixNano()) + _, err = admin.Exec(ctx, "CREATE SCHEMA "+schema) + require.NoError(t, err) + t.Cleanup(func() { + _, _ = admin.Exec(ctx, "DROP SCHEMA "+schema+" CASCADE") + _ = admin.Close(ctx) + }) + + cfg, err := pgxpool.ParseConfig(url) + require.NoError(t, err) + cfg.ConnConfig.RuntimeParams["search_path"] = schema + s, err := openWithConfig(ctx, cfg) + require.NoError(t, err) + t.Cleanup(s.Close) + return s +} +``` + +`internal/master/store/agents_integration_test.go`: + +```go +//go:build integration + +package store + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/pin" + "github.com/shcizo/package-updater/internal/protocol" +) + +func TestIdentity_CreatedOnceThenReused(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + k1, err := s.LoadOrCreateIdentity(ctx, pin.GenerateKey) + require.NoError(t, err) + k2, err := s.LoadOrCreateIdentity(ctx, pin.GenerateKey) + require.NoError(t, err) + assert.True(t, k1.Equal(k2), "a restarted master must keep its key or every agent breaks") +} + +func TestMigrationsAreIdempotent(t *testing.T) { + s := openTestStore(t) + require.NoError(t, migrate(context.Background(), s.pool)) +} + +func TestAgents_CreateLookupRevoke(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + + created, err := s.CreateAgent(ctx, "node-2", "hash-2") + require.NoError(t, err) + assert.Equal(t, "node-2", created.Name) + assert.Nil(t, created.RevokedAt) + + found, err := s.AgentByTokenHash(ctx, "hash-2") + require.NoError(t, err) + assert.Equal(t, created.ID, found.ID) + + _, err = s.AgentByTokenHash(ctx, "nope") + assert.ErrorIs(t, err, model.ErrNotFound) + + require.NoError(t, s.RevokeAgent(ctx, created.ID, time.Now())) + found, err = s.AgentByTokenHash(ctx, "hash-2") + require.NoError(t, err) + assert.NotNil(t, found.RevokedAt) +} + +func TestAgents_DuplicateNameIsConflict(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + _, err := s.CreateAgent(ctx, "node-2", "h1") + require.NoError(t, err) + _, err = s.CreateAgent(ctx, "node-2", "h2") + assert.ErrorIs(t, err, model.ErrConflict) +} + +func TestAgents_RecordHelloAndMarkSeen(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + a, err := s.CreateAgent(ctx, "swarm-mgr", "h") + require.NoError(t, err) + + at := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC) + require.NoError(t, s.RecordHello(ctx, a.ID, protocol.Hello{ + Hostname: "swarm-mgr", AgentVersion: "v1", SwarmManager: true, SwarmClusterID: "c1", + }, at)) + require.NoError(t, s.MarkSeen(ctx, a.ID, at.Add(time.Minute))) + + agents, err := s.ListAgents(ctx) + require.NoError(t, err) + require.Len(t, agents, 1) + assert.Equal(t, "swarm-mgr", agents[0].Hostname) + assert.Equal(t, "v1", agents[0].AgentVersion) + assert.True(t, agents[0].SwarmManager) + require.NotNil(t, agents[0].LastSeenAt) + assert.True(t, agents[0].LastSeenAt.Equal(at.Add(time.Minute))) +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `go test ./internal/master/model/ && go test -tags integration ./internal/master/store/` +Expected: FAIL — undefined identifiers. + +- [ ] **Step 3: Write the implementation** + +`internal/master/model/model.go`: + +```go +// Package model holds the master's domain types shared by storage, the agent +// hub and the web UI. +package model + +import ( + "errors" + "strconv" + "strings" + "time" + + "github.com/shcizo/package-updater/internal/protocol" +) + +// Sentinel errors returned by storage. +var ( + ErrNotFound = errors.New("not found") + ErrConflict = errors.New("already exists") +) + +// Agent is a registered agent. Whether it is connected right now is held in +// memory by the hub, not here. +type Agent struct { + ID int64 + Name string + Hostname string + AgentVersion string + SwarmManager bool + LastSeenAt *time.Time + RevokedAt *time.Time + CreatedAt time.Time +} + +// Service is one service as last reported by an agent. +type Service struct { + ID int64 + Kind protocol.Kind + LocationKey string + AgentID *int64 + AgentName string + ComposeProject string + ComposeService string + WorkingDir string + SwarmClusterID string + SwarmService string + ContainerName string + ImageRepo string + ActualRef string + ActualDigest string + Status string + Self bool + UpdateMode string + FirstSeenAt time.Time + UpdatedAt time.Time +} + +// DisplayName is the human-facing name of the service. +func (s Service) DisplayName() string { + switch s.Kind { + case protocol.KindCompose: + return s.ComposeProject + "/" + s.ComposeService + case protocol.KindSwarm: + return s.SwarmService + default: + return s.ContainerName + } +} + +// Session is a logged-in browser session. +type Session struct { + ID string + Subject string + Name string + Email string + CSRFToken string + ExpiresAt time.Time +} + +// LocationKey turns an agent-relative protocol key into the fleet-wide key +// stored in services.location_key. Compose services and containers belong to +// the agent's machine; Swarm services belong to the cluster, so two manager +// agents in one cluster report the same row. +func LocationKey(agentID int64, clusterID, agentKey string) string { + kind, rest, _ := strings.Cut(agentKey, "|") + if kind == string(protocol.KindSwarm) { + return kind + "|" + clusterID + "|" + rest + } + return kind + "|" + strconv.FormatInt(agentID, 10) + "|" + rest +} +``` + +`internal/master/store/migrations/0001_init.sql`: + +```sql +CREATE TABLE master_identity ( + id int PRIMARY KEY CHECK (id = 1), + private_key bytea NOT NULL, + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE agents ( + id bigserial PRIMARY KEY, + name text NOT NULL UNIQUE, + token_hash text NOT NULL UNIQUE, + hostname text NOT NULL DEFAULT '', + agent_version text NOT NULL DEFAULT '', + swarm_manager boolean NOT NULL DEFAULT false, + last_seen_at timestamptz, + revoked_at timestamptz, + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE TABLE services ( + id bigserial PRIMARY KEY, + kind text NOT NULL CHECK (kind IN ('compose', 'swarm', 'container')), + location_key text NOT NULL UNIQUE, + agent_id bigint REFERENCES agents (id), + compose_project text NOT NULL DEFAULT '', + compose_service text NOT NULL DEFAULT '', + working_dir text NOT NULL DEFAULT '', + swarm_cluster_id text NOT NULL DEFAULT '', + swarm_service text NOT NULL DEFAULT '', + container_name text NOT NULL DEFAULT '', + image_repo text NOT NULL, + actual_ref text NOT NULL, + actual_digest text NOT NULL DEFAULT '', + status text NOT NULL, + self boolean NOT NULL DEFAULT false, + update_mode text NOT NULL DEFAULT 'manual' CHECK (update_mode IN ('manual', 'auto')), + first_seen_at timestamptz NOT NULL, + updated_at timestamptz NOT NULL, + removed_at timestamptz +); + +CREATE TABLE sessions ( + id text PRIMARY KEY, + subject text NOT NULL, + name text NOT NULL DEFAULT '', + email text NOT NULL DEFAULT '', + csrf_token text NOT NULL, + expires_at timestamptz NOT NULL +); +``` + +`internal/master/store/store.go`: + +```go +// Package store persists the master's state in Postgres. +package store + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// Store is the Postgres-backed storage for the master. +type Store struct { + pool *pgxpool.Pool +} + +// Open connects to Postgres and applies pending migrations. +func Open(ctx context.Context, url string) (*Store, error) { + cfg, err := pgxpool.ParseConfig(url) + if err != nil { + return nil, fmt.Errorf("parse DATABASE_URL: %w", err) + } + return openWithConfig(ctx, cfg) +} + +func openWithConfig(ctx context.Context, cfg *pgxpool.Config) (*Store, error) { + pool, err := pgxpool.NewWithConfig(ctx, cfg) + if err != nil { + return nil, fmt.Errorf("connect postgres: %w", err) + } + if err := migrate(ctx, pool); err != nil { + pool.Close() + return nil, err + } + return &Store{pool: pool}, nil +} + +// Close releases the connection pool. +func (s *Store) Close() { s.pool.Close() } + +// Ping checks that Postgres answers. Used by /healthz. +func (s *Store) Ping(ctx context.Context) error { return s.pool.Ping(ctx) } +``` + +`internal/master/store/migrate.go`: + +```go +package store + +import ( + "context" + "embed" + "fmt" + "io/fs" + + "github.com/jackc/pgx/v5/pgxpool" +) + +//go:embed migrations/*.sql +var migrationsFS embed.FS + +// migrate applies every embedded migration not yet recorded in +// schema_migrations, in file-name order, each in its own transaction. +func migrate(ctx context.Context, pool *pgxpool.Pool) error { + if _, err := pool.Exec(ctx, `CREATE TABLE IF NOT EXISTS schema_migrations ( + version text PRIMARY KEY, + applied_at timestamptz NOT NULL DEFAULT now() + )`); err != nil { + return fmt.Errorf("create schema_migrations: %w", err) + } + entries, err := fs.ReadDir(migrationsFS, "migrations") + if err != nil { + return fmt.Errorf("read migrations: %w", err) + } + for _, e := range entries { + name := e.Name() + var applied bool + if err := pool.QueryRow(ctx, + `SELECT EXISTS (SELECT 1 FROM schema_migrations WHERE version = $1)`, name, + ).Scan(&applied); err != nil { + return fmt.Errorf("check migration %s: %w", name, err) + } + if applied { + continue + } + sql, err := migrationsFS.ReadFile("migrations/" + name) + if err != nil { + return fmt.Errorf("read migration %s: %w", name, err) + } + if err := applyMigration(ctx, pool, name, string(sql)); err != nil { + return err + } + } + return nil +} + +func applyMigration(ctx context.Context, pool *pgxpool.Pool, name, sql string) error { + tx, err := pool.Begin(ctx) + if err != nil { + return fmt.Errorf("begin migration %s: %w", name, err) + } + defer func() { _ = tx.Rollback(ctx) }() + // Exec without arguments uses the simple protocol, which allows the + // several statements a migration file contains. + if _, err := tx.Exec(ctx, sql); err != nil { + return fmt.Errorf("migration %s: %w", name, err) + } + if _, err := tx.Exec(ctx, `INSERT INTO schema_migrations (version) VALUES ($1)`, name); err != nil { + return fmt.Errorf("record migration %s: %w", name, err) + } + if err := tx.Commit(ctx); err != nil { + return fmt.Errorf("commit migration %s: %w", name, err) + } + return nil +} +``` + +`internal/master/store/identity.go`: + +```go +package store + +import ( + "context" + "crypto/ed25519" + "crypto/x509" + "errors" + "fmt" +) + +// LoadOrCreateIdentity returns the master's private key, creating and +// storing one with generate on first start. Keeping it in Postgres means it +// survives container recreation and is covered by database backups; losing +// it would force every agent to be reconfigured with a new MASTER_KEY. +func (s *Store) LoadOrCreateIdentity(ctx context.Context, generate func() (ed25519.PrivateKey, error)) (ed25519.PrivateKey, error) { + fresh, err := generate() + if err != nil { + return nil, err + } + der, err := x509.MarshalPKCS8PrivateKey(fresh) + if err != nil { + return nil, fmt.Errorf("marshal master key: %w", err) + } + if _, err := s.pool.Exec(ctx, + `INSERT INTO master_identity (id, private_key) VALUES (1, $1) ON CONFLICT (id) DO NOTHING`, der, + ); err != nil { + return nil, fmt.Errorf("store master key: %w", err) + } + var stored []byte + if err := s.pool.QueryRow(ctx, `SELECT private_key FROM master_identity WHERE id = 1`).Scan(&stored); err != nil { + return nil, fmt.Errorf("load master key: %w", err) + } + parsed, err := x509.ParsePKCS8PrivateKey(stored) + if err != nil { + return nil, fmt.Errorf("parse master key: %w", err) + } + key, ok := parsed.(ed25519.PrivateKey) + if !ok { + return nil, errors.New("stored master key is not ed25519") + } + return key, nil +} +``` + +`internal/master/store/agents.go`: + +```go +package store + +import ( + "context" + "errors" + "fmt" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgconn" + + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/protocol" +) + +const agentColumns = `id, name, hostname, agent_version, swarm_manager, last_seen_at, revoked_at, created_at` + +func scanAgent(row pgx.Row) (model.Agent, error) { + var a model.Agent + err := row.Scan(&a.ID, &a.Name, &a.Hostname, &a.AgentVersion, &a.SwarmManager, &a.LastSeenAt, &a.RevokedAt, &a.CreatedAt) + if errors.Is(err, pgx.ErrNoRows) { + return model.Agent{}, model.ErrNotFound + } + return a, err +} + +// CreateAgent registers an agent. Returns model.ErrConflict if the name is taken. +func (s *Store) CreateAgent(ctx context.Context, name, tokenHash string) (model.Agent, error) { + a, err := scanAgent(s.pool.QueryRow(ctx, + `INSERT INTO agents (name, token_hash) VALUES ($1, $2) RETURNING `+agentColumns, name, tokenHash)) + var pgErr *pgconn.PgError + if errors.As(err, &pgErr) && pgErr.Code == "23505" { + return model.Agent{}, model.ErrConflict + } + if err != nil { + return model.Agent{}, fmt.Errorf("create agent: %w", err) + } + return a, nil +} + +// ListAgents returns all agents, including revoked ones, ordered by name. +func (s *Store) ListAgents(ctx context.Context) ([]model.Agent, error) { + rows, err := s.pool.Query(ctx, `SELECT `+agentColumns+` FROM agents ORDER BY name`) + if err != nil { + return nil, fmt.Errorf("list agents: %w", err) + } + defer rows.Close() + var out []model.Agent + for rows.Next() { + a, err := scanAgent(rows) + if err != nil { + return nil, fmt.Errorf("scan agent: %w", err) + } + out = append(out, a) + } + return out, rows.Err() +} + +// AgentByTokenHash finds the agent a token belongs to. Revoked agents are +// returned too; the caller decides. Returns model.ErrNotFound if none. +func (s *Store) AgentByTokenHash(ctx context.Context, hash string) (model.Agent, error) { + return scanAgent(s.pool.QueryRow(ctx, `SELECT `+agentColumns+` FROM agents WHERE token_hash = $1`, hash)) +} + +// RevokeAgent marks the agent revoked. Its token stops working immediately; +// closing a live connection is the hub's job. +func (s *Store) RevokeAgent(ctx context.Context, id int64, at time.Time) error { + if _, err := s.pool.Exec(ctx, + `UPDATE agents SET revoked_at = $2 WHERE id = $1 AND revoked_at IS NULL`, id, at); err != nil { + return fmt.Errorf("revoke agent: %w", err) + } + return nil +} + +// RecordHello stores what the agent reported about itself on connect. +func (s *Store) RecordHello(ctx context.Context, id int64, h protocol.Hello, at time.Time) error { + if _, err := s.pool.Exec(ctx, + `UPDATE agents SET hostname = $2, agent_version = $3, swarm_manager = $4, last_seen_at = $5 WHERE id = $1`, + id, h.Hostname, h.AgentVersion, h.SwarmManager, at); err != nil { + return fmt.Errorf("record hello: %w", err) + } + return nil +} + +// MarkSeen updates last_seen_at, called when an agent disconnects. +func (s *Store) MarkSeen(ctx context.Context, id int64, at time.Time) error { + if _, err := s.pool.Exec(ctx, `UPDATE agents SET last_seen_at = $2 WHERE id = $1`, id, at); err != nil { + return fmt.Errorf("mark agent seen: %w", err) + } + return nil +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `go test ./internal/master/... && go test -tags integration ./internal/master/store/` +Expected: PASS (integration tests SKIP if `DATABASE_URL` is unset — run them against the local Postgres above before committing). + +- [ ] **Step 5: Commit** + +```bash +git add go.mod go.sum internal/master/model internal/master/store +git commit -m "feat(master): postgres store with migrations, master identity and agents" +``` + +--- + +### Task 9: Store — services and sessions + +**Files:** +- Create: `internal/master/store/services.go`, `internal/master/store/sessions.go` +- Test: `internal/master/store/services_integration_test.go`, `internal/master/store/sessions_integration_test.go` + +**Interfaces:** +- Consumes: `model.*`, `model.LocationKey` (Task 8); `protocol.*` (Task 2); `imageref.Parse` (Task 1). +- Produces: + - `(*Store).ApplySnapshot(ctx, agentID int64, h protocol.Hello, services []protocol.Service, at time.Time) error` + - `(*Store).UpsertService(ctx, agentID int64, clusterID string, svc protocol.Service, at time.Time) error` + - `(*Store).RemoveService(ctx, agentID int64, clusterID, agentKey string, at time.Time) error` + - `(*Store).ListServices(ctx) ([]model.Service, error)` — not removed; Compose/containers first, by agent name then key; Swarm last. + - `(*Store).CreateSession(ctx, s model.Session) error` + - `(*Store).GetSession(ctx, id string, now time.Time) (model.Session, error)` — `ErrNotFound` if missing or expired + - `(*Store).DeleteSession(ctx, id string) error` + +Snapshot rules: every service in the snapshot is upserted (`removed_at` cleared). Afterwards, rows *in this agent's scope* that were not in the snapshot get `removed_at`. Scope: Compose and container rows with `agent_id = agentID`; Swarm rows with this cluster ID, only if the agent is a manager. Swarm services from a non-manager are ignored. + +- [ ] **Step 1: Write the failing tests** + +`services_integration_test.go`: + +```go +//go:build integration + +package store + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/protocol" +) + +var ( + t0 = time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC) + slusk = protocol.Service{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", WorkingDir: "/srv/slusk", Image: "gitea.shcizo.se/shcizo/slusk:1.4.2", Digest: "sha256:d1", Status: "running"} + adhoc = protocol.Service{Kind: protocol.KindContainer, ContainerName: "adhoc", Image: "redis:alpine", Status: "stopped"} + gitea = protocol.Service{Kind: protocol.KindSwarm, SwarmService: "gitea_gitea", Image: "gitea/gitea:1.27.3@sha256:87a6", Digest: "sha256:87a6", Status: "running"} + worker = protocol.Hello{Hostname: "node-2"} + manager = protocol.Hello{Hostname: "swarm-mgr", SwarmManager: true, SwarmClusterID: "c1"} +) + +func names(svcs []model.Service) []string { + var out []string + for _, s := range svcs { + out = append(out, s.DisplayName()) + } + return out +} + +func newAgent(t *testing.T, s *Store, name string) int64 { + t.Helper() + a, err := s.CreateAgent(context.Background(), name, "hash-"+name) + require.NoError(t, err) + return a.ID +} + +func TestApplySnapshot_InsertsAsManualWithAgentName(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + id := newAgent(t, s, "node-2") + + require.NoError(t, s.ApplySnapshot(ctx, id, worker, []protocol.Service{slusk}, t0)) + + got, err := s.ListServices(ctx) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, "manual", got[0].UpdateMode) + assert.Equal(t, "node-2", got[0].AgentName) + assert.Equal(t, "gitea.shcizo.se/shcizo/slusk", got[0].ImageRepo) + assert.Equal(t, "gitea.shcizo.se/shcizo/slusk:1.4.2", got[0].ActualRef) + assert.Equal(t, "sha256:d1", got[0].ActualDigest) + assert.Equal(t, "compose|"+itoa(id)+"|slusk|web", got[0].LocationKey) +} + +func TestApplySnapshot_RemovedWhileOfflineThenBack(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + id := newAgent(t, s, "node-2") + + require.NoError(t, s.ApplySnapshot(ctx, id, worker, []protocol.Service{slusk, adhoc}, t0)) + before, err := s.ListServices(ctx) + require.NoError(t, err) + + require.NoError(t, s.ApplySnapshot(ctx, id, worker, []protocol.Service{slusk}, t0.Add(time.Hour))) + got, err := s.ListServices(ctx) + require.NoError(t, err) + assert.Equal(t, []string{"slusk/web"}, names(got)) + + require.NoError(t, s.ApplySnapshot(ctx, id, worker, []protocol.Service{slusk, adhoc}, t0.Add(2*time.Hour))) + after, err := s.ListServices(ctx) + require.NoError(t, err) + require.Len(t, after, 2) + for i := range before { + assert.Equal(t, before[i].ID, after[i].ID, "a returning service keeps its row") + assert.True(t, before[i].FirstSeenAt.Equal(after[i].FirstSeenAt)) + } +} + +func TestApplySnapshot_TwoManagersShareSwarmRowsAndKeepOwnCompose(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + a := newAgent(t, s, "swarm-mgr") + b := newAgent(t, s, "swarm-mgr-2") + + require.NoError(t, s.ApplySnapshot(ctx, a, manager, []protocol.Service{gitea, slusk}, t0)) + require.NoError(t, s.ApplySnapshot(ctx, b, manager, []protocol.Service{gitea}, t0)) + + got, err := s.ListServices(ctx) + require.NoError(t, err) + assert.ElementsMatch(t, []string{"slusk/web", "gitea_gitea"}, names(got), + "one swarm row, and b's snapshot must not remove a's compose service") +} + +func TestApplySnapshot_IgnoresSwarmFromWorker(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + id := newAgent(t, s, "node-2") + require.NoError(t, s.ApplySnapshot(ctx, id, worker, []protocol.Service{gitea}, t0)) + got, err := s.ListServices(ctx) + require.NoError(t, err) + assert.Empty(t, got) +} + +func TestUpsertAndRemoveService(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + id := newAgent(t, s, "swarm-mgr") + + require.NoError(t, s.UpsertService(ctx, id, "c1", gitea, t0)) + changed := gitea + changed.Status = "degraded" + require.NoError(t, s.UpsertService(ctx, id, "c1", changed, t0.Add(time.Minute))) + + got, err := s.ListServices(ctx) + require.NoError(t, err) + require.Len(t, got, 1) + assert.Equal(t, "degraded", got[0].Status) + assert.Equal(t, "swarm|c1|gitea_gitea", got[0].LocationKey) + + require.NoError(t, s.RemoveService(ctx, id, "c1", gitea.Key(), t0.Add(2*time.Minute))) + got, err = s.ListServices(ctx) + require.NoError(t, err) + assert.Empty(t, got) +} + +func TestListServices_SwarmLast(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + id := newAgent(t, s, "swarm-mgr") + require.NoError(t, s.ApplySnapshot(ctx, id, manager, []protocol.Service{gitea, slusk}, t0)) + got, err := s.ListServices(ctx) + require.NoError(t, err) + assert.Equal(t, []string{"slusk/web", "gitea_gitea"}, names(got)) +} +``` + +Add at the bottom of `services_integration_test.go`: + +```go +func itoa(n int64) string { return strconv.FormatInt(n, 10) } +``` + +and `"strconv"` to its imports. + +`sessions_integration_test.go`: + +```go +//go:build integration + +package store + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/master/model" +) + +func TestSessions(t *testing.T) { + s := openTestStore(t) + ctx := context.Background() + now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC) + sess := model.Session{ID: "s1", Subject: "sub", Name: "Samuel", Email: "s@example.com", CSRFToken: "c1", ExpiresAt: now.Add(time.Hour)} + + require.NoError(t, s.CreateSession(ctx, sess)) + got, err := s.GetSession(ctx, "s1", now) + require.NoError(t, err) + assert.Equal(t, "Samuel", got.Name) + assert.Equal(t, "c1", got.CSRFToken) + + _, err = s.GetSession(ctx, "s1", now.Add(2*time.Hour)) + assert.ErrorIs(t, err, model.ErrNotFound, "expired sessions are not returned") + + require.NoError(t, s.DeleteSession(ctx, "s1")) + _, err = s.GetSession(ctx, "s1", now) + assert.ErrorIs(t, err, model.ErrNotFound) +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `go test -tags integration ./internal/master/store/` +Expected: FAIL — undefined methods. + +- [ ] **Step 3: Write the implementation** + +`services.go`: + +```go +package store + +import ( + "context" + "fmt" + "time" + + "github.com/jackc/pgx/v5/pgconn" + + "github.com/shcizo/package-updater/internal/imageref" + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/protocol" +) + +type execer interface { + Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error) +} + +const upsertServiceSQL = ` +INSERT INTO services ( + kind, location_key, agent_id, compose_project, compose_service, working_dir, + swarm_cluster_id, swarm_service, container_name, image_repo, actual_ref, actual_digest, + status, self, first_seen_at, updated_at) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $15) +ON CONFLICT (location_key) DO UPDATE SET + agent_id = EXCLUDED.agent_id, + working_dir = EXCLUDED.working_dir, + image_repo = EXCLUDED.image_repo, + actual_ref = EXCLUDED.actual_ref, + actual_digest = EXCLUDED.actual_digest, + status = EXCLUDED.status, + self = EXCLUDED.self, + updated_at = EXCLUDED.updated_at, + removed_at = NULL` + +func upsertService(ctx context.Context, db execer, agentID int64, clusterID string, svc protocol.Service, at time.Time) error { + clusterCol := "" + if svc.Kind == protocol.KindSwarm { + clusterCol = clusterID + } + _, err := db.Exec(ctx, upsertServiceSQL, + string(svc.Kind), model.LocationKey(agentID, clusterID, svc.Key()), agentID, + svc.ComposeProject, svc.ComposeService, svc.WorkingDir, + clusterCol, svc.SwarmService, svc.ContainerName, + imageref.Parse(svc.Image).Repo, svc.Image, svc.Digest, + svc.Status, svc.Self, at) + if err != nil { + return fmt.Errorf("upsert service %s: %w", svc.Key(), err) + } + return nil +} + +// ApplySnapshot makes the stored inventory for this agent's scope equal to +// services: everything listed is upserted, everything in scope but not +// listed is marked removed. Scope is the agent's own Compose services and +// containers, plus the cluster's Swarm services if the agent is a manager. +func (s *Store) ApplySnapshot(ctx context.Context, agentID int64, h protocol.Hello, services []protocol.Service, at time.Time) error { + tx, err := s.pool.Begin(ctx) + if err != nil { + return fmt.Errorf("begin snapshot: %w", err) + } + defer func() { _ = tx.Rollback(ctx) }() + + keys := make([]string, 0, len(services)) + for _, svc := range services { + if svc.Kind == protocol.KindSwarm && !h.SwarmManager { + continue + } + if err := upsertService(ctx, tx, agentID, h.SwarmClusterID, svc, at); err != nil { + return err + } + keys = append(keys, model.LocationKey(agentID, h.SwarmClusterID, svc.Key())) + } + if _, err := tx.Exec(ctx, ` + UPDATE services SET removed_at = $1 + WHERE removed_at IS NULL + AND NOT (location_key = ANY($2)) + AND ((kind IN ('compose', 'container') AND agent_id = $3) + OR (kind = 'swarm' AND $4 AND swarm_cluster_id = $5))`, + at, keys, agentID, h.SwarmManager, h.SwarmClusterID); err != nil { + return fmt.Errorf("mark removed services: %w", err) + } + if err := tx.Commit(ctx); err != nil { + return fmt.Errorf("commit snapshot: %w", err) + } + return nil +} + +// UpsertService stores one changed or new service. +func (s *Store) UpsertService(ctx context.Context, agentID int64, clusterID string, svc protocol.Service, at time.Time) error { + return upsertService(ctx, s.pool, agentID, clusterID, svc, at) +} + +// RemoveService marks the service with the agent-relative key as removed. +func (s *Store) RemoveService(ctx context.Context, agentID int64, clusterID, agentKey string, at time.Time) error { + if _, err := s.pool.Exec(ctx, + `UPDATE services SET removed_at = $2 WHERE location_key = $1 AND removed_at IS NULL`, + model.LocationKey(agentID, clusterID, agentKey), at); err != nil { + return fmt.Errorf("remove service: %w", err) + } + return nil +} + +// ListServices returns all services not marked removed: machine-bound +// services grouped by agent name first, Swarm services last. +func (s *Store) ListServices(ctx context.Context) ([]model.Service, error) { + rows, err := s.pool.Query(ctx, ` + SELECT s.id, s.kind, s.location_key, s.agent_id, COALESCE(a.name, ''), + s.compose_project, s.compose_service, s.working_dir, s.swarm_cluster_id, + s.swarm_service, s.container_name, s.image_repo, s.actual_ref, s.actual_digest, + s.status, s.self, s.update_mode, s.first_seen_at, s.updated_at + FROM services s LEFT JOIN agents a ON a.id = s.agent_id + WHERE s.removed_at IS NULL + ORDER BY s.kind = 'swarm', COALESCE(a.name, ''), s.location_key`) + if err != nil { + return nil, fmt.Errorf("list services: %w", err) + } + defer rows.Close() + var out []model.Service + for rows.Next() { + var m model.Service + var kind string + if err := rows.Scan(&m.ID, &kind, &m.LocationKey, &m.AgentID, &m.AgentName, + &m.ComposeProject, &m.ComposeService, &m.WorkingDir, &m.SwarmClusterID, + &m.SwarmService, &m.ContainerName, &m.ImageRepo, &m.ActualRef, &m.ActualDigest, + &m.Status, &m.Self, &m.UpdateMode, &m.FirstSeenAt, &m.UpdatedAt); err != nil { + return nil, fmt.Errorf("scan service: %w", err) + } + m.Kind = protocol.Kind(kind) + out = append(out, m) + } + return out, rows.Err() +} +``` + +`sessions.go`: + +```go +package store + +import ( + "context" + "errors" + "fmt" + "time" + + "github.com/jackc/pgx/v5" + + "github.com/shcizo/package-updater/internal/master/model" +) + +// CreateSession stores a new session and deletes expired ones, so the table +// never grows without bound. +func (s *Store) CreateSession(ctx context.Context, sess model.Session) error { + if _, err := s.pool.Exec(ctx, `DELETE FROM sessions WHERE expires_at < now()`); err != nil { + return fmt.Errorf("delete expired sessions: %w", err) + } + if _, err := s.pool.Exec(ctx, + `INSERT INTO sessions (id, subject, name, email, csrf_token, expires_at) VALUES ($1, $2, $3, $4, $5, $6)`, + sess.ID, sess.Subject, sess.Name, sess.Email, sess.CSRFToken, sess.ExpiresAt); err != nil { + return fmt.Errorf("create session: %w", err) + } + return nil +} + +// GetSession returns the session unless it is missing or expired at now, +// in which case it returns model.ErrNotFound. +func (s *Store) GetSession(ctx context.Context, id string, now time.Time) (model.Session, error) { + var m model.Session + err := s.pool.QueryRow(ctx, + `SELECT id, subject, name, email, csrf_token, expires_at FROM sessions WHERE id = $1 AND expires_at > $2`, + id, now).Scan(&m.ID, &m.Subject, &m.Name, &m.Email, &m.CSRFToken, &m.ExpiresAt) + if errors.Is(err, pgx.ErrNoRows) { + return model.Session{}, model.ErrNotFound + } + if err != nil { + return model.Session{}, fmt.Errorf("get session: %w", err) + } + return m, nil +} + +// DeleteSession removes a session (logout). +func (s *Store) DeleteSession(ctx context.Context, id string) error { + if _, err := s.pool.Exec(ctx, `DELETE FROM sessions WHERE id = $1`, id); err != nil { + return fmt.Errorf("delete session: %w", err) + } + return nil +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `go test ./... && go test -tags integration ./internal/master/store/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/master/store +git commit -m "feat(master): store service inventory snapshots and web sessions" +``` + +--- + +### Task 10: Change broker + +**Files:** +- Create: `internal/master/events/broker.go` +- Test: `internal/master/events/broker_test.go` + +**Interfaces:** +- Consumes: nothing. +- Produces: `type Topic string` with `Services`, `Agents`; `func NewBroker() *Broker`; `func (b *Broker) Subscribe() (<-chan Topic, func())`; `func (b *Broker) Publish(t Topic)`. + +- [ ] **Step 1: Write the failing test** + +```go +package events + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestBroker_DeliversToAllSubscribers(t *testing.T) { + b := NewBroker() + a, cancelA := b.Subscribe() + defer cancelA() + c, cancelC := b.Subscribe() + defer cancelC() + + b.Publish(Services) + assert.Equal(t, Services, <-a) + assert.Equal(t, Services, <-c) +} + +func TestBroker_CancelStopsDelivery(t *testing.T) { + b := NewBroker() + ch, cancel := b.Subscribe() + cancel() + cancel() // safe to call twice + b.Publish(Agents) + assert.Empty(t, ch) +} + +func TestBroker_SlowSubscriberDoesNotBlockPublish(t *testing.T) { + b := NewBroker() + _, cancel := b.Subscribe() + defer cancel() + for range 1000 { + b.Publish(Services) // must return even though nobody reads + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/master/events/` +Expected: FAIL — undefined identifiers. + +- [ ] **Step 3: Write minimal implementation** + +```go +// Package events fans out "something changed" signals from the agent hub to +// open browser streams. +package events + +import "sync" + +// Topic names what changed. Subscribers refetch that part of the UI. +type Topic string + +// Topics. +const ( + Services Topic = "services" + Agents Topic = "agents" +) + +// Broker is an in-process publish/subscribe hub. +type Broker struct { + mu sync.Mutex + subs map[chan Topic]struct{} +} + +// NewBroker returns an empty Broker. +func NewBroker() *Broker { + return &Broker{subs: make(map[chan Topic]struct{})} +} + +// Subscribe returns a channel of topics and a cancel function that must be +// called when the subscriber goes away. Cancel is safe to call twice. +func (b *Broker) Subscribe() (<-chan Topic, func()) { + ch := make(chan Topic, 16) + b.mu.Lock() + b.subs[ch] = struct{}{} + b.mu.Unlock() + var once sync.Once + return ch, func() { + once.Do(func() { + b.mu.Lock() + delete(b.subs, ch) + b.mu.Unlock() + }) + } +} + +// Publish signals t to every subscriber. A subscriber whose buffer is full is +// skipped rather than blocking the agent hub; it already has signals queued +// and will refetch current state when it processes them. +func (b *Broker) Publish(t Topic) { + b.mu.Lock() + defer b.mu.Unlock() + for ch := range b.subs { + select { + case ch <- t: + default: + } + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `go test -race ./internal/master/events/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/master/events +git commit -m "feat(master): in-process broker for live UI change signals" +``` + +--- + +### Task 11: Agent hub + +**Files:** +- Create: `internal/master/hub/hub.go` +- Test: `internal/master/hub/hub_test.go` + +**Interfaces:** +- Consumes: `agenttoken.Hash` (Task 3); `protocol.*` (Task 2); `model.Agent`, `model.ErrNotFound` (Task 8); `events.Topic`, `events.Services`, `events.Agents` (Task 10). +- Produces: + - `type Store interface { AgentByTokenHash; RecordHello; ApplySnapshot; UpsertService; RemoveService; MarkSeen }` (signatures as in Tasks 8–9; `*store.Store` satisfies it) + - `type Publisher interface { Publish(events.Topic) }` + - `func New(store Store, pub Publisher, log *slog.Logger) *Hub` + - `func (h *Hub) ServeHTTP(w http.ResponseWriter, r *http.Request)` — mount at `GET /agent` + - `func (h *Hub) Online(agentID int64) bool` + - `func (h *Hub) Disconnect(agentID int64)` + +Behaviour: missing/unknown/revoked token → `401` before upgrading. First message must be `hello`, otherwise the connection is closed with a policy violation. A new connection for the same agent replaces the old one (old one closed). On disconnect of the current connection: `MarkSeen` and publish `Agents`. Hello publishes `Agents`; snapshot/upsert/remove publish `Services`. Swarm services from a non-manager agent are ignored. + +- [ ] **Step 1: Write the failing test** + +```go +package hub + +import ( + "context" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + + "github.com/coder/websocket" + "github.com/coder/websocket/wsjson" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/agenttoken" + "github.com/shcizo/package-updater/internal/master/events" + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/protocol" +) + +type fakeStore struct { + mu sync.Mutex + agents map[string]model.Agent // by token hash + hellos []protocol.Hello + snapshots [][]protocol.Service + upserts []string // "clusterID|key" + removes []string // "clusterID|key" + seen []int64 +} + +func newFakeStore(agents ...model.Agent) *fakeStore { + f := &fakeStore{agents: map[string]model.Agent{}} + for _, a := range agents { + f.agents[agenttoken.Hash("pua_"+a.Name)] = a + } + return f +} + +func (f *fakeStore) AgentByTokenHash(_ context.Context, hash string) (model.Agent, error) { + f.mu.Lock() + defer f.mu.Unlock() + a, ok := f.agents[hash] + if !ok { + return model.Agent{}, model.ErrNotFound + } + return a, nil +} + +func (f *fakeStore) RecordHello(_ context.Context, _ int64, h protocol.Hello, _ time.Time) error { + f.mu.Lock() + defer f.mu.Unlock() + f.hellos = append(f.hellos, h) + return nil +} + +func (f *fakeStore) ApplySnapshot(_ context.Context, _ int64, _ protocol.Hello, s []protocol.Service, _ time.Time) error { + f.mu.Lock() + defer f.mu.Unlock() + f.snapshots = append(f.snapshots, s) + return nil +} + +func (f *fakeStore) UpsertService(_ context.Context, _ int64, cluster string, s protocol.Service, _ time.Time) error { + f.mu.Lock() + defer f.mu.Unlock() + f.upserts = append(f.upserts, cluster+"|"+s.Key()) + return nil +} + +func (f *fakeStore) RemoveService(_ context.Context, _ int64, cluster, key string, _ time.Time) error { + f.mu.Lock() + defer f.mu.Unlock() + f.removes = append(f.removes, cluster+"|"+key) + return nil +} + +func (f *fakeStore) MarkSeen(_ context.Context, id int64, _ time.Time) error { + f.mu.Lock() + defer f.mu.Unlock() + f.seen = append(f.seen, id) + return nil +} + +func (f *fakeStore) count(get func(*fakeStore) int) func() int { + return func() int { + f.mu.Lock() + defer f.mu.Unlock() + return get(f) + } +} + +type fakePub struct { + mu sync.Mutex + topics []events.Topic +} + +func (p *fakePub) Publish(t events.Topic) { + p.mu.Lock() + defer p.mu.Unlock() + p.topics = append(p.topics, t) +} + +func (p *fakePub) has(t events.Topic) bool { + p.mu.Lock() + defer p.mu.Unlock() + for _, x := range p.topics { + if x == t { + return true + } + } + return false +} + +var ( + node2 = model.Agent{ID: 2, Name: "node-2"} + mgr = model.Agent{ID: 1, Name: "swarm-mgr"} + revoked = model.Agent{ID: 3, Name: "old", RevokedAt: ptr(time.Now())} +) + +func ptr[T any](v T) *T { return &v } + +func newHub(t *testing.T, store *fakeStore) (*Hub, *fakePub, *httptest.Server) { + t.Helper() + pub := &fakePub{} + h := New(store, pub, slog.New(slog.NewTextHandler(io.Discard, nil))) + srv := httptest.NewServer(h) + t.Cleanup(srv.Close) + return h, pub, srv +} + +func dial(t *testing.T, srv *httptest.Server, token string) (*websocket.Conn, *http.Response, error) { + t.Helper() + header := http.Header{} + if token != "" { + header.Set("Authorization", "Bearer "+token) + } + return websocket.Dial(context.Background(), "ws"+strings.TrimPrefix(srv.URL, "http"), + &websocket.DialOptions{HTTPHeader: header}) +} + +func send(t *testing.T, c *websocket.Conn, typ string, payload any) { + t.Helper() + env, err := protocol.Encode(typ, payload) + require.NoError(t, err) + require.NoError(t, wsjson.Write(context.Background(), c, env)) +} + +func TestHub_RejectsBadTokens(t *testing.T) { + _, _, srv := newHub(t, newFakeStore(node2, revoked)) + for name, token := range map[string]string{"missing": "", "unknown": "pua_nobody", "revoked": "pua_old"} { + t.Run(name, func(t *testing.T) { + _, resp, err := dial(t, srv, token) + require.Error(t, err) + require.NotNil(t, resp) + assert.Equal(t, http.StatusUnauthorized, resp.StatusCode) + }) + } +} + +func TestHub_HelloAndSnapshotReachStore(t *testing.T) { + store := newFakeStore(node2) + h, pub, srv := newHub(t, store) + c, _, err := dial(t, srv, "pua_node-2") + require.NoError(t, err) + defer c.CloseNow() + + send(t, c, protocol.TypeHello, protocol.Hello{Hostname: "node-2"}) + send(t, c, protocol.TypeSnapshot, protocol.Snapshot{Services: []protocol.Service{ + {Kind: protocol.KindContainer, ContainerName: "a", Image: "a:1", Status: "running"}, + }}) + + require.Eventually(t, func() bool { return store.count(func(f *fakeStore) int { return len(f.snapshots) })() == 1 }, time.Second, 5*time.Millisecond) + assert.True(t, h.Online(node2.ID)) + assert.True(t, pub.has(events.Agents)) + assert.True(t, pub.has(events.Services)) +} + +func TestHub_LargeSnapshotArrivesIntact(t *testing.T) { + store := newFakeStore(node2) + _, _, srv := newHub(t, store) + c, _, err := dial(t, srv, "pua_node-2") + require.NoError(t, err) + defer c.CloseNow() + + services := make([]protocol.Service, 500) + for i := range services { + services[i] = protocol.Service{ + Kind: protocol.KindContainer, ContainerName: strings.Repeat("x", 20) + string(rune('a'+i%26)) + strings.Repeat("y", i%7), + Image: "registry.example.com/some/long/image/name:v1.2.3@sha256:" + strings.Repeat("0", 64), Status: "running", + } + } + send(t, c, protocol.TypeHello, protocol.Hello{Hostname: "node-2"}) + send(t, c, protocol.TypeSnapshot, protocol.Snapshot{Services: services}) + + require.Eventually(t, func() bool { return store.count(func(f *fakeStore) int { return len(f.snapshots) })() == 1 }, time.Second, 5*time.Millisecond) + store.mu.Lock() + defer store.mu.Unlock() + assert.Len(t, store.snapshots[0], 500) +} + +func TestHub_UpsertAndRemoveCarryClusterID(t *testing.T) { + store := newFakeStore(mgr) + _, _, srv := newHub(t, store) + c, _, err := dial(t, srv, "pua_swarm-mgr") + require.NoError(t, err) + defer c.CloseNow() + + gitea := protocol.Service{Kind: protocol.KindSwarm, SwarmService: "gitea_gitea", Image: "gitea/gitea:1", Status: "running"} + send(t, c, protocol.TypeHello, protocol.Hello{Hostname: "swarm-mgr", SwarmManager: true, SwarmClusterID: "c1"}) + send(t, c, protocol.TypeUpsert, protocol.Upsert{Service: gitea}) + send(t, c, protocol.TypeRemove, protocol.Remove{Key: gitea.Key()}) + + require.Eventually(t, func() bool { return store.count(func(f *fakeStore) int { return len(f.removes) })() == 1 }, time.Second, 5*time.Millisecond) + store.mu.Lock() + defer store.mu.Unlock() + assert.Equal(t, []string{"c1|swarm|gitea_gitea"}, store.upserts) + assert.Equal(t, []string{"c1|swarm|gitea_gitea"}, store.removes) +} + +func TestHub_IgnoresSwarmUpsertFromWorker(t *testing.T) { + store := newFakeStore(node2) + _, _, srv := newHub(t, store) + c, _, err := dial(t, srv, "pua_node-2") + require.NoError(t, err) + defer c.CloseNow() + + send(t, c, protocol.TypeHello, protocol.Hello{Hostname: "node-2"}) + send(t, c, protocol.TypeUpsert, protocol.Upsert{Service: protocol.Service{Kind: protocol.KindSwarm, SwarmService: "x"}}) + send(t, c, protocol.TypeUpsert, protocol.Upsert{Service: protocol.Service{Kind: protocol.KindContainer, ContainerName: "y"}}) + + require.Eventually(t, func() bool { return store.count(func(f *fakeStore) int { return len(f.upserts) })() == 1 }, time.Second, 5*time.Millisecond) + store.mu.Lock() + defer store.mu.Unlock() + assert.Equal(t, []string{"|container|y"}, store.upserts) +} + +func TestHub_ClosesConnectionThatSkipsHello(t *testing.T) { + store := newFakeStore(node2) + _, _, srv := newHub(t, store) + c, _, err := dial(t, srv, "pua_node-2") + require.NoError(t, err) + defer c.CloseNow() + + send(t, c, protocol.TypeSnapshot, protocol.Snapshot{}) + _, _, err = c.Read(context.Background()) + assert.Equal(t, websocket.StatusPolicyViolation, websocket.CloseStatus(err)) +} + +func TestHub_NewConnectionReplacesOld(t *testing.T) { + store := newFakeStore(node2) + h, _, srv := newHub(t, store) + first, _, err := dial(t, srv, "pua_node-2") + require.NoError(t, err) + defer first.CloseNow() + send(t, first, protocol.TypeHello, protocol.Hello{Hostname: "node-2"}) + require.Eventually(t, func() bool { return h.Online(node2.ID) }, time.Second, 5*time.Millisecond) + + second, _, err := dial(t, srv, "pua_node-2") + require.NoError(t, err) + defer second.CloseNow() + + _, _, err = first.Read(context.Background()) + assert.Error(t, err, "the replaced connection must be closed") + assert.True(t, h.Online(node2.ID), "the agent stays online on its new connection") +} + +func TestHub_DisconnectCutsLiveConnection(t *testing.T) { + store := newFakeStore(node2) + h, pub, srv := newHub(t, store) + c, _, err := dial(t, srv, "pua_node-2") + require.NoError(t, err) + defer c.CloseNow() + send(t, c, protocol.TypeHello, protocol.Hello{Hostname: "node-2"}) + require.Eventually(t, func() bool { return h.Online(node2.ID) }, time.Second, 5*time.Millisecond) + + h.Disconnect(node2.ID) + + _, _, err = c.Read(context.Background()) + assert.Error(t, err) + require.Eventually(t, func() bool { return !h.Online(node2.ID) }, time.Second, 5*time.Millisecond) + require.Eventually(t, func() bool { return store.count(func(f *fakeStore) int { return len(f.seen) })() == 1 }, time.Second, 5*time.Millisecond) + assert.True(t, pub.has(events.Agents)) +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/master/hub/` +Expected: FAIL — undefined identifiers. + +- [ ] **Step 3: Write minimal implementation** + +```go +// Package hub accepts agent connections, authenticates them, and turns their +// messages into stored inventory and live UI signals. +package hub + +import ( + "context" + "errors" + "fmt" + "log/slog" + "net/http" + "strings" + "sync" + "time" + + "github.com/coder/websocket" + "github.com/coder/websocket/wsjson" + + "github.com/shcizo/package-updater/internal/agenttoken" + "github.com/shcizo/package-updater/internal/master/events" + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/protocol" +) + +// readLimit caps one incoming message. The library default of 32 KiB is too +// small for a full snapshot from a busy machine. +const readLimit = 1 << 20 + +// Store is the storage the hub writes to. *store.Store satisfies it. +type Store interface { + AgentByTokenHash(ctx context.Context, hash string) (model.Agent, error) + RecordHello(ctx context.Context, agentID int64, h protocol.Hello, at time.Time) error + ApplySnapshot(ctx context.Context, agentID int64, h protocol.Hello, services []protocol.Service, at time.Time) error + UpsertService(ctx context.Context, agentID int64, clusterID string, svc protocol.Service, at time.Time) error + RemoveService(ctx context.Context, agentID int64, clusterID, agentKey string, at time.Time) error + MarkSeen(ctx context.Context, agentID int64, at time.Time) error +} + +// Publisher receives "something changed" signals. *events.Broker satisfies it. +type Publisher interface { + Publish(t events.Topic) +} + +// Hub is the master's endpoint for agents. +type Hub struct { + store Store + pub Publisher + log *slog.Logger + now func() time.Time + + mu sync.Mutex + conns map[int64]*websocket.Conn +} + +// New returns a Hub. Mount it at GET /agent on the TLS listener. +func New(store Store, pub Publisher, log *slog.Logger) *Hub { + return &Hub{store: store, pub: pub, log: log, now: time.Now, conns: make(map[int64]*websocket.Conn)} +} + +// ServeHTTP authenticates the agent's token and runs its connection. +func (h *Hub) ServeHTTP(w http.ResponseWriter, r *http.Request) { + token, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ") + if !ok || token == "" { + http.Error(w, "unauthorized", http.StatusUnauthorized) + return + } + agent, err := h.store.AgentByTokenHash(r.Context(), agenttoken.Hash(token)) + if err != nil && !errors.Is(err, model.ErrNotFound) { + h.log.Error("agent_lookup_failed", "error", err) + } + if err != nil || agent.RevokedAt != nil { + http.Error(w, "unauthorized", http.StatusUnauthorized) + return + } + + c, err := websocket.Accept(w, r, nil) + if err != nil { + return + } + c.SetReadLimit(readLimit) + h.register(agent.ID, c) + defer h.unregister(agent.ID, c) + + s := &session{hub: h, agent: agent} + err = s.run(r.Context(), c) + h.log.Info("agent_disconnected", "agent", agent.Name, "reason", err) +} + +// Online reports whether the agent has a live connection. +func (h *Hub) Online(agentID int64) bool { + h.mu.Lock() + defer h.mu.Unlock() + _, ok := h.conns[agentID] + return ok +} + +// Disconnect closes the agent's live connection, if any. Call it after +// revoking the agent so the revocation takes effect immediately. +func (h *Hub) Disconnect(agentID int64) { + h.mu.Lock() + c := h.conns[agentID] + h.mu.Unlock() + if c != nil { + c.CloseNow() + } +} + +func (h *Hub) register(id int64, c *websocket.Conn) { + h.mu.Lock() + old := h.conns[id] + h.conns[id] = c + h.mu.Unlock() + if old != nil { + old.CloseNow() + } +} + +func (h *Hub) unregister(id int64, c *websocket.Conn) { + h.mu.Lock() + current := h.conns[id] == c + if current { + delete(h.conns, id) + } + h.mu.Unlock() + if !current { + return // replaced by a newer connection; that one owns the state + } + if err := h.store.MarkSeen(context.Background(), id, h.now()); err != nil { + h.log.Error("mark_seen_failed", "agent_id", id, "error", err) + } + h.pub.Publish(events.Agents) +} + +// session is one agent connection's state. +type session struct { + hub *Hub + agent model.Agent + hello *protocol.Hello +} + +func (s *session) run(ctx context.Context, c *websocket.Conn) error { + for { + var env protocol.Envelope + if err := wsjson.Read(ctx, c, &env); err != nil { + return err + } + if s.hello == nil && env.Type != protocol.TypeHello { + _ = c.Close(websocket.StatusPolicyViolation, "hello required first") + return fmt.Errorf("%q before hello", env.Type) + } + if err := s.handle(ctx, env); err != nil { + _ = c.Close(websocket.StatusPolicyViolation, "bad message") + return err + } + } +} + +func (s *session) handle(ctx context.Context, env protocol.Envelope) error { + h := s.hub + switch env.Type { + case protocol.TypeHello: + v, err := protocol.Decode[protocol.Hello](env) + if err != nil { + return err + } + if err := h.store.RecordHello(ctx, s.agent.ID, v, h.now()); err != nil { + return err + } + s.hello = &v + h.log.Info("agent_connected", "agent", s.agent.Name, "hostname", v.Hostname, "swarm_manager", v.SwarmManager) + h.pub.Publish(events.Agents) + case protocol.TypeSnapshot: + v, err := protocol.Decode[protocol.Snapshot](env) + if err != nil { + return err + } + if err := h.store.ApplySnapshot(ctx, s.agent.ID, *s.hello, v.Services, h.now()); err != nil { + return err + } + h.pub.Publish(events.Services) + case protocol.TypeUpsert: + v, err := protocol.Decode[protocol.Upsert](env) + if err != nil { + return err + } + if v.Service.Kind == protocol.KindSwarm && !s.hello.SwarmManager { + return nil + } + if err := h.store.UpsertService(ctx, s.agent.ID, s.hello.SwarmClusterID, v.Service, h.now()); err != nil { + return err + } + h.pub.Publish(events.Services) + case protocol.TypeRemove: + v, err := protocol.Decode[protocol.Remove](env) + if err != nil { + return err + } + if err := h.store.RemoveService(ctx, s.agent.ID, s.hello.SwarmClusterID, v.Key, h.now()); err != nil { + return err + } + h.pub.Publish(events.Services) + default: + return fmt.Errorf("unknown message type %q", env.Type) + } + return nil +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `go test -race ./internal/master/hub/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/master/hub +git commit -m "feat(master): agent hub that authenticates agents and stores their inventory" +``` + +--- + +### Task 12: Web authentication — OIDC, sessions, CSRF + +**Files:** +- Create: `internal/master/webauth/provider.go`, `internal/master/webauth/auth.go` +- Test: `internal/master/webauth/auth_test.go` + +**Interfaces:** +- Consumes: `model.Session`, `model.ErrNotFound` (Task 8). +- Produces: + - `type Claims struct { Subject, Name, Email string; Groups []string }` + - `type Provider interface { AuthCodeURL(state, verifier string) string; Exchange(ctx, code, verifier string) (Claims, error) }` + - `type OIDCConfig struct { Issuer, ClientID, ClientSecret, RedirectURL string }`, `func NewOIDCProvider(ctx, cfg OIDCConfig) (Provider, error)` + - `type Store interface { CreateSession(ctx, model.Session) error; GetSession(ctx, id string, now time.Time) (model.Session, error); DeleteSession(ctx, id string) error }` + - `func New(p Provider, s Store, allowedGroup string, secureCookies bool, log *slog.Logger) *Auth` + - Handlers `(*Auth).Login`, `(*Auth).Callback`, `(*Auth).Logout` (`http.HandlerFunc` signatures) + - Middleware `(*Auth).Require(next http.Handler) http.Handler` + - `func SessionFrom(ctx context.Context) (model.Session, bool)` + +Add the dependencies: `go get github.com/coreos/go-oidc/v3@latest golang.org/x/oauth2@latest`. + +- [ ] **Step 1: Write the failing test** + +```go +package webauth + +import ( + "context" + "errors" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/master/model" +) + +type fakeProvider struct { + claims Claims + err error + gotVer string +} + +func (p *fakeProvider) AuthCodeURL(state, _ string) string { + return "https://auth.example/authorize?state=" + url.QueryEscape(state) +} + +func (p *fakeProvider) Exchange(_ context.Context, _ string, verifier string) (Claims, error) { + p.gotVer = verifier + return p.claims, p.err +} + +type fakeStore struct { + mu sync.Mutex + sessions map[string]model.Session +} + +func newFakeStore() *fakeStore { return &fakeStore{sessions: map[string]model.Session{}} } + +func (f *fakeStore) CreateSession(_ context.Context, s model.Session) error { + f.mu.Lock() + defer f.mu.Unlock() + f.sessions[s.ID] = s + return nil +} + +func (f *fakeStore) GetSession(_ context.Context, id string, now time.Time) (model.Session, error) { + f.mu.Lock() + defer f.mu.Unlock() + s, ok := f.sessions[id] + if !ok || !s.ExpiresAt.After(now) { + return model.Session{}, model.ErrNotFound + } + return s, nil +} + +func (f *fakeStore) DeleteSession(_ context.Context, id string) error { + f.mu.Lock() + defer f.mu.Unlock() + delete(f.sessions, id) + return nil +} + +func newAuth(p *fakeProvider, s *fakeStore) *Auth { + return New(p, s, "updater-admins", true, slog.New(slog.NewTextHandler(io.Discard, nil))) +} + +// startLogin runs /auth/login and returns the flow cookie and the state. +func startLogin(t *testing.T, a *Auth) (*http.Cookie, string) { + t.Helper() + rec := httptest.NewRecorder() + a.Login(rec, httptest.NewRequest(http.MethodGet, "/auth/login", nil)) + require.Equal(t, http.StatusFound, rec.Code) + loc, err := url.Parse(rec.Header().Get("Location")) + require.NoError(t, err) + cookies := rec.Result().Cookies() + require.Len(t, cookies, 1) + return cookies[0], loc.Query().Get("state") +} + +func callback(a *Auth, flow *http.Cookie, state string) *httptest.ResponseRecorder { + req := httptest.NewRequest(http.MethodGet, "/auth/callback?code=c&state="+url.QueryEscape(state), nil) + req.AddCookie(flow) + rec := httptest.NewRecorder() + a.Callback(rec, req) + return rec +} + +func TestLogin_SetsFlowCookieAndRedirects(t *testing.T) { + a := newAuth(&fakeProvider{}, newFakeStore()) + flow, state := startLogin(t, a) + assert.True(t, flow.HttpOnly) + assert.True(t, flow.Secure) + assert.NotEmpty(t, state) +} + +func TestCallback_StateMismatchRejected(t *testing.T) { + a := newAuth(&fakeProvider{}, newFakeStore()) + flow, _ := startLogin(t, a) + rec := callback(a, flow, "forged") + assert.Equal(t, http.StatusBadRequest, rec.Code) +} + +func TestCallback_UserOutsideGroupForbidden(t *testing.T) { + store := newFakeStore() + a := newAuth(&fakeProvider{claims: Claims{Subject: "u", Groups: []string{"family"}}}, store) + flow, state := startLogin(t, a) + rec := callback(a, flow, state) + assert.Equal(t, http.StatusForbidden, rec.Code) + assert.Empty(t, store.sessions) +} + +func TestCallback_ExchangeFailure(t *testing.T) { + a := newAuth(&fakeProvider{err: errors.New("bad code")}, newFakeStore()) + flow, state := startLogin(t, a) + assert.Equal(t, http.StatusUnauthorized, callback(a, flow, state).Code) +} + +func TestCallback_CreatesSession(t *testing.T) { + store := newFakeStore() + p := &fakeProvider{claims: Claims{Subject: "u1", Name: "Samuel", Groups: []string{"updater-admins"}}} + a := newAuth(p, store) + flow, state := startLogin(t, a) + rec := callback(a, flow, state) + + require.Equal(t, http.StatusFound, rec.Code) + assert.Equal(t, "/", rec.Header().Get("Location")) + assert.NotEmpty(t, p.gotVer, "the PKCE verifier from the flow cookie must reach the exchange") + var sessionCookie *http.Cookie + for _, c := range rec.Result().Cookies() { + if c.Name == sessionCookieName { + sessionCookie = c + } + } + require.NotNil(t, sessionCookie) + assert.True(t, sessionCookie.HttpOnly) + assert.Equal(t, http.SameSiteLaxMode, sessionCookie.SameSite) + s, ok := store.sessions[sessionCookie.Value] + require.True(t, ok) + assert.Equal(t, "Samuel", s.Name) + assert.NotEmpty(t, s.CSRFToken) +} + +func protectedRequest(method, target string, sessionID string) *http.Request { + req := httptest.NewRequest(method, target, nil) + if sessionID != "" { + req.AddCookie(&http.Cookie{Name: sessionCookieName, Value: sessionID}) + } + return req +} + +func TestRequire(t *testing.T) { + store := newFakeStore() + store.sessions["live"] = model.Session{ID: "live", Name: "Samuel", CSRFToken: "tok", ExpiresAt: time.Now().Add(time.Hour)} + store.sessions["old"] = model.Session{ID: "old", CSRFToken: "tok", ExpiresAt: time.Now().Add(-time.Hour)} + a := newAuth(&fakeProvider{}, store) + + var gotName string + next := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + s, _ := SessionFrom(r.Context()) + gotName = s.Name + w.WriteHeader(http.StatusNoContent) + }) + h := a.Require(next) + + cases := []struct { + name string + req *http.Request + wantCode int + wantHeader [2]string + }{ + {"no session GET redirects to login", protectedRequest(http.MethodGet, "/", ""), http.StatusFound, [2]string{"Location", "/auth/login"}}, + {"expired session on htmx fragment", func() *http.Request { + r := protectedRequest(http.MethodGet, "/fragments/services", "old") + r.Header.Set("HX-Request", "true") + return r + }(), http.StatusUnauthorized, [2]string{"HX-Redirect", "/auth/login"}}, + {"no session POST", protectedRequest(http.MethodPost, "/agents", ""), http.StatusUnauthorized, [2]string{}}, + {"POST without CSRF", protectedRequest(http.MethodPost, "/agents", "live"), http.StatusForbidden, [2]string{}}, + {"POST with CSRF header", func() *http.Request { + r := protectedRequest(http.MethodPost, "/agents", "live") + r.Header.Set("X-CSRF-Token", "tok") + return r + }(), http.StatusNoContent, [2]string{}}, + {"POST with CSRF form field", func() *http.Request { + r := httptest.NewRequest(http.MethodPost, "/agents", strings.NewReader("csrf_token=tok")) + r.Header.Set("Content-Type", "application/x-www-form-urlencoded") + r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "live"}) + return r + }(), http.StatusNoContent, [2]string{}}, + {"GET with session", protectedRequest(http.MethodGet, "/", "live"), http.StatusNoContent, [2]string{}}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + rec := httptest.NewRecorder() + h.ServeHTTP(rec, c.req) + assert.Equal(t, c.wantCode, rec.Code) + if c.wantHeader[0] != "" { + assert.Equal(t, c.wantHeader[1], rec.Header().Get(c.wantHeader[0])) + } + }) + } + assert.Equal(t, "Samuel", gotName) +} + +func TestLogout_DeletesSession(t *testing.T) { + store := newFakeStore() + store.sessions["live"] = model.Session{ID: "live", CSRFToken: "tok", ExpiresAt: time.Now().Add(time.Hour)} + a := newAuth(&fakeProvider{}, store) + + req := protectedRequest(http.MethodPost, "/auth/logout", "live") + req.Header.Set("X-CSRF-Token", "tok") + rec := httptest.NewRecorder() + a.Require(http.HandlerFunc(a.Logout)).ServeHTTP(rec, req) + + assert.Equal(t, http.StatusOK, rec.Code) + assert.Empty(t, store.sessions) +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/master/webauth/` +Expected: FAIL — undefined identifiers. + +- [ ] **Step 3: Write the implementation** + +`provider.go`: + +```go +package webauth + +import ( + "context" + "errors" + "fmt" + + "github.com/coreos/go-oidc/v3/oidc" + "golang.org/x/oauth2" +) + +// Claims is what the master needs from the identity provider. +type Claims struct { + Subject string + Name string + Email string + Groups []string +} + +// Provider performs the OIDC authorization code flow with PKCE. Declared as +// an interface so handlers can be tested without a real identity provider. +type Provider interface { + AuthCodeURL(state, verifier string) string + Exchange(ctx context.Context, code, verifier string) (Claims, error) +} + +// OIDCConfig configures the real provider. +type OIDCConfig struct { + Issuer string // e.g. https://auth.example.com/application/o/package-updater/ + ClientID string + ClientSecret string + RedirectURL string // PUBLIC_URL + /auth/callback +} + +type oidcProvider struct { + oauth *oauth2.Config + verifier *oidc.IDTokenVerifier +} + +// NewOIDCProvider discovers the issuer's endpoints. It makes a network call, +// so the master fails at startup if Authentik is unreachable. +func NewOIDCProvider(ctx context.Context, cfg OIDCConfig) (Provider, error) { + p, err := oidc.NewProvider(ctx, cfg.Issuer) + if err != nil { + return nil, fmt.Errorf("oidc discovery: %w", err) + } + return &oidcProvider{ + oauth: &oauth2.Config{ + ClientID: cfg.ClientID, + ClientSecret: cfg.ClientSecret, + Endpoint: p.Endpoint(), + RedirectURL: cfg.RedirectURL, + Scopes: []string{oidc.ScopeOpenID, "profile", "email"}, + }, + verifier: p.Verifier(&oidc.Config{ClientID: cfg.ClientID}), + }, nil +} + +func (p *oidcProvider) AuthCodeURL(state, verifier string) string { + return p.oauth.AuthCodeURL(state, oauth2.S256ChallengeOption(verifier)) +} + +func (p *oidcProvider) Exchange(ctx context.Context, code, verifier string) (Claims, error) { + tok, err := p.oauth.Exchange(ctx, code, oauth2.VerifierOption(verifier)) + if err != nil { + return Claims{}, fmt.Errorf("exchange code: %w", err) + } + raw, ok := tok.Extra("id_token").(string) + if !ok { + return Claims{}, errors.New("token response has no id_token") + } + idt, err := p.verifier.Verify(ctx, raw) + if err != nil { + return Claims{}, fmt.Errorf("verify id_token: %w", err) + } + var c struct { + Name string `json:"name"` + Email string `json:"email"` + Groups []string `json:"groups"` + } + if err := idt.Claims(&c); err != nil { + return Claims{}, fmt.Errorf("read claims: %w", err) + } + return Claims{Subject: idt.Subject, Name: c.Name, Email: c.Email, Groups: c.Groups}, nil +} +``` + +`auth.go`: + +```go +// Package webauth logs people into the master UI through OIDC (Authentik), +// keeps their sessions, and protects state-changing requests with a CSRF +// token. +package webauth + +import ( + "context" + "crypto/rand" + "crypto/subtle" + "encoding/base64" + "errors" + "log/slog" + "net/http" + "slices" + "strings" + "time" + + "golang.org/x/oauth2" + + "github.com/shcizo/package-updater/internal/master/model" +) + +const ( + sessionCookieName = "pu_session" + flowCookieName = "pu_oidc_flow" + sessionTTL = 12 * time.Hour +) + +// Store keeps sessions. *store.Store satisfies it. +type Store interface { + CreateSession(ctx context.Context, s model.Session) error + GetSession(ctx context.Context, id string, now time.Time) (model.Session, error) + DeleteSession(ctx context.Context, id string) error +} + +// Auth holds the login handlers and the session middleware. +type Auth struct { + provider Provider + store Store + allowedGroup string + secure bool + log *slog.Logger + now func() time.Time +} + +// New returns an Auth. Only members of allowedGroup may log in. secureCookies +// should be true whenever the UI is served over HTTPS. +func New(p Provider, s Store, allowedGroup string, secureCookies bool, log *slog.Logger) *Auth { + return &Auth{provider: p, store: s, allowedGroup: allowedGroup, secure: secureCookies, log: log, now: time.Now} +} + +// Login starts the OIDC flow. State and PKCE verifier travel in a short-lived +// cookie scoped to /auth/; tampering with it can only break the user's own +// login. +func (a *Auth) Login(w http.ResponseWriter, r *http.Request) { + state := randomString() + verifier := oauth2.GenerateVerifier() + http.SetCookie(w, &http.Cookie{ + Name: flowCookieName, Value: state + "." + verifier, Path: "/auth/", + MaxAge: 600, HttpOnly: true, Secure: a.secure, SameSite: http.SameSiteLaxMode, + }) + http.Redirect(w, r, a.provider.AuthCodeURL(state, verifier), http.StatusFound) +} + +// Callback finishes the OIDC flow, checks group membership and creates the +// session. +func (a *Auth) Callback(w http.ResponseWriter, r *http.Request) { + flow, err := r.Cookie(flowCookieName) + if err != nil { + http.Error(w, "login flow expired, start again", http.StatusBadRequest) + return + } + state, verifier, ok := strings.Cut(flow.Value, ".") + if !ok || subtle.ConstantTimeCompare([]byte(r.URL.Query().Get("state")), []byte(state)) != 1 { + http.Error(w, "invalid login state", http.StatusBadRequest) + return + } + http.SetCookie(w, &http.Cookie{Name: flowCookieName, Path: "/auth/", MaxAge: -1}) + + claims, err := a.provider.Exchange(r.Context(), r.URL.Query().Get("code"), verifier) + if err != nil { + a.log.Warn("login_failed", "error", err) + http.Error(w, "login failed", http.StatusUnauthorized) + return + } + if !slices.Contains(claims.Groups, a.allowedGroup) { + a.log.Warn("login_forbidden", "subject", claims.Subject, "required_group", a.allowedGroup) + http.Error(w, "forbidden: you are not a member of the group allowed to use package-updater", http.StatusForbidden) + return + } + + sess := model.Session{ + ID: randomString(), Subject: claims.Subject, Name: claims.Name, Email: claims.Email, + CSRFToken: randomString(), ExpiresAt: a.now().Add(sessionTTL), + } + if err := a.store.CreateSession(r.Context(), sess); err != nil { + a.log.Error("create_session_failed", "error", err) + http.Error(w, "internal error", http.StatusInternalServerError) + return + } + http.SetCookie(w, &http.Cookie{ + Name: sessionCookieName, Value: sess.ID, Path: "/", Expires: sess.ExpiresAt, + HttpOnly: true, Secure: a.secure, SameSite: http.SameSiteLaxMode, + }) + a.log.Info("login", "subject", claims.Subject, "name", claims.Name) + http.Redirect(w, r, "/", http.StatusFound) +} + +// Logout ends the session. Mount behind Require so it is CSRF-protected. +func (a *Auth) Logout(w http.ResponseWriter, r *http.Request) { + if s, ok := SessionFrom(r.Context()); ok { + if err := a.store.DeleteSession(r.Context(), s.ID); err != nil { + a.log.Error("delete_session_failed", "error", err) + } + } + http.SetCookie(w, &http.Cookie{Name: sessionCookieName, Path: "/", MaxAge: -1}) + _, _ = w.Write([]byte("Logged out.")) +} + +// Require lets a request through only with a valid session, and for +// anything but GET/HEAD only with the session's CSRF token in the +// X-CSRF-Token header or the csrf_token form field. htmx requests without a +// session get 401 + HX-Redirect so the whole page navigates to the login, +// instead of the login page being swapped into a fragment. +func (a *Auth) Require(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + sess, err := a.session(r) + if err != nil && !errors.Is(err, model.ErrNotFound) { + a.log.Error("session_lookup_failed", "error", err) + http.Error(w, "internal error", http.StatusInternalServerError) + return + } + if err != nil { + switch { + case r.Header.Get("HX-Request") == "true": + w.Header().Set("HX-Redirect", "/auth/login") + w.WriteHeader(http.StatusUnauthorized) + case r.Method == http.MethodGet: + http.Redirect(w, r, "/auth/login", http.StatusFound) + default: + http.Error(w, "unauthorized", http.StatusUnauthorized) + } + return + } + if r.Method != http.MethodGet && r.Method != http.MethodHead { + token := r.Header.Get("X-CSRF-Token") + if token == "" { + token = r.FormValue("csrf_token") + } + if subtle.ConstantTimeCompare([]byte(token), []byte(sess.CSRFToken)) != 1 { + http.Error(w, "invalid CSRF token", http.StatusForbidden) + return + } + } + next.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), sessionKey{}, sess))) + }) +} + +func (a *Auth) session(r *http.Request) (model.Session, error) { + c, err := r.Cookie(sessionCookieName) + if err != nil { + return model.Session{}, model.ErrNotFound + } + return a.store.GetSession(r.Context(), c.Value, a.now()) +} + +type sessionKey struct{} + +// SessionFrom returns the session Require stored in ctx. +func SessionFrom(ctx context.Context) (model.Session, bool) { + s, ok := ctx.Value(sessionKey{}).(model.Session) + return s, ok +} + +func randomString() string { + var b [32]byte + _, _ = rand.Read(b[:]) + return base64.RawURLEncoding.EncodeToString(b[:]) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `go test ./internal/master/webauth/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add go.mod go.sum internal/master/webauth +git commit -m "feat(master): OIDC login with group check, sessions and CSRF protection" +``` + +--- + +### Task 13: Web UI — overview, agents, live updates + +**Files:** +- Create: `internal/master/web/server.go`, `internal/master/web/views.go`, `internal/master/web/templates/layout.html`, `internal/master/web/templates/overview.html`, `internal/master/web/templates/agents.html`, `internal/master/web/templates/agent_created.html`, `internal/master/web/static/app.css`, `internal/master/web/static/htmx.min.js`, `internal/master/web/static/htmx-ext-sse.js` +- Test: `internal/master/web/server_test.go` + +**Interfaces:** +- Consumes: `model.*` (Task 8); `events.Topic`, `events.Services`, `events.Agents` (Task 10); `webauth.SessionFrom` (Task 12); `agenttoken.Generate`, `agenttoken.Hash` (Task 3); `imageref.Parse` (Task 1). +- Produces: + - `type Store interface { ListServices; ListAgents; CreateAgent; RevokeAgent }` (`*store.Store` satisfies it) + - `type Hub interface { Online(id int64) bool; Disconnect(id int64) }` (`*hub.Hub` satisfies it) + - `type Subscriber interface { Subscribe() (<-chan events.Topic, func()) }` (`*events.Broker` satisfies it) + - `type Config struct { AgentURL, MasterKey, AgentImage string }` + - `func New(store Store, hub Hub, sub Subscriber, cfg Config, log *slog.Logger) (*Server, error)` + - `func (s *Server) Register(mux *http.ServeMux, protect func(http.Handler) http.Handler)` + +Routes: `GET /static/` (public), and behind `protect`: `GET /{$}`, `GET /fragments/services`, `GET /agents`, `GET /fragments/agents`, `POST /agents`, `POST /agents/{id}/revoke`, `GET /events`. + +The visual design is a placeholder: plain tables, dark-first CSS. Claude Design drafts replace the CSS and markup later without changing handlers. + +- [ ] **Step 1: Vendor htmx** + +```bash +mkdir -p internal/master/web/static +curl -fsSL -o internal/master/web/static/htmx.min.js https://unpkg.com/htmx.org@2.0.4/dist/htmx.min.js +curl -fsSL -o internal/master/web/static/htmx-ext-sse.js https://unpkg.com/htmx-ext-sse@2.2.2/sse.js +head -c 200 internal/master/web/static/htmx.min.js +``` + +Expected: both downloads succeed; the head output is minified JavaScript, not an HTML error page. + +- [ ] **Step 2: Write the failing test** + +```go +package web + +import ( + "bufio" + "context" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "net/url" + "regexp" + "strings" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/agenttoken" + "github.com/shcizo/package-updater/internal/master/events" + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/protocol" +) + +type fakeStore struct { + mu sync.Mutex + services []model.Service + agents []model.Agent + created map[string]string // name → token hash + revokedID int64 +} + +func (f *fakeStore) ListServices(context.Context) ([]model.Service, error) { return f.services, nil } +func (f *fakeStore) ListAgents(context.Context) ([]model.Agent, error) { return f.agents, nil } + +func (f *fakeStore) CreateAgent(_ context.Context, name, hash string) (model.Agent, error) { + f.mu.Lock() + defer f.mu.Unlock() + if _, taken := f.created[name]; taken { + return model.Agent{}, model.ErrConflict + } + f.created[name] = hash + return model.Agent{ID: 9, Name: name}, nil +} + +func (f *fakeStore) RevokeAgent(_ context.Context, id int64, _ time.Time) error { + f.revokedID = id + return nil +} + +type fakeHub struct { + online map[int64]bool + disconnected int64 +} + +func (h *fakeHub) Online(id int64) bool { return h.online[id] } +func (h *fakeHub) Disconnect(id int64) { h.disconnected = id } + +func newTestServer(t *testing.T) (*httptest.Server, *fakeStore, *fakeHub, *events.Broker) { + t.Helper() + agentID := int64(2) + store := &fakeStore{ + created: map[string]string{"node-2": "x"}, + services: []model.Service{ + {Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", AgentID: &agentID, AgentName: "node-2", + ActualRef: "gitea.shcizo.se/shcizo/slusk:1.4.2", Status: "running", UpdateMode: "manual"}, + {Kind: protocol.KindSwarm, SwarmService: "jellyfin_jellyfin", + ActualRef: "lscr.io/linuxserver/jellyfin@sha256:0f42497a69fa0441bfd5f9d6bba8694f2a656ec984e571d0ac04c6dd91250039", Status: "degraded", UpdateMode: "manual"}, + {Kind: protocol.KindContainer, ContainerName: "updater-agent", AgentID: &agentID, AgentName: "node-2", + ActualRef: "pu:1", Status: "running", UpdateMode: "manual", Self: true}, + }, + agents: []model.Agent{ + {ID: 2, Name: "node-2", Hostname: "node-2"}, + {ID: 3, Name: "node-3"}, + {ID: 4, Name: "old", RevokedAt: ptr(time.Now())}, + }, + } + hub := &fakeHub{online: map[int64]bool{2: true}} + broker := events.NewBroker() + s, err := New(store, hub, broker, Config{ + AgentURL: "wss://10.0.0.5:8090/agent", MasterKey: "sha256:abc", AgentImage: "gitea.shcizo.se/shcizo/package-updater:latest", + }, slog.New(slog.NewTextHandler(io.Discard, nil))) + require.NoError(t, err) + mux := http.NewServeMux() + s.Register(mux, func(h http.Handler) http.Handler { return h }) + srv := httptest.NewServer(mux) + t.Cleanup(srv.Close) + return srv, store, hub, broker +} + +func ptr[T any](v T) *T { return &v } + +func get(t *testing.T, srv *httptest.Server, path string) (int, string) { + t.Helper() + resp, err := http.Get(srv.URL + path) + require.NoError(t, err) + defer resp.Body.Close() + body, err := io.ReadAll(resp.Body) + require.NoError(t, err) + return resp.StatusCode, string(body) +} + +func noRedirectClient() *http.Client { + return &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }} +} + +func TestOverview_GroupsServicesPerMachineAndSwarm(t *testing.T) { + srv, _, _, _ := newTestServer(t) + code, body := get(t, srv, "/") + assert.Equal(t, http.StatusOK, code) + assert.Contains(t, body, " 12 { + r.Digest = "sha256:" + hex[:12] + "…" + } + return r.String() +} + +type agentRow struct { + model.Agent + Online bool +} +``` + +`server.go`: + +```go +// Package web serves the master's htmx UI. +package web + +import ( + "context" + "embed" + "errors" + "fmt" + "html/template" + "log/slog" + "net/http" + "regexp" + "strconv" + "strings" + "time" + + "github.com/shcizo/package-updater/internal/agenttoken" + "github.com/shcizo/package-updater/internal/master/events" + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/master/webauth" +) + +//go:embed templates/*.html static/* +var assets embed.FS + +var agentNamePattern = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`) + +// Store is the storage the UI reads and writes. *store.Store satisfies it. +type Store interface { + ListServices(ctx context.Context) ([]model.Service, error) + ListAgents(ctx context.Context) ([]model.Agent, error) + CreateAgent(ctx context.Context, name, tokenHash string) (model.Agent, error) + RevokeAgent(ctx context.Context, id int64, at time.Time) error +} + +// Hub reports and controls live agent connections. *hub.Hub satisfies it. +type Hub interface { + Online(agentID int64) bool + Disconnect(agentID int64) +} + +// Subscriber delivers change signals. *events.Broker satisfies it. +type Subscriber interface { + Subscribe() (<-chan events.Topic, func()) +} + +// Config holds values shown in the "add agent" snippet. +type Config struct { + AgentURL string // what agents dial, e.g. wss://10.0.0.5:8090/agent + MasterKey string // the master's fingerprint + AgentImage string +} + +// Server renders the UI. +type Server struct { + store Store + hub Hub + sub Subscriber + cfg Config + log *slog.Logger + now func() time.Time + pages map[string]*template.Template +} + +// New parses the templates. Each page is its own template set so every page +// can define its own "content" block. +func New(store Store, hub Hub, sub Subscriber, cfg Config, log *slog.Logger) (*Server, error) { + funcs := template.FuncMap{"shortRef": shortRef} + pages := map[string]*template.Template{} + for _, page := range []string{"overview.html", "agents.html", "agent_created.html"} { + t, err := template.New(page).Funcs(funcs).ParseFS(assets, "templates/layout.html", "templates/"+page) + if err != nil { + return nil, fmt.Errorf("parse %s: %w", page, err) + } + pages[page] = t + } + return &Server{store: store, hub: hub, sub: sub, cfg: cfg, log: log, now: time.Now, pages: pages}, nil +} + +// Register adds the UI routes to mux. protect wraps every route except +// static assets, normally with webauth's Require. +func (s *Server) Register(mux *http.ServeMux, protect func(http.Handler) http.Handler) { + mux.Handle("GET /static/", http.FileServerFS(assets)) + route := func(pattern string, h http.HandlerFunc) { mux.Handle(pattern, protect(h)) } + route("GET /{$}", s.overview) + route("GET /fragments/services", s.servicesFragment) + route("GET /agents", s.agents) + route("GET /fragments/agents", s.agentsFragment) + route("POST /agents", s.createAgent) + route("POST /agents/{id}/revoke", s.revokeAgent) + route("GET /events", s.events) +} + +type pageData struct { + Title string + CSRF string + Groups []serviceGroup + Agents []agentRow + Error string + Created model.Agent + Snippet string +} + +func (s *Server) base(r *http.Request, title string) pageData { + sess, _ := webauth.SessionFrom(r.Context()) + return pageData{Title: title, CSRF: sess.CSRFToken} +} + +func (s *Server) render(w http.ResponseWriter, status int, page, block string, data pageData) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.WriteHeader(status) + if err := s.pages[page].ExecuteTemplate(w, block, data); err != nil { + s.log.Error("render_failed", "page", page, "block", block, "error", err) + } +} + +func (s *Server) serviceData(r *http.Request) (pageData, error) { + data := s.base(r, "Services") + services, err := s.store.ListServices(r.Context()) + if err != nil { + return data, err + } + data.Groups = groupServices(services) + return data, nil +} + +func (s *Server) overview(w http.ResponseWriter, r *http.Request) { + data, err := s.serviceData(r) + if err != nil { + s.fail(w, err) + return + } + s.render(w, http.StatusOK, "overview.html", "layout", data) +} + +func (s *Server) servicesFragment(w http.ResponseWriter, r *http.Request) { + data, err := s.serviceData(r) + if err != nil { + s.fail(w, err) + return + } + s.render(w, http.StatusOK, "overview.html", "services", data) +} + +func (s *Server) agentData(r *http.Request) (pageData, error) { + data := s.base(r, "Agents") + agents, err := s.store.ListAgents(r.Context()) + if err != nil { + return data, err + } + for _, a := range agents { + data.Agents = append(data.Agents, agentRow{Agent: a, Online: s.hub.Online(a.ID)}) + } + return data, nil +} + +func (s *Server) agents(w http.ResponseWriter, r *http.Request) { + data, err := s.agentData(r) + if err != nil { + s.fail(w, err) + return + } + s.render(w, http.StatusOK, "agents.html", "layout", data) +} + +func (s *Server) agentsFragment(w http.ResponseWriter, r *http.Request) { + data, err := s.agentData(r) + if err != nil { + s.fail(w, err) + return + } + s.render(w, http.StatusOK, "agents.html", "agents", data) +} + +func (s *Server) createAgent(w http.ResponseWriter, r *http.Request) { + name := strings.TrimSpace(r.FormValue("name")) + if !agentNamePattern.MatchString(name) { + s.agentsWithError(w, r, http.StatusBadRequest, "Agent names use lowercase letters, digits and dashes.") + return + } + token, err := agenttoken.Generate() + if err != nil { + s.fail(w, err) + return + } + agent, err := s.store.CreateAgent(r.Context(), name, agenttoken.Hash(token)) + if errors.Is(err, model.ErrConflict) { + s.agentsWithError(w, r, http.StatusConflict, fmt.Sprintf("An agent named %q already exists.", name)) + return + } + if err != nil { + s.fail(w, err) + return + } + data := s.base(r, "Agent created") + data.Created = agent + data.Snippet = agentSnippet(s.cfg, token) + s.render(w, http.StatusOK, "agent_created.html", "layout", data) +} + +func (s *Server) agentsWithError(w http.ResponseWriter, r *http.Request, status int, msg string) { + data, err := s.agentData(r) + if err != nil { + s.fail(w, err) + return + } + data.Error = msg + s.render(w, status, "agents.html", "layout", data) +} + +func (s *Server) revokeAgent(w http.ResponseWriter, r *http.Request) { + id, err := strconv.ParseInt(r.PathValue("id"), 10, 64) + if err != nil { + http.Error(w, "invalid agent id", http.StatusBadRequest) + return + } + if err := s.store.RevokeAgent(r.Context(), id, s.now()); err != nil { + s.fail(w, err) + return + } + s.hub.Disconnect(id) + http.Redirect(w, r, "/agents", http.StatusSeeOther) +} + +// events streams change signals as Server-Sent Events. htmx's SSE extension +// turns "event: services" into a refetch of the services fragment. +func (s *Server) events(w http.ResponseWriter, r *http.Request) { + rc := http.NewResponseController(w) + w.Header().Set("Content-Type", "text/event-stream") + w.Header().Set("Cache-Control", "no-cache") + w.Header().Set("X-Accel-Buffering", "no") // stop nginx from buffering the stream + + ch, cancel := s.sub.Subscribe() + defer cancel() + + if _, err := fmt.Fprint(w, ": connected\n\n"); err != nil { + return + } + if err := rc.Flush(); err != nil { + s.log.Error("sse_flush_unsupported", "error", err) + return + } + keepalive := time.NewTicker(25 * time.Second) + defer keepalive.Stop() + for { + var msg string + select { + case <-r.Context().Done(): + return + case t := <-ch: + msg = fmt.Sprintf("event: %s\ndata: changed\n\n", t) + case <-keepalive.C: + msg = ": keepalive\n\n" + } + if _, err := fmt.Fprint(w, msg); err != nil { + return + } + if err := rc.Flush(); err != nil { + return + } + } +} + +func (s *Server) fail(w http.ResponseWriter, err error) { + s.log.Error("request_failed", "error", err) + http.Error(w, "internal error", http.StatusInternalServerError) +} + +// agentSnippet is the compose file shown once when an agent is added. +func agentSnippet(cfg Config, token string) string { + return fmt.Sprintf(`services: + updater-agent: + image: %s + entrypoint: ["/usr/local/bin/updater-agent"] + restart: unless-stopped + environment: + MASTER_URL: %s + MASTER_KEY: %s + AGENT_TOKEN: %s + volumes: + - /var/run/docker.sock:/var/run/docker.sock +`, cfg.AgentImage, cfg.AgentURL, cfg.MasterKey, token) +} +``` + +`templates/layout.html`: + +```html +{{define "layout"}} + + + + + {{.Title}} · package-updater + + + + + + +
{{template "content" .}}
+ +{{end}} +``` + +`templates/overview.html`: + +```html +{{define "content"}} +

Services

+
+ {{template "services" .}} +
+{{end}} + +{{define "services"}} +{{range .Groups}} +
+

{{.Title}}

+ + + + {{range .Services}} + + + + + + + {{end}} + +
ServiceImageStatusMode
{{.DisplayName}}{{if .Self}} managed outside{{end}}{{shortRef .ActualRef}}{{.Status}}{{.UpdateMode}}
+
+{{else}} +

No services reported yet. Add an agent to get started.

+{{end}} +{{end}} +``` + +`templates/agents.html`: + +```html +{{define "content"}} +

Agents

+
+ + + +
+{{if .Error}}

{{.Error}}

{{end}} +
+ {{template "agents" .}} +
+{{end}} + +{{define "agents"}} + + + + {{range .Agents}} + + + + + + + + + + {{else}} + + {{end}} + +
NameHostnameStateVersionSwarm managerLast seen
{{.Name}}{{.Hostname}}{{if .RevokedAt}}revoked{{else if .Online}}online{{else}}offline{{end}}{{.AgentVersion}}{{if .SwarmManager}}yes{{end}}{{with .LastSeenAt}}{{.Format "2006-01-02 15:04"}}{{end}}{{if not .RevokedAt}} +
+ + +
+ {{end}}
No agents yet.
+{{end}} +``` + +`templates/agent_created.html`: + +```html +{{define "content"}} +

Agent "{{.Created.Name}}" created

+

The token is shown only once. Save this as docker-compose.yml on the machine and run docker compose up -d.

+
{{.Snippet}}
+

Back to agents

+{{end}} +``` + +`static/app.css`: + +```css +:root { --bg: #111418; --fg: #e6e8eb; --muted: #8a929c; --line: #262b31; --ok: #3fb950; --warn: #d29922; --bad: #f85149; --accent: #58a6ff; } +@media (prefers-color-scheme: light) { + :root { --bg: #ffffff; --fg: #1f2328; --muted: #59636e; --line: #d1d9e0; --ok: #1a7f37; --warn: #9a6700; --bad: #cf222e; --accent: #0969da; } +} +* { box-sizing: border-box; } +body { margin: 0; background: var(--bg); color: var(--fg); font: 14px/1.5 system-ui, sans-serif; } +nav { display: flex; gap: 16px; align-items: center; padding: 12px 16px; border-bottom: 1px solid var(--line); } +nav form { margin-left: auto; } +main { padding: 16px; max-width: 1200px; margin: 0 auto; } +a { color: var(--accent); } +table { width: 100%; border-collapse: collapse; margin-bottom: 24px; } +th, td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--line); } +th { color: var(--muted); font-weight: 500; } +code, pre { font: 13px ui-monospace, monospace; } +pre { padding: 12px; border: 1px solid var(--line); overflow-x: auto; } +.badge { font-size: 12px; color: var(--muted); border: 1px solid var(--line); padding: 0 6px; border-radius: 8px; } +.status-running { color: var(--ok); } +.status-degraded, .status-unknown { color: var(--warn); } +.status-stopped { color: var(--bad); } +.error { color: var(--bad); } +.warning { color: var(--warn); } +form.inline { display: flex; gap: 8px; margin-bottom: 16px; } +@media (max-width: 600px) { table { display: block; overflow-x: auto; } } +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `go test ./internal/master/web/` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add internal/master/web +git commit -m "feat(master): htmx UI with live service overview and agent management" +``` + +--- + +### Task 14: Master binary and config + +**Files:** +- Create: `internal/master/config/config.go`, `cmd/master/main.go` +- Modify: `Dockerfile` (add `master:updater-master` back to the build loop from Task 7) +- Test: `internal/master/config/config_test.go` + +**Interfaces:** +- Consumes: everything from Tasks 3, 8–13. +- Produces: `config.Load() (*config.Config, error)` with fields `DatabaseURL, PublicURL, AgentURL, AgentImage, OIDCIssuer, OIDCClientID, OIDCClientSecret, AllowedGroup, Port, AgentPort, LogLevel string`; binary `/usr/local/bin/updater-master`. + +Environment: `DATABASE_URL`, `PUBLIC_URL` (e.g. `https://updater.example.com`), `AGENT_URL` (`wss://10.0.0.5:8090/agent`), `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `ALLOWED_GROUP` — all required. `PORT` (default `8080`), `AGENT_PORT` (default `8090`), `AGENT_IMAGE` (default `gitea.shcizo.se/shcizo/package-updater:latest`), `LOG_LEVEL` (default `info`). + +- [ ] **Step 1: Write the failing test** + +```go +package config + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +var required = map[string]string{ + "DATABASE_URL": "postgres://u:p@db/pu", + "PUBLIC_URL": "https://updater.example.com", + "AGENT_URL": "wss://10.0.0.5:8090/agent", + "OIDC_ISSUER": "https://auth.example.com/application/o/pu/", + "OIDC_CLIENT_ID": "id", + "OIDC_CLIENT_SECRET": "secret", + "ALLOWED_GROUP": "updater-admins", +} + +func setAll(t *testing.T) { + for k, v := range required { + t.Setenv(k, v) + } + for _, k := range []string{"PORT", "AGENT_PORT", "AGENT_IMAGE", "LOG_LEVEL"} { + t.Setenv(k, "") + } +} + +func TestLoad_Defaults(t *testing.T) { + setAll(t) + cfg, err := Load() + require.NoError(t, err) + assert.Equal(t, "8080", cfg.Port) + assert.Equal(t, "8090", cfg.AgentPort) + assert.Equal(t, "gitea.shcizo.se/shcizo/package-updater:latest", cfg.AgentImage) + assert.Equal(t, "info", cfg.LogLevel) + assert.Equal(t, "updater-admins", cfg.AllowedGroup) +} + +func TestLoad_EachRequiredVariable(t *testing.T) { + for key := range required { + t.Run(key, func(t *testing.T) { + setAll(t) + t.Setenv(key, "") + _, err := Load() + assert.ErrorContains(t, err, key+" is required") + }) + } +} + +func TestLoad_URLFormats(t *testing.T) { + cases := []struct{ key, value, wantErr string }{ + {"PUBLIC_URL", "updater.example.com", "PUBLIC_URL must start with http:// or https://"}, + {"AGENT_URL", "ws://10.0.0.5:8090/agent", "AGENT_URL must start with wss://"}, + } + for _, c := range cases { + t.Run(c.key, func(t *testing.T) { + setAll(t) + t.Setenv(c.key, c.value) + _, err := Load() + assert.ErrorContains(t, err, c.wantErr) + }) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `go test ./internal/master/config/` +Expected: FAIL — `undefined: Load`. + +- [ ] **Step 3: Write the implementation** + +`internal/master/config/config.go`: + +```go +// Package config loads the master's configuration from environment variables. +package config + +import ( + "fmt" + "os" + "strings" +) + +// Config holds the master's runtime configuration. +type Config struct { + DatabaseURL string + PublicURL string // where people reach the UI, without trailing slash + AgentURL string // what agents dial, shown in the add-agent snippet + AgentImage string + OIDCIssuer string + OIDCClientID string + OIDCClientSecret string + AllowedGroup string + Port string + AgentPort string + LogLevel string +} + +// Load reads and validates the master configuration, failing fast on +// anything missing so a half-configured master never starts. +func Load() (*Config, error) { + cfg := &Config{ + DatabaseURL: os.Getenv("DATABASE_URL"), + PublicURL: strings.TrimSuffix(os.Getenv("PUBLIC_URL"), "/"), + AgentURL: os.Getenv("AGENT_URL"), + AgentImage: getenvDefault("AGENT_IMAGE", "gitea.shcizo.se/shcizo/package-updater:latest"), + OIDCIssuer: os.Getenv("OIDC_ISSUER"), + OIDCClientID: os.Getenv("OIDC_CLIENT_ID"), + OIDCClientSecret: os.Getenv("OIDC_CLIENT_SECRET"), + AllowedGroup: os.Getenv("ALLOWED_GROUP"), + Port: getenvDefault("PORT", "8080"), + AgentPort: getenvDefault("AGENT_PORT", "8090"), + LogLevel: getenvDefault("LOG_LEVEL", "info"), + } + for _, r := range []struct{ name, value string }{ + {"DATABASE_URL", cfg.DatabaseURL}, + {"PUBLIC_URL", cfg.PublicURL}, + {"AGENT_URL", cfg.AgentURL}, + {"OIDC_ISSUER", cfg.OIDCIssuer}, + {"OIDC_CLIENT_ID", cfg.OIDCClientID}, + {"OIDC_CLIENT_SECRET", cfg.OIDCClientSecret}, + {"ALLOWED_GROUP", cfg.AllowedGroup}, + } { + if r.value == "" { + return nil, fmt.Errorf("%s is required", r.name) + } + } + if !strings.HasPrefix(cfg.PublicURL, "http://") && !strings.HasPrefix(cfg.PublicURL, "https://") { + return nil, fmt.Errorf("PUBLIC_URL must start with http:// or https://") + } + if !strings.HasPrefix(cfg.AgentURL, "wss://") { + return nil, fmt.Errorf("AGENT_URL must start with wss://") + } + return cfg, nil +} + +func getenvDefault(key, fallback string) string { + if v := os.Getenv(key); v != "" { + return v + } + return fallback +} +``` + +`cmd/master/main.go`: + +```go +// Command updater-master serves the package-updater UI and accepts agent +// connections. +package main + +import ( + "context" + "crypto/tls" + "errors" + "fmt" + "net/http" + "os" + "os/signal" + "strings" + "syscall" + "time" + + "github.com/shcizo/package-updater/internal/logging" + "github.com/shcizo/package-updater/internal/master/config" + "github.com/shcizo/package-updater/internal/master/events" + "github.com/shcizo/package-updater/internal/master/hub" + "github.com/shcizo/package-updater/internal/master/store" + "github.com/shcizo/package-updater/internal/master/web" + "github.com/shcizo/package-updater/internal/master/webauth" + "github.com/shcizo/package-updater/internal/pin" +) + +// Build info is injected at link time via -ldflags="-X main.version=...". +var version = "dev" + +func main() { + if err := run(); err != nil { + fmt.Fprintln(os.Stderr, "fatal:", err) + os.Exit(1) + } +} + +func run() error { + cfg, err := config.Load() + if err != nil { + return err + } + logger := logging.New(cfg.LogLevel) + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + st, err := store.Open(ctx, cfg.DatabaseURL) + if err != nil { + return err + } + defer st.Close() + + key, err := st.LoadOrCreateIdentity(ctx, pin.GenerateKey) + if err != nil { + return err + } + cert, err := pin.Certificate(key) + if err != nil { + return err + } + fingerprint, err := pin.Fingerprint(key.Public()) + if err != nil { + return err + } + logger.Info("master_starting", "version", version, "master_key", fingerprint) + + broker := events.NewBroker() + agentHub := hub.New(st, broker, logger) + + provider, err := webauth.NewOIDCProvider(ctx, webauth.OIDCConfig{ + Issuer: cfg.OIDCIssuer, ClientID: cfg.OIDCClientID, ClientSecret: cfg.OIDCClientSecret, + RedirectURL: cfg.PublicURL + "/auth/callback", + }) + if err != nil { + return err + } + auth := webauth.New(provider, st, cfg.AllowedGroup, strings.HasPrefix(cfg.PublicURL, "https://"), logger) + + ui, err := web.New(st, agentHub, broker, web.Config{ + AgentURL: cfg.AgentURL, MasterKey: fingerprint, AgentImage: cfg.AgentImage, + }, logger) + if err != nil { + return err + } + + mux := http.NewServeMux() + mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { + if err := st.Ping(r.Context()); err != nil { + http.Error(w, "database unreachable", http.StatusServiceUnavailable) + return + } + _, _ = w.Write([]byte("ok")) + }) + mux.HandleFunc("GET /auth/login", auth.Login) + mux.HandleFunc("GET /auth/callback", auth.Callback) + mux.Handle("POST /auth/logout", auth.Require(http.HandlerFunc(auth.Logout))) + ui.Register(mux, auth.Require) + + // No WriteTimeout on either server: SSE streams and agent connections + // are long-lived by design. + webSrv := &http.Server{Addr: ":" + cfg.Port, Handler: mux, ReadHeaderTimeout: 10 * time.Second} + + agentMux := http.NewServeMux() + agentMux.Handle("GET /agent", agentHub) + agentSrv := &http.Server{ + Addr: ":" + cfg.AgentPort, Handler: agentMux, ReadHeaderTimeout: 10 * time.Second, + TLSConfig: &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS13}, + } + + errCh := make(chan error, 2) + go func() { errCh <- webSrv.ListenAndServe() }() + go func() { errCh <- agentSrv.ListenAndServeTLS("", "") }() + logger.Info("listening", "web", webSrv.Addr, "agents", agentSrv.Addr) + + select { + case <-ctx.Done(): + logger.Info("shutdown_signal") + case err := <-errCh: + if !errors.Is(err, http.ErrServerClosed) { + return err + } + } + shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + _ = webSrv.Shutdown(shutdownCtx) + _ = agentSrv.Shutdown(shutdownCtx) + return nil +} +``` + +`Dockerfile`: put `master:updater-master` back into the build loop so it reads `for cmd in server:package-updater master:updater-master agent:updater-agent; do`. + +- [ ] **Step 4: Run tests and build** + +Run: `go vet ./... && go test ./... && go build ./cmd/master ./cmd/agent ./cmd/server && rm -f master agent server && docker build -t package-updater:dev .` +Expected: everything passes. If Docker is not available locally, skip `docker build` and say so in the task report. + +- [ ] **Step 5: Commit** + +```bash +git add internal/master/config cmd/master Dockerfile +git commit -m "feat(master): add updater-master binary wiring store, hub, auth and UI" +``` + +--- + +### Task 15: End-to-end agent ↔ master over pinned TLS + +**Files:** +- Create: `internal/e2e/agent_master_test.go` + +**Interfaces:** +- Consumes: `pin.*` (Task 3), `agenttoken.Hash` (Task 3), `conn.New`, `conn.Config` (Task 6), `hub.New`, `hub.Store` (Task 11), `events.NewBroker` (Task 10), `model.*` (Task 8), `protocol.*` (Task 2). +- Produces: nothing (test only). + +- [ ] **Step 1: Write the test** + +```go +// Package e2e runs a real agent connection against a real hub over TLS with +// a pinned key, in one process, with in-memory storage and fake Docker. +package e2e + +import ( + "context" + "crypto/tls" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/shcizo/package-updater/internal/agent/conn" + "github.com/shcizo/package-updater/internal/agenttoken" + "github.com/shcizo/package-updater/internal/master/events" + "github.com/shcizo/package-updater/internal/master/hub" + "github.com/shcizo/package-updater/internal/master/model" + "github.com/shcizo/package-updater/internal/pin" + "github.com/shcizo/package-updater/internal/protocol" +) + +type memStore struct { + mu sync.Mutex + agents map[string]model.Agent + lookups int + services map[string]protocol.Service +} + +func (m *memStore) AgentByTokenHash(_ context.Context, hash string) (model.Agent, error) { + m.mu.Lock() + defer m.mu.Unlock() + m.lookups++ + a, ok := m.agents[hash] + if !ok { + return model.Agent{}, model.ErrNotFound + } + return a, nil +} + +func (m *memStore) RecordHello(context.Context, int64, protocol.Hello, time.Time) error { return nil } + +func (m *memStore) ApplySnapshot(_ context.Context, _ int64, _ protocol.Hello, s []protocol.Service, _ time.Time) error { + m.mu.Lock() + defer m.mu.Unlock() + m.services = map[string]protocol.Service{} + for _, x := range s { + m.services[x.Key()] = x + } + return nil +} + +func (m *memStore) UpsertService(_ context.Context, _ int64, _ string, s protocol.Service, _ time.Time) error { + m.mu.Lock() + defer m.mu.Unlock() + m.services[s.Key()] = s + return nil +} + +func (m *memStore) RemoveService(_ context.Context, _ int64, _, key string, _ time.Time) error { + m.mu.Lock() + defer m.mu.Unlock() + delete(m.services, key) + return nil +} + +func (m *memStore) MarkSeen(context.Context, int64, time.Time) error { return nil } + +func (m *memStore) keys() []string { + m.mu.Lock() + defer m.mu.Unlock() + var out []string + for k := range m.services { + out = append(out, k) + } + return out +} + +type staticSource struct { + mu sync.Mutex + services []protocol.Service +} + +func (s *staticSource) Hello(context.Context) (protocol.Hello, error) { + return protocol.Hello{Hostname: "node-2"}, nil +} + +func (s *staticSource) Snapshot(context.Context) ([]protocol.Service, error) { + s.mu.Lock() + defer s.mu.Unlock() + return append([]protocol.Service(nil), s.services...), nil +} + +func (s *staticSource) set(v ...protocol.Service) { + s.mu.Lock() + defer s.mu.Unlock() + s.services = v +} + +func startMaster(t *testing.T, store *memStore) (url, fingerprint string) { + t.Helper() + key, err := pin.GenerateKey() + require.NoError(t, err) + cert, err := pin.Certificate(key) + require.NoError(t, err) + fingerprint, err = pin.Fingerprint(key.Public()) + require.NoError(t, err) + + mux := http.NewServeMux() + mux.Handle("GET /agent", hub.New(store, events.NewBroker(), slog.New(slog.NewTextHandler(io.Discard, nil)))) + srv := httptest.NewUnstartedServer(mux) + srv.TLS = &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS13} + srv.StartTLS() + t.Cleanup(srv.Close) + return "wss" + strings.TrimPrefix(srv.URL, "https") + "/agent", fingerprint +} + +func runAgent(t *testing.T, url, fingerprint, token string, src conn.Source) *conn.Agent { + t.Helper() + a := conn.New(conn.Config{URL: url, Token: token, Fingerprint: fingerprint, Version: "e2e", + MinBackoff: 10 * time.Millisecond, MaxBackoff: 50 * time.Millisecond, + }, src, slog.New(slog.NewTextHandler(io.Discard, nil))) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { _ = a.Run(ctx); close(done) }() + t.Cleanup(func() { cancel(); <-done }) + return a +} + +var ( + web = protocol.Service{Kind: protocol.KindCompose, ComposeProject: "slusk", ComposeService: "web", Image: "slusk:1", Status: "running"} + cache = protocol.Service{Kind: protocol.KindContainer, ContainerName: "cache", Image: "redis:alpine", Status: "running"} +) + +func TestInventoryFlowsFromAgentToStore(t *testing.T) { + store := &memStore{agents: map[string]model.Agent{agenttoken.Hash("pua_ok"): {ID: 2, Name: "node-2"}}} + url, fp := startMaster(t, store) + src := &staticSource{} + src.set(web) + agent := runAgent(t, url, fp, "pua_ok", src) + + require.Eventually(t, func() bool { return len(store.keys()) == 1 }, 2*time.Second, 10*time.Millisecond) + + src.set(cache) + agent.Notify() + require.Eventually(t, func() bool { + k := store.keys() + return len(k) == 1 && k[0] == "container|cache" + }, 2*time.Second, 10*time.Millisecond) +} + +func TestAgentWithWrongKeyNeverReachesHub(t *testing.T) { + store := &memStore{agents: map[string]model.Agent{agenttoken.Hash("pua_ok"): {ID: 2, Name: "node-2"}}} + url, _ := startMaster(t, store) + other, err := pin.GenerateKey() + require.NoError(t, err) + wrong, err := pin.Fingerprint(other.Public()) + require.NoError(t, err) + + runAgent(t, url, wrong, "pua_ok", &staticSource{}) + time.Sleep(200 * time.Millisecond) + store.mu.Lock() + defer store.mu.Unlock() + assert.Zero(t, store.lookups, "TLS must fail before the token is ever presented") +} + +func TestRevokedAgentIsRejected(t *testing.T) { + store := &memStore{agents: map[string]model.Agent{ + agenttoken.Hash("pua_old"): {ID: 3, Name: "old", RevokedAt: func() *time.Time { n := time.Now(); return &n }()}, + }} + url, fp := startMaster(t, store) + src := &staticSource{} + src.set(web) + runAgent(t, url, fp, "pua_old", src) + + require.Eventually(t, func() bool { + store.mu.Lock() + defer store.mu.Unlock() + return store.lookups >= 2 + }, 2*time.Second, 10*time.Millisecond, "the agent keeps retrying") + assert.Empty(t, store.keys()) +} +``` + +- [ ] **Step 2: Run the test** + +Run: `go test -race ./internal/e2e/` +Expected: PASS. If a test fails, the bug is in Tasks 3, 6 or 11 — fix it there with a regression test in that package, not by changing this test. + +- [ ] **Step 3: Commit** + +```bash +git add internal/e2e +git commit -m "test: agent to master inventory over pinned TLS end to end" +``` + +--- + +### Task 16: Documentation and deployment example + +**Files:** +- Create: `deploy/master.stack.yml` +- Modify: `CLAUDE.md` (Commands, Architecture, Gotchas, References), `README.md` (new section) + +**Interfaces:** none. + +- [ ] **Step 1: Write `deploy/master.stack.yml`** + +```yaml +# Example Swarm stack for the package-updater master (phase 1: inventory only). +# Deploy on the manager: docker stack deploy -c master.stack.yml package-updater-master +# +# Before deploying: +# 1. In Postgres: CREATE USER package_updater ...; CREATE DATABASE package_updater OWNER package_updater; +# 2. In Authentik: create an OAuth2/OpenID provider + application with redirect URI +# https://updater.example.com/auth/callback, and a group "updater-admins". +# 3. docker secret create pu_oidc_client_secret - ; docker secret create pu_database_url - +services: + master: + image: gitea.shcizo.se/shcizo/package-updater:latest + entrypoint: ["/bin/sh", "-c", "DATABASE_URL=$$(cat /run/secrets/pu_database_url) OIDC_CLIENT_SECRET=$$(cat /run/secrets/pu_oidc_client_secret) exec /usr/local/bin/updater-master"] + environment: + PUBLIC_URL: https://updater.example.com + AGENT_URL: wss://10.0.0.5:8090/agent + OIDC_ISSUER: https://auth.example.com/application/o/package-updater/ + OIDC_CLIENT_ID: package-updater + ALLOWED_GROUP: updater-admins + secrets: + - pu_database_url + - pu_oidc_client_secret + ports: + # Agents connect here directly over the internal network, not via the proxy. + - target: 8090 + published: 8090 + mode: host + networks: + - proxy + deploy: + placement: + constraints: [node.role == manager] + +secrets: + pu_database_url: + external: true + pu_oidc_client_secret: + external: true + +networks: + proxy: + external: true +``` + +- [ ] **Step 2: Update `CLAUDE.md`** + +In **Commands**, add after the `go build ./cmd/server` line: + +```bash +go build ./cmd/master ./cmd/agent # new master/agent binaries (phase 1: inventory) +DATABASE_URL=postgres://... go test -tags integration ./internal/master/store/ # Postgres tests +``` + +In **Architecture**, add: + +```markdown +- `cmd/master`, `cmd/agent` — master/agent architecture (spec `2026-10-12-master-agent-design.md`), phase 1 = inventory only. `cmd/server` keeps doing updates until phase 3. +- `internal/protocol` — agent↔master WebSocket messages +- `internal/pin` — master key pair, self-signed cert, fingerprint-pinned TLS client +- `internal/agenttoken`— per-agent tokens (`pua_` prefix, only the SHA-256 hash is stored) +- `internal/imageref` — split `repo:tag@digest` +- `internal/agent/...` — inventory collection, Docker event debounce, connection loop +- `internal/master/...`— Postgres store, agent hub, change broker, OIDC auth, htmx UI +``` + +In **Gotchas**, add: + +```markdown +- **WebSocket read limit is raised to 1 MiB on both sides.** coder/websocket defaults to 32 KiB, which a full snapshot from a busy machine exceeds — symptom is an agent stuck in a reconnect loop. +- **The master's key lives in Postgres (`master_identity`).** Wiping the database means every agent needs a new `MASTER_KEY`. +- **SSE responses must flush through every wrapper.** `web.events` uses `http.ResponseController`; a middleware that wraps `ResponseWriter` without `Unwrap()` silently breaks live updates. Nginx needs `X-Accel-Buffering: no` (set by the handler). +- **The "stateless" and "label gate" design intents above describe `cmd/server` only.** The master is stateful by design and gates updates per service in the UI (spec 2026-10-12). +``` + +In **References**, add: + +```markdown +- Master/agent spec: `docs/superpowers/specs/2026-10-12-master-agent-design.md` +- Phase 1 plan: `docs/superpowers/plans/2026-10-12-master-agent-phase-1-inventory.md` +``` + +- [ ] **Step 3: Update `README.md`** + +Add a section before **Quick start**: + +```markdown +## Master/agent (in progress) + +A master with one agent per machine is replacing the single-instance webhook +model; see the [design spec](docs/superpowers/specs/2026-10-12-master-agent-design.md). +Phase 1 delivers a live inventory: every Compose stack, Swarm service and +container on every machine, behind Authentik login. Updates still go through +the existing `/update` webhook until phase 3. + +- Master: `updater-master` in the same image, example stack in `deploy/master.stack.yml`. +- Agents: add one per machine on the master's **Agents** page; it shows a + ready-to-run compose file with the agent's token and the master's key + fingerprint. +``` + +- [ ] **Step 4: Verify** + +Run: `go test ./... && go vet ./...` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add deploy CLAUDE.md README.md +git commit -m "docs: document master/agent phase 1 and add master stack example" +``` + +--- + +## Manual smoke test (after Task 16) + +Not automatable here — needs a real Authentik and Docker hosts. Run once before calling phase 1 done: + +1. Deploy the master as in `deploy/master.stack.yml`; its log prints `master_key=sha256:…`. +2. Log in. A user outside `ALLOWED_GROUP` gets 403. If a user *inside* the group also gets 403, Authentik is not sending `groups` in the ID token — check the `profile` scope mapping in Authentik. +3. Add an agent for `swarm-mgr` and one for the machine running `slusk`; run the snippets. +4. Overview shows Swarm services once (not per node), `slusk` under its machine, and the agents marked "managed outside". +5. `docker service scale =0` on the manager → the row turns `stopped` within a few seconds without reloading. +6. Revoke an agent → it shows `revoked` and its log shows the connection dropping and `401` on retry. diff --git a/docs/superpowers/specs/2026-10-12-master-agent-design.md b/docs/superpowers/specs/2026-10-12-master-agent-design.md new file mode 100644 index 0000000..ce50afc --- /dev/null +++ b/docs/superpowers/specs/2026-10-12-master-agent-design.md @@ -0,0 +1,373 @@ +# Master/Agent Architecture — Design + +**Date:** 2026-10-12 +**Status:** Approved in brainstorming, pending written-spec review + +## Problem + +package-updater today is one instance per Docker daemon: a webhook (`POST /update`) +finds opted-in containers by label and runs `docker compose` or `docker service update`. +It is stateless and has no UI. + +The goal now is a tool that replaces Watchtower and Shepherd across the whole fleet: + +- One overview of every service on every machine — Swarm services and Compose stacks, + which can coexist on the same machine. +- Per-service choice, made in the UI, between **manual** (show that an update exists, + a human clicks) and **auto** (apply as soon as a new version is detected). +- Detect new versions both from CI (webhook) and by polling registries, so images we do + not build ourselves are covered too. +- See available and completed updates, pick any of the recently seen versions + (which also gives rollback). +- Live updates in the UI. + +Building this ourselves instead of adopting Komodo or Portainer is a deliberate choice: +it is a hobby project, and owning it means it works exactly the way we want. + +## Environment and assumptions + +- Several machines on one internal network, reachable from each other by IP. +- One machine (`swarm-mgr`) is a Swarm manager; most services are Swarm stacks + (`docker-stack.yml`). At least one Compose stack exists (`slusk`). +- A central Postgres already runs and will be used. +- Authentik runs and is used per application via OIDC (not via the reverse proxy). +- A reverse proxy fronts the web UI. +- Stack files are version-controlled. Image references are ad hoc: a mix of `repo:tag`, + `repo:tag@digest` and bare `repo@digest`. Many digests were written by a Claude instance + on the servers, not by an automated process. +- Shepherd and Watchtower currently run and will be removed (see Migration). + +## Decisions + +### Topology: one master, one agent per machine + +- **Master** (one instance): Postgres, web UI, OIDC login, webhook, registry polling, + job decisions. Runs as a Swarm service on `swarm-mgr`. +- **Agent** (one per machine): owns that machine's Docker daemon via `docker.sock`. + Reports inventory and runs update jobs. Runs via plain `docker compose` on each host + (not a global Swarm service), because each agent has its own token and a global + service would hand every node the same environment. + The agent mounts `docker.sock` and `STACKS_ROOT` **read-write** (today it is `:ro`), + because it writes `compose.override.yaml` files. + +### Connection: agent dials master over WebSocket, live + +The agent opens a long-lived WebSocket to the master and keeps it alive with reconnect +and backoff. Inventory changes are pushed live from Docker events; the master forwards +them to browsers. Chosen over the master polling agents over HTTP because live updates +are an explicit goal. + +Messages are JSON in a common envelope `{type, id, payload}` with one Go type per +message kind, shared in `internal/protocol`. JSON over WebSocket rather than gRPC: +same tooling as the browser side, debuggable with `websocat`, no protobuf toolchain. + +### Trust in both directions + +Commands flow from master to agent and can restart anything on that machine, so the +agent must authenticate the master, not only the other way round. + +- On first start the master generates a key pair and stores it in Postgres + (`master_identity`), so it survives container recreation and is covered by DB backups. +- The master serves the agent endpoint on its own port with TLS, using a self-signed + certificate for that key. No reverse proxy, no CA. +- The agent is configured with `MASTER_URL`, `MASTER_KEY` and `AGENT_TOKEN`. + `MASTER_KEY` is the SHA-256 fingerprint of the master's public key. The agent skips + normal chain verification and instead accepts the connection only if the + certificate's public key hashes to `MASTER_KEY` (same idea as SSH `known_hosts`). + The TLS handshake proves the master holds the private key. +- `AGENT_TOKEN` is per agent, created in the UI ("Add agent", which shows a ready + compose snippet including token and `MASTER_KEY`). Only its hash is stored. + Agents can be revoked individually. A leaked token lets someone impersonate an agent + (feed false inventory), not command machines. +- Losing the master key means every agent needs a new `MASTER_KEY`. + +Plain `ws://` with a signed handshake was rejected: anyone able to hijack the TCP +stream after the handshake could inject commands. + +### State: the master remembers, agents report reality + +Agents are the truth for what is running. Postgres holds what the master must remember: +desired version, update mode, version history and job history. This deliberately +retires the "stateless" property for the master; agents stay stateless. + +### Security gate moves from labels to the UI + +The opt-in label no longer decides what may be updated; the per-service mode in the UI +does. Consequences: + +- Every newly discovered service starts as `manual`. Nothing is ever `auto` by default. +- Access to the UI is equivalent to being able to restart everything, so login is the + most important control (see Authentication). +- The `STACKS_ROOT` path check stays on the agent as an independent guard for Compose, + and matters more now that the agent needs write access there (override files). +- The master cannot reliably tell which service is its own Postgres or Authentik, so + they are not locked automatically. Putting them on `auto` is the operator's risk and + is documented. + +### Changing the version of a service + +| Kind | How the version is changed | What can revert it | +|---|---|---| +| Compose | Agent writes the service's image into `compose.override.yaml` next to the stack's compose file (the agent owns this file, preserving other services' entries in it), then `docker compose pull ` + `up -d `. `docker compose` loads the override automatically, so manual `docker compose up -d` keeps the new version. | Deleting the override file. | +| Swarm | `service update` via the Docker API with `repo:tag@digest` (the agent resolves the digest; without it Swarm does not re-pull when the tag is unchanged). | Re-running `docker stack deploy` with a file holding an older reference. | + +The updater always writes `repo:tag@digest`. The stack's own files in git are never +edited. Writing version bumps back to git (GitOps style) was considered and deferred: +it needs repo write access, YAML rewriting and conflict handling — a different kind of +tool. It can be added later as a step after a successful update. + +### Drift + +A service drifts when `actual_ref` differs from `desired_ref`. A service the updater has +never touched has no `desired_ref` and cannot drift. Drift is **never corrected +automatically**, not even in `auto` mode, because a manual `stack deploy` with a new tag +may be intentional. The UI shows the drift and offers "restore desired" or +"accept running". + +### Service identity and moves + +A service row has its own ID. Location is an attribute with a unique index: +Compose = agent + project + service; Swarm = cluster ID + service name (independent of +which manager agent reports it); plain container = agent + container name. + +Identity without the agent was rejected: the same stack legitimately runs on several +machines (the agent itself, node exporters), and they would be merged into one history. +Automatic merging was rejected for the same reason. + +When a Compose service disappears on one agent and exactly one service with the same +project and service name appears on another, the UI suggests "Moved from A?" with a +**Merge** action: the old row is moved to the new location with its history, mode and +`desired_ref`, and the duplicate is removed. Merging can also be done by hand. + +### Versions: what the system has seen + +Every version the system sees is recorded in `image_versions`, from three sources: +webhook (every build CI announces), polling (every new digest under a followed tag) and +inventory (whatever is running). The service page lists the 10 most recent versions for +the service's repo; picking one creates a manual job to exactly that reference. Picking +an older one is a rollback. + +`available_*` on the service remains as "the newest suggestion", which drives the +"update available" badge and `auto` mode. + +| Reference format | Polling | Webhook | Pick a version | +|---|---|---|---| +| `repo:tag` | yes | yes | yes | +| `repo:tag@digest` | yes (follows the tag) | yes | yes | +| `repo@digest` | no — nothing to follow | yes (own images) | yes (seen versions) | + +Services without a tag are flagged "follows no tag" in the UI. Recommended convention +for stack files: `repo:tag@digest`, tag = what to follow. + +## Data model (Postgres) + +**`master_identity`** — single row: private key, created_at. + +**`agents`** — `id`, `name` (set when adding), `token_hash`, `hostname`, +`agent_version`, `is_swarm_manager` (reported), `last_seen_at`, `revoked_at`, `created_at`. +Online/offline is held in memory only. + +**`services`** +- `id` +- `kind`: `compose` | `swarm` | `container` (plain containers are shown but cannot be + updated in v1) +- location: `agent_id`, `compose_project`, `compose_service`, `swarm_cluster_id`, + `swarm_service`, `container_name` (populated per kind; unique index per kind) +- `image_repo` — normalised, used to match webhooks and polling +- `update_mode`: `manual` (default) | `auto` +- `desired_ref` +- `actual_ref`, `actual_digest`, `status` — last known, kept so the UI has something to + show while an agent is offline +- `available_ref`, `available_digest`, `available_source` (`webhook` | `poll`), + `available_at` +- `last_poll_error` +- `first_seen_at`, `removed_at` — disappeared services are marked, not deleted + +**`image_versions`** — `repo`, `tag`, `digest`, `source` (`webhook` | `poll` | `running`), +`first_seen_at`; unique on (repo, tag, digest). + +**`update_jobs`** — `id`, `service_id`, `from_ref`, `to_ref`, +`trigger` (`manual` | `webhook` | `poll`), `requested_by` (OIDC subject for manual), +`status` (`queued` | `running` | `succeeded` | `failed`), `output`, `error`, +`created_at`, `started_at`, `finished_at`. + +**`sessions`** — web sessions for OIDC logins. + +## Flows + +### Inventory + +1. On connect the agent sends `hello` (hostname, version, Swarm manager or not, cluster + ID) and a full `snapshot`. +2. The master diffs the snapshot against that agent's rows: new services are inserted as + `manual`, missing ones get `removed_at`, changed ones get new `actual_*`. Running + references are recorded in `image_versions` with source `running`. +3. The agent subscribes to Docker events (container start/stop/die/destroy, service + update), debounces bursts (a Compose restart emits many events), re-inspects the + affected item and sends `upsert` or `remove`. +4. Swarm services are reported only by agents on manager nodes. Agents skip containers + that belong to Swarm tasks, so a service is not listed once per node. +5. The master pushes changes to browsers over SSE. + +### Update + +1. Triggers: a click in the UI, the webhook, or polling. Webhook and polling always set + `available_*` and record the version; they create a job only if the service is `auto`. +2. The master inserts an `update_jobs` row and sends `job.start` to the owning agent + (for Swarm, a connected manager agent). +3. The agent puts the job in its FIFO queue (single worker, as today) and executes it as + described under "Changing the version of a service". +4. The agent sends `job.progress` and `job.result` with output. On success the master + sets `desired_ref` and clears `available_*` if that version is now running. + +### Failure handling + +| Situation | Behaviour | +|---|---| +| Agent offline when a job is created | Job fails immediately with "agent offline". It is not stored for later — a job that runs hours later surprises more than it helps. `available_*` remains, so the user can retry. | +| Connection drops mid-job | The agent finishes the job (its queue is local) and keeps results in memory until the master acknowledges them. They are resent after reconnect along with the snapshot. | +| Master restarts mid-job | Jobs in `running` are reconciled when the agent reconnects; jobs the agent does not know are marked `failed` ("lost"). | +| New trigger while a job for the same service is queued or running | No new job; only `available_*` is updated. Two jobs never race on one service. | +| Job exceeds `UPDATE_TIMEOUT` | The agent cancels it, as today. | +| Polling fails for one image | Error stored in `last_poll_error` and shown; other images continue. | + +### Self-protection + +The agent reports its own container and the master knows its own service. Both are shown +as "updated manually outside the system", and the master refuses to create jobs for +them. An agent recreating itself would kill the job it is running. + +## Registry polling + +- The master polls, not the agents: credentials live in one place and identical + references are polled once. +- Every distinct `repo:tag` in use is polled, including `repo:tag@digest`. Bare + `repo@digest` and locally built images without a registry digest are skipped and + shown as such. +- Mechanism: `HEAD /v2//manifests/`; the returned digest is compared with + `actual_digest`. Use `go-containerregistry` — it handles Docker Hub's token auth and + multi-arch indexes. +- Interval `POLL_INTERVAL`, default 6h. Believed not to count against Docker Hub pull + limits for HEAD requests — verify during implementation. +- Credentials: a Docker `config.json` (as written by `docker login`) mounted into the + master. No credential UI, no encryption in Postgres. + +## Webhook + +- `POST /update` with `{image, tag}` and a bearer token — the same contract as today, so + the existing Gitea action works unchanged; only its `endpoint` moves to the master. +- `image` is matched against `image_repo` across all agents using today's normalisation + (`NormaliseImage`). Each match gets `available_ref = repo:tag` (source `webhook`) and + the version is recorded; `auto` services get a job. +- The response waits for the `auto` jobs to finish (bounded by `UPDATE_TIMEOUT`), and + returns 5xx if any failed or timed out, so CI still surfaces a broken deploy. `manual` matches are reported as "marked", not as + errors. +- No match returns **404**, which catches typos in CI configuration. +- The webhook never creates services. The first deployment of a stack is done by hand. + +## Authentication and UI + +### Authentication + +Three separate mechanisms that never mix: + +| Who | How | Where | +|---|---|---| +| People | OIDC authorization code flow with PKCE against Authentik (`coreos/go-oidc`, `x/oauth2`) | Web UI, via reverse proxy | +| CI | Shared bearer token | `POST /update`, via reverse proxy | +| Agents | Per-agent token + pinned master key over TLS | Dedicated agent port, internal network | + +- Sessions are stored in Postgres; the cookie holds a session ID (`HttpOnly`, `Secure`, + `SameSite=Lax`). +- The master also requires membership of `ALLOWED_GROUP` from the token's `groups` + claim. Restricting the application in Authentik alone is not enough: one misconfigured + binding would hand every Authentik user the ability to restart the fleet. +- All state-changing UI requests carry a CSRF token. +- `/healthz`, `/metrics` and `/version` stay unauthenticated, as today. + +### UI + +Server-rendered Go templates with htmx; live updates via htmx's SSE support. Embedded in +the master binary with `go:embed`. No Node build step. Visual design is drafted +separately; this spec defines what each page contains. + +1. **Overview** — all services grouped per machine (Swarm services per cluster). Each + shows status, running version, the manual/auto toggle, and badges for + *update available*, *drifted* and *follows no tag*. Live. +2. **Service** — the 10 most recent versions with an *Update* action, job history with + logs, drift actions (restore / accept), and *Merge* for moves. +3. **Agents** — online/offline, version. *Add agent* produces a compose snippet with + token and `MASTER_KEY`. *Revoke*. +4. **Jobs** — full history; running jobs stream their log live. + +## Code layout + +- `cmd/master`, `cmd/agent` — replace `cmd/server`. +- `internal/protocol` — message types shared by master and agent. +- `internal/agent/...` — today's `discovery` and `updater` (queue, Compose and Swarm + executors) move here, plus inventory, Docker events and the connection loop. +- `internal/master/...` — agent hub, jobs, polling, webhook, OIDC, storage. +- `web/` — templates and static assets. + +Existing conventions continue: interfaces defined at the consumer, hand-written fakes, +nil-able `*Metrics`, Conventional Commits. + +## Testing + +- `go test ./...` must keep passing without Docker or Postgres. +- Master logic (mode decisions, drift, merge, webhook matching, job reconciliation) is + tested against a storage interface with a hand-written fake. +- Postgres storage is tested by integration tests behind the `integration` build tag, + requiring `DATABASE_URL`. +- Master and agent are tested together in-process: an `httptest` TLS server, a real + WebSocket client and a fake Docker. Covers the `MASTER_KEY` handshake (including a + wrong key being rejected), reconnect with snapshot, and jobs completed while + disconnected. +- Table-driven tests with testify. + +## Migration from today's deployment + +1. Create the database and user in Postgres; register the application and group in + Authentik. +2. Deploy the master. +3. **Remove Shepherd and Watchtower.** Running alongside, they would update services + behind the master's back and every such update would show up as drift. +4. Add one agent per machine through the UI. Inventory fills in; everything is `manual`. +5. Point the Gitea action's `endpoint` at the master. Contract unchanged. +6. Remove today's package-updater instance. +7. Convert bare `repo@digest` references to `repo:tag@digest`, and set `auto` where + wanted. Recommended: record the convention in the self-hosted repo's `CLAUDE.md` + so Claude instances on the servers stop writing bare digests. + +`CLAUDE.md` in this repo is updated as part of the implementation: the "stateless" and +"token AND opt-in label AND STACKS_ROOT" design-intent entries no longer describe the +system, and the `MODE` entry is replaced by the agent model. + +## Implementation phases + +Each phase ends with something that works on its own. + +1. **Protocol, agent and master with inventory.** Pinned-key TLS, per-agent tokens, + snapshot and live events, Postgres storage, OIDC login, Overview and Agents pages. + The overview works; nothing can be updated yet. +2. **Jobs and manual update.** Job protocol, agent queue and executors (Compose override + file, Swarm with digest), job history, Service and Jobs pages, self-protection. +3. **Webhook and auto mode.** `POST /update` on the master, matching, waiting for + auto jobs, mode toggle. +4. **Polling and version history.** `image_versions`, registry polling, version picker + and rollback. +5. **Drift and merge.** Drift detection and actions, move suggestion and merge. + +## Out of scope (later specs) + +- Detecting newer version *tags* for images we do not build (e.g. `gitea:1.27.3` → + `1.27.4`, `postgres:17.10`, `loki:3.4.2`). Requires listing tags and ordering them; + major upgrades such as Postgres 17 → 18 must never be automatic. Revisit after + checking the other two servers' stack files. +- Notifications. +- Export/import of a fleet configuration to restore a setup. The `services` table + already holds what an export needs. +- Writing version bumps back to git. +- Self-update of master and agents (`internal/selfupdate` is the starting point). +- Updating plain (non-Compose, non-Swarm) containers. +- Credential management UI for registries.