package-updater/docs/superpowers/plans/2026-08-04-multi-server-fanout.md

# Multi-Server Deployment & CI Fan-Out Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Land the finished Swarm support on `main`, document the one-instance-per-server deployment model, and make the Gitea composite action post `/update` to every server in a fleet instead of a single endpoint.

**Architecture:** No Go code changes. Part 1 is a merge plus documentation reframing — `MODE=compose|swarm` stays exclusive per instance, because each instance owns exactly one Docker daemon. Part 2 changes `gitea-action/action.yml` so its `endpoint` input accepts a newline-separated list, looping over every URL and continuing past failures so one dead host cannot stop the rest of the fleet from updating.

**Tech Stack:** Go 1.26.3 (unchanged, no code edits), Bash inside a Gitea composite action, `curl`, `jq`.

## Global Constraints

- **No Go source changes in this plan.** If a task seems to need one, stop and re-read the spec — the design explicitly rejects composite Finder/Executor work.
- `MODE` stays `compose` or `swarm` only. Do not add a `both` value.
- Do not weaken Compose mode's three-factor gate (token AND opt-in label AND `STACKS_ROOT` prefix).
- Existing single-URL usages of the action must keep working unchanged — a single URL is a one-element list. Do not add a separate input for the list form.
- Separator for the endpoint list is **newline**, never comma. URLs may legally contain commas.
- Conventional Commits (`feat:`, `fix:`, `docs:`, `ci:`, `build:`, `chore:`).
- Full test suite (`go test ./...`) must be green at the end of every task that touches the repo.

---

### Task 1: Merge Swarm support to `main`

**Files:**
- No files edited. This task merges branch `worktree-docker-swarm-support` (10 commits) into `main`.

**Interfaces:**
- Consumes: nothing.
- Produces: `main` gains `internal/discovery/swarm.go`, `internal/updater/swarm_executor.go`, `config.Config.Mode`, `discovery.Job.Image`, `discovery.Job.ServiceID`, and `api.ResultDTO.ServiceID`. Task 2 edits documentation files that this merge brings in (`README.md` "Swarm mode" section, `CLAUDE.md` swarm entries).

- [ ] **Step 1: Confirm the branch is where you think it is**

Run:
```bash
cd /Users/samuelenocsson/dev/package-updater
git log --oneline main..worktree-docker-swarm-support
```

Expected: exactly 10 commits, oldest `8f999c6 docs: add docker swarm support implementation plan`, newest `db84a2f fix(api): plumb request tag into discovery so swarm mode deploys the requested version`.

If the list differs, stop and report — someone has moved the branch since this plan was written.

- [ ] **Step 2: Verify the branch is green before merging**

Run:
```bash
cd /Users/samuelenocsson/dev/package-updater/.claude/worktrees/docker-swarm-support
go build ./... && go test ./...
```

Expected: build succeeds, all packages PASS. Do not proceed on a red branch — fix or report first.

- [ ] **Step 3: Merge into `main`**

```bash
cd /Users/samuelenocsson/dev/package-updater
git checkout main
git merge --no-ff worktree-docker-swarm-support -m "$(cat <<'EOF'
Merge branch 'worktree-docker-swarm-support'

Adds MODE=compose|swarm. Each instance owns one Docker daemon and runs
one paradigm; see docs/superpowers/specs/2026-08-04-multi-server-fanout-design.md
for why this exclusivity is the intended design.
EOF
)"
```

`--no-ff` is deliberate: it keeps the ten swarm commits grouped under one merge commit, matching how PRs #1 and #2 already appear in this repo's history.

Note: `docs/superpowers/plans/2026-07-04-docker-swarm-support.md` currently exists as an *untracked* file on `main` and is *committed* on the swarm branch. The merge brings in the tracked copy. If git refuses the merge with "untracked working tree file would be overwritten", delete the untracked copy first (`rm docs/superpowers/plans/2026-07-04-docker-swarm-support.md`) — the branch's committed version is identical and authoritative.

- [ ] **Step 4: Verify `main` is green after the merge**

Run:
```bash
go build ./... && go test ./...
```

Expected: build succeeds, all packages PASS. This is the first time the swarm code and the post-swarm `main` commits (`CLAUDE.md`, CI workflow) are compiled together on `main`.

- [ ] **Step 5: Push**

```bash
git push origin main
```

---

### Task 2: Document the deployment topology

**Files:**
- Modify: `README.md` (the version merged in Task 1 — it already contains a "Swarm mode" section at lines ~30-47, a `MODE` row in the config table at line ~71, and a "Known v1 gaps" bullet about single-host at lines ~103-105)
- Modify: `CLAUDE.md` (the "Design intent" section, which after Task 1 contains a "Compose mode and Swarm mode are selected once per deployment via `MODE`" entry)

**Interfaces:**
- Consumes: the merged documentation from Task 1.
- Produces: nothing consumed by later tasks. Task 3 documents the action separately in `gitea-action/README.md`.

There is no test for documentation. Verification is reading it back and confirming the claims match the code in `cmd/server/main.go` and `internal/config/config.go`.

- [ ] **Step 1: Rebase the working branch onto the updated `main`**

```bash
cd /Users/samuelenocsson/dev/package-updater
git checkout feat/multi-server-fanout
git rebase main
```

Expected: clean rebase. The branch currently holds one commit (`docs: design for multi-server deployment and CI fan-out`) touching only a new spec file, so there is nothing to conflict with.

- [ ] **Step 2: Add a deployment-topology section to `README.md`**

Insert this section immediately **after** the "Swarm mode" section and **before** "## Quick start":

```markdown
## Deploying across multiple servers

An instance can only reach the Docker daemon it is configured against. It cannot
update stacks on other machines. The deployment model follows from that:

**One instance per server.** Each server runs its own package-updater against its
own local Docker socket, with `MODE` set to whatever that machine runs:

| Server | `MODE` | Updates |
|---|---|---|
| Swarm manager node | `swarm` | All Swarm services in the cluster |
| Standalone Compose host | `compose` | Compose stacks on that host |

**CI fans out.** The [Gitea composite action](gitea-action/README.md) takes a list of
endpoints and posts `/update` to every instance, so one workflow run reaches the
whole fleet. Each instance answers for its own machine; there is no aggregated
cross-server response and no instance coordinates any other.

This is why `MODE` is exclusive rather than a mode that handles both at once: an
instance that could do both would still only reach one daemon, so the extra
generality buys nothing.
```

- [ ] **Step 3: Reword the `MODE` row in the configuration table**

In `README.md`'s configuration table, replace this row:

```markdown
| `MODE` | no | `compose` | `compose` or `swarm`. Selects the update mechanism for the whole deployment; not mixed per-request. |
```

with:

```markdown
| `MODE` | no | `compose` | `compose` or `swarm`. Which paradigm *this instance's* Docker daemon runs. See [Deploying across multiple servers](#deploying-across-multiple-servers). |
```

- [ ] **Step 4: Replace the "single host only" gap bullet**

In `README.md`'s "Known v1 gaps" section, replace this bullet:

```markdown
- **Single host only in Compose mode**. Swarm mode (`MODE=swarm`) is the
  multi-node path, but only from a manager node's point of view — the updater
  itself still needs manager API access (see "Swarm mode" above).
```

with:

```markdown
- **One instance reaches one daemon.** An instance never updates another server;
  fleets run one instance per server with CI fanning out to all of them (see
  "Deploying across multiple servers"). Swarm mode is the exception in that a
  single manager-node instance covers the whole cluster.
```

- [ ] **Step 5: Reword the design-intent entry in `CLAUDE.md`**

In `CLAUDE.md`'s "Design intent (do not break without discussion)" section, replace this entry:

```markdown
- **Compose mode and Swarm mode are selected once per deployment via `MODE`**, never
  mixed at request time. Swarm mode's security gate is opt-in label only — there is
  no STACKS_ROOT-equivalent path check, since Swarm services have no local compose
  file. Don't add one; don't weaken Compose mode's three-factor gate to match.
```

with:

```markdown
- **`MODE` is exclusive because each instance owns exactly one Docker daemon.**
  A composite "handle both at once" mode has been considered and rejected: it would
  still only reach one daemon, so it buys nothing. Fleets run one instance per
  server and CI fans out. Reopen this only if a Swarm manager node starts running
  standalone Compose stacks locally. Rationale and rejected alternatives:
  `docs/superpowers/specs/2026-08-04-multi-server-fanout-design.md`.
- **Swarm mode's security gate is opt-in label only** — there is no
  STACKS_ROOT-equivalent path check, since Swarm services have no local compose
  file. Don't add one; don't weaken Compose mode's three-factor gate to match.
```

Splitting the original entry in two is deliberate: the mode-exclusivity rationale and the Swarm security-gate rule are separate rules that were sharing a bullet, and only the first one is changing.

- [ ] **Step 6: Verify the anchor link resolves**

The `MODE` table row links to `#deploying-across-multiple-servers`. Confirm the heading added in Step 2 is exactly `## Deploying across multiple servers` — GitHub/Gitea derive the anchor by lowercasing and replacing spaces with hyphens, so any wording drift silently breaks the link.

Run: `grep -n "^## Deploying across multiple servers" README.md`
Expected: one match.

- [ ] **Step 7: Commit**

```bash
git add README.md CLAUDE.md
git commit -m "docs: describe one-instance-per-server topology and why MODE is exclusive"
```

---

### Task 3: Fan out to multiple endpoints in the Gitea action

**Files:**
- Modify: `gitea-action/action.yml` (whole `runs.steps` block, lines 17-37, and the `endpoint` input description at lines 5-6)
- Modify: `gitea-action/README.md` (usage example, inputs table, failure-modes section)

**Interfaces:**
- Consumes: nothing from earlier tasks.
- Produces: the action's `endpoint` input accepts one URL (unchanged behaviour) or several separated by newlines. No other input changes name, type, or meaning.

- [ ] **Step 1: Rewrite `gitea-action/action.yml`**

Replace the entire file with:

```yaml
name: "Deploy via package-updater"
description: "Notifies one or more package-updater instances to pull & restart a service"
inputs:
  endpoint:
    description: "Full URL to /update. Give several, one per line, to update a fleet."
    required: true
  image:
    description: "Image reference without tag (e.g. registry.example.com/myapp)"
    required: true
  tag:
    description: "Tag that was just pushed (for logging)"
    required: false
    default: ""
  token:
    description: "Bearer token for package-updater"
    required: true
runs:
  using: "composite"
  steps:
    - name: Trigger update
      shell: bash
      env:
        ENDPOINTS: ${{ inputs.endpoint }}
        IMAGE: ${{ inputs.image }}
        TAG: ${{ inputs.tag }}
        TOKEN: ${{ inputs.token }}
      run: |
        # No `set -e`: a failing endpoint must not abort the loop, or one dead
        # server leaves the rest of the fleet un-updated and hides which hosts
        # actually succeeded.
        set -uo pipefail

        payload=$(jq -nc --arg image "$IMAGE" --arg tag "$TAG" \
          '{image: $image, tag: $tag}')

        attempted=0
        failed=0

        while IFS= read -r endpoint; do
          # URLs never contain whitespace, so stripping all of it safely
          # handles indentation, blank lines and CRLF line endings.
          endpoint=$(printf '%s' "$endpoint" | tr -d '[:space:]')
          [ -z "$endpoint" ] && continue

          attempted=$((attempted + 1))
          echo "--- $endpoint"

          if ! response=$(curl -sS -w "\n%{http_code}" \
            -X POST "$endpoint" \
            -H "Authorization: Bearer $TOKEN" \
            -H "Content-Type: application/json" \
            -d "$payload"); then
            echo "unreachable"
            failed=$((failed + 1))
            continue
          fi

          body=$(printf '%s' "$response" | head -n -1)
          code=$(printf '%s' "$response" | tail -n 1)
          echo "HTTP $code"
          printf '%s' "$body" | jq . || printf '%s\n' "$body"

          if [ "$code" -ge 400 ]; then
            failed=$((failed + 1))
          fi
        done <<< "$ENDPOINTS"

        if [ "$attempted" -eq 0 ]; then
          echo "no endpoints given"
          exit 1
        fi
        if [ "$failed" -gt 0 ]; then
          echo "$failed of $attempted endpoint(s) failed"
          exit 1
        fi
        echo "all $attempted endpoint(s) updated"
```

Three changes beyond the loop itself, each load-bearing:

1. **All inputs moved into `env:`.** A multi-line `${{ inputs.endpoint }}` interpolated directly into the script body would break it syntactically. Moving the others too keeps one consistent style in a short script, and stops a value containing shell metacharacters from being executed.
2. **`jq -nc` builds the payload** instead of hand-interpolating into a JSON string. A tag containing a quote previously produced malformed JSON and a confusing 400.
3. **`set -e` dropped**, `-uo pipefail` kept. This is the whole point of the task — see the comment in the script.

- [ ] **Step 2: Verify the loop continues past a failure**

This is the behaviour that justifies the task, so test it directly rather than trusting the code by inspection.

Start a server that answers POST with 200:

```bash
python3 -c "
import http.server
class H(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(200)
        self.send_header('Content-Type','application/json')
        self.end_headers()
        self.wfile.write(b'{\"matched\":1,\"results\":[]}')
    def log_message(self, *a): pass
http.server.HTTPServer(('127.0.0.1',8099), H).serve_forever()
" &
echo $! > /tmp/fanout-test-server.pid
```

Write the script body to a runnable file. This must be the **exact** text of the
`run:` block from Step 1, dedented — copy it, do not retype it, or you are testing
something other than what ships:

```bash
cat > /tmp/fanout-test.sh <<'SCRIPT'
set -uo pipefail

payload=$(jq -nc --arg image "$IMAGE" --arg tag "$TAG" \
  '{image: $image, tag: $tag}')

attempted=0
failed=0

while IFS= read -r endpoint; do
  endpoint=$(printf '%s' "$endpoint" | tr -d '[:space:]')
  [ -z "$endpoint" ] && continue

  attempted=$((attempted + 1))
  echo "--- $endpoint"

  if ! response=$(curl -sS -w "\n%{http_code}" \
    -X POST "$endpoint" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "$payload"); then
    echo "unreachable"
    failed=$((failed + 1))
    continue
  fi

  body=$(printf '%s' "$response" | head -n -1)
  code=$(printf '%s' "$response" | tail -n 1)
  echo "HTTP $code"
  printf '%s' "$body" | jq . || printf '%s\n' "$body"

  if [ "$code" -ge 400 ]; then
    failed=$((failed + 1))
  fi
done <<< "$ENDPOINTS"

if [ "$attempted" -eq 0 ]; then
  echo "no endpoints given"
  exit 1
fi
if [ "$failed" -gt 0 ]; then
  echo "$failed of $attempted endpoint(s) failed"
  exit 1
fi
echo "all $attempted endpoint(s) updated"
SCRIPT
```

Now execute it with a list where the middle endpoint is dead (port 1 always refuses connections):

```bash
ENDPOINTS="http://127.0.0.1:8099/update
http://127.0.0.1:1/update
http://127.0.0.1:8099/update" \
IMAGE="registry.example.com/myapp" \
TAG="abc123" \
TOKEN="dummy" \
bash /tmp/fanout-test.sh
echo "exit code: $?"
```

Expected output shape:
```
--- http://127.0.0.1:8099/update
HTTP 200
{ "matched": 1, "results": [] }
--- http://127.0.0.1:1/update
unreachable
--- http://127.0.0.1:8099/update
HTTP 200
{ "matched": 1, "results": [] }
1 of 3 endpoint(s) failed
exit code: 1
```

Three things must all hold: the **third** endpoint was attempted (proves the loop continued), the exit code is **1**, and each endpoint's outcome is visible. If the third `---` line is missing, `set -e` crept back in.

- [ ] **Step 3: Verify a single endpoint still works unchanged**

```bash
ENDPOINTS="http://127.0.0.1:8099/update" \
IMAGE="registry.example.com/myapp" TAG="abc123" TOKEN="dummy" \
bash /tmp/fanout-test.sh
echo "exit code: $?"
```

Expected: one `--- ` line, `HTTP 200`, `all 1 endpoint(s) updated`, exit code 0. This is the existing-usage regression check.

- [ ] **Step 4: Verify blank lines and indentation are tolerated**

YAML block scalars routinely produce leading indentation and a trailing newline, so this is the realistic input shape, not an edge case.

```bash
ENDPOINTS="  http://127.0.0.1:8099/update

   http://127.0.0.1:8099/update
" \
IMAGE="registry.example.com/myapp" TAG="abc123" TOKEN="dummy" \
bash /tmp/fanout-test.sh
echo "exit code: $?"
```

Expected: exactly two `--- http://127.0.0.1:8099/update` lines with no stray whitespace in the URL, `all 2 endpoint(s) updated`, exit code 0.

- [ ] **Step 5: Clean up the test server**

```bash
kill "$(cat /tmp/fanout-test-server.pid)" && rm -f /tmp/fanout-test-server.pid /tmp/fanout-test.sh
```

- [ ] **Step 6: Update `gitea-action/README.md`**

Replace the usage example's `with:` block so it shows the fleet form:

```yaml
      - uses: gitea.example.com/shcizo/package-updater/gitea-action@v1
        with:
          endpoint: |
            https://updater-swarm.example.com/update
            https://updater-web01.example.com/update
            https://updater-web02.example.com/update
          image: registry.example.com/${{ gitea.repository }}
          tag: ${{ gitea.sha }}
          token: ${{ secrets.UPDATER_TOKEN }}
```

Change the `endpoint` row in the inputs table to:

```markdown
| `endpoint` | yes | — | Full URL to `/update`. Several may be given, one per line, to update a fleet. |
```

Replace the "Failure modes" section with:

```markdown
## Failure modes

Every endpoint is attempted, even when an earlier one fails — otherwise one dead
server would leave the rest of the fleet un-updated, and the CI log would not show
which hosts actually succeeded.

The step exits non-zero if any endpoint returned 4xx/5xx or was unreachable. The
log lists each endpoint with its HTTP status and response body, so a partial
deploy is visible at a glance.
```

Add this note after the `UPDATER_TOKEN` sentence:

```markdown
All instances in the fleet must share the same bearer token, since one `token`
input is sent to every endpoint.
```

- [ ] **Step 7: Commit**

```bash
git add gitea-action/action.yml gitea-action/README.md
git commit -m "feat(action): post /update to every endpoint in a fleet, continuing past failures"
```

- [ ] **Step 8: Push and open a PR**

```bash
git push -u origin feat/multi-server-fanout
```

Then open a PR against `main` (see `issue-tracker-cli` skill for whether this repo uses `gh` or `tea`). The PR covers the spec, the documentation reframing, and the action fan-out; the Swarm merge landed separately in Task 1.