package-updater/docs/superpowers/plans/2026-08-04-multi-server-fanout.md

Multi-Server Deployment & CI Fan-Out Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Land the finished Swarm support on main, document the one-instance-per-server deployment model, and make the Gitea composite action post /update to every server in a fleet instead of a single endpoint.

Architecture: No Go code changes. Part 1 is a merge plus documentation reframing — MODE=compose|swarm stays exclusive per instance, because each instance owns exactly one Docker daemon. Part 2 changes gitea-action/action.yml so its endpoint input accepts a newline-separated list, looping over every URL and continuing past failures so one dead host cannot stop the rest of the fleet from updating.

Tech Stack: Go 1.26.3 (unchanged, no code edits), Bash inside a Gitea composite action, curl, jq.

Global Constraints

• No Go source changes in this plan. If a task seems to need one, stop and re-read the spec — the design explicitly rejects composite Finder/Executor work.
• MODE stays compose or swarm only. Do not add a both value.
• Do not weaken Compose mode's three-factor gate (token AND opt-in label AND STACKS_ROOT prefix).
• Existing single-URL usages of the action must keep working unchanged — a single URL is a one-element list. Do not add a separate input for the list form.
• Separator for the endpoint list is newline, never comma. URLs may legally contain commas.
• Conventional Commits (feat:, fix:, docs:, ci:, build:, chore:).
• Full test suite (go test ./...) must be green at the end of every task that touches the repo.

---

Task 1: Merge Swarm support to main

Files:

• No files edited. This task merges branch worktree-docker-swarm-support (10 commits) into main.

Interfaces:

• Consumes: nothing.

• Produces: main gains internal/discovery/swarm.go, internal/updater/swarm_executor.go, config.Config.Mode, discovery.Job.Image, discovery.Job.ServiceID, and api.ResultDTO.ServiceID. Task 2 edits documentation files that this merge brings in (README.md "Swarm mode" section, CLAUDE.md swarm entries).

• Step 1: Confirm the branch is where you think it is

Run:

cd /Users/samuelenocsson/dev/package-updater
git log --oneline main..worktree-docker-swarm-support

Expected: exactly 10 commits, oldest 8f999c6 docs: add docker swarm support implementation plan, newest db84a2f fix(api): plumb request tag into discovery so swarm mode deploys the requested version.

If the list differs, stop and report — someone has moved the branch since this plan was written.

• Step 2: Verify the branch is green before merging

Run:

cd /Users/samuelenocsson/dev/package-updater/.claude/worktrees/docker-swarm-support
go build ./... && go test ./...

Expected: build succeeds, all packages PASS. Do not proceed on a red branch — fix or report first.

• Step 3: Merge into main

cd /Users/samuelenocsson/dev/package-updater
git checkout main
git merge --no-ff worktree-docker-swarm-support -m "$(cat <<'EOF'
Merge branch 'worktree-docker-swarm-support'

Adds MODE=compose|swarm. Each instance owns one Docker daemon and runs
one paradigm; see docs/superpowers/specs/2026-08-04-multi-server-fanout-design.md
for why this exclusivity is the intended design.
EOF
)"

--no-ff is deliberate: it keeps the ten swarm commits grouped under one merge commit, matching how PRs #1 and #2 already appear in this repo's history.

Note: docs/superpowers/plans/2026-07-04-docker-swarm-support.md currently exists as an untracked file on main and is committed on the swarm branch. The merge brings in the tracked copy. If git refuses the merge with "untracked working tree file would be overwritten", delete the untracked copy first (rm docs/superpowers/plans/2026-07-04-docker-swarm-support.md) — the branch's committed version is identical and authoritative.

• Step 4: Verify main is green after the merge

Run:

go build ./... && go test ./...

Expected: build succeeds, all packages PASS. This is the first time the swarm code and the post-swarm main commits (CLAUDE.md, CI workflow) are compiled together on main.

• Step 5: Push

git push origin main

---

Task 2: Document the deployment topology

Files:

• Modify: README.md (the version merged in Task 1 — it already contains a "Swarm mode" section at lines ~30-47, a MODE row in the config table at line ~71, and a "Known v1 gaps" bullet about single-host at lines ~103-105)
• Modify: CLAUDE.md (the "Design intent" section, which after Task 1 contains a "Compose mode and Swarm mode are selected once per deployment via MODE" entry)

Interfaces:

• Consumes: the merged documentation from Task 1.
• Produces: nothing consumed by later tasks. Task 3 documents the action separately in gitea-action/README.md.

There is no test for documentation. Verification is reading it back and confirming the claims match the code in cmd/server/main.go and internal/config/config.go.

• Step 1: Rebase the working branch onto the updated main

cd /Users/samuelenocsson/dev/package-updater
git checkout feat/multi-server-fanout
git rebase main

Expected: clean rebase. The branch currently holds one commit (docs: design for multi-server deployment and CI fan-out) touching only a new spec file, so there is nothing to conflict with.

• Step 2: Add a deployment-topology section to README.md

Insert this section immediately after the "Swarm mode" section and before "## Quick start":

## Deploying across multiple servers

An instance can only reach the Docker daemon it is configured against. It cannot
update stacks on other machines. The deployment model follows from that:

**One instance per server.** Each server runs its own package-updater against its
own local Docker socket, with `MODE` set to whatever that machine runs:

| Server | `MODE` | Updates |
|---|---|---|
| Swarm manager node | `swarm` | All Swarm services in the cluster |
| Standalone Compose host | `compose` | Compose stacks on that host |

**CI fans out.** The [Gitea composite action](gitea-action/README.md) takes a list of
endpoints and posts `/update` to every instance, so one workflow run reaches the
whole fleet. Each instance answers for its own machine; there is no aggregated
cross-server response and no instance coordinates any other.

This is why `MODE` is exclusive rather than a mode that handles both at once: an
instance that could do both would still only reach one daemon, so the extra
generality buys nothing.

• Step 3: Reword the MODE row in the configuration table

In README.md's configuration table, replace this row:

| `MODE` | no | `compose` | `compose` or `swarm`. Selects the update mechanism for the whole deployment; not mixed per-request. |

with:

| `MODE` | no | `compose` | `compose` or `swarm`. Which paradigm *this instance's* Docker daemon runs. See [Deploying across multiple servers](#deploying-across-multiple-servers). |

• Step 4: Replace the "single host only" gap bullet

In README.md's "Known v1 gaps" section, replace this bullet:

- **Single host only in Compose mode**. Swarm mode (`MODE=swarm`) is the
  multi-node path, but only from a manager node's point of view — the updater
  itself still needs manager API access (see "Swarm mode" above).

with:

- **One instance reaches one daemon.** An instance never updates another server;
  fleets run one instance per server with CI fanning out to all of them (see
  "Deploying across multiple servers"). Swarm mode is the exception in that a
  single manager-node instance covers the whole cluster.

• Step 5: Reword the design-intent entry in CLAUDE.md

In CLAUDE.md's "Design intent (do not break without discussion)" section, replace this entry:

- **Compose mode and Swarm mode are selected once per deployment via `MODE`**, never
  mixed at request time. Swarm mode's security gate is opt-in label only — there is
  no STACKS_ROOT-equivalent path check, since Swarm services have no local compose
  file. Don't add one; don't weaken Compose mode's three-factor gate to match.

with:

- **`MODE` is exclusive because each instance owns exactly one Docker daemon.**
  A composite "handle both at once" mode has been considered and rejected: it would
  still only reach one daemon, so it buys nothing. Fleets run one instance per
  server and CI fans out. Reopen this only if a Swarm manager node starts running
  standalone Compose stacks locally. Rationale and rejected alternatives:
  `docs/superpowers/specs/2026-08-04-multi-server-fanout-design.md`.
- **Swarm mode's security gate is opt-in label only** — there is no
  STACKS_ROOT-equivalent path check, since Swarm services have no local compose
  file. Don't add one; don't weaken Compose mode's three-factor gate to match.

Splitting the original entry in two is deliberate: the mode-exclusivity rationale and the Swarm security-gate rule are separate rules that were sharing a bullet, and only the first one is changing.

• Step 6: Verify the anchor link resolves

The MODE table row links to #deploying-across-multiple-servers. Confirm the heading added in Step 2 is exactly ## Deploying across multiple servers — GitHub/Gitea derive the anchor by lowercasing and replacing spaces with hyphens, so any wording drift silently breaks the link.

Run: grep -n "^## Deploying across multiple servers" README.md Expected: one match.

• Step 7: Commit

git add README.md CLAUDE.md
git commit -m "docs: describe one-instance-per-server topology and why MODE is exclusive"

---

Task 3: Fan out to multiple endpoints in the Gitea action

Files:

• Modify: gitea-action/action.yml (whole runs.steps block, lines 17-37, and the endpoint input description at lines 5-6)
• Modify: gitea-action/README.md (usage example, inputs table, failure-modes section)

Interfaces:

• Consumes: nothing from earlier tasks.

• Produces: the action's endpoint input accepts one URL (unchanged behaviour) or several separated by newlines. No other input changes name, type, or meaning.

• Step 1: Rewrite gitea-action/action.yml

Replace the entire file with:

name: "Deploy via package-updater"
description: "Notifies one or more package-updater instances to pull & restart a service"
inputs:
  endpoint:
    description: "Full URL to /update. Give several, one per line, to update a fleet."
    required: true
  image:
    description: "Image reference without tag (e.g. registry.example.com/myapp)"
    required: true
  tag:
    description: "Tag that was just pushed (for logging)"
    required: false
    default: ""
  token:
    description: "Bearer token for package-updater"
    required: true
runs:
  using: "composite"
  steps:
    - name: Trigger update
      shell: bash
      env:
        ENDPOINTS: ${{ inputs.endpoint }}
        IMAGE: ${{ inputs.image }}
        TAG: ${{ inputs.tag }}
        TOKEN: ${{ inputs.token }}
      run: |
        # No `set -e`: a failing endpoint must not abort the loop, or one dead
        # server leaves the rest of the fleet un-updated and hides which hosts
        # actually succeeded.
        set -uo pipefail

        payload=$(jq -nc --arg image "$IMAGE" --arg tag "$TAG" \
          '{image: $image, tag: $tag}')

        attempted=0
        failed=0

        while IFS= read -r endpoint; do
          # URLs never contain whitespace, so stripping all of it safely
          # handles indentation, blank lines and CRLF line endings.
          endpoint=$(printf '%s' "$endpoint" | tr -d '[:space:]')
          [ -z "$endpoint" ] && continue

          attempted=$((attempted + 1))
          echo "--- $endpoint"

          if ! response=$(curl -sS -w "\n%{http_code}" \
            -X POST "$endpoint" \
            -H "Authorization: Bearer $TOKEN" \
            -H "Content-Type: application/json" \
            -d "$payload"); then
            echo "unreachable"
            failed=$((failed + 1))
            continue
          fi

          # Split on the last newline: curl's -w appended the status code
          # there. Pure bash — `head -n -1` is GNU-only and fails on
          # BSD/macOS, so the fan-out could not be tested locally.
          code="${response##*$'\n'}"
          body="${response%$'\n'*}"
          echo "HTTP $code"
          printf '%s' "$body" | jq . || printf '%s\n' "$body"

          if [ "$code" -ge 400 ]; then
            failed=$((failed + 1))
          fi
        done <<< "$ENDPOINTS"

        if [ "$attempted" -eq 0 ]; then
          echo "no endpoints given"
          exit 1
        fi
        if [ "$failed" -gt 0 ]; then
          echo "$failed of $attempted endpoint(s) failed"
          exit 1
        fi
        echo "all $attempted endpoint(s) updated"

Four changes beyond the loop itself, each load-bearing:

1. All inputs moved into env:. A multi-line ${{ inputs.endpoint }} interpolated directly into the script body would break it syntactically. Moving the others too keeps one consistent style in a short script, and stops a value containing shell metacharacters from being executed.
2. jq -nc builds the payload instead of hand-interpolating into a JSON string. A tag containing a quote previously produced malformed JSON and a confusing 400.
3. set -e dropped, -uo pipefail kept. This is the whole point of the task — see the comment in the script.
4. head -n -1 replaced by bash parameter expansion. The existing action used head -n -1, which is GNU-only — it fails on BSD/macOS with illegal line count. That made the fan-out logic impossible to test on a developer machine, so the portability fix is what makes Steps 2-4 runnable at all. It also drops two subprocesses per endpoint.

• Step 2: Verify the loop continues past a failure

This is the behaviour that justifies the task, so test it directly rather than trusting the code by inspection.

Start a server that answers POST with 200:

python3 -c "
import http.server
class H(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(200)
        self.send_header('Content-Type','application/json')
        self.end_headers()
        self.wfile.write(b'{\"matched\":1,\"results\":[]}')
    def log_message(self, *a): pass
http.server.HTTPServer(('127.0.0.1',8099), H).serve_forever()
" &
echo $! > /tmp/fanout-test-server.pid

Write the script body to a runnable file. This must be the exact text of the run: block from Step 1, dedented — copy it, do not retype it, or you are testing something other than what ships:

cat > /tmp/fanout-test.sh <<'SCRIPT'
set -uo pipefail

payload=$(jq -nc --arg image "$IMAGE" --arg tag "$TAG" \
  '{image: $image, tag: $tag}')

attempted=0
failed=0

while IFS= read -r endpoint; do
  endpoint=$(printf '%s' "$endpoint" | tr -d '[:space:]')
  [ -z "$endpoint" ] && continue

  attempted=$((attempted + 1))
  echo "--- $endpoint"

  if ! response=$(curl -sS -w "\n%{http_code}" \
    -X POST "$endpoint" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "$payload"); then
    echo "unreachable"
    failed=$((failed + 1))
    continue
  fi

  code="${response##*$'\n'}"
  body="${response%$'\n'*}"
  echo "HTTP $code"
  printf '%s' "$body" | jq . || printf '%s\n' "$body"

  if [ "$code" -ge 400 ]; then
    failed=$((failed + 1))
  fi
done <<< "$ENDPOINTS"

if [ "$attempted" -eq 0 ]; then
  echo "no endpoints given"
  exit 1
fi
if [ "$failed" -gt 0 ]; then
  echo "$failed of $attempted endpoint(s) failed"
  exit 1
fi
echo "all $attempted endpoint(s) updated"
SCRIPT

Now execute it with a list where the middle endpoint is dead (port 1 always refuses connections):

ENDPOINTS="http://127.0.0.1:8099/update
http://127.0.0.1:1/update
http://127.0.0.1:8099/update" \
IMAGE="registry.example.com/myapp" \
TAG="abc123" \
TOKEN="dummy" \
bash /tmp/fanout-test.sh
echo "exit code: $?"

Expected output shape:

--- http://127.0.0.1:8099/update
HTTP 200
{ "matched": 1, "results": [] }
--- http://127.0.0.1:1/update
unreachable
--- http://127.0.0.1:8099/update
HTTP 200
{ "matched": 1, "results": [] }
1 of 3 endpoint(s) failed
exit code: 1

Three things must all hold: the third endpoint was attempted (proves the loop continued), the exit code is 1, and each endpoint's outcome is visible. If the third --- line is missing, set -e crept back in.

• Step 3: Verify a single endpoint still works unchanged

ENDPOINTS="http://127.0.0.1:8099/update" \
IMAGE="registry.example.com/myapp" TAG="abc123" TOKEN="dummy" \
bash /tmp/fanout-test.sh
echo "exit code: $?"

Expected: one --- line, HTTP 200, all 1 endpoint(s) updated, exit code 0. This is the existing-usage regression check.

• Step 4: Verify blank lines and indentation are tolerated

YAML block scalars routinely produce leading indentation and a trailing newline, so this is the realistic input shape, not an edge case.

ENDPOINTS="  http://127.0.0.1:8099/update

   http://127.0.0.1:8099/update
" \
IMAGE="registry.example.com/myapp" TAG="abc123" TOKEN="dummy" \
bash /tmp/fanout-test.sh
echo "exit code: $?"

Expected: exactly two --- http://127.0.0.1:8099/update lines with no stray whitespace in the URL, all 2 endpoint(s) updated, exit code 0.

• Step 5: Clean up the test server

kill "$(cat /tmp/fanout-test-server.pid)" && rm -f /tmp/fanout-test-server.pid /tmp/fanout-test.sh

• Step 6: Update gitea-action/README.md

Replace the usage example's with: block so it shows the fleet form:

      - uses: gitea.example.com/shcizo/package-updater/gitea-action@v1
        with:
          endpoint: |
            https://updater-swarm.example.com/update
            https://updater-web01.example.com/update
            https://updater-web02.example.com/update
          image: registry.example.com/${{ gitea.repository }}
          tag: ${{ gitea.sha }}
          token: ${{ secrets.UPDATER_TOKEN }}

Change the endpoint row in the inputs table to:

| `endpoint` | yes | — | Full URL to `/update`. Several may be given, one per line, to update a fleet. |

Replace the "Failure modes" section with:

## Failure modes

Every endpoint is attempted, even when an earlier one fails — otherwise one dead
server would leave the rest of the fleet un-updated, and the CI log would not show
which hosts actually succeeded.

The step exits non-zero if any endpoint returned 4xx/5xx or was unreachable. The
log lists each endpoint with its HTTP status and response body, so a partial
deploy is visible at a glance.

Add this note after the UPDATER_TOKEN sentence:

All instances in the fleet must share the same bearer token, since one `token`
input is sent to every endpoint.

• Step 7: Commit

git add gitea-action/action.yml gitea-action/README.md
git commit -m "feat(action): post /update to every endpoint in a fleet, continuing past failures"

• Step 8: Push and open a PR

git push -u origin feat/multi-server-fanout

Then open a PR against main (see issue-tracker-cli skill for whether this repo uses gh or tea). The PR covers the spec, the documentation reframing, and the action fan-out; the Swarm merge landed separately in Task 1.