package-updater/docs/superpowers/specs/2026-05-22-package-updater-design.md

# package-updater — Design Spec

**Date:** 2026-05-22
**Status:** Approved (brainstorm complete, ready for implementation planning)
**Author:** Samuel Enocsson

---

## 1. Purpose

A small HTTP service that receives webhook-style notifications from Gitea Actions (or any CI) and triggers a pull + restart of the corresponding Docker Compose service on the host. It fills the gap between Watchtower (polling, no CI integration) and full GitOps (Argo CD / Flux) for a self-hosted, single-host environment.

**Concrete trigger:** Gitea workflow builds and pushes `registry.example.com/myapp:v1.2.3` to the container registry, then calls `POST /update` with the image name. The service finds the matching Compose-managed container(s) on the host, pulls the new image, and restarts the relevant service(s).

Watchtower remains in place for third-party images that the user does not build themselves.

## 2. Non-Goals (v1)

- Multi-host orchestration (single Docker host only)
- Real rollback (relies on Compose's "keep old container if new fails to start")
- Persistent state, audit history, or web UI (logs live in Loki)
- Per-repo / per-image API keys (single shared bearer token)
- Wildcards/regex in image matching
- Notifications outside of Gitea (workflow failure is visible there already)
- Rate limiting
- TLS termination (handled by Nginx Proxy Manager in front of the service)

## 3. Architecture

```
┌──────────────────┐     POST /update      ┌─────────────────────┐
│ Gitea Actions    │ ─────────────────────▶│ package-updater     │
│ (composite       │   Bearer <token>      │ (Go service,        │
│  action)         │   {image, tag}        │  container)         │
└──────────────────┘                       └──────────┬──────────┘
                                                      │
                                          docker.sock │ (mounted)
                                                      ▼
                                           ┌─────────────────────┐
                                           │ Docker daemon       │
                                           │ + running containers│
                                           │ + Compose stacks    │
                                           └─────────────────────┘
```

### 3.1 High-level flow

1. Gitea workflow builds + pushes `registry.example.com/myapp:v1.2.3`.
2. Workflow calls `POST /update` with `{image, tag}` + bearer token.
3. Service validates token (constant-time compare).
4. Service lists Docker containers (running + stopped), filters by:
   - Image name match (tag-agnostic)
   - Opt-in label `se.shcizo.auto-update=true`
5. For each match, reads Compose's built-in labels to find `working_dir`, `config_files`, `service`, `project`.
6. Path safety check: refuse jobs whose `working_dir` is not inside `STACKS_ROOT`.
7. Deduplicates `(project, service, config_files)` and enqueues one job per unique tuple.
8. Single FIFO worker executes jobs synchronously: `docker compose -f <file> -p <project> pull <service>` then `up -d <service>`.
9. Response aggregates per-job results.

### 3.2 Design principles

- **Stateless** — no database, no config file beyond environment variables. The Docker daemon is the source of truth.
- **Idempotent** — re-running the same update is safe. Zero matches returns `200` with an empty result list.
- **Fail-safe** — per-job failures do not stop other jobs. Compose's normal behaviour preserves the old container if the new one fails to start.
- **Defense in depth** — two independent gates (bearer token AND opt-in label) must both pass before any container is touched.

## 4. API Contract

### 4.1 `POST /update`

**Headers:**
```
Authorization: Bearer <api-key>
Content-Type: application/json
```

**Request body:**
```json
{
  "image": "registry.example.com/myapp",
  "tag": "v1.2.3"
}
```

- `image` — **required**. Image reference without tag. Matched against container image (tag-agnostic).
- `tag` — **optional**. For logging/audit only. The actual tag pulled is governed by the `image:` line in the Compose file.

**Response codes:**

| Status | Meaning |
|---|---|
| `200 OK` | All jobs succeeded (or zero matches) |
| `207 Multi-Status` | Mixed success/failure across jobs |
| `400 Bad Request` | Missing or invalid `image` field |
| `401 Unauthorized` | Missing or invalid token |
| `500 Internal Server Error` | Docker daemon unreachable, or all jobs failed |

**Success response body:**
```json
{
  "request_id": "0d8c4b9e-7a1f-4d2c-8b1a-3c5e6f7a8b9d",
  "image": "registry.example.com/myapp",
  "tag": "v1.2.3",
  "matched": 2,
  "results": [
    {
      "project": "myapp-prod",
      "service": "web",
      "compose_file": "/home/shcizo/self-hosted/myapp-prod/docker-compose.yml",
      "status": "updated",
      "duration_ms": 4231
    },
    {
      "project": "myapp-staging",
      "service": "web",
      "compose_file": "/home/shcizo/self-hosted/myapp-staging/docker-compose.yml",
      "status": "failed",
      "error": "pull: manifest unknown",
      "duration_ms": 812
    }
  ]
}
```

**`status` values:** `updated`, `failed`, `refused` (path outside `STACKS_ROOT`), `timeout`.

### 4.2 `GET /healthz`

Returns `200 OK` if the process is alive AND `cli.Ping(ctx)` against the Docker socket succeeds. Used by container healthcheck and external uptime monitoring.

### 4.3 `GET /metrics`

Prometheus exposition format. No auth (internal network only).

### 4.4 `GET /version`

Returns build info (version, commit, build time).

## 5. Discovery & Matching

### 5.1 Container lookup

```go
containers, err := cli.ContainerList(ctx, container.ListOptions{All: true})
```

`All: true` so we also catch crashed/stopped containers we may want to restart.

### 5.2 Image matching

Tag-agnostic exact match after normalisation:

```
input:     "registry.example.com/myapp"
container: "registry.example.com/myapp:v1.2.2@sha256:abc..."
           → strip tag and digest → "registry.example.com/myapp"
           → MATCH
```

Case-sensitive. No wildcards or regex (YAGNI).

### 5.3 Opt-in filter

Container must have label `se.shcizo.auto-update=true`. Anything else (`false`, missing, other value) is silently excluded.

### 5.4 Compose label extraction

| Label | Used for |
|---|---|
| `com.docker.compose.project` | `-p <project>` flag |
| `com.docker.compose.project.working_dir` | Working directory for `docker compose` exec |
| `com.docker.compose.project.config_files` | `-f <file>` flag(s) (comma-separated list) |
| `com.docker.compose.service` | Which service to pull/restart |

If any of these labels are missing (container not started by Compose), the container is excluded with a warning log. This should not happen in practice — the whole tool is Compose-centric — but we don't crash.

### 5.5 Path safety check

Before running any `docker compose` command, verify that `working_dir` is inside `STACKS_ROOT` (default: `/home/shcizo/self-hosted`). If not, the job is marked `refused` and logged at warning level. This is a defense-in-depth measure against forged or unexpected Docker labels.

### 5.6 Deduplication

Multiple containers may belong to the same `(project, service, config_files)` tuple (e.g. Compose `deploy.replicas > 1`). Collapse to a single job using:

```
dedupKey = project + "|" + service + "|" + sorted(config_files)
```

### 5.7 Concurrency

A single global FIFO queue with one worker. The HTTP handler blocks until all jobs for the request are complete, so the caller receives the result synchronously. The single-worker model gives per-stack mutual exclusion implicitly — no race conditions between concurrent updates against the same stack.

## 6. Self-Update Handling

When the service receives an update whose image matches its own running container, it must finish writing the HTTP response before exec'ing `docker compose up -d` against itself (otherwise the response is lost when the container is replaced).

**Implementation:** detect self-update at job dispatch time. For self-update jobs, the worker writes the HTTP response, flushes, closes the connection, then waits ~1 second before invoking the `up -d` command. All other jobs execute normally.

**Acceptable residual risk:** if a buggy version is pushed that fails to start, the running version dies (Compose can't restart what won't start) and the user must SSH in and run `docker compose up -d` manually with a previous tag. This is the same risk any self-updating system carries; mitigation is a deploy discipline issue, not a code one.

## 7. Security Model

### 7.1 Authentication

- Single bearer token read from env var `UPDATER_API_KEY` at startup. Service refuses to start if it is missing or empty (fail-fast).
- Token compared with `crypto/subtle.ConstantTimeCompare` to defeat timing attacks.
- Token is never logged.

### 7.2 Authorisation

A container is eligible for update only if it has both:
- An image name matching the request, AND
- The opt-in label `se.shcizo.auto-update=true`.

Both gates are independent. Compromising either alone does not allow an attacker to trigger an update.

### 7.3 Network exposure

The service binds `0.0.0.0:8080` inside the container and is reached via Nginx Proxy Manager which terminates TLS and forwards on the internal Docker network. The service itself does not handle TLS.

### 7.4 Socket access

The service mounts `/var/run/docker.sock` and therefore has effective root on the host. This is the same trust model as Watchtower. The opt-in label is the second gate that limits blast radius from API-layer abuse.

## 8. Deployment

### 8.1 Service's own Compose stack

Path: `/home/shcizo/self-hosted/package-updater/docker-compose.yml`

```yaml
services:
  package-updater:
    image: registry.example.com/package-updater:latest
    container_name: package-updater
    restart: unless-stopped
    environment:
      - UPDATER_API_KEY=${UPDATER_API_KEY}
      - STACKS_ROOT=/home/shcizo/self-hosted
      - LOG_LEVEL=info
      - PORT=8080
      - UPDATE_TIMEOUT=5m
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/shcizo/self-hosted:/home/shcizo/self-hosted:ro
    labels:
      - "se.shcizo.auto-update=true"
    networks:
      - proxy
    healthcheck:
      test: ["CMD", "wget", "-q", "-O-", "http://localhost:8080/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3

networks:
  proxy:
    external: true
```

`.env` next to it (git-ignored):
```
UPDATER_API_KEY=<openssl rand -hex 32>
```

NPM is configured with a proxy host pointing at `package-updater:8080` with TLS.

### 8.2 Dockerfile

Multi-stage, statically-linked Go binary:

```dockerfile
FROM golang:1.23-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/package-updater ./cmd/server

FROM alpine:3.20
RUN apk add --no-cache docker-cli docker-cli-compose ca-certificates wget
COPY --from=build /out/package-updater /usr/local/bin/
ENTRYPOINT ["/usr/local/bin/package-updater"]
```

`docker-cli` and `docker-cli-compose` are installed because we shell out to `docker compose` for pull/up. Going through the CLI handles dependent services and networks correctly out of the box; calling the Docker API directly would require us to replicate that logic.

Expected image size: ~50 MB.

### 8.3 Stacks root convention

All Compose stacks eligible for auto-update must live under `STACKS_ROOT` (default `/home/shcizo/self-hosted`). This is required so that the path stored in `com.docker.compose.project.working_dir` (a host path) resolves to the same path inside the service container. The mount uses identical source and target paths to avoid any path translation logic.

Read-only mount is sufficient — the service only reads Compose files; it never writes them.

## 9. Gitea Action

A reusable composite action lives in a separate repository, e.g. `gitea.example.com/samuel/action-deploy-update`.

`action.yml`:
```yaml
name: "Deploy via package-updater"
description: "Notifies package-updater to pull & restart a service"
inputs:
  endpoint:
    description: "Full URL to /update (e.g. https://updater.example.com/update)"
    required: true
  image:
    description: "Image reference without tag (e.g. registry.example.com/myapp)"
    required: true
  tag:
    description: "Tag that was just pushed (for logging)"
    required: false
    default: ""
  token:
    description: "Bearer token for package-updater"
    required: true
runs:
  using: "composite"
  steps:
    - name: Trigger update
      shell: bash
      env:
        TOKEN: ${{ inputs.token }}
      run: |
        set -euo pipefail
        response=$(curl -sS -w "\n%{http_code}" \
          -X POST "${{ inputs.endpoint }}" \
          -H "Authorization: Bearer $TOKEN" \
          -H "Content-Type: application/json" \
          -d "{\"image\":\"${{ inputs.image }}\",\"tag\":\"${{ inputs.tag }}\"}")
        body=$(echo "$response" | head -n -1)
        code=$(echo "$response" | tail -n 1)
        echo "HTTP $code"
        echo "$body" | jq .
        if [ "$code" -ge 400 ]; then
          exit 1
        fi
```

Consumer workflow (`.gitea/workflows/deploy.yml`):
```yaml
jobs:
  deploy:
    runs-on: ubuntu-latest
    needs: [build-and-push]
    steps:
      - uses: gitea.example.com/samuel/action-deploy-update@v1
        with:
          endpoint: https://updater.example.com/update
          image: registry.example.com/${{ gitea.repository }}
          tag: ${{ gitea.sha }}
          token: ${{ secrets.UPDATER_TOKEN }}
```

`UPDATER_TOKEN` is set as an organisation-level secret so all repos share it.

## 10. Error Handling

| Phase | Possible failure | Behaviour |
|---|---|---|
| Auth | Missing/invalid token | `401`, warning log with client IP |
| Validation | Missing/invalid `image` | `400`, no action |
| Docker daemon | Socket unreachable | `500`, error log |
| Discovery | Zero matches (after opt-in filter) | `200` with `matched: 0`, info log |
| Path safety | `working_dir` outside `STACKS_ROOT` | Skip job with `status: refused`, warning log. Other jobs continue. |
| `docker compose pull` | Manifest unknown, registry auth fail, network timeout | Per-job `status: failed`. No rollback (old container still running). |
| `docker compose up -d` | Validation error, port conflict, etc. | Per-job `status: failed`. Compose retains old container if new fails. |
| Per-job timeout | Job exceeds `UPDATE_TIMEOUT` (default 5m) | Cancel and mark `status: timeout`. |

**Aggregate HTTP status:**
- All jobs `updated` → `200`
- Mixed → `207`
- All jobs `failed`/`refused`/`timeout` → `500`
- `matched: 0` → `200` (idempotent: "nothing to do" is not an error)

No rollback in v1. If you push a buggy image, Compose keeps the old container if the new one fails to start. Real rollback (re-pin previous tag) would require tracking previous tags per service, which we don't.

## 11. Observability

### 11.1 Logs (Loki)

Structured JSON to stdout. Promtail/Alloy already scrapes Docker container logs and forwards to Loki. Example records:

```json
{"time":"2026-05-22T14:32:11Z","level":"info","event":"update_request","request_id":"0d8c4b9e","image":"registry.example.com/myapp","tag":"v1.2.3","client_ip":"10.0.0.5"}
{"time":"2026-05-22T14:32:11Z","level":"info","event":"matched","request_id":"0d8c4b9e","image":"registry.example.com/myapp","matched":2}
{"time":"2026-05-22T14:32:11Z","level":"info","event":"job_start","request_id":"0d8c4b9e","project":"myapp-prod","service":"web"}
{"time":"2026-05-22T14:32:15Z","level":"info","event":"job_complete","request_id":"0d8c4b9e","project":"myapp-prod","service":"web","status":"updated","duration_ms":4231}
{"time":"2026-05-22T14:32:15Z","level":"error","event":"job_complete","request_id":"0d8c4b9e","project":"myapp-staging","service":"web","status":"failed","error":"pull: manifest unknown","duration_ms":812}
{"time":"2026-05-22T14:32:15Z","level":"info","event":"update_response","request_id":"0d8c4b9e","status":207,"matched":2,"updated":1,"failed":1}
```

**Rules:**
- Never log the API key
- Always log client IP for audit
- Every request gets a `request_id` (UUID) that propagates through all related log lines
- High-cardinality fields (`image`, `project`, `service`) stay at top level for LogQL filtering

### 11.2 Metrics (Prometheus)

Exposed at `/metrics`, no auth (internal network).

| Metric | Type | Labels | Purpose |
|---|---|---|---|
| `package_updater_build_info` | Gauge=1 | `version`, `commit` | Dashboard version display |
| `package_updater_http_requests_total` | Counter | `endpoint`, `status_code` | Traffic + error rate |
| `package_updater_update_jobs_total` | Counter | `project`, `service`, `status` | Deploy counts per stack, success/fail |
| `package_updater_update_duration_seconds` | Histogram | `project`, `service` | Pull+up duration; detect regressions |
| `package_updater_queue_depth` | Gauge | — | Sustained backlog indicator |
| `package_updater_last_update_timestamp` | Gauge | `project`, `service` | Wall-clock of last successful update; alert on stale stacks |
| `package_updater_docker_ping_up` | Gauge=0\|1 | — | Distinguishes "service up" from "socket broken" |

Dependency: `github.com/prometheus/client_golang`.

**Cardinality note:** `last_update_timestamp` grows with the number of stacks. Fine for <50 stacks. If it ever grows to hundreds, drop the per-service labels or move that metric to a separate exporter.

## 12. Configuration

All configuration is via environment variables.

| Variable | Required | Default | Description |
|---|---|---|---|
| `UPDATER_API_KEY` | **yes** | — | Bearer token. Service refuses to start if missing. |
| `STACKS_ROOT` | no | `/home/shcizo/self-hosted` | Required parent directory for any Compose stack to be eligible. |
| `PORT` | no | `8080` | HTTP listen port. |
| `LOG_LEVEL` | no | `info` | `debug`/`info`/`warn`/`error`. |
| `UPDATE_TIMEOUT` | no | `5m` | Per-job timeout. Go duration string. |
| `OPT_IN_LABEL` | no | `se.shcizo.auto-update` | Label name to check (allows renaming without rebuild). Value must equal `"true"`. |

## 13. Repository Layout (planned)

```
package-updater/
├── cmd/
│   └── server/
│       └── main.go              # entrypoint
├── internal/
│   ├── api/                     # HTTP handlers, middleware (auth, request_id, logging)
│   ├── discovery/               # Docker container lookup + Compose label parsing
│   ├── updater/                 # Job queue + worker + docker compose exec
│   ├── selfupdate/              # Self-update detection + deferred exec
│   ├── config/                  # Env var loading + validation
│   └── metrics/                 # Prometheus collectors
├── docs/
│   └── superpowers/
│       └── specs/
│           └── 2026-05-22-package-updater-design.md
├── Dockerfile
├── docker-compose.example.yml   # template users can copy
├── go.mod
├── go.sum
└── README.md
```

The layout follows the standard Go project structure: `cmd/` for entry points, `internal/` for non-exported packages organised by responsibility. Each `internal/` package owns one concept and exposes a small interface; this keeps files focused and individually testable.

## 14. Testing Strategy

- **Unit tests** for: image-name normalisation, Compose label parsing, dedup logic, path safety check, request validation.
- **Integration tests** for the discovery layer using `github.com/testcontainers/testcontainers-go` to spin up a real Docker-in-Docker environment with a known Compose stack, verify discovery + matching end-to-end.
- **HTTP handler tests** with `httptest` covering auth (200/401), validation (400), happy path (200), partial failure (207), zero match (200).
- **Self-update test** verifies response is fully written and connection closed before the deferred exec fires (use a fake exec'er injected via interface).
- **No end-to-end test** that actually exercises `docker compose pull` against a registry — that becomes a manual verification step on first deploy.

## 15. Open Questions / Future Considerations

These are deliberately out of scope for v1 but noted as plausible additions:

- Per-repo / per-image API keys (`keys.json` with image-pattern scopes)
- Real rollback via tag pinning (requires state)
- Notification plug-ins (Discord, Slack, email)
- Dry-run mode (`?dry_run=true`)
- Rate limiting (token bucket per client IP or per token)
- Multi-worker concurrency with per-stack mutex
- Audit history UI

## 16. Acceptance Criteria for v1

- [ ] `POST /update` with a known image triggers `docker compose pull` + `up -d` for all matching opt-in services on the host.
- [ ] Containers without the opt-in label are never touched, even with a valid token.
- [ ] Requests without a valid bearer token receive `401` and trigger no Docker action.
- [ ] Zero matches returns `200` with `matched: 0`.
- [ ] Partial failure across multiple matches returns `207` with per-job results.
- [ ] Service updates itself successfully (HTTP response is fully delivered before the container is replaced).
- [ ] Logs in Loki are filterable by `event`, `level`, `project`, `service`, `status`, `request_id`.
- [ ] Prometheus successfully scrapes `/metrics`; all listed metrics are present.
- [ ] Healthcheck returns `200` when Docker socket is reachable, fails otherwise.
- [ ] Gitea composite action, used from a real workflow, triggers an end-to-end deploy and surfaces failure to the workflow when something goes wrong.