Files

T

shcizo 9832b2b190 docs: complete se.enocsson to se.shcizo rename in spec

2026-05-22 11:35:33 +02:00

21 KiB

Raw Blame History

package-updater — Design Spec

Date: 2026-05-22 Status: Approved (brainstorm complete, ready for implementation planning) Author: Samuel Enocsson

1. Purpose

A small HTTP service that receives webhook-style notifications from Gitea Actions (or any CI) and triggers a pull + restart of the corresponding Docker Compose service on the host. It fills the gap between Watchtower (polling, no CI integration) and full GitOps (Argo CD / Flux) for a self-hosted, single-host environment.

Concrete trigger: Gitea workflow builds and pushes registry.example.com/myapp:v1.2.3 to the container registry, then calls POST /update with the image name. The service finds the matching Compose-managed container(s) on the host, pulls the new image, and restarts the relevant service(s).

Watchtower remains in place for third-party images that the user does not build themselves.

2. Non-Goals (v1)

Multi-host orchestration (single Docker host only)
Real rollback (relies on Compose's "keep old container if new fails to start")
Persistent state, audit history, or web UI (logs live in Loki)
Per-repo / per-image API keys (single shared bearer token)
Wildcards/regex in image matching
Notifications outside of Gitea (workflow failure is visible there already)
Rate limiting
TLS termination (handled by Nginx Proxy Manager in front of the service)

3. Architecture

┌──────────────────┐     POST /update      ┌─────────────────────┐
│ Gitea Actions    │ ─────────────────────▶│ package-updater     │
│ (composite       │   Bearer <token>      │ (Go service,        │
│  action)         │   {image, tag}        │  container)         │
└──────────────────┘                       └──────────┬──────────┘
                                                      │
                                          docker.sock │ (mounted)
                                                      ▼
                                           ┌─────────────────────┐
                                           │ Docker daemon       │
                                           │ + running containers│
                                           │ + Compose stacks    │
                                           └─────────────────────┘

3.1 High-level flow

Gitea workflow builds + pushes registry.example.com/myapp:v1.2.3.
Workflow calls POST /update with {image, tag} + bearer token.
Service validates token (constant-time compare).
Service lists Docker containers (running + stopped), filters by:
- Image name match (tag-agnostic)
- Opt-in label se.shcizo.auto-update=true
For each match, reads Compose's built-in labels to find working_dir, config_files, service, project.
Path safety check: refuse jobs whose working_dir is not inside STACKS_ROOT.
Deduplicates (project, service, config_files) and enqueues one job per unique tuple.
Single FIFO worker executes jobs synchronously: docker compose -f <file> -p <project> pull <service> then up -d <service>.
Response aggregates per-job results.

3.2 Design principles

Stateless — no database, no config file beyond environment variables. The Docker daemon is the source of truth.
Idempotent — re-running the same update is safe. Zero matches returns 200 with an empty result list.
Fail-safe — per-job failures do not stop other jobs. Compose's normal behaviour preserves the old container if the new one fails to start.
Defense in depth — two independent gates (bearer token AND opt-in label) must both pass before any container is touched.

4. API Contract

4.1 `POST /update`

Headers:

Authorization: Bearer <api-key>
Content-Type: application/json

Request body:

{
  "image": "registry.example.com/myapp",
  "tag": "v1.2.3"
}

image — required. Image reference without tag. Matched against container image (tag-agnostic).
tag — optional. For logging/audit only. The actual tag pulled is governed by the image: line in the Compose file.

Response codes:

Status	Meaning
`200 OK`	All jobs succeeded (or zero matches)
`207 Multi-Status`	Mixed success/failure across jobs
`400 Bad Request`	Missing or invalid `image` field
`401 Unauthorized`	Missing or invalid token
`500 Internal Server Error`	Docker daemon unreachable, or all jobs failed

Success response body:

{
  "request_id": "0d8c4b9e-7a1f-4d2c-8b1a-3c5e6f7a8b9d",
  "image": "registry.example.com/myapp",
  "tag": "v1.2.3",
  "matched": 2,
  "results": [
    {
      "project": "myapp-prod",
      "service": "web",
      "compose_file": "/home/shcizo/self-hosted/myapp-prod/docker-compose.yml",
      "status": "updated",
      "duration_ms": 4231
    },
    {
      "project": "myapp-staging",
      "service": "web",
      "compose_file": "/home/shcizo/self-hosted/myapp-staging/docker-compose.yml",
      "status": "failed",
      "error": "pull: manifest unknown",
      "duration_ms": 812
    }
  ]
}

status values: updated, failed, refused (path outside STACKS_ROOT), timeout.

4.2 `GET /healthz`

Returns 200 OK if the process is alive AND cli.Ping(ctx) against the Docker socket succeeds. Used by container healthcheck and external uptime monitoring.

4.3 `GET /metrics`

Prometheus exposition format. No auth (internal network only).

4.4 `GET /version`

Returns build info (version, commit, build time).

5. Discovery & Matching

5.1 Container lookup

containers, err := cli.ContainerList(ctx, container.ListOptions{All: true})

All: true so we also catch crashed/stopped containers we may want to restart.

5.2 Image matching

Tag-agnostic exact match after normalisation:

input:     "registry.example.com/myapp"
container: "registry.example.com/myapp:v1.2.2@sha256:abc..."
           → strip tag and digest → "registry.example.com/myapp"
           → MATCH

Case-sensitive. No wildcards or regex (YAGNI).

5.3 Opt-in filter

Container must have label se.shcizo.auto-update=true. Anything else (false, missing, other value) is silently excluded.

5.4 Compose label extraction

Label	Used for
`com.docker.compose.project`	`-p <project>` flag
`com.docker.compose.project.working_dir`	Working directory for `docker compose` exec
`com.docker.compose.project.config_files`	`-f <file>` flag(s) (comma-separated list)
`com.docker.compose.service`	Which service to pull/restart

If any of these labels are missing (container not started by Compose), the container is excluded with a warning log. This should not happen in practice — the whole tool is Compose-centric — but we don't crash.

5.5 Path safety check

Before running any docker compose command, verify that working_dir is inside STACKS_ROOT (default: /home/shcizo/self-hosted). If not, the job is marked refused and logged at warning level. This is a defense-in-depth measure against forged or unexpected Docker labels.

5.6 Deduplication

Multiple containers may belong to the same (project, service, config_files) tuple (e.g. Compose deploy.replicas > 1). Collapse to a single job using:

dedupKey = project + "|" + service + "|" + sorted(config_files)

5.7 Concurrency

A single global FIFO queue with one worker. The HTTP handler blocks until all jobs for the request are complete, so the caller receives the result synchronously. The single-worker model gives per-stack mutual exclusion implicitly — no race conditions between concurrent updates against the same stack.

6. Self-Update Handling

When the service receives an update whose image matches its own running container, it must finish writing the HTTP response before exec'ing docker compose up -d against itself (otherwise the response is lost when the container is replaced).

Implementation: detect self-update at job dispatch time. For self-update jobs, the worker writes the HTTP response, flushes, closes the connection, then waits ~1 second before invoking the up -d command. All other jobs execute normally.

Acceptable residual risk: if a buggy version is pushed that fails to start, the running version dies (Compose can't restart what won't start) and the user must SSH in and run docker compose up -d manually with a previous tag. This is the same risk any self-updating system carries; mitigation is a deploy discipline issue, not a code one.

7. Security Model

7.1 Authentication

Single bearer token read from env var UPDATER_API_KEY at startup. Service refuses to start if it is missing or empty (fail-fast).
Token compared with crypto/subtle.ConstantTimeCompare to defeat timing attacks.
Token is never logged.

7.2 Authorisation

A container is eligible for update only if it has both:

An image name matching the request, AND
The opt-in label se.shcizo.auto-update=true.

Both gates are independent. Compromising either alone does not allow an attacker to trigger an update.

7.3 Network exposure

The service binds 0.0.0.0:8080 inside the container and is reached via Nginx Proxy Manager which terminates TLS and forwards on the internal Docker network. The service itself does not handle TLS.

7.4 Socket access

The service mounts /var/run/docker.sock and therefore has effective root on the host. This is the same trust model as Watchtower. The opt-in label is the second gate that limits blast radius from API-layer abuse.

8. Deployment

8.1 Service's own Compose stack

Path: /home/shcizo/self-hosted/package-updater/docker-compose.yml

services:
  package-updater:
    image: registry.example.com/package-updater:latest
    container_name: package-updater
    restart: unless-stopped
    environment:
      - UPDATER_API_KEY=${UPDATER_API_KEY}
      - STACKS_ROOT=/home/shcizo/self-hosted
      - LOG_LEVEL=info
      - PORT=8080
      - UPDATE_TIMEOUT=5m
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/shcizo/self-hosted:/home/shcizo/self-hosted:ro
    labels:
      - "se.shcizo.auto-update=true"
    networks:
      - proxy
    healthcheck:
      test: ["CMD", "wget", "-q", "-O-", "http://localhost:8080/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3

networks:
  proxy:
    external: true

.env next to it (git-ignored):

UPDATER_API_KEY=<openssl rand -hex 32>

NPM is configured with a proxy host pointing at package-updater:8080 with TLS.

8.2 Dockerfile

Multi-stage, statically-linked Go binary:

FROM golang:1.23-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/package-updater ./cmd/server

FROM alpine:3.20
RUN apk add --no-cache docker-cli docker-cli-compose ca-certificates wget
COPY --from=build /out/package-updater /usr/local/bin/
ENTRYPOINT ["/usr/local/bin/package-updater"]

docker-cli and docker-cli-compose are installed because we shell out to docker compose for pull/up. Going through the CLI handles dependent services and networks correctly out of the box; calling the Docker API directly would require us to replicate that logic.

Expected image size: ~50 MB.

8.3 Stacks root convention

All Compose stacks eligible for auto-update must live under STACKS_ROOT (default /home/shcizo/self-hosted). This is required so that the path stored in com.docker.compose.project.working_dir (a host path) resolves to the same path inside the service container. The mount uses identical source and target paths to avoid any path translation logic.

Read-only mount is sufficient — the service only reads Compose files; it never writes them.

9. Gitea Action

A reusable composite action lives in a separate repository, e.g. gitea.example.com/samuel/action-deploy-update.

action.yml:

name: "Deploy via package-updater"
description: "Notifies package-updater to pull & restart a service"
inputs:
  endpoint:
    description: "Full URL to /update (e.g. https://updater.example.com/update)"
    required: true
  image:
    description: "Image reference without tag (e.g. registry.example.com/myapp)"
    required: true
  tag:
    description: "Tag that was just pushed (for logging)"
    required: false
    default: ""
  token:
    description: "Bearer token for package-updater"
    required: true
runs:
  using: "composite"
  steps:
    - name: Trigger update
      shell: bash
      env:
        TOKEN: ${{ inputs.token }}
      run: |
        set -euo pipefail
        response=$(curl -sS -w "\n%{http_code}" \
          -X POST "${{ inputs.endpoint }}" \
          -H "Authorization: Bearer $TOKEN" \
          -H "Content-Type: application/json" \
          -d "{\"image\":\"${{ inputs.image }}\",\"tag\":\"${{ inputs.tag }}\"}")
        body=$(echo "$response" | head -n -1)
        code=$(echo "$response" | tail -n 1)
        echo "HTTP $code"
        echo "$body" | jq .
        if [ "$code" -ge 400 ]; then
          exit 1
        fi

Consumer workflow (.gitea/workflows/deploy.yml):

jobs:
  deploy:
    runs-on: ubuntu-latest
    needs: [build-and-push]
    steps:
      - uses: gitea.example.com/samuel/action-deploy-update@v1
        with:
          endpoint: https://updater.example.com/update
          image: registry.example.com/${{ gitea.repository }}
          tag: ${{ gitea.sha }}
          token: ${{ secrets.UPDATER_TOKEN }}

UPDATER_TOKEN is set as an organisation-level secret so all repos share it.

10. Error Handling

Phase	Possible failure	Behaviour
Auth	Missing/invalid token	`401`, warning log with client IP
Validation	Missing/invalid `image`	`400`, no action
Docker daemon	Socket unreachable	`500`, error log
Discovery	Zero matches (after opt-in filter)	`200` with `matched: 0`, info log
Path safety	`working_dir` outside `STACKS_ROOT`	Skip job with `status: refused`, warning log. Other jobs continue.
`docker compose pull`	Manifest unknown, registry auth fail, network timeout	Per-job `status: failed`. No rollback (old container still running).
`docker compose up -d`	Validation error, port conflict, etc.	Per-job `status: failed`. Compose retains old container if new fails.
Per-job timeout	Job exceeds `UPDATE_TIMEOUT` (default 5m)	Cancel and mark `status: timeout`.

Aggregate HTTP status:

All jobs updated → 200
Mixed → 207
All jobs failed/refused/timeout → 500
matched: 0 → 200 (idempotent: "nothing to do" is not an error)

No rollback in v1. If you push a buggy image, Compose keeps the old container if the new one fails to start. Real rollback (re-pin previous tag) would require tracking previous tags per service, which we don't.

11. Observability

11.1 Logs (Loki)

Structured JSON to stdout. Promtail/Alloy already scrapes Docker container logs and forwards to Loki. Example records:

{"time":"2026-05-22T14:32:11Z","level":"info","event":"update_request","request_id":"0d8c4b9e","image":"registry.example.com/myapp","tag":"v1.2.3","client_ip":"10.0.0.5"}
{"time":"2026-05-22T14:32:11Z","level":"info","event":"matched","request_id":"0d8c4b9e","image":"registry.example.com/myapp","matched":2}
{"time":"2026-05-22T14:32:11Z","level":"info","event":"job_start","request_id":"0d8c4b9e","project":"myapp-prod","service":"web"}
{"time":"2026-05-22T14:32:15Z","level":"info","event":"job_complete","request_id":"0d8c4b9e","project":"myapp-prod","service":"web","status":"updated","duration_ms":4231}
{"time":"2026-05-22T14:32:15Z","level":"error","event":"job_complete","request_id":"0d8c4b9e","project":"myapp-staging","service":"web","status":"failed","error":"pull: manifest unknown","duration_ms":812}
{"time":"2026-05-22T14:32:15Z","level":"info","event":"update_response","request_id":"0d8c4b9e","status":207,"matched":2,"updated":1,"failed":1}

Rules:

Never log the API key
Always log client IP for audit
Every request gets a request_id (UUID) that propagates through all related log lines
High-cardinality fields (image, project, service) stay at top level for LogQL filtering

11.2 Metrics (Prometheus)

Exposed at /metrics, no auth (internal network).

Metric	Type	Labels	Purpose
`package_updater_build_info`	Gauge=1	`version`, `commit`	Dashboard version display
`package_updater_http_requests_total`	Counter	`endpoint`, `status_code`	Traffic + error rate
`package_updater_update_jobs_total`	Counter	`project`, `service`, `status`	Deploy counts per stack, success/fail
`package_updater_update_duration_seconds`	Histogram	`project`, `service`	Pull+up duration; detect regressions
`package_updater_queue_depth`	Gauge	—	Sustained backlog indicator
`package_updater_last_update_timestamp`	Gauge	`project`, `service`	Wall-clock of last successful update; alert on stale stacks
`package_updater_docker_ping_up`	Gauge=0\|1	—	Distinguishes "service up" from "socket broken"

Dependency: github.com/prometheus/client_golang.

Cardinality note: last_update_timestamp grows with the number of stacks. Fine for <50 stacks. If it ever grows to hundreds, drop the per-service labels or move that metric to a separate exporter.

12. Configuration

All configuration is via environment variables.

Variable	Required	Default	Description
`UPDATER_API_KEY`	yes	—	Bearer token. Service refuses to start if missing.
`STACKS_ROOT`	no	`/home/shcizo/self-hosted`	Required parent directory for any Compose stack to be eligible.
`PORT`	no	`8080`	HTTP listen port.
`LOG_LEVEL`	no	`info`	`debug`/`info`/`warn`/`error`.
`UPDATE_TIMEOUT`	no	`5m`	Per-job timeout. Go duration string.
`OPT_IN_LABEL`	no	`se.shcizo.auto-update`	Label name to check (allows renaming without rebuild). Value must equal `"true"`.

13. Repository Layout (planned)

package-updater/
├── cmd/
│   └── server/
│       └── main.go              # entrypoint
├── internal/
│   ├── api/                     # HTTP handlers, middleware (auth, request_id, logging)
│   ├── discovery/               # Docker container lookup + Compose label parsing
│   ├── updater/                 # Job queue + worker + docker compose exec
│   ├── selfupdate/              # Self-update detection + deferred exec
│   ├── config/                  # Env var loading + validation
│   └── metrics/                 # Prometheus collectors
├── docs/
│   └── superpowers/
│       └── specs/
│           └── 2026-05-22-package-updater-design.md
├── Dockerfile
├── docker-compose.example.yml   # template users can copy
├── go.mod
├── go.sum
└── README.md

The layout follows the standard Go project structure: cmd/ for entry points, internal/ for non-exported packages organised by responsibility. Each internal/ package owns one concept and exposes a small interface; this keeps files focused and individually testable.

14. Testing Strategy

Unit tests for: image-name normalisation, Compose label parsing, dedup logic, path safety check, request validation.
Integration tests for the discovery layer using github.com/testcontainers/testcontainers-go to spin up a real Docker-in-Docker environment with a known Compose stack, verify discovery + matching end-to-end.
HTTP handler tests with httptest covering auth (200/401), validation (400), happy path (200), partial failure (207), zero match (200).
Self-update test verifies response is fully written and connection closed before the deferred exec fires (use a fake exec'er injected via interface).
No end-to-end test that actually exercises docker compose pull against a registry — that becomes a manual verification step on first deploy.

15. Open Questions / Future Considerations

These are deliberately out of scope for v1 but noted as plausible additions:

Per-repo / per-image API keys (keys.json with image-pattern scopes)
Real rollback via tag pinning (requires state)
Notification plug-ins (Discord, Slack, email)
Dry-run mode (?dry_run=true)
Rate limiting (token bucket per client IP or per token)
Multi-worker concurrency with per-stack mutex
Audit history UI

16. Acceptance Criteria for v1

POST /update with a known image triggers docker compose pull + up -d for all matching opt-in services on the host.
Containers without the opt-in label are never touched, even with a valid token.
Requests without a valid bearer token receive 401 and trigger no Docker action.
Zero matches returns 200 with matched: 0.
Partial failure across multiple matches returns 207 with per-job results.
Service updates itself successfully (HTTP response is fully delivered before the container is replaced).
Logs in Loki are filterable by event, level, project, service, status, request_id.
Prometheus successfully scrapes /metrics; all listed metrics are present.
Healthcheck returns 200 when Docker socket is reachable, fails otherwise.
Gitea composite action, used from a real workflow, triggers an end-to-end deploy and surfaces failure to the workflow when something goes wrong.

21 KiB Raw Blame History