Deploy & Release

The backend is a single Go binary deployed to two Hetzner boxes via systemd + bash-over-SSH — no orchestrator, no CI push-to-server, no blue/green. Frontends deploy separately via GitHub Actions → Cloudflare. Docker runs only the data plane, never the app.

The release runbook (backend change → sandbox → prod)

Pre-flight (local): merge to master, CI must be green; if schema changed, go run main.go migrate create <name>, write a real Down and a backward-compatible Up (there is no automatic DB rollback).

Sandbox (94.130.137.222): git pull → docker exec local_postgres pg_isready -U octopus -d octopus_sandbox → sudo ./deploy/update-octopus-sandbox.sh → smoke-test https://sandbox-api.octopuscards.io/status. If the client changed, sudo ./deploy/update-octopus-sandbox-client.sh.

Prod (65.109.119.112): git pull → pg_isready …-d octopus → sudo ./deploy/update-octopus.sh → verify api/status + warden (302→/admin/login) + journalctl -u octopus -f. Client change → sudo ./deploy/update-octopus-client.sh.

Frontends deploy themselves on the master push (path-filtered GitHub Actions) — except the client portal, which is the one frontend deployed by hand over SSH (steps above).

The deploy/ directory

Root-level (not in deploy/): docker-compose.yaml (data plane), otel-collector-config.yaml.

Build & artifact

• On-server native build — every script runs plain go build -o <target> on the box (Go installed by init.sh). No cross-compile, no Docker image, no CI artifact shipped.
• No build metadata: bare go build, no -ldflags, no version/commit embedding. You cannot tell which commit is running from the binary.
• Update scripts build to a temp path with a trap cleanup, then cp to /opt/octopus/octopus only after a successful build + migration — the running binary is never clobbered mid-build.
• Embedded (//go:embed, main.go:58): SQL migrations, Jet views, email templates, assets/**.
• NOT embedded (fragile): seeder JSON (database/seeder/data/*.json) is read by relative path at runtime, so every deploy cps it to /opt/octopus/database/seeder/data/ and runs with cwd /opt/octopus. Flagged in-code as tech debt pending //go:embed.

systemd services

Five units, all User=octopus, Type=simple, Restart=always, RestartSec=5, LimitNOFILE=65535.

• Backend units are hardened: NoNewPrivileges, PrivateTmp, ProtectSystem=strict, ProtectHome, ReadWritePaths=/var/log/octopus /opt/octopus/uploads.
• Sandbox is deliberately not After=octopus.service (a prod restart won't cascade) and sets JOB_LOGS_DIR because ProtectSystem=strict makes the prod log dir read-only to it.
• Client units run Node, not Bun at runtime — a Bun getPrototypeOf shim bug crashes Next.js SSR; Bun is used only for bun install / bun run build. See use-bun-not-npm.
• PROD vs SANDBOX: ports 8081↔8082 / 3001↔3002; DB octopus↔octopus_sandbox; cache prefix none↔sandbox:; queue exchange default↔app.sandbox; distinct JWT_SECRET/APP_KEY.

nginx

HTTP-only in the repo; certbot injects :443 at install. Both prod vhosts proxy 127.0.0.1:8081 (octopus_backend, keepalive 32); sandbox → :8082.

Migrations in deploy

• Goose v3 via the binary's migrate subcommand; migrations are embedded in the binary. up uses WithAllowMissing() to tolerate out-of-order timestamps from parallel branches.
• The update scripts run migrate up after installing the new binary, before starting the service (as octopus with env sourced); a migration failure triggers automatic binary rollback. Standalone: sudo -u octopus /opt/octopus/octopus migrate up.
• Sandbox update refuses to migrate if PG_DB=octopus (prod-DB guard).
• D1 (grasshopper) migrations are a separate world — Drizzle SQL applied by Wrangler in CI (wrangler d1 migrations apply grasshopper --remote). See Cloudflare.

Env & secrets on the servers

• Backend env files (systemd EnvironmentFile, single source of truth): PROD /etc/octopus/octopus.env, SANDBOX /etc/octopus-sandbox/octopus.env (mode 640, root:octopus).
• Generated once by the setup scripts: JWT_SECRET=openssl rand -base64 64, APP_KEY=openssl rand -base64 32, PG_PASS=openssl rand -base64 24 (sandbox reads the PG password from the running container). Prod/sandbox secrets are deliberately distinct so tokens/ciphertext don't cross environments.
• Env drift repair: update scripts append any missing keys with defaults (ensure_env_key) and validate a required set; never overwrite existing values.
• There is no committed env template — the "template" is the heredoc inside each setup script.

See Secrets & Config for the full store map. New leaks found in this pass:

Docker data plane

Root docker-compose.yaml, brought up by setup-octopus-service.sh. Update scripts never touch compose — restarting containers mid-update has caused outages, so they only pg_isready-check and fail loudly.

Data persisted to ./.data/*. The replica (deploy/replica/) streams from the primary on a separate machine.

Frontend deploys (contrast)

The Go backend and the client portal are the only things not deployed by CI. Note two competing docs deploy paths (Cloudflare Pages CI vs nginx-static on box) — ambiguous source of truth.

Rollback & health

• Health: GET /status → HealthCheck (routes.go:47), given a no-log fast path in every vhost. Update scripts poll localhost:<PORT>/status for 30s + check systemctl is-active.
• Automated rollback: current binary backed up to /opt/octopus/octopus.backup.<ts> (last 5 kept). On migrate or health failure, rollback() stops, restores the backup, restarts, re-checks.
• Manual: systemctl stop octopus && cp /opt/octopus/octopus.backup.<ts> /opt/octopus/octopus && systemctl start octopus; down-migrate only if the schema is incompatible.
• Logs: journalctl -u octopus -f + /var/log/octopus/octopus.log; nginx /var/log/nginx/octopus-{api,warden}.*.
• Zero-downtime: none — stop→migrate→start on every deploy; Restart=always covers crashes, not deploys.

Risk register

Key files

• deploy/{init,setup-octopus-service,update-octopus,setup-octopus-sandbox-service,update-octopus-sandbox}.sh
• deploy/{octopus,octopus-sandbox,octopus-client,octopus-sandbox-client}.service, deploy/octopus.logrotate
• deploy/{nginx-octopus,nginx-octopus-sandbox,nginx-octopus-client}.conf, deploy/nginx/{signoz,octopus-docs,mypopupgiftcards}.conf
• docker-compose.yaml, otel-collector-config.yaml, deploy/replica/docker-compose.yaml
• CLI/migrations: main.go (Kong subcommands, //go:embed), database/migration/migration.go
• CI: .github/workflows/{test-backend-api,deploy-grasshopper,deploy-octopus-docs,deploy-octopus-website,deploy-mocky-balboa}.yml