Operations & Runbooks

Day-to-day operating procedures — deploying, rolling back, reading logs, checking health, and safe database investigation.

Practical runbooks for keeping Octopus healthy. All commands run on the origin servers (PROD 65.109.119.112, SANDBOX 94.130.137.222) as a sudo-capable user.

The two rules, again

No ad-hoc INSERT/UPDATE/DELETE on prod. SELECT for investigation; changes go through the app or migrations.
Migrations are applied by a human, deliberately, with a backup. Never wire auto-migrate.

Health checks

# Backend (prod / sandbox)
curl -fsS https://api.octopuscards.io/status
curl -fsS https://sandbox-api.octopuscards.io/status
# On-box, direct to the process:
curl -fsS http://localhost:8081/status     # prod backend
curl -fsS http://localhost:8082/status     # sandbox backend
curl -fsSI http://localhost:3001/          # prod client portal

# Data plane
docker ps                                   # local_postgres / local_valkey / local_rabbitmq / local_otel_collector
docker exec local_postgres pg_isready -U octopus

Frontends on Cloudflare (grasshopper, website, docs, mocky) deploy automatically via GitHub Actions on push to master under their path. The Go backend and the Node client portal deploy via the on-box scripts below.

Backend (Go):

cd /path/to/repo               # the checkout on the server
sudo ./deploy/update-octopus.sh            # PROD  → :8081, health-checked, auto-rollback
sudo ./deploy/update-octopus-sandbox.sh    # SANDBOX → :8082 (refuses if PG_DB=octopus)

update-octopus.sh builds, backs up the running binary, stops the service, installs, runs migrate up, restarts, and health-checks :8081/status — rolling back automatically if the build, migration, or health check fails. It never touches docker compose.

Client portal (Next.js):

sudo ./deploy/update-octopus-client.sh          # PROD  → :3001
sudo ./deploy/update-octopus-sandbox-client.sh  # SANDBOX → :3002

Remember NEXT_PUBLIC_* are build-time inlined — an env change requires this rebuild, not just a restart.

Service control & logs

sudo systemctl status  octopus            # or octopus-sandbox / octopus-client / octopus-sandbox-client
sudo systemctl restart octopus
sudo journalctl -u octopus -f             # live (systemd also appends to the files below)

# App logs
tail -f /var/log/octopus/octopus.log
tail -f /var/log/octopus/octopus-client.log
tail -f /var/log/octopus-sandbox/octopus.log
# Per-cron-execution logs
ls /var/log/octopus/job_executions/           # prod
ls /var/log/octopus-sandbox/job_executions/   # sandbox

# nginx
tail -f /var/log/nginx/octopus-api.error.log
sudo nginx -t && sudo systemctl reload nginx

/var/log/octopus rotates daily (keep 14) via octopus.logrotate. /var/log/octopus-sandbox has no logrotate config — watch disk on the sandbox box, or add one.

Rolling back the backend

update-octopus.sh keeps the last 5 binary backups as ${BINARY}.backup.<timestamp>:

sudo systemctl stop octopus
sudo cp /opt/octopus/octopus.backup.<timestamp> /opt/octopus/octopus
sudo systemctl start octopus
curl -fsS http://localhost:8081/status

If a migration caused the problem, roll the code back first, then decide on the DB: go run main.go migrate status to see state, and migrate down only after a backup and deliberate review. Migrations are one-way in spirit — prefer forward fixes.

Running / triggering jobs manually

cd /opt/octopus && sudo -u octopus bash -c 'set -a; . /etc/octopus/octopus.env; set +a; \
  ./octopus cron inventory-pump'                     # run one cron now
sudo -u octopus ... ./octopus pump-inventory -c      # rebuild Valkey inventory pools

Cron execution history is in the job_executions table and surfaced in the admin UI (warden.octopuscards.io/admin → Job Executions).

Cache & queue

docker exec -it local_valkey valkey-cli
#   KEYS 'sandbox:*'            # sandbox keys are prefixed
#   SCAN 0 MATCH 'pool:*'       # inventory pools

# RabbitMQ mgmt UI: http://<host>:15672  (octopus/octopus) — but note NO workers consume today

Remember the queue is dormant — background work is cron-driven (see Jobs).

Observability (SigNoz)

UI: https://telemetry.octopuscards.io (proxies to SigNoz on :8080).
Ingest: https://otel.octopuscards.io/v1/{traces,metrics,logs} (→ collector :4318).
Turn app telemetry on with OTEL_ENABLED=true + OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_EXPORTER_OTLP_TOKEN in the env file, then restart.
Traces are W3C-propagated, so a claim on the Grasshopper worker and its downstream Octopus API calls share one trace.
Helper scripts: scripts/check-signoz.sh, scripts/test-trace-export.sh.

Safe database investigation

docker exec -it local_postgres psql -U octopus -d octopus          # prod
docker exec -it local_postgres psql -U octopus -d octopus_sandbox  # sandbox

SELECT-only. Useful starting points: orders + order_items (+ order_item_responses) for fulfilment issues, recharges for topups, esim_orders for eSIM, vendor_attributes for a vendor's config, job_executions for cron history. See the data model.

Common incidents

Symptom	First checks
Orders stuck PENDING	Is the relevant retry cron running? (`pending-order-retry`, `topup-order-retry`, `esim-order-retry`). Check `job_executions` + vendor `vendor_webhook_log`. Is the vendor `is_active`? Balance sync healthy?
Claim page failing	Grasshopper worker logs in the CF dashboard; is `ENCRYPTION_KEY` set (worker crashes without it)? Is the Octopus JWT in `octopus_token` KV valid? Rate-limiter tripped?
Client portal 5xx	`octopus-client.service` status + `octopus-client.log`. Confirm it's running on Node, not Bun.
Telemetry missing	`OTEL_ENABLED`, collector container up, `otel.octopuscards.io` reachable, token valid.
TLS renewal fails	Grey-cloud the CF record, run `certbot --nginx -d <host>`, re-proxy.
Sandbox disk full	`/var/log/octopus-sandbox` (no logrotate) — truncate/rotate.

Danger zone

deploy/uninstall-octopus-sandbox.sh tears down the entire sandbox (units, dirs, DB octopus_sandbox, sandbox:* cache keys, app.sandbox exchange). It requires --yes-i-am-sure, supports --dry-run, and has hard guards against touching prod. Never run the prod equivalent casually — there isn't one, by design.