Troubleshooting Runbooks

This is the page you open when a ticket comes in. Each section starts from the words a client or ops person actually says, walks the real code path that produces that symptom, gives you the read-only SELECTs to pinpoint the cause, and says who can fix it (self-heals on a cron / needs an admin action / needs a config change / needs a deploy).

It is the deep companion to the Operations "Common incidents" table — that table is the 10-second version; this is the trace. For the inventory of what's structurally broken (not incident response), see Known Issues.

How to triage anything here

Four moves, in order, before you touch any specific runbook:

1. Name the surface. Voucher/gift-card? → orders/order_items. Top-up/recharge? → recharges. eSIM? → esim_orders. Login? → client_users/clients. Money? → wallets/forex_values. The tables differ per surface and so do the crons.
2. Read the row, not the dashboard. Pull the single order/user/wallet row and read its status + sub_status + retry_count + retry_after + failure_*. Almost every "stuck" answer is visible there.
3. Decide: mid-flight vs stuck. Most PENDING things are legitimately mid-flight and self-heal within a minute or two. A thing is only stuck when its retry counter is exhausted or a terminal item is trapped under a non-terminal order. Each runbook draws that line precisely.
4. Pull the vendor evidence. The *_responses tables (order_item_responses, recharge_responses, esim_order_responses) and the webhook logs hold the raw upstream request/response — that's the ground truth for "what did the supplier actually say."

Symptom index

Recharge is not complete

A direct top-up (mobile/gaming recharge) or an eSIM order was placed but never reached its delivered state. These are two different tables with two different crons — identify which first.

• Top-up → table recharges, status enum RechargeStatus (database/models/enum.go:319), recovered by the topup-order-retry cron (every minute).
• eSIM → table esim_orders, status enum EsimOrderStatus + EsimOrderSubStatus (database/models/esim_order.go:14, :39), recovered by the esim-order-retry cron (every minute).

The happy terminal is DELIVERED. Everything else is either mid-flight (self-heals) or trapped (needs admin).

Diagnostic checklist

1. Which table? Look up by reference_number (the server-minted KSUID = the vendor merchant reference) or client_reference (the client's idempotency key). If it's in recharges, it's a top-up; if esim_orders, it's an eSIM.
2. Read status + retry state. For top-up: status, retry_count, retry_after, failure_code, failure_reason, failure_is_user_fixable, vendor_order_id (database/repo/recharge.go:456). For eSIM add sub_status, activation_code IS NOT NULL, iccid IS NOT NULL (database/migration/20260414000001_create_esim_tables.sql).
3. User-fixable failure? If failure_is_user_fixable = true, the input (player id / mobile number) was wrong — the retry handler force-FAILs and refunds on the next tick (http/handler/topup.go:1589). It is not recoverable by retrying; the customer must re-order with correct details.
4. Recreate budget exhausted? retry_count >= 5 means the double-bill guard tripped (topupRecreateBudget/EsimRecreateBudget = 5, http/handler/topup.go:1550, services/esim_order.go:53). The order is parked at PENDING for an admin — the cron re-schedules a long backoff (eSIM: now + 6h) but never re-purchases. Needs AdminResetRecharge/AdminRetryRecharge (http/handler/admin_recharge_actions.go).
5. eSIM airtight mid-state? An eSIM at PENDING / VENDOR_ORDER_CREATED with a non-null vendor_order_id but null activation_code is the normal state while the delivery webhook has landed but the code hasn't been polled yet. The esim.delivered webhook is deliberately a nudge (mapped to PROCESSING, never terminal — services/external_vendors/esim/octopus/octopus_esim_vendor.go:474) because the webhook can't carry the activation code; the every-minute esim-order-retry cron re-polls by vendor_order_id and the poll response carries the code, flipping it to DELIVERED / VENDOR_CODE_FETCHED (services/esim_order.go:805). Expect it to clear within ~1 min. If it doesn't, the cron isn't running or the poll keeps returning no code.
6. Otherwise mid-flight. retry_count < 5 and retry_after in the near future = the cron will retry with exponential backoff. Confirm topup-order-retry / esim-order-retry actually ran (job_executions).
7. Pull the vendor evidence. recharge_responses (filter recharge_id) or esim_order_responses (filter esim_order_id) hold the raw upstream request/response by request_type (CREATE_ORDER, GET_ORDER_STATUS, PURCHASE_ESIM…). Inbound top-up webhooks land in direct_topup_webhooks (status, raw_payload, error_message).

Read-only queries

-- Top-up: is it stuck, mid-flight, or parked-for-admin?
SELECT id, reference_number, client_reference, status, sub_status,
       vendor_id, vendor_order_id, retry_count, retry_after,
       failure_code, failure_reason, failure_is_user_fixable,
       status_text, created_at, updated_at
FROM recharges
WHERE reference_number = :ref AND deleted_at IS NULL;

-- eSIM: has the airtight-rule poll fetched the code yet?
SELECT id, reference_number, status, sub_status, vendor_id, vendor_order_id,
       (activation_code IS NOT NULL) AS has_code,
       (iccid IS NOT NULL) AS has_iccid,
       activation_status, retry_count, retry_after,
       failure_code, failure_reason, created_at
FROM esim_orders
WHERE reference_number = :ref AND deleted_at IS NULL;

-- What did the supplier actually return?  (top-up)
SELECT id, recharge_id, request_type, request, response, created_at
FROM recharge_responses
WHERE recharge_id = :recharge_id
ORDER BY created_at DESC;

Order is stuck in PENDING

A voucher / gift-card order (table orders) is not progressing. The order-level status is a roll-up of its items computed by getOrderDeliveryStatus (http/handler/create_voucher_order.go:272), and the recovery cron is pending-order-retry (every minute).

Diagnostic checklist

1. Confirm the cron is alive. pending-order-retry runs every minute only when the vouchers feature is enabled (main.go:181). No cron → nothing moves. Check job_executions.
2. Read the order's retry state. orders has retry_count, retry_after, delivered_quantity, refunded_quantity, refunded_amount — but no error column (database/models/order.go:20). The only free-text error lives on the items.
3. Inspect the items — this is where the truth is. order_items.status / sub_status / failure_reason (the only error text in the whole flow, database/models/order_item.go:51) / vendor_order_id. An item at FAILED / VENDOR_ORDER_FAILED under a PENDING order is trapped: failed items are excluded from the next retry's pending-item filter (create_voucher_order.go:2251), so they're never retried and never refunded. Needs an admin refund/cancel.
4. Check the caps. The cron selects status='PENDING' AND retry_after <= now() AND retry_count <= 10 (database/repo/order.go:1541). retry_count > 10 → silently abandoned (parked at PENDING, no alert). For bulk orders, a separate recreate budget of 5 (http/handler/bulk_vendor_order_checkpoints.go:695) parks the order "for admin" when the supplier keeps losing the order.
5. Distinguish mid-flight from stuck. Mid-flight = retry_count <= 10 and retry_after near future (backoff window: 1, 2, 4, 8, 16 min — ScheduleOrderRetry :3653). Large orders (quantity > 5) are async by design and sit at PENDING/Initial briefly with retry_count=0 (create_voucher_order.go:832).
6. Pull the vendor audit. order_item_responses (by order_item_id) records every AUTH/CREATE/GET_ORDER_STATUS/GET_VOUCHERS exchange, written outside the caller's transaction so it survives rollbacks (create_voucher_order.go:3490).
7. Rule out money problems. Insufficient balance is caught synchronously at create (400, no order row is ever written — create_voucher_order.go:711), so it is not a stuck-PENDING cause. If an order exists, the wallet was already debited.

Read-only queries

-- The prime "stuck" set: PENDING orders the cron will no longer retry
SELECT id, client_id, reference_code, status, sub_status,
       quantity, delivered_quantity, retry_count, retry_after,
       remarks, created_at, updated_at
FROM orders
WHERE deleted_at IS NULL AND status = 'PENDING'
  AND (retry_count > 10 OR retry_after < now() - interval '30 minutes')
ORDER BY created_at;

-- Item-level truth: look for FAILED / VENDOR_ORDER_FAILED, missing vendor_order_id, failure_reason
SELECT id, order_id, vendor_id, status, sub_status, failure_reason,
       vendor_order_id, (code IS NOT NULL) AS has_code, created_at, updated_at
FROM order_items
WHERE order_id = :order_id AND deleted_at IS NULL
ORDER BY id;

-- Classify a single order: due now / backing off / abandoned
SELECT id, status, sub_status, retry_count, retry_after,
       (retry_count <= 10 AND retry_after <= now()) AS due_now,
       (retry_count <= 10 AND retry_after >  now()) AS backing_off,
       (retry_count >  10)                          AS abandoned
FROM orders WHERE id = :order_id;

-- What the supplier returned for the failing item
SELECT oir.request_type, oir.request, oir.response, oir.created_at
FROM order_item_responses oir
JOIN order_items oi ON oi.id = oir.order_item_id
WHERE oi.order_id = :order_id
ORDER BY oir.created_at DESC;

The cron cannot auto-FAIL or auto-refund a voucher order — exhausted orders are silently parked and require an admin action (see Admin Order Actions).

Product not visible in the client catalog

A product that should be sellable doesn't appear in a client's catalog (/api/v1 list or the portal). Visibility is a gauntlet of gates in Repository.GetProducts → applyClientProductFilters (database/repo/product.go:22, :1470). The single most common cause is an inactive vendor.

For a product to be visible to a client, all of these must hold:

Diagnostic checklist

1. Master product active & not deleted. products.is_active = true AND deleted_at IS NULL. In the list path this is a hard filter (product.go:1533); in the detail-by-id path is_active isn't re-checked, so a direct fetch can still return an inactive product — trust the list filter.
2. Client is ACTIVE. If clients.status != 'ACTIVE' the request never reaches the catalog query — it's rejected at auth (middleware/auth.go:93, Client.IsActive() database/models/client.go:48). That's a login/access problem, not a catalog one.
3. At least one usable vendor_product (the big one). The product is hidden if a NOT EXISTS over its vendor_products finds nothing with vp.is_active = true AND vp.deleted_at IS NULL AND vendor.is_active = true (GetProductsWithoutActiveVendorProducts, product.go:1633, :1688). A vendor turned inactive (vendors.is_active = false) hides every product for which it's the only mapping — this is the #1 real cause. Globally-inactive vendors come from GetInactiveVendorIDs (database/repo/vendor.go:225).
4. Vendor not blacklisted for this client. An active row in client_vendor_blacklists for the product's only vendor hides it (product.go:1584).
5. Product not blacklisted for this client. An active row in client_product_blacklists for (client_id, product_id) hides it (Phase 2, product.go:1503).
6. Query-param filters. country_id, currency_id, category, and search (name/category ILIKE) only exclude when the portal actually sends them (product.go:112).
7. Suspect the cache last. Per-client results are cached 10–30 min, plus a products_without_vendors cache (10 min, product.go:1641). A fix (re-activating a vendor) can look un-applied until the TTL expires or clearProductCaches fires.

Read-only queries

-- One query that reproduces the whole hide decision for this client+product
SELECT p.id, p.name, p.is_active, p.deleted_at,
  EXISTS (
    SELECT 1 FROM vendor_products vp JOIN vendors v ON v.id = vp.vendor_id
    WHERE vp.product_id = p.id AND vp.is_active AND vp.deleted_at IS NULL
      AND v.is_active AND v.deleted_at IS NULL
      AND vp.vendor_id NOT IN (
        SELECT vendor_id FROM client_vendor_blacklists
        WHERE client_id = :client_id AND is_active AND deleted_at IS NULL)
  ) AS has_sellable_vendor_product,
  EXISTS (
    SELECT 1 FROM client_product_blacklists cpb
    WHERE cpb.client_id = :client_id AND cpb.product_id = p.id
      AND cpb.is_active AND cpb.deleted_at IS NULL
  ) AS is_client_blacklisted
FROM products p WHERE p.id = :product_id;
-- Visible iff: is_active=true, deleted_at NULL, has_sellable_vendor_product=true, is_client_blacklisted=false

-- Why is there no sellable mapping?  Inspect the product's vendors
SELECT vp.id, vp.vendor_id, v.name AS vendor_name,
       vp.is_active AS vp_active, vp.deleted_at AS vp_deleted,
       v.is_active  AS vendor_active, v.deleted_at AS vendor_deleted
FROM vendor_products vp JOIN vendors v ON v.id = vp.vendor_id
WHERE vp.product_id = :product_id;

Cannot log in to the client panel

A user can't get into the octopus-client Next.js portal. Split the failure into credentials/account rejected by the backend vs login succeeded but the cookie never stuck. The full auth flow is documented in Client Portal · Auth & 2FA; this section is the failure catalog.

Backend rejected the credentials

The login handler ClientLoginHandler (http/handler/client_auth.go:18) checks in this order, with distinct messages:

Note the case difference: client_users.status is lowercase (active), clients.status is UPPERCASE (ACTIVE). "Account not active" and "Organization not active" are the two top "correct password still won't log in" causes.

Login succeeded (200) but the user bounces back to /login

The backend logged Successful client portal login (client_auth.go:137) but the browser never stored or never sends the cookie, so the next GET /client/api/me is a 401 and the app bounces to login. Cookies are octopus_access (Path=/client) and octopus_refresh (Path=/client/auth), set by setAuthCookies with config from GetClientCookieConfig (config/cookie.go:30). Causes:

1. COOKIE_DOMAIN mismatch — set to a domain the portal isn't served under (app.octopuscards.io vs a sandbox host, or .octopuscards.io while the portal runs on a different registrable domain). The browser silently drops the Set-Cookie. No backend error.
2. Secure=true over HTTP — production forces Secure=true (config/cookie.go:36); reached over plain HTTP (proxy misconfig) the cookie is dropped.
3. Cross-domain SameSite=Lax — if the API host and portal host are different registrable domains, the withCredentials XHR is cross-site and Lax cookies aren't sent at all. Works only when API and portal share a registrable domain (subdomains are fine). The code never sets SameSite=None.
4. CORS success ≠ cookie success — a cross-domain request can pass CORS (web/domains.go allows mypopupgiftcards.com, octopuscards.io, octopusrewards.com + subdomains) yet still fail to persist the cookie because of #3.

2FA and passkey failure modes

• TOTP code rejected (client_2fa.go:85, 401 Invalid verification code): the top real cause is server clock skew > ±30s (pquerna totp.Validate, skew 1). Also an inconsistent two_factor_enabled=true but two_factor_secret IS NULL state makes every code fail with the same generic error — the user must fall back to a backup code.
• Pending 2FA expired (client_2fa.go:34, 401 2FA session expired): the client_2fa_pending intermediate token is 5-minute TTL (utils/jwt.go:224).
• Passkey fails (services/webauthn.go:302): almost always WEBAUTHN_RP_ID / WEBAUTHN_ORIGINS not matching the current domain — the RP is baked in at credential creation, so credentials made under one domain cannot be used under another (after any domain/subdomain/scheme change).

Read-only queries

-- The two "correct password but blocked" gates, in one shot
SELECT cu.id, cu.email, cu.status AS user_status,
       (cu.password IS NOT NULL) AS has_password,
       cu.two_factor_enabled, (cu.two_factor_secret IS NOT NULL) AS has_2fa_secret,
       cu.deleted_at, c.status AS client_status
FROM client_users cu JOIN clients c ON c.id = cu.client_id
WHERE cu.email = :email;
-- Need: user_status='active' (lowercase), client_status='ACTIVE' (uppercase), deleted_at NULL

-- Session/token state (revoked or expired refresh = forced re-login)
SELECT id, user_id, token_type, is_revoked, expires_at, created_at
FROM auth_tokens WHERE user_id = :user_id
ORDER BY created_at DESC LIMIT 10;

-- 2FA recovery headroom
SELECT count(*) FROM client_backup_codes
WHERE client_user_id = :user_id AND used_at IS NULL;

Cannot pay for a product in this currency

Placing an order fails because of the product's currency. Octopus is a prepaid wallet model: a client holds one wallet per currency (wallets_client_currency_unique (client_id, currency_id)), and an order in currency X paid from a wallet in currency Y needs an FX rate X → Y. The failure is almost always a missing FX rate.

Diagnostic checklist

1. Find the product's currency. products.currency_id → currencies.currency (ISO code). The currencies table has id, currency, numeric_code, precision, is_active, deleted_at (database/models/currency.go:8).
2. Can a wallet be resolved? ResolveWalletForOrder prefers a wallet in the product's currency, then falls back to the client's default-currency wallet (clients.currency) (services/wallet_resolver.go:90). If neither exists → ErrWalletNotFound → the voucher handler returns 404 Wallet not found (create_voucher_order.go:681).
3. Same currency? If the resolved wallet currency equals the product currency, no FX is needed — skip to the balance check.
4. Cross-currency → FX rate required (the usual culprit). computeCharges looks up forex_values for product → wallet via GetForexRateForVendor → GetForexValue (database/repo/forex.go:56). A missing row returns (nil, nil) (silent), which becomes forex rate not available for X to Y conversion and surfaces as 400 Failed to compute order charges (create_voucher_order.go:1273). For top-up/eSIM the equivalent is ErrForexNotAvailable (services/order_charges.go:14).
5. Balance check. Converted TotalPayable vs wallets.amount: app pre-check returns 400 Insufficient funds in your wallet (create_voucher_order.go:715); the stored proc also hard-guards with RAISE EXCEPTION 'Insufficient funds…' under a FOR UPDATE lock.
6. Voucher vs top-up/eSIM cross-currency rules differ. Voucher auto-converts (RequireCurrencyMatch=false). Top-up/eSIM set RequireCurrencyMatch=true for an explicitly chosen wallet, so a mismatched explicit wallet is rejected with ErrWalletCurrencyMismatch ("wallet currency does not match variant currency", services/wallet_resolver.go:83).

Read-only queries

-- Product currency + whether that currency row is present
SELECT p.id, p.name, p.currency_id, c.currency, c.is_active, c.deleted_at
FROM products p LEFT JOIN currencies c ON c.id = p.currency_id
WHERE p.id = :product_id;

-- Does the client hold a wallet in the product's currency?  What's its balance?
SELECT w.id AS wallet_id, cur.currency, w.amount, w.type, w.deleted_at
FROM wallets w JOIN currencies cur ON cur.id = w.currency_id
WHERE w.client_id = :client_id AND w.deleted_at IS NULL;

-- The usual culprit: is there an active FX rate product_ccy -> wallet_ccy?
SELECT id, source_currency, destination_currency, value, created_at
FROM forex_values
WHERE source_currency = :product_ccy AND destination_currency = :wallet_ccy
  AND deleted_at IS NULL
ORDER BY id DESC LIMIT 1;   -- zero rows => "forex rate not available for X to Y conversion"

-- Confirm the debit actually recorded the currency + FX (post-mortem)
SELECT t.id, cur.currency, t.amount, t.source_currency, t.destination_currency,
       t.forex_rate, t.conversion_charges, t.status, t.created_at
FROM transactions t JOIN currencies cur ON cur.id = t.currency_id
WHERE t.client_id = :client_id
ORDER BY t.id DESC LIMIT 20;

forex_values rows are manual admin entries — there's no automatic rate feed and no staleness/TTL check (the newest non-deleted row wins regardless of age). A missing pair is fixed by an admin adding the rate, not by anything self-healing. See Wallet & Ledger.

When a runbook runs out

If none of the five fits, fall back to the Operations page for service health, logs, and the cron-run history, and to Metrics & Tracing to follow a single request's trace end-to-end (a claim on Grasshopper and its downstream Octopus calls share one W3C trace). Structural defects — the ones these runbooks keep pointing at — are consolidated in Known Issues.