Top-up & eSIM Flows
The two single-unit order resources alongside the voucher engine — recharges and esim_orders. Their sync/hybrid-sync flows, the recreate-budget recovery model, the delivered=PROCESSING nudge, retry crons, and where they diverge from vouchers.
Top-up (mobile recharge) and eSIM are single-unit, non-fan-out resources living alongside the voucher engine. Unlike vouchers (processOrder fans one order into N order_items across vendors), each creates exactly one row (recharges / esim_orders) and talks to exactly one vendor.
Read Order Lifecycle first for the voucher engine; this page documents where top-up/eSIM parallel it and where they diverge. The money mechanics are in Wallet & Ledger; the vendor adapters in Vendor Adapters.
Top-up (mobile recharge)
- Entry:
Handler.CreateTopupOrder(topup.go:421) →services.CreateTopupOrder(topup_order.go:91). Fully synchronous — the vendor call runs inline in the request (contrast eSIM). - input_data validation (
validateTopupInputFields,topup.go:1104): rejects unknown/extra keys, enforces required + 500-char cap, per-type checks (phoneE.164-normalized,number,selectagainst options), optionalValidationRegexwith a ReDoS guard (skips patterns >200 chars). ⚠️ It mutatesinputDatain place to store the normalized phone. - Create tx (
topup_order.go:191): debit wallet (or claim voucher if prepaid) →CreateRecharge(Type=TOPUP,Status=PENDING,Currency=walletCurrency— must matchAmountcurrency or FX refund routing breaks) → storerecharge_input_values. Commit. - Vendor submission (inline, 45s):
CreateOrder; classify the response — a vendor error with no structured status stays PENDING so the cron polls; a 2xx terminal decline propagates failure fields. Persisted in a second transaction. rechargesstate machine (enum.go:319):PENDING / DELIVERED / FAILED / CANCELLED / RECHARGED. Terminal ={DELIVERED, FAILED, CANCELLED}. No sub-status column — progress isStatus+StatusText+ failure fields +RetryCount/RetryAfter.
Dead status values
RECHARGED is defined and validated but never written anywhere in production. And refreshTopupOrderStatusFromVendor guards on a "PROCESSING" string literal (topup.go:711) that is not a RechargeStatus member and never persisted for recharges — a dead branch / copy artifact.
topup_unit (topup_unit.go) is a pure display helper — resolves the in-game currency unit ("Diamonds", "UC") by precedence: admin override → vendor par_value_currency → regex inference from the name, guarding against showing a bare ISO money code as a gaming unit.
eSIM
- Entry:
Handler.CreateEsimOrder(esim.go:28) →services.CreateEsimOrder(esim_order.go:121). Hybrid-sync: quantity forced to 1; the handler passescontext.WithoutCancel(ctx)so the dispatch goroutine outlives the request, applies its ownesimHandlerDeadline = 60s, and always returns HTTP 200 — clients branch onstatus, not HTTP code. - Phase 1 (sync tx): mirrors top-up, plus denormalizes the product snapshot and sets
SubStatus=INITIAL,ActivationStatus=NOT_INSTALLED,RetryAfter = now+90s(so the retry cron can't race the goroutine). ⚠️ClientDiscountis already a percent here — top-up passes a 0–1 fraction ×100. Easy-to-miss divergence. - Phase 2 (detached goroutine):
DispatchEsimOrder(esim_order.go:400) — the reusable engine shared by the initial goroutine, the retry cron, and admin retry.- First-dispatch fast-path (
retry==0 && no vendor_order_id) skips recovery → straight totryRecreateEsimOrder. - Two-phase poll-before-recreate: poll by
vendor_order_id, then bymerchant_ref(whereErrOrderNotFoundis definitive → safe to recreate), then recreate withEsimRecreateBudget = 5(retry_count bumped beforePurchaseEsimso an accept-then-404 loop can't run forever).ErrDuplicateMerchantRef(DT One 1007001) → fetch existing and adopt.
- First-dispatch fast-path (
persistDispatchOutcome(esim_order.go:732): row-lock (LockEsimOrderForUpdate) + terminal short-circuit — the load-bearing guard against double-refund and status regression. Activation code + ICCID are encrypted before storage. Auto-refund on FAILED runs inside the locked tx — if the credit fails, the whole transition rolls back, so there's never a FAILED order without a refund.
The delivered=PROCESSING airtight rule
The Octopus eSIM webhook adapter maps esim.delivered → EsimStatusProcessing (not terminal). The delivered webhook can't carry the activation code, so treating it as terminal DELIVERED would strand the order without its deliverable. Instead it's a "poll now" nudge — the order stays PENDING and the cron's GetOrderStatus (which does return the code) completes it as DELIVERED/VENDOR_CODE_FETCHED. Pinned by a test asserting "delivered webhook must NOT be terminal." DT One folds SUBMITTED/CONFIRMED → PROCESSING similarly. See Octopus eSIM adapter.
- Lifecycle:
RefreshEsimInstallationStatus(install-poll cron) setsACTIVE+ firesesim.installed/esim.activatedonce per genuine transition;EsimExpiryJobflipsACTIVE→EXPIRED+ firesesim.depleted. - State machine (
esim_order.go): statusPENDING/DELIVERED/FAILED/CANCELLED; sub-statusINITIAL / VENDOR_ORDER_PENDING / VENDOR_ORDER_CREATED / VENDOR_CODE_FETCHED / VENDOR_ORDER_FAILED(⚠️REFUNDED/NOT_REFUNDED/COMPLETEDare defined but never assigned); activation statusNOT_INSTALLED/ACTIVE/EXPIRED. Delivery artifact is the encryptedactivation_code(LPA/QR) +ICCID, reveal-once viaMarkEsimOrderRevealed.
Retry crons
Top-up and eSIM each have their own dedicated retry cron — the voucher pending-order-retry is not used by either.
| Cron | Schedule | Job | Drives |
|---|---|---|---|
topup-order-retry | * * * * * | TopupOrderRetryJob | ProcessTopupRechargeRetry per pending recharge |
esim-order-retry | * * * * * | EsimOrderRetryJob | DispatchEsimOrder per pending order |
esim-installation-poll | */5 * * * * | EsimInstallationPollJob | install/activate refresh |
esim-expiry | 0 2 * * * | EsimExpiryJob | ACTIVE→EXPIRED + esim.depleted |
Backoff: top-up 1<<retryCount min capped at 60; eSIM esimRetryBackoff = 1m/5m/30m/2h/6h.
Budget-vs-cron-max mismatch differs between the two
Top-up cron MaxRetryCount=10 is higher than the recreate budget of 5 — a budget-exhausted, parked recharge (retry_count 5–10) is still swept and re-parked with long backoff. eSIM cron max == budget == 5, so a parked eSIM order at retry_count 5 falls out of the eligibility query entirely (retry_count < 5 is false) and requires admin reset. Different operational behavior for "stuck" orders.
Idempotency
Identical contract on both: optional client client_reference, server-minted KSUID reference_number (internal — vendor merchant_order_id + refund routing). Pre-check + unique partial index on (client_id, client_reference) → hard 400 ErrDuplicateClientRef, no silent replay. Vendor-side dedup reuses reference_number as merchant_order_id; ErrDuplicateMerchantRef (SEAGM 20135 / DT One 1007001) is caught and converted to fetch-existing.
Concurrency hazard (top-up webhook)
The top-up webhook path is not concurrency-safe
ProcessDirectTopUpWebhook (vendor_webhook_service.go:282) writes status with no row lock, no terminal short-circuit, and no auto-refund on FAILED — unlike both the eSIM webhook and the top-up cron path (applyTopupVendorStatus), which are hardened. Consequences:
- A webhook racing the retry cron can overwrite a terminal status (status regression).
- A webhook-driven FAILED leaves the recharge un-refunded.
The eSIM webhook (ProcessEsimWebhook) is the reference: fully locked, terminal short-circuit, auto-refund inside the lock. Bring the top-up webhook up to parity.
Webhook auth also differs: top-up uses HMAC signature (VerifyWebhookSignature) with a vendor_webhook_logs idempotency key; eSIM uses a URL path token (constant-time, since DT One callbacks have no signature) and relies on the terminal short-circuit inside the lock (its webhook-log insert sets no idempotency key).
Voucher vs top-up vs eSIM
| Aspect | Voucher | Top-up | eSIM |
|---|---|---|---|
| Fan-out | Yes (N order_items) | No (single unit) | No (Quantity→1) |
| Table | orders + order_items | recharges | esim_orders |
| Sync model | engine-driven | synchronous inline | hybrid-sync (goroutine + 60s deadline, always 200) |
| Sub-status | yes | none | yes + activation status |
| Delivery artifact | voucher code(s) | recharge applied to MSISDN | encrypted activation_code (QR) + ICCID, reveal-once |
| Vendor line | per-item, multi-vendor | single, direct_topup factory | single, esim factory |
| Recovery | prefetch/inventory + per-item retry | recreate-budget 5, 2-phase | recreate-budget 5, 2-phase |
| Terminal-write lock | engine | cron yes; webhook NO | dispatch and webhook lock + short-circuit |
| Auto-refund on FAILED | engine | cron path only (not webhook) | dispatch + webhook, inside lock |
| Prepaid burn | n/a | RedeemResourceTopup | RedeemResourceEsim |
| Notifications | OrderNotifier | topup.delivered/failed/cancelled | esim.delivered/failed/cancelled + lifecycle installed/activated/depleted |
Findings summary
| Severity | Finding | Where |
|---|---|---|
| High | Top-up webhook: no row-lock / terminal short-circuit / auto-refund → status regression + un-refunded failures | vendor_webhook_service.go:282 |
| Low | RechargeStatusRecharged dead (never written) | enum.go:319 |
| Low | "PROCESSING" literal guard for recharges is a dead branch | topup.go:711 |
| Low | eSIM sub-statuses REFUNDED/NOT_REFUNDED/COMPLETED defined but never assigned | esim_order.go:45 |
| Low | validateTopupInputFields mutates caller's inputData map in place | topup.go:1161 |
| Low | Two-transaction window in top-up create (PENDING commit, then vendor status in tx2) — crash-safe by design | topup_order.go:386 |
| Low | eSIM esimHandlerDeadline (60s) < esimDispatchTimeout (90s) relies on WithoutCancel wiring | esim.go:24, esim_order.go:33 |
Key files
- Services:
services/{topup_order,esim_order}.go,services/vendor_webhook_service.go,services/{recharge_notifier,esim_notifier}.go - Handlers:
http/handler/{topup,topup_unit,esim,esim_webhook,direct_topup_webhook}.go - Models:
database/models/{recharge,esim_order,enum}.go - Jobs/scheduler:
jobs/{topup_order_retry,esim_order_retry,esim_installation_poll,esim_expiry}_job.go+ matchingscheduler/*_task.go - Vendors:
services/external_vendors/{direct_topup,esim}/{factory,contract}.go
Order Lifecycle
The end-to-end voucher order engine — single vs bulk, single- vs multi-vendor, sync vs async, the checkpoint state machine, retries, the recreate protocol, money, and webhooks.
State Machines
Every order/resource state machine — voucher orders (order-level + item checkpoints + voucher links), recharge (mobile & gaming), eSIM (status/sub-status/activation), and payouts — with enum tables, transition tables, diagrams, and the dead-state catalog.