Notifications & Email

Every customer-facing notification fans out to two channels: transactional email and outbound client webhooks. There are three per-family notifiers (voucher / topup / eSIM), one shared outbound-webhook engine, and a strict terminal-only firing rule.

Architecture

All notifier dispatch runs in separate fire-and-forget goroutines with a detached context.Background() so request cancellation can't abort a send.

The terminal-only rule (verified)

Notifications fire only at terminal (status[, sub_status]) pairs. Each notifier maps status→kind and no-ops otherwise; call sites re-load the row post-commit and gate on IsFinal() before notifying.

In-flight states (VENDOR_ORDER_CREATED, provisioning, recharge PENDING/RECHARGED) never notify. The only non-terminal events emitted at all are the eSIM post-delivery lifecycle webhooks (installed / activated / depleted) — genuine one-shot state transitions, webhook-only.

Email generation & sending

Transport — selected by EMAIL_PROVIDER env (default sendgrid):

Interface email.MessageDeliveryService.DeliverMessage(ctx, MessageContent); global singleton via InitialiseEmail / GetEmailProvider.

Templates — Jet (github.com/gofiber/template/jet/v2). Dev reads views/email from disk; prod uses an embedded FS. RenderEmailTemplates(base, data) always renders both base.txt and base.html, both attached to the message. Data is map[string]any; Jet requires capitalised field access.

Voucher email builder & the 6 permutations

BuildVoucherEmailData (services/voucher_email.go:61) reloads order+items+product, decrypts voucher fields (decryptOrPassthrough, degrades to empty on failure so one bad row can't fail the email), and returns a bundle with a Cleanup() for any temp attachment.

The 6 permutations are the single-voucher delivered layouts, driven by which of {Code, Pin, ClaimURL} are present (rendered in views/email/voucher/delivered.html.jet via hasCode/hasPIN/hasClaimURL): code-only; code+PIN; claim+PIN; claim-only; PIN-only; code+PIN+claim. Design mock: views/email/preview/voucher_permutations.html.

Excel (services/excel_generator.go, excelize v2): columns Product Name / Denomination / Code / Pin / Voucher Link / Reference / Expires At. Written to /tmp/octopus-orders/…xlsx, removed by CleanupExcelFile via the caller's defer content.Cleanup().

eSIM & recharge builders

• eSIM (esim_email.go:68): delivered template gets activationCode, iccid, and a QR image URL (https://api.qrserver.com/...). Failed/cancelled include refundAmount.
• Recharge (recharge_email.go:61): masks the destination — phone as +CC AA ***NNNN, IDs keep prefix+last4. RechargeClaimURL = https://claim.octopuscards.io for user-fixable retries.
• ⚠️ No voucher/failed template exists (the voucher builder rejects any kind but Delivered/Partial/Cancelled), while recharge and eSIM both have failed templates. Asymmetric.

Email design system

Shared layout per family: views/email/{voucher,esim,recharge}/_layout.{html,txt}.jet (each family has its own copy of the same token set).

• Palette / elevation (light inline, dark via prefers-color-scheme + Outlook [data-ogsc]): text text-ink/text-slate/text-mist; surfaces body-bg → card-bg → surface-bg/surface-bg-strong (3-tier elevation); accent brand blue #3BA8D9.
• State→color via notice-* classes: notice-orange (warning), notice-red (failure), notice-amber (caution), notice-slate (neutral), each with a -strong foreground.
• Fonts: DM Serif Display (headings), Nunito (body), DM Mono (codes), with MSO/Outlook fallbacks. Accordions via <details>.

Outbound client webhooks

Registration (client portal)

http/handler/client_webhooks.go, CRUD over web_hook_urls:

• POST /client/api/webhooks — requires HTTPS; the signing key is generated server-side (GenerateWebhookSecret), never accepted from the client, returned once. Optional events validated against IsValidClientWebhookEvent.
• POST /:id/rotate-secret — new key, old invalid immediately. POST /:id/test — synthetic type:"test" payload, not persisted.
• WebHookURL.SubscribesToEvent treats an empty events list as "all events" (legacy rows subscribe to everything).

Payload & signing

Envelope models.WebhookPayload: { id: "evt_...", type, created_at, data{} } (webhook_service.go:91). SendWebhook (webhook_sender.go:41) POSTs JSON with headers X-Webhook-ID, X-Event-ID, X-Event-Type, X-Timestamp, and X-Signature = HMAC-SHA256(payload, token) hex. The signing key is never transmitted; success = HTTP 2xx.

Queue → delivery → retry

• Cron: WebhookTask (*/1 * * * *, batch 50) → ProcessPendingDeliveries. Bounded fan-out of 10 via errgroup; one bad delivery never cancels the batch.
• Claim: FOR UPDATE SKIP LOCKED flips pending→processing and increments attempt_count in one UPDATE — prevents duplicate sends across overlapping ticks / instances.
• Retry/backoff: exponential 1<<(attempt-1) minutes = 1, 2, 4, 8, 16; at attempt >= MaxAttempts → markFailed (terminal failed, no separate DLQ).
• Crash recovery: ReapStuckProcessingDeliveries re-queues processing rows older than 3 min without resetting attempt_count (a poison endpoint can't loop forever).
• Idempotency: stable event_id (evt_...) in body + X-Event-ID, constant across retries and across all URLs of one event. Gap: no dedup on enqueue — a double-fired trigger queues two deliveries with different event_ids; dedup relies on single-fire callers + the client honoring event_id.

Payout client webhooks are a separate task/service (scheduler/payout_webhook_task.go) — and are currently never enqueued (see Payouts).

Inbound vs outbound

Two entirely separate systems:

• Inbound (vendor → Octopus): services/vendor_webhook_service.go receives async fulfilment callbacks, updates internal state, and on terminal transition triggers the outbound notifiers. This is where vendor topology lives — logged to vendor_webhook_logs, firewalled from customer output.
• Outbound (Octopus → client): everything on this page. Never carries vendor identity.

The notifier layer is the boundary: inbound processors call notifiers; notifiers emit vendor-free payloads/emails.

Event → notification matrix

Constants: database/models/webhook.go:85; subscribable set ClientWebhookEventTypes (:119).

Delivery guarantees

• Webhooks: at-least-once, up to 5 attempts with backoff; terminal failed is the dead-letter state (no DLQ table); crash recovery via reaper; client-side idempotency via stable event_id.
• Emails: best-effort fire-and-forget — no retry, no persistence. A failed send is logged and dropped; manual resend (ResendOrderEmail → sendOrderDeliveryEmail) is the only recovery.

Dead & divergent code

1. OrderNotifier fully implemented but unwired — live voucher path is the inline create_voucher_order.go/order.go code. Two generations coexist; pick one.
2. Voucher partial/cancelled emails never sent in production despite templates + builder support.
3. No voucher/failed template while recharge/eSIM have one.
4. SMTP drops attachments (smtp.go:210).
5. Stale comment "FAILED is no longer a status" (order_notifier.go:115) — it still is.
6. Vendor-leak vectors: preview mock copy + dead OrderItemWebhook.VendorID/VendorName.
7. No enqueue-side webhook dedup.
8. factory.go NewEmailService is a parallel unused provider constructor (live path is email.go:createProvider).

Key files

• Notifiers: services/{order_notifier,esim_notifier,recharge_notifier,esim_notification}.go
• Email builders: services/{voucher_email,esim_email,recharge_email,excel_generator}.go
• Transport: email/{email,sendgrid,smtp,templates}.go; templates views/email/
• Outbound webhooks: services/{webhook_service,webhook_sender}.go; http/handler/client_webhooks.go; scheduler/webhook_task.go; database/repo/webhook_delivery.go
• Inbound: services/vendor_webhook_service.go
• Models: database/models/{webhook,web_hook_url}.go