Test Catalog
The complete inventory of test cases across every system — the 291 Go suites (150 repo with the three-branch sqlmock/faultDB/cachedCache harness, the 10 vendor orchestration suites / ~226 subtests, unit/utils/email), the 25 Grasshopper vitest files (~330 cases), what CI actually runs vs skips, the notable security assertions, and the concrete coverage gaps.
The inventory of what is tested, per system. This complements Testing (which is the harness/how-to) and the Seed Catalog (the fixtures these tests run against). For the Grasshopper-specific view see also Grasshopper API.
Big picture
| System | Suites | Real infra | In CI? |
|---|---|---|---|
Go database/repo | 150 files | Postgres + Valkey | ✅ Database job |
Go test/clientapi (core) | ~40 files | Postgres (+Valkey in CI) | ✅ Client API job |
Go test/clientapi orchestration | 10 suites / ~226 subtests | Postgres + mocky-balboa / octopusfake | ✅ Orchestration job |
Go services/external_vendors (detail) | 26 files (contract/soak) | mocky-balboa | ✅ Vendor-contract job |
Go utils | 15 files (pure) | none | ✅ Utils job |
Go database/models | 10 files (pure) | none | ✅ Models job |
Go cache | 4 files | Valkey | ✅ Cache job |
Go services, email, middleware, jobs, http/handler, services/shopify | ~22 files | mostly pure | ❌ not in CI |
Go test/{admin,clientportal,flows,frontend,load} | ~10 files | Postgres / k6 | ❌ not in CI |
| Grasshopper vitest | 25 files / ~330 cases | workerd + real D1/KV | ❌ no test workflow at all |
Referenced-but-absent test docs
Confirmed on disk: there is no database/repo/TESTING.md, no automation/scripts/run-go-tests.sh, and no make setup/Makefile. The only README under test/ is test/testdata/README.md. All run/setup knowledge lives in per-file doc comments and the CI YAML — earlier memory notes that cite those files are stale.
Go — unit & repository tests
The three-branch repo harness
Every database/repo function is expected to ship happy-path + error branch + cache path, realized by three reusable doubles:
| Double | Where | Exercises |
|---|---|---|
| real DB + tx-rollback | <domain>_test.go (package repo_test) | happy path against real Postgres; StartTransaction + defer tx.Rollback() isolation |
mockDB (sqlmock) + faultDB | sqlmock_*_internal_test.go, fault_injection_internal_test.go:23 | error branches — forced DB/scan failures via per-op fail flags; faultCache for cache errors |
cachedCache | sqlmock_cache_hits_internal_test.go:16 | cache-hit path — canned JSON on Retrieve, sqlmock asserted not hit; reconstructs the exact prod cache key |
The *_coverage_test.go files (~45 of the 150) are the dedicated branch-completion companions. Shared vocabulary lives in test/testhelpers — TestMainSetup, SkipIfShort, RequirePostgres (which resets all Postgres sequences once per run via a DO $$ block, db.go:81), assertions.go, factory.go. No golden files anywhere — email/template tests assert via strings.Contains needles.
Repo coverage by domain (150 files)
| Domain | Files | Notes |
|---|---|---|
| Client | ~22 | densest domain — client/user/CPD/blacklists/topup+esim variants/dashboard/backup-codes |
| Vendor & catalog | ~13 | vendor/attributes/catalog/product/discount-audit/discount-import/webhook-log |
| Orders | ~9 | order/item/response/webhook/esim-order + sqlmock internals |
| Payouts | ~10 | payout/beneficiary/event/provider/scheduled/config/webhook — well covered |
| Topup/eSIM/recharge variants | ~11 | variant/field/region/input-field + recharge log/response/input-value |
| Catalog reference | ~11 | product/category/country/currency/forex/timezone/topup+esim product |
| Auth/admin/user | ~8 | admin-user/session/backup-code, auth-token, passkey, login-attempt, ip-whitelist |
| Webhook | ~5 | webhook/delivery/url/direct-topup/order-webhook |
| Jobs/prefetch/export | ~10 | job-execution/prefetch/import/export/temp-tables/file |
| Inventory, wallet/ledger, cart, shopify-repo, voucher-link, G2A, dashboard | ~21 | G2A + dashboard are in-package mock suites |
Pure & service unit suites
utils(15, best-covered pure pkg,-race+ coverage in CI) — money precision (money_test.go:140— the0.1+0.2classic, accumulated rounding, %-of-%), discount/charges (discount_calc_test.gowith property-basedTestMergeRangesProperties+ native fuzzFuzzMergeRanges/FuzzFindBestDiscount), encryption (AES round-trip, per-encryption nonce uniqueness, thread-safety:266), JWT + G2A JWT (token uniqueness, cross-use rejection, tamper/expiry), PIN randomness, webhook-secret generation (200 iters,whsec_prefix, len 70, zero duplicates).services(7, NOT in CI) —inventory_test.goproves FEFO (reserved earlier-expiry pool drained before general,:329) + multi-vendor split;webhook_sender_test.goverifiesX-Signature == HMAC-SHA256(body, token)and that the token isn't leaked in a header;*_emailJet renders.email(3, NOT in CI) — voucher/recharge/esim Jet templates, HTML + text variants, substring assertions.database/models(10, pure) — enumScan/Value/IsValidround-trips (only 10 tests for 87 model files — thin).cache(4, Valkey),middleware(1 — onlyg2a_auth),jobs(1 — only prefetch phase-6 backoff),http/handler(9),services/shopify(2).
Go — integration, orchestration & E2E (test/)
The orchestration harness (two faking tracks)
DB-backed E2E driving real orders through the Fiber app + order service + retry cron + webhook handler + wallet ledger against real Postgres, with the vendor faked two ways:
- Track A — mocky-balboa (Cloudflare Worker mock) for real third parties (Runa, Neo, Wupex, iRewardify, SEAGM, DTOne). Chaos via the
/_chaosAPI driven bytesthelpers/mockychaos(modesone_shot/count/probabilistic/always; failuresstatus/delay_then_fail/malformed/drop;MustHaveFiredcatches tests that silently hit the happy path). - Track B —
octopusfake(in-processhttptest) for the federated self-vendors (OCTO_TOPUP/OCTO_ESIM): programmable auth + chaos engine + opt-in order engines (create→poll→deliver lifecycle) + HMAC-SHA256 webhook signing so inbound verification is exercised.
All orchestration files carry //go:build orchestration; testhelpers/vendor_binding rebinds a vendor row to the mock (per-vendor mutex), cronrunner steps a single order's retry without waiting on gocron, and test/shared/concurrency.go RunConcurrentN fires N goroutines released simultaneously for real race pressure.
Orchestration suites (~226 subtests)
| Suite | Vendor / kind | Track | Subtests | Coverage shape |
|---|---|---|---|---|
runa | Runa voucher (async link — reference) | mocky | 39 | happy+vendor-logging, 503/timeout/flaky-capped/persistent-500, never-terminal retries, 4 malformed kinds, drop+429, async poll→DELIVERED, signed webhook (nudge not deliver), concurrent-same-ref dedup, retry-cap, link opt-out, ShouldReCreate budget, admin deliver/reject/bypass |
neo | NeoCurrency voucher (sync reference) | mocky | 35 | Runa skeleton but sync inline + GetOrdersRfid reconciliation |
wupex | Wupex voucher | mocky | 34 | Runa skeleton + insufficient-stock |
irewardify | iRewardify voucher | mocky | 40 | largest — full skeleton + all reprocess/recreate/admin/invariant |
seagm_topup | SEAGM topup | mocky | 17 | recovery contract: sentinels, two-phase poll, recreate budget 5, auto-refund, row-lock idempotency |
dtone_topup | DTOne topup | mocky | 16 | same recovery contract |
dtone_esim | DTOne eSIM | mocky | 16 | eSIM recovery contract mirror (activation-code, webhook-token) |
octopus_esim | OCTO_ESIM (self) | octopusfake | 10 | debit-once, cron-poll activation, webhook-is-nudge, auto-refund-once, bad-sig/bad-path-token rejected, cron-vs-webhook money-conservation |
octopus_topup | OCTO_TOPUP (self) | octopusfake | 8 | debit-once, webhook/cron delivery, auto-refund, chaos recreate, idempotent |
g2a | G2A inbound (Octopus is server) | GH httptest stub | 10 | OAuth, bearer-gate, reservation lifecycle, idempotent-order/409, expired-410, concurrent no-double-mint, mint-failure no-code-leak |
Every suite has explicit concurrency/money-conservation subtests (ConcurrentSameRef, TwoCronTicks_RefundOnce, no_double_mint). No orchestration suite for TRS, EPIN, or GH voucher — GH is exercised only indirectly as the mint stub inside the G2A suite.
The rest of test/
test/clientapinon-orchestration (~40 files, in CI) — the bulk integration layer: core API (auth/orders/charges1779 L/wallets/transactions/payouts/catalog), money-leak/conservation per product type (voucher/topup/esim_money_leak— no-footprint-on-invalid, exactly-one-debit, ledger-sum==balance-delta, 100-order audit), dedup (+ fuzz), validation (+ native Go fuzz targets with a seed corpus).test/clientportal—password_reset_test.go(9 e2e: no account-existence leak, single-use token) + a placeholder.test/admin— scaffolding + placeholder only; no real admin handler tests.test/flows— product data-integrity, inventory-allocation flow (inventory_allocation_final_test.go800 L).test/frontend/pages— Jet status pages.test/load/g2a/g2a_load.js— a k6 script;MODE=journeymints real codes + debits the wallet (staging only, no CI guard).
Grasshopper — vitest (25 files, ~330 cases)
Runs inside the real workerd runtime via @cloudflare/vitest-pool-workers with real D1/KV/RateLimit bindings from wrangler.jsonc; singleWorker + isolatedStorage for determinism, migrations applied per-suite via TEST_MIGRATIONS, injected dummy secrets (always-pass Turnstile, 32-byte ENCRYPTION_KEY, RATE_LIMIT_FAIL_OPEN='').
| Layer / dir | Files | ~Cases | Highlights |
|---|---|---|---|
test/utils/ (L1) | 8 | ~92 | crypto AES-GCM round-trip + fresh-IV + tamper-throws; hash NIST vectors; claim-nonce code-binding/expiry/URL-safe; rate-limit fail-open matrix ({success:false} never forgiven); circuit-breaker 5-fail/half-open; octopus-client two-tier cache + 401-refresh + 5xx-fail/4xx-ok |
test/api/ (L2) | 9 | ~137 | verify-code (nonce minted/empty), submit-direct (fail-closed, cross-code nonce anti-replay, generic wrong-PIN, atomic gift-card, dial-prefix, 4xx/5xx branches), status (recovery/recreate, token-expiry HX-Refresh, FAILED releases voucher), notify, contact (24), landing, vouchers (30 — constant-time token), voucher-types (37 — ESIM exclusion, idempotent sync) |
test/integration/ (L3) | 2 | ~12 | full stack SELF.fetch security headers; admin-write→public-read contracts |
test/boundary/ (L3) | 1 | ~20 | numeric-ID edges never-500, length caps, pagination clamps, poll-counter n clamp |
test/chaos/ (L3) | 2 | ~14 | malformed Octopus responses classify-not-crash; breaker opens at 5; concurrent double-claim |
test/fuzz/ (L1, fast-check) | 2 | ~20 | crypto round-trip/wrong-secret; nonce never-forged/never-crashes/anti-replay |
test/smoke/ (L3) | 1 | 5 | infra sanity |
A production bug is pinned by a witness test, not fixed
test/chaos/concurrent_claims.test.ts:143 asserts the buggy behaviour: the topup reservation UPDATE (src/api/claim.ts:849) gates on is_redeemed=false but never sets it true, so N concurrent submits fire multiple Octopus order calls — the test asserts octopusCalls > 1 with a TODO(BUG) note, and the correct single-call test is it.skipped. Plain gift-card claims are single-winner-safe; topup/eSIM are not.
What CI actually runs
.github/workflows/test-backend-api.yml: lint → build → parallel {cache, utils, models, database, clientapi, vendor-contract, orchestration}, all with -race -count=1 -covermode=atomic, Postgres 16 + Valkey 7.2. The orchestration/contract jobs hit deployed mocky, whose chaos rules are global — concurrent CI runs can interfere (a documented flakiness source). Grasshopper's only workflows are deploy-grasshopper{,-sandbox}.yml, which deploy without running vitest.
Notable security-relevant assertions
- Constant-time compares — admin token (prefix rejected), PIN (generic "invalid code or PIN", no which-was-wrong leak), nonce/poll-token (unit + fuzz).
- Fail-closed / fail-open — nonce fail-closed when secret unset; rate-limit
{success:false}never forgiven, throw forgiven only when explicitly enabled. - Anti-replay — nonce bound to code hash (unit + fuzz + endpoint); voucher single-winner claim;
TwoCronTicks_RefundOncemoney conservation. - Idempotency —
client_referencerecreate only on definitive notFound; sync upsert; webhook idempotent + bad-signature/bad-path-token rejected (octopusfake HMAC). - Crypto — encryption round-trip + thread-safety (Go) + fresh-IV + wrong-secret-throws (both sides); webhook-secret zero-duplicates over 200 iters.
Coverage gaps (consolidated)
| Area | Reality |
|---|---|
http/handler | 74 source files, 9 tests — admin_ui.go (5993 L), create_voucher_order.go (3785 L), topup.go, admin_* all untested; not in CI |
jobs | 25 files, 1 test — only prefetch phase-6 backoff; payout_processor_job, shopify jobs, inventory pump untested; not in CI |
services | 36 files, 7 tests — payout engine (service + job) has no tests; vendor-catalog, esim-order, webhook-service untested; not in CI |
middleware | 14 files, 1 test — only g2a_auth; auth/csrf/ip-whitelist/feature-flag untested; not in CI |
database/models | 87 files, 10 tests — most Scan/Value uncovered |
| Grasshopper | no L4 e2e (test/e2e/ absent), no CI, pinned concurrency bug, RateLimit branches only stub-tested, low fuzz iteration counts |
| Vendors | no orchestration for TRS / EPIN / GH voucher; vendor-contract CI only SEAGM/DTOne/Runa/Neo |
| CI blind spots | test/{admin,clientportal,flows,frontend} are real suites but not invoked by any workflow |
| Skips that hide coverage | admin-API reprocess subtests skip without a session cookie; G2A order/concurrency skips without a GH-seeded product |
Key files
- Repo harness:
database/repo/{main_test,fault_injection_internal_test,sqlmock_cache_hits_internal_test}.go,test/testhelpers/* - Orchestration:
test/clientapi/*_orchestration_test.go,test/octopusfake/,test/testhelpers/{vendor_binding,cronrunner,mockychaos},test/shared/concurrency.go,test/testenv/ - Grasshopper:
frontend/grasshopper/test/**,vitest.config.ts - CI:
.github/workflows/test-backend-api.yml
Test Infrastructure
How the codebase is tested — the test/ tree, repo integration tests with fault injection, the octopusfake in-process upstream, the mocky-balboa-driven vendor orchestration suites, concurrency/goleak, CI, and how to add a new vendor suite.
Vendor Adapter Tests
The build-tagged test layer colocated with the vendor adapters (services/external_vendors/**) — the unit/contract/soak/sandbox/fuzz matrix, the per-vendor coverage table, how contract tests bind to mocky-balboa and drive chaos, the two goleak soak tests, and the concrete gaps (TRS/GH/iRewardify untested, SEAGM money-mapping fixed-but-unverified, the triple FailureInfo declaration, the untested Svix verifier).