Test Infrastructure
How the codebase is tested — the test/ tree, repo integration tests with fault injection, the octopusfake in-process upstream, the mocky-balboa-driven vendor orchestration suites, concurrency/goleak, CI, and how to add a new vendor suite.
Testing spans five layers: co-located unit tests, real-Postgres repository tests, the test/clientapi/ client-API + orchestration suites, load tests, and fuzz targets. Two purpose-built harnesses carry the heavy work — octopusfake (in-process Octopus upstream) and mocky-balboa (a Cloudflare-worker mock vendor with a chaos API).
Two referenced-but-absent tooling paths
CLAUDE.md points at automation/scripts/run-go-tests.sh and the pre-push hook mentions make setup — neither exists (there is no automation/ dir and no Makefile). Use the raw go test invocations below. The pre-push hook itself (.githooks/pre-push) is real and runs golangci-lint on changed Go files; enable it with git config core.hooksPath .githooks.
Layers & how to run
| Layer | Location | Run |
|---|---|---|
| Unit | co-located *_test.go (cache, utils, models, services, handlers) | go test ./cache/... ./utils/... ./database/models/... |
| Repository integration | database/repo/*_test.go (150 files) | needs Postgres: go test ./database/repo/... (CI adds -race -timeout=15m) |
| Client-API + money/dedup | test/clientapi/*_test.go | go test ./test/clientapi/... |
| Vendor orchestration | test/clientapi/*_orchestration_test.go | build-tagged: go test -tags orchestration ./test/clientapi/... (needs mocky-balboa up) |
| Fuzz | *_fuzz_test.go (auth, esim/topup/voucher dedup & validation, G2A) | go test -run=xxx -fuzz=FuzzX ./... |
| Load | test/load/ (e.g. test/load/g2a/g2a_load.js) | k6 / JS runner |
| goleak soak | services/external_vendors/{direct_topup,esim}/octopus/*_soak_test.go | go test ./services/external_vendors/.../octopus/... |
Race detector (-race) is used throughout CI. testhelpers.SkipIfShort(t) (test/testhelpers/db.go:53) lets long-running/DB tests be skipped with -short.
Repository test patterns
Two complementary styles:
- Real-Postgres integration — the bulk of
database/repo/*_test.go. CI spins up apostgresservice, runs migrations + seeders, thengo test ./database/repo/.... Isolation is per-test (transaction/cleanup helpers intest/testhelpers/db.go). - Fault injection (
database/repo/fault_injection_internal_test.go) — afaultDBwraps a realDatabaseClientand forwards everything until a fail-flag is flipped, then returns a configured error. This unlocks the error branches integration tests can't reach (connection failures, scan errors, malformed JSON).newFaultRepobuilds aRepositoryaround it. Combined with cache-fault helpers, this is how the repo layer hits the "every function needs happy + error + cache-path tests" bar.
database/repo/TESTING.md is referenced in CLAUDE.md but does not exist — the patterns live in the *_internal_test.go files and test/testhelpers/ instead.
testhelpers
test/testhelpers/ — shared infrastructure imported by the suites:
| File | Provides |
|---|---|
db.go | DB setup/teardown, SkipIfShort |
factory.go, seeddata.go | fixture builders + seed data |
http.go, config.go, assertions.go | request helpers, config, custom asserts |
wallet.go | wallet/balance test helpers |
mocky.go | MockyBaseURL() ($MOCKY_BASE_URL, default http://localhost:8788), MockyVendorURL(), MockyChaosURL() |
vendor_binding.go | BindVendorToMocky — points a vendor's host attribute at mocky's per-vendor sub-app, sets is_active, and restores both on cleanup; per-vendor locks so parallel tests don't clobber each other |
mockychaos/client.go | typed client for mocky's /_chaos API — arm/clear failure rules; NewForTest resets to a clean baseline and restores the seeded library on teardown |
cronrunner/runner.go | one-shot cron/retry job invokers (e.g. PendingVoucherOrderRetryOnce) to step a PENDING order through its lifecycle without sleeping. Separate subpackage because it imports jobs (which pulls in every vendor adapter) — keeping it out of testhelpers avoids import cycles |
test/shared/ adds concurrency.go (RunConcurrentN — fires N goroutines released simultaneously for race/idempotency tests, CountSuccess), isolation.go, suites.go, config_groups.go, fuzz.go, auth.go.
octopusfake — in-process Octopus upstream
test/octopusfake/octopusfake.go is a programmable net/http/httptest fake of the Octopus public /api/v1 used to test the federated Octopus vendor adapters (OCTO / OCTO_TOPUP / OCTO_ESIM) deterministically — including chaos, concurrency, fuzz and load that can't run against the live sandbox. It is not mocky-balboa: no separate service, just a server you start and program from a Go test.
- Core: auth (
/auth/login,/auth/refresh),/api/v1/wallets, a chaos-rule engine (Arm(Rule)— first armed rule matching method + path-contains applies; malformed-body kinds, consumption modes), a request recorder, and HMAC webhook signing. - Order engines (opt-in):
EnableTopupOrderEngine()/EnableEsimOrderEngine()give fake orders a real create → status-poll → deliver lifecycle;SetTopupBehavior(createStatus, pollStatus, failureCode)models async PENDING→DELIVERED and failure paths. The eSIM engine rendersactivation_code/icciddeliverables. SetHandler(method, path, h)overrides any exact route for bespoke scenarios.- Drives the goleak soak tests in
services/external_vendors/{direct_topup,esim}/octopus/*_soak_test.go(the OCTO family uses octopusfake instead of a mocky route — see Octopus adapters).
Vendor orchestration harness
test/clientapi/<vendor>_orchestration_test.go (build tag orchestration) drives real orders end-to-end through a vendor pointed at mocky-balboa, exercising chaos, malformed responses, async polling, signed webhooks, idempotency, retry caps, the link path, validation, and vendor logging. Runa is the ~28-subtest reference.
Ten suites exist today:
runa · wupex · neo · irewardify (voucher)
seagm_topup · dtone_topup (top-up)
dtone_esim · octopus_esim · octopus_topup (esim / octo)
g2a (inbound channel)Coverage gaps
No orchestration suite for TRS, EpinForce, or Grasshopper on the voucher side. TRS is the least-covered adapter overall (no tests, no mock route — see TRS). G2A and Runa are the best-covered. The money-leak suites do cover all three resources ({voucher,topup,esim}_money_leak_test.go).
Adding a new vendor suite (checklist)
- Add a per-vendor sub-app + routes to mocky-balboa (
frontend/mocky-balboa/src/index.ts) that mimic the real vendor's contract. - Seed chaos rules for that route (4xx/5xx/timeout/malformed/drop) in mocky's
chaos_rules. - In the test,
BindVendorToMocky("CODE", "mocky-path")(fromtesthelpers) to point the vendor'shostat mocky and restore on cleanup. - Use
mockychaos.NewForTest(t)to reset to a clean baseline, thenArmspecific rules per subtest. - Drive orders through the real client API; step retries with
cronrunnerone-shot helpers instead of sleeping. - Assert idempotency with
shared.RunConcurrentN(N concurrent same-client_referencerequests → exactly one durable effect). - Gate the file with
//go:build orchestrationand mirror the Runa suite's structure (chaos, malformed, async, webhook, idempotency, retry-cap, validation, logging).
See the vendor chaos + tests rule — every new adapter ships with a mocky route (incl. chaos hooks) plus unit + contract + service tests in the same PR.
mocky-balboa
frontend/mocky-balboa/ — a Cloudflare Worker (Hono + D1) mock vendor server with a /_chaos management API. Adapters are pointed at it by overwriting their host vendor-attribute (via BindVendorToMocky); tests arm failure scenarios through the mockychaos client. Base URL resolves from $MOCKY_BASE_URL (default http://localhost:8788; CI sets https://mocky-balboa.octopuscards.io). Deploys via GitHub Actions (deploy-mocky-balboa.yml) — see Cloudflare. Uses bun, not npm.
CI
.github/workflows/test-backend-api.yml — parallel jobs:
| Job | Runs |
|---|---|
| Lint & Static Check | golangci-lint |
| Cache Testing | go test -race -count=1 ./cache/... |
| Utils Testing | go test -race -coverprofile ./utils/... |
| Models Testing | go test -race -coverprofile ./database/models/... |
| Build Binary | go build -o octopus . → uploads artifact |
| Database Testing | Postgres service → download binary → migrate + seeders → go test -race -timeout=15m -coverprofile ./database/repo/... |
Frontend/worker deploys are separate workflows (grasshopper, mocky-balboa, docs, website — see Deploy & Release). The orchestration-tagged suites are not obviously wired into this CI file (they need mocky-balboa reachable) — verify whether they run in CI or only locally/against the deployed mock.
Findings
| Finding | Where |
|---|---|
automation/scripts/run-go-tests.sh (CLAUDE.md) and make setup (pre-push) are referenced but absent | — |
database/repo/TESTING.md referenced but absent | CLAUDE.md |
| No orchestration suite for TRS / EpinForce / Grasshopper; TRS wholly untested | test/clientapi/ |
| Orchestration suites may not run in the backend CI workflow (mocky dependency) | test-backend-api.yml |
Key files
- Harnesses:
test/octopusfake/octopusfake.go,frontend/mocky-balboa/src/ - Helpers:
test/testhelpers/{db,factory,mocky,vendor_binding}.go,test/testhelpers/mockychaos/client.go,test/testhelpers/cronrunner/runner.go - Shared:
test/shared/{concurrency,isolation,suites}.go - Fault injection:
database/repo/fault_injection_internal_test.go - Reference suite:
test/clientapi/runa_orchestration_test.go - CI:
.github/workflows/test-backend-api.yml; hook.githooks/pre-push
Troubleshooting Runbooks
Symptom-first playbooks for the tickets you'll actually get — "recharge not complete", "order stuck PENDING", "product not showing in the catalog", "can't log in", "can't pay in this currency". Each is a decision tree, an ordered file:line diagnostic checklist, the read-only SELECTs to run, and who can actually fix it.
Test Catalog
The complete inventory of test cases across every system — the 291 Go suites (150 repo with the three-branch sqlmock/faultDB/cachedCache harness, the 10 vendor orchestration suites / ~226 subtests, unit/utils/email), the 25 Grasshopper vitest files (~330 cases), what CI actually runs vs skips, the notable security assertions, and the concrete coverage gaps.