Mocky-Balboa — Vendor Mock
The Cloudflare Worker (Hono + D1) that impersonates all seven third-party vendor APIs so the orchestration and contract tests run without touching live vendors — its per-vendor emulation, async-delivery modes (Svix/signed/unsigned webhooks vs poll), the chaos engine and its MustHaveFired self-deletion contract, the D1 schema and seeded credentials, deploy, and the shared-state fragility.
Mocky-balboa is a single Cloudflare Worker (Hono + one D1 database) that impersonates every real third-party vendor API so the Octopus backend's orchestration and vendor-contract tests run deterministically without touching live vendors. It self-describes as Vendor Mock Server v1.6.0 fronting [SEAGM, DT One, Runa, NeoCurrency, Wupex, iRewardify, EpinForce] (src/index.ts:36). Everything below is relative to frontend/mocky-balboa/.
This is "Track A" of the two faking tracks — the real third parties. The federated OCTO self-vendors use a separate in-process octopusfake instead. See also Frontends → mocky-balboa.
What it is & how Octopus points at it
Each real vendor row in Octopus carries a host attribute. The Go helper BindVendorToMocky (test/testhelpers/vendor_binding.go:63) rewrites that host to MockyVendorURL(path) (e.g. http://localhost:8788/runa/v2), forces is_active=true, and registers a t.Cleanup to restore it — so once bound, every vendor.CreateOrder/catalog call the adapter makes hits this worker's per-vendor sub-app instead of the real API.
- Local:
wrangler devon port 8788 (defaultMOCKY_BASE_URL). - Deployed: CI sets
MOCKY_BASE_URL=https://mocky-balboa.octopuscards.io— but note that hostname exists only in a Go comment; there is noroutes/custom_domainblock inwrangler.jsonc, so the custom domain is dashboard-managed and won't be re-established by a redeploy.
Architecture & middleware
Hono app (src/index.ts:21): cors() → a Drizzle-setup middleware that puts drizzle(c.env.DB) on context → seven vendor sub-apps (/seagm, /dtone, /runa, /neocurrency, /wupex, /irewardify, /epinforce) → the /_chaos management API mounted last.
Chaos runs BEFORE auth — deliberately
Every vendor sub-app wires chaosMiddleware(vendor) before its auth middleware (runa.ts:32,37), so failure injection can fire even on unauthenticated probes. Both chaos and auth exempt the /_chaos and /_mock/... paths so the test seams stay deterministic.
Per-vendor emulation
| Vendor | Auth | Delivery | Notable |
|---|---|---|---|
SEAGM (routes/seagm.ts, 2085 L) | HMAC-SHA256 — legacy query-param (uid/timestamp/signature, ±120s) and topup header (X-UID/X-Timestamp/X-Sign) | card orders instant; recharge/topup async signed callback (3s legacy / 20s topup) | two API surfaces; _MOCK_FAIL/_MOCK_HANG order-id suffixes force terminal/stuck |
DT One (routes/dtone.ts, 1025 L) | HTTP Basic | unsigned callback (500ms setTimeout) | service_id 1=mobile, 13=eSIM (same sub-app); dup external_id→409 1007001; eSIM ICCID+LPA: PIN ripens over polls |
Runa (routes/runa.ts, 1049 L) | X-Api-Key | Svix-signed webhook (~3s) or sync | idempotency cache, X-Execution-Mode, cursor catalog, CloudFront markdown mirror |
NeoCurrency (routes/neocurrency.ts, 597 L) | Bearer (from /get-token) | synchronous — codes returned inline, polled recovery via get-orders-rfid | no webhooks; requires_prepaid_client_id for Bancorp/Pathward |
Wupex (routes/wupex.ts, 566 L) | x-api-key | synchronous inline codes | {data, status} envelope; "File not found" literal for not-found (Go substring-matches) |
iRewardify (routes/irewardify.ts, 888 L) | Bearer JWT | unsigned webhook + optional auto-fulfil | multivariant (mongo_id+item_id); webhook_secret column exists but unused (prod unsigned) |
EpinForce (routes/epinforce.ts, 1061 L) | Bearer apikt_ | poll-only + auto-fulfil (3s seed) | stateful server-side cart (unique per user → concurrent contention); codes delivered only as XLSX/ZIP files via lib/xlsx.ts |
Each vendor exposes /_mock/... seams (complete/:id, fail/:id, webhook-url, auto-fulfill, seed/delete rows) that let a contract test drive create→complete deterministically without waiting on the scheduler or chaos. EpinForce is the outlier — its /_mock/* seams require auth.
Async-delivery modes differ per vendor — a common source of confusion
Runa = Svix-signed webhook · SEAGM = signed payment-order callback · DTOne = unsigned callback (and uses a bare setTimeout, not waitUntil, so it may be unreliable on the deployed runtime) · iRewardify = unsigned webhook + auto-fulfil · EpinForce = poll-only + auto-fulfil · Neo & Wupex = fully synchronous (codes inline, no webhook).
The chaos engine
Three parts: middleware/chaos.ts (injection), routes/_chaos.ts (management API), and the chaos_rules D1 table (migration 0009_chaos_rules.sql). This is what the Go mockychaos client drives.
Rule model
| Field | Meaning |
|---|---|
vendor | one of 7, or * wildcard |
method / path_substring | match filters (null = any); path is matched after stripping the /<vendor> mount prefix |
mode | one_shot · count · always · probabilistic |
probability / remaining_count | probabilistic threshold / consumption counter (also an optional cap on probabilistic) |
failure_type | status · delay_then_pass · delay_then_fail · malformed · drop · jitter |
status_code/status_body, delay_ms, malformed_kind (truncated_json/html/empty/invalid_utf8/wrong_schema), jitter_min/max_ms | per-type params |
label, enabled | LIKE-matchable label; enabled=0 = library/template |
Firing & the MustHaveFired contract
Rules matching (vendor OR '*') AND enabled=1 are loaded in id order; the first that passes method+path match and shouldFire wins and short-circuits. The key mechanic:
A fired one-shot deletes itself — that's how the Go assertions work
For consuming modes (one_shot/count/capped-probabilistic), firing decrements remaining_count; when it hits 0 the row is DELETED (chaos.ts:84). So the Go mockychaos.MustHaveFired(t, id) lists rules and fails if the id is still present — "still there = never matched, the test observed the happy-path baseline." MustNotHaveFired is the inverse. This catches tests that silently exercised no chaos.
Failure application: status (custom code/body + X-Chaos-* headers), delay_then_pass/delay_then_fail (sleep then continue/fail), malformed (truncated/html/empty/raw-0xff invalid-utf8/wrong-schema), drop (throw mid-request → bare 500, the closest Workers analog to a TCP RST), jitter (random delay then continue).
/_chaos management API
GET / (list+docs), POST / (create), GET /:vendor, GET /library, POST /:id/enable|disable, bulk enable|disable {label_like}, enable-all|disable-all, DELETE /:id|/vendor/:vendor|/, and POST /preset/:name (outage/flaky/slow/malformed-storm/drop-storm/unauth-storm).
/_chaos has no authentication at all
Anyone who can reach the deployed worker can arm outages/drops against it. Combined with the global rule state below, this is the biggest operational fragility.
D1 schema & seed
~34 tables, grouped by vendor (users/products/orders/codes per vendor, plus SEAGM's card+recharge split, DTOne operators/transactions, Runa idempotency_cache, EpinForce carts/deliveries) and the cross-cutting chaos_rules.
seed.sql is a 1.26 MB committed artifact run by wrangler d1 execute mocky --file=seed.sql. Contents: 74 dtone_products + 16 dtone_operators, 154 seagm_recharge_types + 30 card types, 12 runa products, 11 neo brands, 10 wupex, 15 irewardify variants, 9 epinforce — and ~201 chaos rows (27 INSERT statements), all low-probability probabilistic rules armed by default.
Seeded credentials must match the Octopus vendor seeders
Mock auth silently 401s if the Octopus vendor seeders drift from these:
| Vendor | Credential |
|---|---|
| SEAGM | uid SEAGM_TEST_001 / sk_test_seagm_secret_key_12345, webhook …/webhooks/direct-topups/SEAGM |
| DT One | Basic dtone_test / dtone_secret_123 |
| Runa | xx_test_runa_playground_123, whsec whsec_MfKQ9r8tZlF3q+bJvR2GkXcPdS5TmYwA6HnV1xJ4eUo= |
| NeoCurrency | neo_local_client_id / neo_local_client_secret / neo_local_password (token via /get-token) |
| Wupex | test-api-key, merchant WUPX_TEST_MERCHANT |
| iRewardify | token irw_test_access_token_local_mock, auto_fulfill_ms=3000 |
| EpinForce | apikt_epinforce_local_mock_token, auto_fulfill_ms=3000 |
src/db/seed.ts is dead code (its own header says so) — it duplicates SEAGM data and is wired to no caller; the live seed is seed.sql. drizzle-seed is a devDependency but unused by the seed path.
Deploy & infra
wrangler.jsonc: worker mocky-balboa, D1 binding DB→mocky (6dad79fb-…), vars.APP_ENV=local, no routes/custom_domain/compat_flags. Scripts: deploy (wrangler deploy --minify), db:reset (local wipe+migrate+seed), db:seed[:remote], migrate:local|remote, db:reset:remote. 19 migrations 0000→0018 (VoucherKart dropped at 0008; auto-fulfil added for EpinForce 0017 and iRewardify 0018).
scripts/reset-remote.sh deletes+recreates the remote D1, auto-rewrites database_id in wrangler.jsonc via sed, re-migrates+seeds, then redeploys (a recreated D1 gets a new id the live worker must rebind).
How tests consume it
MOCKY_BASE_URL default http://localhost:8788, CI → deployed. BindVendorToMocky rewrites the host (Runa passes /v2 suffix, others ""); a per-vendor sync.Mutex serialises same-vendor host mutations; SetVendorAttribute aligns creds to the seeded values; SetVendorIsLinkEnabled(false) forces Runa off the link path so CreateOrder actually runs and chaos on /v2/order can fire.
Global chaos state → concurrent CI runs interfere
Chaos rules are global rows in the shared D1. mockychaos.ResetForTest flips enabled globally (DisableAll/EnableAll). Against the deployed mocky, one job's disable/arm affects another job's requests — chaos was designed for a per-run local D1. This is the documented flakiness source for the vendor-contract and orchestration CI jobs.
Findings
| # | Finding |
|---|---|
| 1 | /_chaos is fully unauthenticated — anyone reaching the worker can arm outages |
| 2 | Global chaos state on the shared deployment — concurrent CI runs interfere |
| 3 | No deployed route in config — mocky-balboa.octopuscards.io is dashboard-only, absent from wrangler.jsonc |
| 4 | 1.26 MB hand-maintained seed.sql — error-prone; src/db/seed.ts is stale dead code that could mislead |
| 5 | Committed test secrets in seed.sql — low-risk but must stay in lockstep with the Octopus vendor seeders |
| 6 | DTOne webhook uses bare setTimeout (not waitUntil) — may not fire after the response on the deployed runtime |
| 7 | "dtone-esim" is not a separate mount — it's service_id=13 products inside /dtone |
| 8 | OCTO self-vendors are not here — served by the separate in-process octopusfake |
Key files
- Entry/routing:
src/index.ts; middlewaresrc/middleware/{chaos,auth}.ts; chaos APIsrc/routes/_chaos.ts - Vendor routes:
src/routes/{seagm,dtone,runa,neocurrency,wupex,irewardify,epinforce}.ts - Libs:
src/lib/{svix_sign,xlsx}.ts; schemasrc/db/schema.ts; seedseed.sql(+ deadsrc/db/seed.ts) - Config:
wrangler.jsonc,package.json,scripts/reset-remote.sh,drizzle/migrations/0000…0018 - Go side:
test/testhelpers/{mocky.go,vendor_binding.go,mockychaos/client.go}
Vendor Adapter Tests
The build-tagged test layer colocated with the vendor adapters (services/external_vendors/**) — the unit/contract/soak/sandbox/fuzz matrix, the per-vendor coverage table, how contract tests bind to mocky-balboa and drive chaos, the two goleak soak tests, and the concrete gaps (TRS/GH/iRewardify untested, SEAGM money-mapping fixed-but-unverified, the triple FailureInfo declaration, the untested Svix verifier).
Seed Catalog
Every database seeder in the platform — the framework (Seeder interface, runner, seed all/fresh/list/info, the load-bearing cache flush, and the absence of any prod guard), all 41 registered seeders in dependency order, the 11 unregistered test-invoked orchestration seeders that wire vendors→tests, the no-op Grasshopper D1 seed, and the committed dev credentials worth rotating.