Orchestration Test Harness

---
name: Phase 3 vendor orchestration test harness
description: End-to-end orchestration test harness in test/clientapi that drives real voucher orders through vendor adapters pointed at mocky-balboa with chaos injection — Runa is the reference implementation, pattern is reusable for Neo/Wupex/SEAGM/DT One/etc.
type: project
originSessionId: 4a756616-78f2-4e8f-a906-fd34e36a1824
---
A full chaos-engineering orchestration test harness is in place under `test/`, gated by the `orchestration` build tag. **Runa is the reference implementation** at `test/clientapi/runa_orchestration_test.go` (28 subtests, ~13s runtime). When extending to a new vendor (Neo, Wupex, SEAGM, DT One), copy the structure verbatim and only swap vendor-specific knobs.

**Why:** Critical voucher/topup platform; vendor outages are the leading cause of customer-visible failures. Phase 3 covers the service layer above the per-vendor adapters built in Phases 1–2. It complements (not replaces) the per-vendor contract tests required by [feedback_vendor_chaos_and_tests.md](feedback_vendor_chaos_and_tests.md).

**How to apply:**

## Running

```bash
go test -tags=orchestration ./test/clientapi/... -run TestRuna_Orchestration -v
# Override mocky URL (defaults to http://localhost:8788):
MOCKY_BASE_URL=https://mocky-balboa.octopuscards.io go test -tags=orchestration ...
```

VS Code: `.vscode/settings.json` sets `go.buildTags: "contract,orchestration"` so gopls indexes these files.

## Key infrastructure (already built)

- **`test/testhelpers/mockychaos`** — typed client for mocky's `/_chaos` API. `NewForTest(t)` resets seeded chaos + restores on teardown; `ArmForTest(t, Rule{...})` deletes on teardown. `MustHaveFired` / `MustNotHaveFired` assert one_shot rules were/weren't consumed.
- **`test/testhelpers/cronrunner`** — wraps retry jobs out of an import cycle. `ProcessSingleVoucherOrder(t, ctx, id)` for surgical retry of one order; `PendingVoucherOrderRetryOnce(t, ctx)` for the batch tick (honors retry_count cap of 10). Also `TopupOrderRetryOnce`, `EsimOrderRetryOnce`.
- **`test/testhelpers/vendor_binding.go`** — `BindVendorToMocky(t, repo, code, mockyPath, suffix)` repoints the vendor's `host` attribute at mocky for the test (per-vendor mutex, full restore on cleanup). `SetVendorAttribute(t, repo, vendorID, key, value)` upserts api_key/webhook_secret. `SetVendorIsLinkEnabled` flips `vendor.is_link_enabled` for forcing the link or non-link path.
- **`test/shared/concurrency.go`** — `RunConcurrentN(t, n, fn)` for synchronized concurrent fires + panic recovery (idempotency tests).
- **`frontend/mocky-balboa/src/middleware/chaos.ts`** — supports modes `always`, `one_shot`, `count`, `probabilistic` (extended with optional `remaining_count` cap so probabilistic tests are bounded). Failure types: `status`, `delay_then_pass`, `delay_then_fail`, `malformed` (kinds: truncated_json/html/empty/invalid_utf8/wrong_schema), `drop` (TCP RST analog), `jitter`. Management endpoints under `/_chaos` and `/_mock/*` are exempt from chaos themselves.
- **`database/seeder/runa_vendor_product_seeder.go`** — pattern for a dedicated zero-inventory test product. Registered AFTER `InventorySeeder` so it lands without inventory rows, forcing orders through `vendor.CreateOrder` rather than the inventory cache. Replicate per vendor.

## Subtest structure (28 categories, mirror per vendor)

1. **Happy path** — order placement reaches vendor, asserts AUTH+PRODUCT_AVAILABILITY+CREATE logs persisted.
2. **Resiliency** — vendor 503 / 504 / 2s-delay-then-fail / persistent 500.
3. **Eventual delivery** — retry job recovers after vendor recovers.
4. **Concurrency idempotency** — `RunConcurrentN(t, 10)` with same ref → exactly 1 order row.
5. **Auth failure** — wrong API key; CREATE must NOT be logged but AUTH must be.
6. **Probabilistic flakiness with cap** — 70% fail capped at 3 actual failures, retry job recovers within N attempts.
7. **Always-retry policy** — `insufficient_funds` and `out_of_stock` are NEVER terminal; both must leave order PENDING + retry-eligible (the adapter's `TerminalForItem` is intentionally ignored by the orchestrator for these).
8. **Robustness** — 4 malformed kinds (truncated_json, html, empty, wrong_schema), 1 drop (TCP RST), 1 rate-limit 429.
9. **Async lifecycle** — status polling resolves PENDING→DELIVERED via `mockyForceCompleteRunaOrder` helper; GET 404 stays retryable; **signed Svix webhook** nudges retry_after, then poll → DELIVERED.
10. **Idempotency replay** — sequential same-ref → 400 Duplicate; cross-client same-ref also rejected (refs are GLOBAL by the handler dedup at `create_voucher_order.go:569`, not client-scoped — pin this behavior).
11. **Lifecycle/policy** — retry cap honored by batch job (`retry_count <= 10`), link-path doesn't call vendor at placement (requires AND of client+vendor+product `IsLinkEnabled` flags — must flip ALL three).
12. **Validation boundary** — denomination below min / above max → 400 with no vendor calls; multi-quantity (qty=3) produces N items.
13. **Vendor logging** — full lifecycle persists AUTH/AVAILABILITY/CREATE/GET_ORDER_STATUS/GET_VOUCHERS rows; failed CREATE must persist its log too (regression guard for the fix at [create_voucher_order.go:3431](http/handler/create_voucher_order.go#L3431) — `logVendorAPICallBulk` now ignores caller's `*sql.Tx` and uses its own committed tx so failure-path logs survive operation rollback).

## Gotchas (paid for in blood)

- **`IsLink` is the AND of three flags** (client + vendor + product `IsLinkEnabled`). Forcing the link path requires flipping ALL THREE, not just the vendor.
- **Webhook secret must be valid base64** after the `whsec_` prefix; svix.NewWebhook errors otherwise. Match the seeded value in `frontend/mocky-balboa/seed.sql`.
- **Test env must force memory cache.** `test/testenv/env.go` sets `CACHE_TYPE: "memory"` in `testEnvForceOverride` (NOT defaults), otherwise stale Valkey state from prior runs breaks tests.
- **Seeder cache flush.** `database/seeder/seeder.go` calls `flushCache(ctx)` at start of `RunAll` and `RunSeeder`. Without it, `StoreClientWithID`'s cached existence check causes silent INSERT skips after `migrate fresh`.
- **`Runa.GetOrderStatus(nil)`** must return `types.ErrOrderNotFound`, not a generic error — otherwise `checkExistingIndividualOrder` treats it as fatal and skips CreateOrder retries.
- **Vendor logs use their own tx.** Don't pass the operation's `*sql.Tx` to logging — use `nil` so the log row commits independently.