Metrics & Tracing
The OpenTelemetry → SigNoz observability catalog — every custom metric with its type/unit/attributes, the tracing conventions, log fields, the export pipeline, and the dimension-compliance gaps and dead metrics.
Full observability catalog: OpenTelemetry SDK → OTLP/HTTP → SigNoz (traces + metrics + logs share one endpoint/token). OTLP push only — there is no Prometheus scrape endpoint. For the operational view see Jobs & Observability; this is the signal catalog.
Init (main.go:127, gated on OTEL_ENABLED): InitTracer → InitMetricsExporter → metrics.InitInstruments() → InitLogExporter. Env vars in Environment → Telemetry.
Metrics inventory
Every metric name is prefixed with SERVICE_NAME (prefix(), instruments.go:15), so the real name is e.g. octopus.business.orders.created. Attributes are applied at record time.
HTTP / infra (middleware/metrics.go)
| Metric | Type | Unit | Attributes |
|---|---|---|---|
http.server.requests | Int64Counter | {request} | route_group, method, status_code, route |
http.server.request.duration | Float64Histogram | s | route_group, method, status_code, route |
http.server.active_requests | Int64UpDownCounter | {request} | route_group |
route_group ∈ api / admin / client / auth / webhook / other; route = registered pattern (not raw URL).
Business metrics (business.*)
| Metric | Type | Attributes | Recorded at |
|---|---|---|---|
business.orders.created | Int64Counter | voucher: type/status/sub_status (repo/order.go:735); topup/esim: +vendor_id/client_id/product_id/currency | 3 sites |
business.orders.amount | Float64Histogram | topup/esim: type/status/vendor_id/client_id/product_id/currency | 2 sites |
business.orders.items | Int64Histogram | topup/esim same set | 2 sites |
business.vouchers.issued | Int64Counter | vendor_id, order_id (no product_id) | create_voucher_order.go:3006 |
business.vouchers.failed | Int64Counter | — | 💀 never recorded |
business.topups.processed | Int64Counter | type/status/vendor_id/client_id/product_id/currency | topup_order.go:281 |
business.topups.amount | Float64Histogram | same | topup_order.go:282 |
business.payouts.created | Int64Counter | status, client_id, currency | payout_service.go:184/200 |
business.payouts.amount | Float64Histogram | status, client_id, currency | payout_service.go:201 |
business.wallet.transactions | Int64Counter | type, client_id, currency_id | repo/wallet.go:483 |
business.wallet.transaction_amount | Float64Histogram | type, client_id, currency_id | repo/wallet.go:484 |
business.vendor_api.requests | Int64Counter | — | 💀 never recorded |
business.vendor_api.latency | Float64Histogram | — | 💀 never recorded |
business.inventory.allocated | Int64Counter | product_id (no vendor/client_id) | inventory.go:824 |
business.inventory.depleted | Int64Counter | product_id | inventory.go:831 |
business.g2a.reservations_created | Int64Counter | client_id | g2a_reservation.go:165 |
business.g2a.orders_created | Int64Counter | client_id, status | g2a_order.go:237 |
business.g2a.codes_delivered | Int64Counter | client_id, product_id ✅ | g2a_order.go:240 |
business.g2a.notifications_received | Int64Counter | client_id, type | g2a.go:188 |
The metrics-dimensions rule is only partly honored
The rule requires business metrics to carry client_id + product_id for Grafana/SigNoz drill-down. Compliance:
- ✅ Full:
topups.*,orders.*(topup/esim paths),g2a.codes_delivered. - ⚠️ client_id only:
payouts.*(no product concept),wallet.*,g2a.reservations_created/orders_created/notifications_received. - ❌ product_id only, no client_id:
inventory.allocated/depleted. - ❌ neither:
orders.createdon the voucher/repo path (type/status/sub_status only),vouchers.issued(vendor_id/order_id only).
So voucher order creation — the flagship KPI — records without client_id or product_id, and does not record amount/items at all (only the topup/esim paths do).
Non-OTel metrics (in-process, not exported)
metrics/metrics.go has two in-memory trackers surfaced only via the admin dashboard API — never pushed to OTel or Prometheus:
- ServerMetrics — uptime %, avg API response time (fed by
RecordAPIResponsefrom the tracing middleware). - CronMetrics — per-job LastRun/NextRun/Duration/Status, persisted to Redis
cron:metrics:jobs(24h TTL). This is the "cron Prometheus metrics" the scheduler mentions — but it's Go structs + Redis, not Prometheus. See Cron Catalog.
Tracing
utils/tracer.go InitTracer:
- Exporter:
otlptracehttptoOTEL_EXPORTER_OTLP_ENDPOINT(token headersignoz-access-token; TLS off whenINSECURE_MODE=true). - Sampler:
AlwaysSample()— 100% sampling. - Propagation: W3C
TraceContext+Baggage— so traces continue across services (e.g. Grasshopper → backend viatraceparent). - Global
utils.Tracerdefaults to a noop tracer so uninitialized paths don't nil-panic.
Span convention — manual, no generic wrapper: ctx, span := utils.Tracer.Start(ctx, "<name>"); defer span.End(). ~1480 call sites. Names are layer.operation, kebab-case op:
| Prefix | Sites | Layer |
|---|---|---|
repo.* | ~780 | repository/DB (dominant) |
handler.* | ~223 | HTTP handlers |
service.* / services.* / inventory-service / prefetch-service | ~40 | service layer (inconsistent prefix) |
admin.*, admin-api.*, webhook.*, shopify.*, webauthn.* | — | domain handlers |
*-job, processor, execution-tracker | — | scheduler/jobs |
HTTP entry span comes from otelfiber.Middleware() (the request-root span); the custom TracingMiddleware deliberately creates no extra span — it reads the otelfiber span's IDs and enriches the logger.
Span attributes are set explicitly in scheduler tasks (task_id, client_count) but there is no generic tenant_id/client_id span attribute injected in middleware. Tenant/client context rides in metric labels and log fields, not span attrs.
Logs
Zap (middleware/logging.go). Logger(ctx) pulls a request-enriched *zap.SugaredLogger from context, else builds a default JSON logger (ISO8601 timestamp, lowercase level, msg, caller; initial fields pid + service_name).
Per-request fields (TracingMiddleware): trace_id, span_id, method, path, ip. Request ID = X-Request-Id or trace_id fallback (echoed back). Debug dumps mask sensitive headers (authorization/cookie/x-api-key/…) and truncate the body to 4096 bytes; the completion log Request completed carries status/duration_ms/response_size.
OTel log export (utils/log_exporter.go): otlploghttp to the same endpoint, bridged via otelzap.NewCore teed with stdout — logs go to both stdout and SigNoz. Known cosmetic caveat: otelzap can't read trace/span IDs from Go ctx, so they're duplicated as Zap fields (correlation still works). LOG_LEVEL gates debug.
Export pipeline
| Concern | Value |
|---|---|
| Master switch | OTEL_ENABLED (default false, fatal if unparseable) |
| Endpoint | OTEL_EXPORTER_OTLP_ENDPOINT (default localhost:4318) |
| Auth | OTEL_EXPORTER_OTLP_TOKEN → signoz-access-token header |
| TLS | INSECURE_MODE (default true → no TLS) |
| Metric interval | OTEL_METRIC_EXPORT_INTERVAL (default 60s, floor 10s), PeriodicReader |
| Resource | service.name=SERVICE_NAME; service.version=SERVICE_VERSION (metrics/logs only); environment+deployment.environment=ENVIRONMENT |
There is no SIGNOZ_ENDPOINT var — SigNoz is reached via the OTLP endpoint+token. See Infrastructure → telemetry (SigNoz UI on the sandbox box).
No committed dashboards — no Grafana/SigNoz dashboard JSON in-repo (only the signoz.conf reverse proxy + a health-check script). The admin "dashboard" is an application feature (ServerMetrics/CronMetrics), not an observability dashboard. Drill-down dimensions are enforced only by convention at record sites (see the gaps above).
Findings
| Category | Detail |
|---|---|
| Dead metrics | vouchers.failed, vendor_api.requests, vendor_api.latency — defined, never recorded |
| Missing dims | voucher orders.created (no client_id/product_id), vouchers.issued, inventory.* (no client_id), several g2a.* (no product_id) |
| Doc-vs-actual | instrument struct comments wrong: vouchers.issued doc says vendor,product_id (actual vendor_id,order_id); wallet.* doc type,source (actual type,client_id,currency_id); orders.* source label never emitted |
| Naming inconsistency | span prefix service. vs services. vs inventory-service.; vendor_id vs vendor; currency vs currency_id; trace resource omits service.version; stale "Mimir" comment (main.go:135) though backend is SigNoz |
Key files
- Metrics:
metrics/instruments.go,metrics/metrics.go,middleware/metrics.go - Tracing:
utils/tracer.go,middleware/tracing.go - Logs:
middleware/logging.go,utils/log_exporter.go - Export:
utils/metrics_exporter.go; wiringmain.go:120-148,490-499