feat(observability): distributed tracing via OpenTelemetry + Tempo

[agjs]

The third observability pillar. The API now ships OTLP/HTTP spans to a
Tempo backend that runs alongside Prometheus/Loki in the default
observability stack; Grafana has a Tempo datasource with two-way
click-through to Loki, so a slow request found in a metric / log line /
error event can be pivoted to its full trace waterfall in one click.

What's traced

• Incoming HTTP (Elysia) — parent span per request.
• Outgoing fetch / undici — Stripe, Resend, OpenAI, OAuth providers,
  anything you fetch. Each is a child span.
• ioredis / Valkey — BullMQ enqueue + lock operations.
• BullMQ job processing — every worker (account-maintenance,
  email-delivery, notification-dispatch, notification-maintenance,
  web-push-delivery) wraps its processJob in withQueueSpan,
  producing a queue.<name>.process span with messaging.* attributes
  (queue name, job id, attempt #). Failures record the exception.
• DB queries — opt-in via the new withDbSpan helper. postgres-js
  has no upstream OTel auto-instrumentation, so callers wrap hot-path
  queries explicitly with db.statement + custom attributes.
• Browser → API trace continuation — already in place via Sentry's
  browserTracingIntegration; the OTel SDK on the API now picks up
  the sentry-trace / traceparent headers and continues the trace.

apps/api

• src/instrument.ts (new) — side-effect import at the top of
  index.ts. Initialises the OTel SDK before any other module
  imports so the auto-instrumentations patch their targets at load
  time. Without this, http/ioredis/undici load unpatched and no spans
  are recorded.
• src/config/otel/otel.ts (new) — NodeSDK wrapper. OTLP/HTTP
  exporter, getNodeAutoInstrumentations with fs + dns disabled
  (they generate enormous span volume and drown the signal), service
  • version + deployment.environment resource attributes. No-op when
    OTEL_EXPORTER_OTLP_ENDPOINT is empty.
• src/config/env/{schema,validate}.ts — new env vars:
  OTEL_EXPORTER_OTLP_ENDPOINT (default empty),
  OTEL_SERVICE_NAME (default boringstack-api).
• src/config/logger/logger.ts — Pino mixin now reads the active
  span from @opentelemetry/api (works regardless of which SDK
  created the span) rather than directly from Sentry. Filters out
  the OTel sentinel 00000000… trace_id so log records outside a
  span context don't ship a useless zero id.
• src/lib/tracing/ (new) — withQueueSpan (BullMQ wrapper) +
  withDbSpan (opt-in DB wrapper) + barrel index.ts. Both record
  exceptions on the span via getErrorMessage to match the
  no-error-stringify lint rule.
• 5 worker files — wired through withQueueSpan at the Worker
  constructor's processor argument.

infra/compose

• New compose/tempo/tempo.yml — single-binary Tempo config, OTLP
  HTTP/gRPC receivers, 24h block retention, local storage at
  /var/tempo.
• docker-compose.observability.yml — new tempo service on the
  observability profile, mounted config, tempo_data volume,
  resource limits (TEMPO_LIMITS_CPUS/MEMORY). Grafana now
  depends_on: tempo.
• docker-compose.yml — api-dev and api services declare
  OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4318 and
  OTEL_SERVICE_NAME with sensible defaults.
• grafana/provisioning/datasources/datasources.yml — new Tempo
  datasource + derivedFields on Loki (clickable trace_id link
  opens Tempo) + tracesToLogsV2 on Tempo (span → matching Loki
  lines).
• .env.example — documents the new OTEL_* vars.

apps/docs

• topics/tracing.mdx (new) — full topic page: why tracing, what's
  auto-instrumented vs opt-in, six concrete use cases (slow-endpoint
  diagnosis, N+1 detection, job lag, benchmarking, log↔trace pivots),
  how to add manual spans, storage + retention + sampling.
• topics/observability.mdx — new "Tempo" entry in the "What ships"
  block.
• astro.config.mjs — sidebar wires Tracing between Observability
  and Alerts.

Tempo image

Pinned by tag (grafana/tempo:2.6.1) rather than digest because the
local Docker daemon was unavailable when this commit was prepared —
follow-up will pin the @sha256 to match the rest of the observability
stack's pattern.

Verification

• apps/api && bun run validate → 997 pass, 2 skip, 0 fail.
• apps/docs && bun run build → 67 pages built (was 66, +1 for
  Tracing topic), pagefind index clean.
• STACK=dev ./dev.sh config --quiet → exit 0 with the new Tempo
  service merged in.

Scope honestly noted: Drizzle's underlying postgres-js driver has
no upstream OTel auto-instrumentation, so DB query spans are opt-in
via withDbSpan. The helper + documented pattern ship; instrumenting
specific service methods is a follow-up exercise as you find slow
queries worth surfacing.

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com