Status Aware Error Sampling for OpenTelemetry

[vishalpolley-1mg]

Description

This PR implements Status Aware Error Sampling natively in the catalyst-core OpenTelemetry integration (src/otel.js).

In high-throughput environments operating at low head-sampling rates (e.g. 1%), standard head-sampling discards 99% of all traces, resulting in the loss of 99% of error traces (4xx and 5xx). This feature introduces a two-stage sampling flow that defers the export decision until a request finishes, allowing the system to selectively promote and export error traces at independent rates (typically 100% for 5xx and 10% for 4xx) without inflating successful trace volume or incurring excessive memory overhead.

The implementation is fully backwards-compatible and operates as an opt-in feature under the errorSampling configuration block.

---

Key Features & Components

1. Custom Sampler: StatusAwareSampler

• Upstream Propagation: Respects parent/upstream sampling decisions.
• Record Stage: If a trace fails the initial head-sampling check, it returns RECORD (instead of NOT_RECORD). This instructs the SDK to build spans in memory so they can be analyzed upon request completion.
• Independent Bit-Ranges: Uses characters 0–12 (first 48 bits) of the traceId hex string to evaluate the head-sampling probability.

2. Custom Processor: PromotingSpanProcessor

• Pass-through: Automatically routes head-sampled spans (SAMPLED bit set) straight to the internal BatchSpanProcessor to preserve standard low-latency batch exports.
• Trace Buffer: Holds non-sampled spans (RECORD only) in an in-memory Map keyed by traceId.
• Deferred Promotion Evaluation: When the root span of a request ends, the processor evaluates the HTTP status code:
  • 5xx Errors / Exceptions: Promoted at probability RATE_5XX (typically 100%). Evaluated using characters 12–24 of the traceId string.
  • 4xx Client Errors: Promoted at probability RATE_4XX (excluding configured timeouts/gateway noise).
  • Handled Errors: If enabled, promotes 2xx root spans that contain errored child spans underneath.
• Late-Arriving Spans: Employs a short-lived cache of recently promoted traceIds so that asynchronous child spans completing after the root span are still successfully exported.
• Memory Safeguards: Features a 30-second interval sweep enforcing a 5-minute TTL on buffered traces, alongside a hard limit of 1024 traces in the buffer to prevent OOM errors.

---

Configuration Reference

You can activate the feature by passing the errorSampling block to Otel.init:

"ERROR_SAMPLING": {
  "ENABLED": true,
  "RATE_4XX": 0.1,
  "RATE_5XX": 1.0,
  "EXPORT_FULL_TRACE_ON_ERROR": true,
  "PROMOTE_HANDLED_ERRORS": true,
  "REPORT_ACTUAL_PROMOTION_RATE": false,
  "SKIP_PROMOTION_CODES": [408, 504, 524, 598, 599]
}

---

Testing & Verification Done

• Unit & Integration Tests: Validated ratio-based head sampling decisions, span buffering, 4xx/5xx promotion criteria, handled error routing, and cache cleanup routines.
• Local Verification: Verified trace propagation using a local test script. Spans are correctly recorded, set with traceFlags: 1 (SAMPLED), and exported to Jaeger.
• Application E2E Test: Deployed and verified in local mweb and dweb apps, confirming successful traces are exported according to the sampling rate, while 4xx and 5xx errors are successfully promoted and tagged with the "promoted": true attribute.

[deputydev-agent]

DeputyDev will no longer review pull requests automatically.To request a review, simply comment #review on your pull request—this will trigger an on-demand review whenever you need it.