Tail Sampling Race Lab

An executable XSLT lab showing how late error spans can lose a race against an OpenTelemetry Collector tail-sampling decision. Metrics and logs still report the failure, but the error trace disappears before anyone can investigate it.

The default scenario models one minute of checkout traffic:

• 60,000 traces enter the Collector.
• 2.5% of requests fail, producing 1,500 error signals.
• The root span arrives immediately.
• The tail sampler decides after 5 seconds.
• A batched downstream error span arrives after 9 seconds.
• A 1% probabilistic policy provides baseline coverage.

At decision time, the sampler has not seen the error status. It drops the trace unless it happens to survive the baseline policy:

1,500 error traces × 1% baseline = 15 retained
1,500 metrics/logs − 15 traces = 1,485 broken correlations

The application failed. The metric exists. The log even carries a trace ID. But 99% of those IDs have no exported error trace behind them.

This is distinct from a generic sampling-rate blind spot. The failure is a decision-timing race: a policy intended to keep every error cannot act on an error span it has not received yet.

Why the race happens

The Collector's tail-sampling processor groups spans by trace ID and evaluates the accumulated trace after decision_wait. A span is late when it arrives after that decision. Late spans inherit an existing decision while it remains known, but they do not change the original policy result.

The obvious fix—wait longer—has a cost. At the default traffic and 16 KiB average trace size:

Strategy	Decision wait	Error coverage	Live trace buffer
Current window	5s	1%	78.1 MiB
Late-span-aware	10s	100%	156.3 MiB

The lab makes that evidence-versus-memory tradeoff explicit instead of treating decision_wait as a guess.

What telemetry.sh reveals

The OTLP-shaped payload correlates:

• lab.tail_sampling.error_signals
• lab.tail_sampling.error_traces_retained
• lab.tail_sampling.correlation_gap
• lab.tail_sampling.buffer_memory
• otelcol_processor_tail_sampling_sampling_late_span_age
• otelcol_processor_tail_sampling_sampling_traces_on_memory

It also includes an error log whose trace ID is deliberately absent from the current-window trace export, plus a complete error trace from the late-span-aware strategy. That makes cross-signal evidence loss directly queryable in telemetry.sh.

Run it

Requirements:

• Node.js 20+
• xsltproc from libxslt

npm ci
npm start

Open http://localhost:8080.

Or run the clean Linux container:

docker compose up --build

Use the XSLT model directly

xsltproc \
  --nonet \
  --stringparam decision_wait_ms 5000 \
  --stringparam error_span_arrival_ms 9000 \
  --stringparam traces_per_minute 60000 \
  --stringparam error_rate_basis_points 250 \
  --stringparam baseline_sample_basis_points 100 \
  --stringparam trace_size_bytes 16384 \
  model/simulate.xsl \
  model/scenario.xml

The Node adapter invokes that stylesheet for every simulation:

npm run simulate
npm run simulate -- --decision-wait-ms 12000
npm run telemetry

HTTP endpoints expose the same model:

curl 'http://localhost:8080/api/simulate?decisionWaitMs=5000'
curl 'http://localhost:8080/api/telemetry?decisionWaitMs=5000'

Collector configuration

collector/tail-sampling.yaml contains the status-code and probabilistic policies represented by the default scenario. The complete pipeline and deployment-specific receivers/exporters are left to the operator.

Runtime proof

Every model and telemetry response includes the libxml/libxslt runtime version, a SHA-256 digest of model/simulate.xsl, and the model path. The Node server does not duplicate the simulation arithmetic.

Verify

npm run check

Tests cover late and on-time spans, zero baseline sampling, bounded inputs, cross-signal OTLP correlation, XSLT runtime proof, and every HTTP endpoint. CI also builds the Alpine Linux container.

References

• OpenTelemetry Collector tail-sampling processor
• Tail-sampling internal telemetry
• xsltproc command-line reference

License

MIT