An interactive OpenTelemetry lab about a slow request whose duration is correct in traces and corrupted before it reaches an application histogram.
The default case converts a three-second duration to nanoseconds and stores it in a signed 32-bit field:
true elapsed 3,000,000,000 ns
32-bit bit pattern 0xB2D05E00
signed value -1,294,967,296 ns
That one cast produces two incompatible answers for the same 1,000 requests:
| Query | Result |
|---|---|
span.duration >= 2s |
50 slow requests |
http.server.request.duration > 2s |
0 slow requests |
first histogram bucket (<= 10ms) |
1,000 requests |
Moving the counter-width control from 32 to 33 bits preserves the default three-second observation and closes the gap.
OpenTelemetry histograms aggregate the measurements applications give them.
They cannot reconstruct information that was lost before Record was called.
The Metrics API says histogram values used for durations are expected to be
non-negative, general semantic conventions recommend seconds for duration
metrics, and OTLP explicit histogram upper bounds are inclusive.
This lab deliberately violates the non-negative expectation in application code, then emits the resulting internally consistent histogram. Because the negative observation is below every positive boundary, it lands in the first bucket. The producer omits the optional histogram sum, following the SDK recommendation for instruments that record negative measurements.
References:
- OpenTelemetry Metrics API
- OpenTelemetry Metrics SDK histogram aggregation
- OpenTelemetry Metrics Data Model
- OpenTelemetry metric units
The histogram alone looks authoritative. telemetry.sh can keep the conflicting evidence close enough to explain it:
- span start and end timestamps from the tracing scope;
- the application histogram from a separate instrumentation scope;
- an exemplar connected to the affected trace and span;
- the signed and true nanosecond values as diagnostic attributes;
- a trace-correlated overflow event and warning log;
- a monotonic count of affected requests.
The lab is about finding the transformation where truth was lost, not blaming the signal that preserved the transformed value.
With Docker:
docker compose up --buildOpen http://localhost:3000.
With Icarus Verilog, Python 3, and Node installed locally:
make check
make runcurl 'http://localhost:3000/api/simulate?requests=1000&slowPct=5&slowDurationMs=3000&counterBits=32'
curl 'http://localhost:3000/api/telemetry?requests=1000&slowPct=5&slowDurationMs=3000&counterBits=32'The first endpoint returns the Verilog experiment and query comparison. The second returns correlated, OTLP-shaped trace, metric, and log payloads.
| Parameter | Range | Meaning |
|---|---|---|
requests |
100–5,000 | Requests represented in the run |
slowPct |
1–25 | Requests in the slow cohort |
slowDurationMs |
2,200–8,000 | True duration of each slow request |
counterBits |
24–40 | Width of the signed nanosecond field |
model/duration_model.v fixed-width duration and cohort model
service/lab.py Verilog adapter and OTLP-shaped signal builder
service/server.py dependency-free HTTP service
public/ interactive logic-analyzer workbench
tests/ model, correlation, and static checks
The experiment is deterministic and local. It opens no outbound connections, uses no credentials, and sends no traffic to another service.
MIT