An interactive OpenTelemetry lab for a small API that records exception events
but sometimes forgets the separate SetStatus(ERROR) call.
At the default settings, the same 1,000 requests produce three defensible but incompatible dashboard answers:
| Query | Result |
|---|---|
status.code = ERROR |
0 errors |
events.name = "exception" |
50 exceptions |
http.response.status_code >= 500 |
20 failures |
The lab makes that disagreement explorable instead of hiding it behind a finished screenshot.
OpenTelemetry defines RecordException as a specialized way to add an event to
a span. Span status is controlled independently by SetStatus, and its default
value is Unset. Current exception guidance shows both calls in the recording
template. Instrumentation that records an exception but omits status therefore
leaves real evidence on a span that a status-only query will not select.
This lab deliberately creates that broken instrumentation pattern. It is not a recommendation to report handled exceptions as failures; use the semantic conventions appropriate to the operation and exception lifecycle.
References:
- OpenTelemetry Tracing API:
RecordExceptionandSetStatus - OpenTelemetry exception recording guidance
- Exception event semantic conventions
- Nix
nix-instantiatereference
A one-field dashboard answers only the question it was asked. telemetry.sh can keep the wider evidence chain together:
- exception events and their attributes;
- explicit span status;
- HTTP outcome;
- resource and instrumentation scope;
- metrics that count both sides of the gap;
- trace-correlated exception logs.
The result is an investigation that can explain why the charts disagree.
The easiest path installs no host dependencies beyond Docker:
docker compose up --buildOpen http://localhost:3000.
If Nix, Python 3, and Node are already installed:
make check
make runFor a UI-only local preview on a machine without Nix:
make previewThe preview uses an explicitly labeled Python mirror of the Nix arithmetic. The container and CI always execute the real Nix model.
curl 'http://localhost:3000/api/simulate?requests=1000&exceptionPct=5&statusCoveragePct=0&handledPct=60'
curl 'http://localhost:3000/api/telemetry?requests=1000&exceptionPct=5&statusCoveragePct=0&handledPct=60'The first endpoint returns the cohort model and query answers. The second returns correlated, OTLP-shaped trace, metric, and log payloads.
| Parameter | Range | Meaning |
|---|---|---|
requests |
100–5,000 | Requests represented by the run |
exceptionPct |
1–20 | Requests with an exception event |
statusCoveragePct |
0–100 | Exception spans explicitly marked ERROR |
handledPct |
0–100 | Exception requests that still return HTTP 200 |
The pure Nix model assigns every request to exactly one cohort and emits totals that the service verifies before exposing telemetry.
model/exception_status.nix deterministic cohort and query model
service/lab.py Nix adapter and correlated OTLP-shaped telemetry
service/server.py dependency-free HTTP service
public/ interactive evidence workbench
tests/ model, telemetry, and static checks
The experiment is deterministic and local. It opens no outbound connections, uses no credentials, and does not generate load against another service.
MIT