OTLP Endpoint Path Lab

The collector is healthy. Metrics arrive. Logs arrive. Traces get a 404.

This interactive lab demonstrates a precise OTLP/HTTP configuration failure: the general endpoint is a base URL that receives an automatically appended signal path, while a signal-specific endpoint takes precedence and is used exactly as configured.

Giving OTEL_EXPORTER_OTLP_TRACES_ENDPOINT the same pathless value as OTEL_EXPORTER_OTLP_ENDPOINT does not preserve the old behavior. It sends traces to the root or collector prefix instead of /v1/traces.

The endpoint resolver and traffic model are written in Pkl, Apple's embeddable configuration language. A dependency-free Python adapter serves a responsive routing workbench and correlated OTLP-shaped traces, metrics, and logs.

Default incident

The deployment contains:

OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example:4318/otel
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://collector.example:4318/otel
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

The resolved requests are:

Signal	Configuration source	Effective path	Result
Traces	signal-specific	/otel	404
Metrics	general	/otel/v1/metrics	200
Logs	general	/otel/v1/logs	200

With a 40% rollout across 12,000 requests producing eight spans each:

Observation	Value
Spans generated	96,000
Spans rejected	38,400
Requests without traces	4,800
Rejected 512-span batches	75
Trace coverage	60%
Healthy signals	2 of 3

This often looks like a sampling or instrumentation regression. It is neither: the exporter is making HTTP requests, and the collector is rejecting the resolved trace URL.

The specification rule

The stable OpenTelemetry OTLP Exporter specification requires OTLP/HTTP exporters to:

1. use each OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT as-is;
2. give the signal-specific option precedence;
3. treat OTEL_EXPORTER_OTLP_ENDPOINT as a base;
4. append v1/traces, v1/metrics, or v1/logs only to that general base.

Authoritative references:

• OTLP Exporter configuration
• OTLP/HTTP request paths
• OTLP SDK configuration

The fix is either to remove the redundant trace override:

OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example:4318/otel

or make the override a complete destination:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://collector.example:4318/otel/v1/traces

Why telemetry.sh helps

Service-level request metrics remain healthy and application logs continue to arrive. A trace-count dashboard alone cannot separate sampling, SDK rollout, and export rejection. The lab correlates:

• effective endpoint source and path policy;
• actual and expected trace URLs;
• per-signal HTTP response status;
• rejected span and batch counts;
• trace coverage by rollout cohort;
• exporter response logs tied to a diagnostic span;
• simultaneous 200 responses for metrics and logs.

It emits:

lab.otlp.trace.coverage
lab.otlp.export.rejected_spans
lab.otlp.export.path_valid
lab.otlp.export.rejected_batches

Telemetry.sh can group lab.otlp.export.path_valid by signal and endpoint source, making the asymmetric routing failure immediately visible.

Run locally

Requirements:

• Pkl 0.32.1
• Python 3.9 or newer
• Node.js (syntax check only)
• GNU Make

make check
make run

Open http://127.0.0.1:3000.

Endpoints:

GET /healthz
GET /api/simulate
GET /api/telemetry

Every experiment control can be supplied as a query parameter:

/api/simulate?requestCount=24000&spansPerRequest=6&rolloutPercent=25&batchSize=256&traceOverrideEnabled=true&overrideIncludesPath=false&prefixEnabled=true

Run with Docker

docker compose up --build

The multi-architecture image installs the pinned official Pkl 0.32.1 binary, runs the complete test suite during the build, and starts the service as an unprivileged user.

Project layout

model/endpoints.pkl  executable endpoint resolver and traffic model
service/lab.py       Pkl adapter and correlated telemetry builder
service/server.py    dependency-free HTTP and static service
public/              interactive routing workbench
tests/               model, telemetry, and static checks

License

MIT