spec(pebble): otel/otlp proxy draft specs

[hpidcock]

Still Drafting

---

Pebble OpenTelemetry Observability

This PR introduces full OpenTelemetry observability support to Pebble across all four OTLP signals: logs, metrics, traces, and profiles (OP0X1–OP0X6).

---

What's changing

Pebble gains four new plan sections — log-targets, metric-targets, trace-targets, and profile-targets — each backed by a local OTLP/HTTP receiver. Services running under Pebble get their OTEL_EXPORTER_OTLP_* environment variables injected automatically at startup. Pebble enriches incoming payloads with resource attributes (service.name, pebble.service, service.instance.id) before forwarding to the configured backend.

In addition, Pebble emits its own traces and metrics covering service lifecycle, health checks, plan updates, changes/tasks, and pebble exec invocations.

flowchart TD
    subgraph Services
        S1[service: web-server]:::accent0
        S2[service: worker]:::accent0
    end

    subgraph Pebble
        R[Local OTLP Receiver\n/v1/services/svc/otlp/...]:::accent2
        E[Enrichment\nservice.name, pebble.service\nservice.instance.id + labels]:::accent2
        SE[Self-emitted telemetry\nlifecycle spans, metrics\nhealth checks, exec]:::accent3
    end

    subgraph Backends
        B1[Remote OTLP backend\ne.g. Tempo / Mimir / Loki]:::accent5
        B2[otel-collector\ntail sampling, fan-out\nprotocol translation]:::accent4
    end

    S1 -- logs/metrics/traces/profiles --> R
    S2 -- logs/metrics/traces/profiles --> R
    R --> E
    E --> B1
    E --> B2
    SE --> B1
    SE --> B2
    B2 --> B1

Loading

---

Signal overview

Spec	Section	Service receives	Pebble self-emits
OP0X1	log-targets	stdout/stderr + OTLP logs	—
OP0X2	metric-targets	OTLP metrics	service state gauges, counters
OP0X3	trace-targets	OTLP spans	lifecycle, health-check, exec spans
OP0X4	profile-targets	OTLP profiles (alpha)	—

Each signal uses a per-service receiver path (/v1/services/{name}/otlp/v1/{signal}) so Pebble can attribute payloads without inspecting the body.

---

Pebble-emitted spans (OP0X6)

flowchart TD
    C[change: kind]:::accent2
    T1[task: kind]:::accent2
    T2[task: kind]:::accent2
    SL[service start/stop/restart]:::accent0
    HC[check: name]:::accent3
    PU[plan update]:::accent5
    EX[exec: command]:::accent4
    EP[external caller\ntraceparent header]:::accent1

    C --> T1
    C --> T2
    T1 --> SL
    T1 --> HC
    EP -. W3C traceparent .-> EX

Loading

Spans scoped to a specific service are only forwarded to trace-targets entries whose services list includes that service. Plan-update and change/task spans are forwarded unconditionally.

---

Delegate pattern (OP0X5)

Running otel-collector as an ordinary Pebble service and pointing all four *-targets at its local port (http://127.0.0.1:12345) enables tail sampling, attribute manipulation, fan-out to multiple backends, and protocol translation — without any changes to Pebble itself.

---

Key design decisions

• OTLP/HTTP JSON only for the local receiver; no gRPC/protobuf codegen required (binary protobuf is also accepted for profiles).
• Env-var injection is non-destructive: Pebble never overwrites a variable already set in the service's environment: block.
• Delivery failures to remote backends are logged and retried with exponential backoff; they never affect service management or the receiver's response to services.
• Profile support is experimental (/v1development/profiles) reflecting the alpha status of the OTel profiling signal.