Skip to content

feat(tracing): opt-in to record the caller's remote parent as a span link - #2450

Open
jeremypng wants to merge 2 commits into
envoyproxy:mainfrom
jeremypng:tracing-remote-parent-as-link
Open

feat(tracing): opt-in to record the caller's remote parent as a span link#2450
jeremypng wants to merge 2 commits into
envoyproxy:mainfrom
jeremypng:tracing-remote-parent-as-link

Conversation

@jeremypng

@jeremypng jeremypng commented Jul 28, 2026

Copy link
Copy Markdown

Description

Fixes #2449.

By default the extproc continues the caller's propagated traceparent, so gateway spans join the calling application's trace. When the caller also instruments its own LLM client (e.g. OpenInference), one request produces two LLM spans in the same trace, and trace backends that group traces into projects — such as Phoenix, whose span ingestion assigns a span to the project of the existing trace row — double-count the request's token/cost rollups in the caller's project. The gateway's own project is also left with no root spans.

This adds AI_GATEWAY_TRACING_REMOTE_PARENT_AS_LINK: when "true", a wrapper on the shared tracer starts every span as the root of a new trace whenever the start context carries a remote span context, recording that context as a span link plus caller.trace_id/caller.span_id attributes (some backends, including Phoenix, do not ingest OTLP links). One seam covers the generic request tracer behind every LLM endpoint and the MCP tracer. Unset, behavior is byte-for-byte unchanged.

OTEL_PROPAGATORS=none is not a substitute: it also disables Inject, so the upstream provider stops receiving any traceparent, MCP _meta propagation dies, and all correlation with the caller is lost.

Notes:

  • Default is unchanged behavior; the feature is pure opt-in and the env var is read once at NewTracingFromEnv.
  • Tests cover: link/root/attribute/carrier semantics with a propagated traceparent, default-off parent continuation, inertness with no incoming context, and the MCP _meta path.
  • Docs: added a section to site/docs/capabilities/observability/tracing.md.
  • We run this in production (Azure OpenAI backend, Phoenix 19.8.0) — verified live that gateway spans land only in the gateway's own Phoenix project as root spans with caller.* correlation attributes, and consumer projects stop double-counting.
  • Happy to move the knob to the tracing config API or rename the env/attributes if maintainers prefer — naming follows the AI_GATEWAY_TRACING_* precedent from feat: add OpenTelemetry GenAI semantic convention for tracing #2405.

…link

By default the extproc continues the caller's propagated traceparent, so
gateway spans join the calling application's trace. When the caller also
instruments its own LLM client (e.g. OpenInference), one request produces
two LLM spans in the same trace, and trace backends that group traces
into projects — such as Phoenix, whose span ingestion assigns a span to
the project of the existing trace row (trace IDs are globally unique
there) — double-count the request's token/cost rollups in the caller's
project. The gateway's own project is also left with no root spans.

Add AI_GATEWAY_TRACING_REMOTE_PARENT_AS_LINK: when "true", a wrapper on
the shared tracer starts every span as the root of a new trace whenever
the start context carries a remote span context, recording that context
as a span link plus caller.trace_id/caller.span_id attributes (some
backends, including Phoenix, do not ingest OTLP links). One seam covers
the generic request tracer behind every LLM endpoint and the MCP tracer.
Unset, behavior is byte-for-byte unchanged.

OTEL_PROPAGATORS=none is not a substitute: it also disables Inject, so
the upstream provider stops receiving any traceparent and MCP _meta
propagation dies, and all correlation with the caller is lost.

Signed-off-by: Jeremy Sanders <23668453+jeremypng@users.noreply.github.com>
@jeremypng
jeremypng requested a review from a team as a code owner July 28, 2026 17:40
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Jul 28, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.86%. Comparing base (f9a03db) to head (bb0c2cc).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2450      +/-   ##
==========================================
- Coverage   84.86%   84.86%   -0.01%     
==========================================
  Files         154      155       +1     
  Lines       22419    22432      +13     
==========================================
+ Hits        19026    19036      +10     
- Misses       2237     2239       +2     
- Partials     1156     1157       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nacx
nacx enabled auto-merge (squash) July 30, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

extproc OpenInference spans join the caller's trace — double-counted token/cost rollups in Phoenix for self-instrumented callers

3 participants