Telemetry docs: rewrite for SDK-spec OTEL variables and semconv metrics (targets 1.0.0-alpha4)

[Rachael-Graham]

Important

Unblocked. kagent#2909 and kagent#2929 both shipped in 1.0.0-alpha4, which is released. The hold in the original description no longer applies and this is actionable now, against alpha5 as the current release.

The KAGENT_ environment rename does not touch this work. alpha5 moved 138 kagent-specific settings under a KAGENT_ prefix, but OTEL_* variables are explicitly excluded — verified in docs/env.md at the alpha5 tag, where OTEL_EXPORTER_OTLP_* and OTEL_TRACES_EXPORTER still appear unprefixed.

One addition from alpha5: kagent#2772 flushes buffered OTEL logs, which affects what a reader sees at the end of a session.

Pages listed below are also being rewritten for the Agent and Session rename in #549. Sequence this after that pass, or expect to edit the same files twice.

What changes

kagent#2909 ("feat(telemetry): configure runtimes with SDK-spec OTEL variables") replaces kagent's own telemetry flags with the standard OpenTelemetry SDK variables, from the chart through to every runtime. It is a breaking change for anyone following the current pages.

Chart values removed: otel.tracing.*, otel.logging.*, otel.captureSensitiveContent, and insecure (the URL scheme decides now).

Replacements: otel.exporter.otlp.{endpoint,protocol,timeout}, otel.traces|metrics|logs.{enabled,endpoint,protocol}, and otel.capture.{messageContent,maxBytes,rawApiBodies}. A per-signal endpoint is now used as given — the chart no longer appends /v1/<signal>.

Environment variables removed: OTEL_TRACING_ENABLED, OTEL_LOGGING_ENABLED, OTEL_EXPORTER_OTLP_*_INSECURE, KAGENT_OTEL_CAPTURE_SENSITIVE_CONTENT, KAGENT_PRE_RESPONSE_TRACE_FLUSH, KAGENT_TRACE_FLUSH_TIMEOUT_MS, and the legacy endpoint aliases. Use OTEL_TRACES|METRICS|LOGS_EXPORTER and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY|NO_CONTENT.

Three behavior changes that need prose, not just a value swap:

• --reuse-values stops working. Upgrades need helm upgrade --reset-then-reuse-values; plain --reuse-values from an older release fails on the new otel.metrics key. Every upgrade snippet in the docs uses the old flag.
• An unset exporter now means on. A runtime started without OTEL_*_EXPORTER exports to localhost, which is the SDK default. Previously it stayed off.
• Baggage is no longer propagated. OTEL_PROPAGATORS=tracecontext, so a caller's baggage does not reach MCP servers or model providers. Opting back in means setting OTEL_PROPAGATORS=tracecontext,baggage in Harness spec.env.

Harness spec.env also now rejects OTEL_SERVICE_NAME, OTEL_SDK_DISABLED, the exporter, endpoint, protocol and timeout variables, and OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT for the Claude and Codex harnesses. The kagent harness overrides them.

Pages affected

Surveyed against 1.x on 2026-09-24:

Page	Why
1.x/observability/tracing.md	otel.tracing.*, captureSensitiveContent
1.x/observability/audit-prompts.md	otel.logging.*, captureSensitiveContent, OTEL_EXPORTER_OTLP, --reuse-values
1.x/agents/agent-harness.md	otel.logging.*, captureSensitiveContent, and the new spec.env rejections
1.x/setup/installation.md	--reuse-values
1.x/agents/agent-memory.md	--reuse-values
1.x/reference/helm.md	helm-docs output — regenerates from the chart, do not hand-edit

Before starting

Confirm kagent#2909 has merged and is in a released chart, then read docs/architecture/telemetry.md in the kagent repo at that release — the PR rewrites it (+54/-36) and it is the authoritative description of the new model.

Done when

• kagent#2909 is merged and in a released chart version the 1.x docs target
• Every removed value and environment variable is gone from the five hand-written pages
• Upgrade snippets use --reset-then-reuse-values
• The three behavior changes above are stated in prose, not just reflected in a values table
• 1.x/reference/helm.md regenerated rather than edited
• Harness spec.env rejections documented on the harness page

[krisztianfekete]

@Rachael-Graham, wow, thanks for catching this before I even had a chance to ping docs!
The PR you are referencing has most of the breaking changes, but if you want to make sure you are tracking this initiative holistically, you can check the OSS epic here: kagent-dev/kagent#2907

[Rachael-Graham]

kagent#2909 merged on 2026-09-24 as ddb9e0ab. Retitled and reworded the blocker: the work targets 1.0.0-alpha4, because v1.0.0-alpha3 was cut at 375fe73a, which precedes the merge. Do not action until a release carries it.

The scope has grown. kagent#2929 landed as f1e46d1a and brings breaking changes the survey above does not cover.

Metrics are renamed. kagent_grpc_server_requests_total and kagent_grpc_server_request_duration_seconds are removed. Read rpc_server_call_duration_seconds_count and the rpc_server_call_duration_seconds histogram instead, with the labels rpc_method and rpc_response_status_code. Any page or example naming the old metrics takes the swap.

Span attributes are removed. gcp.vertex.agent.llm_request and gcp.vertex.agent.llm_response are gone from the Go ADK, along with the invocation span. observability/tracing.md devotes an admonition to those two attributes and tells a reader to set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to false to keep them out of traces. That admonition needs rewriting rather than a value swap.

Turning capture off is not complete protection, and this needs prose. adk-go writes gcp.vertex.agent.tool_call_args and gcp.vertex.agent.tool_response regardless of the capture setting. The fix is google/adk-go#1635, an open upstream dependency on the epic. Until kagent bumps adk-go, a reader who turns capture off still sends tool arguments and tool responses to their backend, and the workaround is dropping those attributes in a Collector.

Related. Tracking the initiative through the epic, kagent-dev/kagent#2907, per @krisztianfekete's note above. "Moving Python off OpenLLMetry" is still open there, which is why the Go and Python runtimes now emit different correlation attributes. That split is recorded on #519, which rewrites the same pages and should follow this issue rather than precede it.

[Rachael-Graham]

Two of the variables in the Environment variables removed list are already handled, so this issue no longer owns them: KAGENT_PRE_RESPONSE_TRACE_FLUSH and KAGENT_TRACE_FLUSH_TIMEOUT_MS.

They were caught by #549's KAGENT_ environment sweep, which validated every KAGENT_* and OTEL_* name on the 1.x pages against the generated docs/env.md at the v1.0.0-alpha7 tag. Neither appears there, and neither exists in the tree.

Fixed on observability/tracing.md in commit ee6e49e on the #549 re-baseline branch. The paragraph now reads:

To avoid losing them, the kagent, codex, and claude runtimes flush their span buffer after each A2A handler returns, before the response completes. The flush is unconditional and waits up to three seconds, and no setting changes either. An agent on the byo runtime flushes only if its own image does, so a conversation's last turn can lose its spans there.

Verified at alpha7: the flush is wired unconditionally in all three Go runtimes, and the timeout is a compile-time FlushTimeout = 3 * time.Second in go/pkg/telemetry/telemetry.go rather than a configurable value. That also corrected a second claim on the page, which said the claude runtime got no flush at all.

Nothing else in this issue is touched. The chart value renames, the per-signal endpoints, the three behavior changes that need prose, and the semconv metrics work are all still open and still scoped as written.