Epic: Enhanced distributed tracing

[psschwei]

Overview

This epic refreshes the tracing telemetry plan to bring it in line with the canonical plugin/hook pattern established by the metrics (#443) and logging (#442) epics, both closed 2026-04-27.

The original plan (2026-02-10) and its nine sub-issues (#469–#477) predate two major refactors that reshape tracing's design surface:

• PR refactor: migrate token metrics to hooks/plugins system #653 (2026-03-18) — migrated token metrics to the hooks/plugins system; established the canonical pattern for telemetry data collection.
• PR refactor!: partition ModelOutputThunk execution metadata into Generat… #908 (2026-04-24) — partitioned ModelOutputThunk execution metadata into GenerationMetadata.

The original sub-issues have been retired and replaced by the phased plan below. See the companion comment on this epic for a feature-level mapping of retired issues to their replacements.

Current state (2026-05-07)

• Tracing lives in mellea/telemetry/tracing.py and mellea/telemetry/backend_instrumentation.py.
• Uses direct inline instrumentation in all 5 backends (~20 call sites) — no plugin/hook usage.
• No tracing_plugins.py; tracing is the only telemetry pillar still on pre-plugin patterns.
• Stores the active backend span in mot._meta["_telemetry_span"] so it survives coroutine boundaries — collides with Epic: ModelOutputThunk structural cleanup #909 item 2.
• Reads raw provider responses for attribute extraction instead of mot.generation (the normalized GenerationMetadata).
• Uses deprecated gen_ai.system attribute (current GenAI semconv is gen_ai.provider.name, which metrics uses). @planetf1's PR feat(telemetry): close five OTel GenAI semantic convention emission gaps (#1035) #1036 is adding dual-emission.
• Env vars split and non-uniform: MELLEA_TRACE_APPLICATION, MELLEA_TRACE_BACKEND, MELLEA_TRACE_CONSOLE. No single umbrella flag. Only honors generic OTEL_EXPORTER_OTLP_ENDPOINT; no OTEL_EXPORTER_OTLP_TRACES_ENDPOINT support.
• Initialized at module import; tests require importlib.reload (vs logging's lazy init).
• Error-path span closure happens in core/base.py:520–541 before the generation_error hook fires — would block a tracing plugin hooking that event.
• Missing span coverage per current semconv: tool calls, streaming events (first-token / chunk / complete), chat content events, gen_ai.conversation.id, most gen_ai.request.* parameters, SpanKind.CLIENT on backend calls.

Scope

Phase 1 — Foundation (serial, blocking)

These must land before Phase 2/3. Likely one PR, separate tracking issues for clarity.

• refactor: move tracing onto plugin/hook pattern #1045 — Refactor tracing onto the plugin/hook pattern. Create mellea/telemetry/tracing_plugins.py, move instrumentation out of backends, read from mot.generation. Naturally resolves Epic: ModelOutputThunk structural cleanup #909 item 2 (removes mot._meta["_telemetry_span"]).
• refactor: rename tracing env vars to plural and align with OTel semconv #1046 — Rename env vars to plural MELLEA_TRACES_* (aligned with OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and MELLEA_METRICS_*), add umbrella enable flag, align attribute names (gen_ai.provider.name), honor OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, switch to lazy init.
• refactor: move tracing error-path closure into generation_error plugin hook #1047 — Move error-path span closure into a generation_error plugin hook so backends and core code no longer touch span state directly.

Phase 2 — Coverage (parallelizable once Phase 1 lands)

• feat: session and act spans via plugin hooks #1048 — Session / act spans via hooks (replaces Add detailed tracing to session operations beyond the current top-level context span #470 and the direct tracing imports in stdlib/session.py / stdlib/functional.py).
• feat(telemetry): migrate stream_with_chunking orchestration span to plugin hooks #1290 — Streaming orchestration span via hooks (added mid-epic; streaming.py was merged after feat: session and act spans via plugin hooks #1048 was filed and was scoped out of PR refactor(telemetry): migrate application spans to plugin hooks #1289).
• feat: tool call spans via tool_post_invoke hook #1049 — Tool call spans via the tool_post_invoke hook (replaces Instrument tool calling to track tool invocations, arguments, and results #474).
• feat: sampling, validator, and formatter spans via plugin hooks #1050 — Sampling / validator / formatter spans via respective hooks (absorbs Instrument formatter operations to provide visibility into template rendering and response parsing #471, Instrument the sampling loop to track each iteration attempt, providing visibility into retry behavior #472, Instrument requirement validators to track validation attempts and failures #473).
• feat: streaming span events #1051 — Streaming span events. Coordinates with @planetf1's streaming-events design (feat(stdlib): add standard streaming event types #902 / PR feat(stdlib): streaming event types, events() iterator, and OTEL bridge (#902) #958), the right home for this work.
• OTel omissions/gaps: chat content events, gen_ai.conversation.id, prompt templates, error status #1035 — OTel GenAI emission gaps. PR feat(telemetry): close five OTel GenAI semantic convention emission gaps (#1035) #1036 lands gaps 1, 2, 4 and auto-closes this issue on merge. The three deferred pieces are tracked as dedicated post-refactor: move tracing onto plugin/hook pattern #1045 issues below.
• feat(telemetry): emit llm.prompt_template.* from tracing plugin (post-#1045) #1067 — Emit llm.prompt_template.* from tracing plugin (gap 3, formatter render path).
• feat(telemetry): emit gen_ai.request.* from tracing plugin (post-#1045) #1068 — Emit gen_ai.request.* from tracing plugin (request params via GENERATION_PRE_CALL hook).
• feat(telemetry): emit gen_ai.input/output.messages content events from tracing plugin (post-#1045) #1073 — Emit gen_ai.input/output.messages content events from tracing plugin (gap 5, content capture).
• feat(telemetry): normalise cache and reasoning token fields in GenerationMetadata across providers #1074 — Normalise cache and reasoning token fields in GenerationMetadata across providers (OpenAI/LiteLLM nested-path extraction).
• feat: CLI root spans #1052 — CLI root spans (refresh of Instrument CLI entrypoints to create root spans for user-initiated operations #469 for the current CLI layout).

Phase 3 — Polish

• refactor: share error taxonomy between metrics and tracing plugins #1053 — Semantic error taxonomy. Reuse classify_error and ERROR_TYPE_* from mellea/telemetry/metrics_plugins.py instead of a parallel telemetry/errors.py (retires Create a standardized error classification system that maps exceptions to semantic error types for consistent error reporting in traces #475).
• test: backend coverage parity audit post-refactor #1054 — Backend coverage parity audit post-refactor (refresh of Verify and expand tracing instrumentation to cover all LLM backends consistently #477).
• feat(telemetry): adopt OTel GenAI semantic convention names for spans and metrics #1209 — Semconv alignment sweep (span names, span kinds, metric renames, additional dimensions).
• docs: add Telemtry example for getting full traces #945 — Docs example for getting full traces.

Acceptance Criteria

• mellea/telemetry/tracing_plugins.py exists and parallels metrics_plugins.py in shape.
• No backend file imports tracing helpers directly.
• mot._meta["_telemetry_span"] is removed.
• Env vars renamed to plural MELLEA_TRACES_* (aligned with OTel standard partner var and MELLEA_METRICS_*). Old MELLEA_TRACE_* names emit deprecation warnings for one release.
• All emitted gen_ai.* attributes match current OTel GenAI semconv (gen_ai.provider.name, not gen_ai.system).
• Span events emitted for streaming milestones (TTFB, first token, complete).
• Tool calls produce spans via tool_post_invoke.
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is respected when set.
• Docs example for full traces lands (docs: add Telemtry example for getting full traces #945).

Coordination

• OTel emission gaps — @planetf1's PR feat(telemetry): close five OTel GenAI semantic convention emission gaps (#1035) #1036 (closing OTel omissions/gaps: chat content events, gen_ai.conversation.id, prompt templates, error status #1035) is compatible to merge as-is. See the PR feat(telemetry): close five OTel GenAI semantic convention emission gaps (#1035) #1036 coordination comment for specifics on rebasing onto Phase 1.
• Streaming events — @planetf1 is also working on streaming event types in feat(stdlib): add standard streaming event types #902 (PRs feat(stdlib): streaming event types, events() iterator, and OTEL bridge (#902) #958 and feat(stdlib): add stream_with_chunking() with per-chunk validation (#901) #942). Phase 2's streaming span events (feat: streaming span events #1051) integrates with that design.
• Hook timing in streaming path — epic feat: streaming validation — per-chunk requirement checking with early exit #891 has an open concern about when GENERATION_POST_CALL fires in the streaming path; whatever resolves that concern will also determine how cleanly a tracing plugin on the same hook can observe streaming generations.
• MOT cleanup — Epic: ModelOutputThunk structural cleanup #909 is on hold pending feat: streaming validation — per-chunk requirement checking with early exit #891 sub-issues (feat(core): add PartialValidationResult with tri-state semantics #898–feat(stdlib): implement stream_with_chunking() with per-chunk validation #901). Phase 1 here closes out Epic: ModelOutputThunk structural cleanup #909 item 2 as a side effect.

Related / Retired

• Retired sub-issues — Instrument CLI entrypoints to create root spans for user-initiated operations #469–Verify and expand tracing instrumentation to cover all LLM backends consistently #477 (closed in favor of the phased plan above; see the mapping comment for feature-level replacement pointers).
• Still valid, kept as-is — docs example (docs: add Telemtry example for getting full traces #945). Emission gaps (OTel omissions/gaps: chat content events, gen_ai.conversation.id, prompt templates, error status #1035) auto-close when PR feat(telemetry): close five OTel GenAI semantic convention emission gaps (#1035) #1036 merges; deferred work tracked as feat(telemetry): emit llm.prompt_template.* from tracing plugin (post-#1045) #1067, feat(telemetry): emit gen_ai.request.* from tracing plugin (post-#1045) #1068, feat(telemetry): emit gen_ai.input/output.messages content events from tracing plugin (post-#1045) #1073.
• Separate non-blocking fix — fix: reconcile pillar env var naming — rename MELLEA_LOG_* to MELLEA_LOGS_* #1055 reconciles pillar env var naming across metrics / logging / tracing.
• Adjacent work — feat: streaming validation — per-chunk requirement checking with early exit #891 streaming epic; Epic: ModelOutputThunk structural cleanup #909 MOT cleanup.