Skip to content

Add TraceRoot Codex observability plugin (v0.1.0)#1

Merged
XinweiHe merged 38 commits into
mainfrom
feat/codex-tracing-plugin
Jun 30, 2026
Merged

Add TraceRoot Codex observability plugin (v0.1.0)#1
XinweiHe merged 38 commits into
mainfrom
feat/codex-tracing-plugin

Conversation

@XinweiHe

Copy link
Copy Markdown
Contributor

Adds the TraceRoot Codex observability plugin (v0.1.0) — a TypeScript plugin that traces OpenAI Codex sessions to TraceRoot.

What it does

Every Codex query becomes one live trace — the agent turn, model calls, tool executions, file edits, and any subagents it spawns — streamed as the agent works and linked to your git repo.

  • One trace per query. A query and everything it does is a single trace; multiple queries in a session are linked by session id (Sessions view).
  • Live. Spans appear and the trace grows as the agent runs (named Codex Turn from the first span).
  • Subagents nested. spawn_agent workers are nested under the spawn tool that created them, in the same trace.
  • No daemon. Stateless one-shot hooks; fail-open (tracing never blocks Codex).

How it works

Stateless Codex hooks (PostToolUse/Stop/SubagentStop) read the rollout transcript and emit OpenTelemetry (protobuf) spans. The key idea is deterministic span ids (hash of session/turn/call ids) so a child span can reference its parent before the parent exists and re-emission is idempotent — a sidecar ledger dedups what's been sent. No background process.

What's captured

  • Agent turn (prompt + final answer, git repo/ref, model)
  • Model steps (input, output incl. reasoning + tool calls, tokens, cache-aware cost)
  • Tools — exec_command, apply_patch (file edits), MCP, search — with args, output, and exit code/status/kind where Codex reports it
  • Subagents, fully nested in the same trace
  • SDK identity (traceroot.sdk.*)

Testing & validation

  • 62 unit tests (vitest), tsc --noEmit clean, committed self-contained dist/index.mjs with a build drift-guard (lint:dist)
  • End-to-end validated against production — simple single-turn and complex multi-agent sessions; confirmed live streaming, correct subagent nesting, non-zero cache-aware cost, and git correlation

Install

codex plugin marketplace add traceroot-ai/traceroot-codex-plugin

Enable in ~/.codex/config.toml and add ~/.codex/traceroot.json with your API key — see the README.

🤖 Generated with Claude Code

XinweiHe and others added 30 commits June 29, 2026 15:46
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…-flush

- README + plugin.json: remove current "subagents" tracing claim (not
  implemented in v0.1.0); add honest note that subagent-subtree tracing
  is planned. SubagentStop hook registration unchanged.
- src/emit.ts: collect emittedIds during the loop; call markSpanEmitted
  for each id AFTER tracing.shutdown() completes so a span is persisted
  to the sidecar only after delivery is confirmed. If shutdown throws,
  marking is skipped and spans will retry on the next hook invocation.
- src/config.ts: guard maxChars against NaN (non-numeric env/JSON value)
  by falling back to 20000 via Number.isFinite check.

All 37 tests pass. Typecheck clean. dist/index.mjs rebuilt.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move src/, test/, hooks/, .codex-plugin/, tsdown.config.ts, and dist/
into plugins/tracing/ to match the marketplace-monorepo structure.
Add .agents/plugins/marketplace.json, pnpm-workspace.yaml, LICENSE
(Apache-2.0), and a minimal .github/workflows/ci.yml. Update root
package.json build/lint:dist scripts, tsconfig.json include paths, and
vitest.config.ts to reflect the new layout. Rebuild bundle at its new
path (plugins/tracing/dist/index.mjs).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…d events

Captures exec kind, completion status, exit code, and error detail from
Codex exec_command_end (and other *_end) event_msg events, surfacing them
as structured fields on ToolCall and in traceroot.span.metadata on TOOL spans.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- planTurnSpans: add optional PlanOpts (seedPrefix/traceId/rootParentSpanId); seed="" default keeps all existing span ids byte-identical (39 existing tests unchanged).
- types: add SubagentRef + Turn.subagents optional field; subagentThreadIds kept for backward compat.
- transcript: capture {threadId, spawnCallId} from collab_agent_spawn_end into turn.subagents.
- subagents: new findSubagentRollout scans sessions/YYYY/MM/DD/ for rollout-*-<threadId>.jsonl.
- emit: recursive emitTurnTree with visited-set (cycle/diamond safe) + STACK_GUARD=32 paranoia; child spans share parent traceId, nested under spawn TOOL span via seedPrefix.
- tests: 18 new tests (planTurnSpans opts, findSubagentRollout, 5 subagent emit assertions incl. idempotency + cycle guard). All 57 tests (39 + 18) pass, typecheck clean, dist rebuilt.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- emit: wrap per-subagent resolve/read/parse/recurse in try/catch so a throwing
  findSubagent/readRollout/parseSession never propagates out of dispatch (fail-open;
  parent spans still flush + get marked). visited.add() stays BEFORE the try so a
  failing/cyclic thread is not retried. Document Deps.findSubagent as should-be-fail-soft
  but not assumed.
- types: make Turn.subagentThreadIds optional (read nowhere); transcript no longer writes it.
- spans: comment that child subagent spans carry the CHILD session id and the backend
  groups the trace off the ROOT span; note future overrideSessionId PlanOpts as the extension.
- test: add fail-soft test (throwing findSubagent → dispatch resolves, parent spans emit).

58 tests pass (57 + 1), typecheck clean, dist rebuilt.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use @opentelemetry/exporter-trace-otlp-proto (was -http/JSON). The Traceroot
  ingest endpoint accepts OTLP protobuf only; JSON was silently 415'd → 0 traces.
  Aligns with the traceroot-ts SDK.
- Parse custom_tool_call/custom_tool_call_output so apply_patch (file edits) and
  other custom tools are captured as TOOL spans (Codex encodes apply_patch as a
  custom_tool_call, not function_call).
- Resolve CODEX_HOME at call time in getConfig (was module-load) so it reflects
  the env and is test-isolatable; config test now isolates via a temp CODEX_HOME.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
e2e against prod surfaced two issues with real multi-agent Codex sessions:

- Subagent detection: real Codex (multi_agent v1) spawns via a `spawn_agent`
  function_call whose function_call_output carries {"agent_id": "<thread>"},
  not the collab_agent_spawn_end event our fixtures assumed. Detect spawn_agent
  and link the child thread (+ spawn call_id) so subagent spans nest under the
  spawn_agent TOOL span, in the same trace.
- Export: BatchSpanProcessor instead of SimpleSpanProcessor. Simple sends one
  POST per span; a burst (a turn's many tools, or a subagent subtree flushed at
  once) lost the tail before the short-lived process exited — confirmed in prod
  (only 1 of 3 subagents landed). Batch coalesces into few large POSTs that
  forceFlush ships reliably; verified 54/54 spans + 3/3 subagents land.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codex fires hooks for subagent sessions too, so each subagent was emitted both
as a standalone top-level trace AND nested under its parent's trace. Detect a
subagent session (session_meta.thread_source === "subagent" / parent_thread_id)
and skip standalone emission — the parent owns the nesting. Keeps one trace per
top-level query with subagents nested inside.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously LLM (model-step) spans had no input and output only when the step
emitted assistant text — so every tool-calling step looked empty in the UI.
Now each LLM span carries:
- input: the user prompt (first step) or the prior step's tool results
- output: {content, reasoning, tool_calls:[{name,args}]} — never empty
- metadata: codex.step_index
and the root AGENT span carries codex.turn_id/thread_id/model/model_provider/
cli_version/tool_call_count. Matches the richness of the reference plugins.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The UI's SDK field reads traceroot.sdk.name/version span attributes (as the
Claude Code plugin sets). We only set the OTLP scope + request headers, so the
field was empty for our traces. Set them on every span via commonTrace.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The root AGENT span ("Codex Turn") was emitted only at task_complete, so during
live tracing the trace had no root yet and the platform named it after the first
span it saw (an LLM step, "gpt-5.5"), then flipped to "Codex Turn" at the end.
Now the trace root is emittable immediately and re-emitted on every hook (the
backend keeps the latest version of a span id), so the trace is named "Codex
Turn" throughout and its end/output refine as the turn progresses.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
XinweiHe and others added 7 commits June 30, 2026 14:08
- emit.ts: pass only the findSubagent function through the recursion instead of
  the whole resolvedDeps struct (emitTurnTree never used buildTracing/getGit);
  removes the resolvedDeps assembly. Public Deps type unchanged (tests inject all).
- types.ts: delete the dead `subagentThreadIds` field (unreleased, read nowhere)
  and the three test assignments.
- sdk.ts: single-source SDK_NAME/SDK_VERSION (were duplicated in exporter.ts and
  spans.ts).
- Verified finding 4: the SubagentStop hook is NOT a no-op — its transcript_path
  is the PARENT session (child under agent_transcript_path), so it re-walks the
  parent to nest a just-finished subagent live. Kept + documented.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…vergence

- parseSession -> parseRollout (pairs with readRollout; it parses rollout lines,
  not a "session")
- findSubagentRollout -> locateSubagentRollout

These two were the only verbatim distinctive names shared with the reference
plugin; the rest of the overlap is Codex's own wire-format vocabulary
(RolloutLine, EventMsgPayload, ToolCall, …) or idiomatic util names, which are
intentionally left as-is.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Subagents are now traced (nested under their spawn tool, same trace) — drop the
"planned for a future release" note. Document live streaming + the "Codex Turn"
naming, richer tool capture (exec/apply_patch/exit codes/status), the
deterministic-id design, and the per-hook roles. Recommend the ~/.codex/
traceroot.json config file as the primary setup since a Codex hook subprocess
does not reliably inherit the shell environment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the full settings table into a collapsible <details>, drop the
implementation-internals 'How it works' section (deterministic ids, sidecar,
per-hook roles) — users need install + config, not the engineering detail.
…s-is

Use the TraceRoot brand capitalization in all prose/display strings (README,
manifest descriptions, author/displayName, LICENSE copyright, hook status
messages, code comments). Package name, env vars (TRACEROOT_*), config files
(traceroot.json), span attributes (traceroot.*), and the domain stay lowercase.
- ci.yml: drop the pinned pnpm 'version: 10' so action-setup reads the version
  from package.json's packageManager (was erroring on Multiple versions of pnpm).
- .gitattributes: mark plugins/tracing/dist/index.mjs linguist-generated + -diff
  so the committed bundle (required for zero-build install) is collapsed in PR
  diffs and excluded from language stats.
- .gitignore: add .env*, coverage/, editor dirs; note that dist/ stays committed.
Mark pnpm-lock.yaml and LICENSE as generated so they fold in the PR diff
and drop out of language stats, leaving the reviewable diff as real code.
@XinweiHe

Copy link
Copy Markdown
Contributor Author

@cubic review it

@cubic-dev-ai

cubic-dev-ai Bot commented Jun 30, 2026

Copy link
Copy Markdown

@cubic review it

@XinweiHe I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 51 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="plugins/tracing/src/emit.ts">

<violation number="1" location="plugins/tracing/src/emit.ts:42">
P2: Nested subagent root spans are suppressed while the subagent is running, so live child tool/LLM spans can arrive without their parent AGENT span. This breaks the advertised live nested subagent view until the child turn completes.</violation>
</file>

You're on the cubic free plan with 11 free PR reviews remaining this month. Upgrade for unlimited reviews.

Re-trigger cubic

Comment thread plugins/tracing/src/config.ts
Comment thread plugins/tracing/src/index.ts
let n = 0;

for (const span of planTurnSpans(sessionMeta, turn, ctx, opts)) {
if (!span.complete) continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Nested subagent root spans are suppressed while the subagent is running, so live child tool/LLM spans can arrive without their parent AGENT span. This breaks the advertised live nested subagent view until the child turn completes.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At plugins/tracing/src/emit.ts, line 42:

<comment>Nested subagent root spans are suppressed while the subagent is running, so live child tool/LLM spans can arrive without their parent AGENT span. This breaks the advertised live nested subagent view until the child turn completes.</comment>

<file context>
@@ -0,0 +1,154 @@
+  let n = 0;
+
+  for (const span of planTurnSpans(sessionMeta, turn, ctx, opts)) {
+    if (!span.complete) continue;
+    // The trace root (AGENT, no parent) is re-emitted on every hook so the trace
+    // is named "Codex Turn" from the first span (not the first LLM step's model
</file context>

Comment thread plugins/tracing/src/sidecar.ts Outdated
…ved-config exit code

- config: host_url/api_key now come only from env or global ~/.codex/traceroot.json,
  never project-local .codex/traceroot.json — a checked-in project config could
  otherwise redirect authenticated uploads (env/global key) to an attacker endpoint.
- sidecar: loadEmittedSpanIds fails open on any read error (not just ENOENT) — an
  unreadable dedup file degrades to 'nothing emitted yet' rather than aborting the
  trace; duplicate emits are idempotent (deterministic ids + backend upsert).
- index: runHook sets process.exitCode from the resolved config so fail_on_error set
  via JSON (not only the env var) is honored; module handler is now a pre-config
  backstop. Rebuilt bundle. 66 tests.
@XinweiHe
XinweiHe merged commit 41c8fc7 into main Jun 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant