Skip to content

feat(orchestration-v2): add native subagent and workflow observability - #4664

Open
shivamhwp wants to merge 61 commits into
t3code/codex-turn-mappingfrom
subagent-obs/05-thread-visibility
Open

feat(orchestration-v2): add native subagent and workflow observability#4664
shivamhwp wants to merge 61 commits into
t3code/codex-turn-mappingfrom
subagent-obs/05-thread-visibility

Conversation

@shivamhwp

@shivamhwp shivamhwp commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

This adds native subagent and workflow observability to Orchestrator V2. It is based directly on the V2 implementation in PR #2829; no V1 runtime or compatibility data source remains in this PR.

Why this PR exists

Orchestrator V2 has its own event model, projection, provider boundary, recovery path, and thread lifecycle. When V2 ships, provider-native child work needs V2-owned identities and activations so the Agents experience stays accurate through reuse, interruption, restart, nesting, and background completion.

Before and after

The comparison below is against the current t3code/codex-turn-mapping V2 base, not main/V1.

V2 base — before This PR — after
Every native child renders as a separate timeline row. Internal child threads also appear in Lineage. One compact CTA represents the spawn batch; live state, settled count, and token totals update in place. Internal provider child threads stay out of the sidebar and Lineage.
V2 base showing separate live subagent rows PR showing a compact live Agents CTA
Completed children remain separate cards in the conversation. The CTA opens one detailed Agents panel with stable rows for role, duration, progress/result, model, tokens, tools, status, and aggregate totals.
V2 base showing separate completed subagent rows PR showing the expanded Agents panel

Responsive web verification at 390×844:

Responsive Agents panel at 390 by 844

What changes

V2 domain and persistence. Reusable subagent identities and immutable per-activation records live in the V2 projection. Migration 050 persists activations, and schema defaults keep older V2 payloads readable without a backfill.

Provider-native projection. Claude, Codex, Cursor, OpenCode, and ACP/Grok emit V2 subagent and activation lifecycle updates, including role, usage, recent activity, workflow membership, activation counts, idle state, and provider-reported nesting.

Reuse and recovery. Reused agents bind to the run that reactivates them. Activation proof prevents stale post-interrupt traffic from attaching to a later run, and recovery retains the cumulative usage baseline.

V2-only client state. The Agents panel derives directly from OrchestrationV2Subagent[]. The V1 activity fold, dual-source fallback, and obsolete tests are removed. V2 timeline items collapse into one CTA per workflow or direct-spawn batch.

Internal thread visibility. One shared V2 predicate controls filtering on server, web, and mobile. Hidden child deep links resolve through the existing targeted projection RPC and redirect to their parent without adding child shells to the global stream. Unfiltered thread refs remain available for cleanup so hidden worktrees are not mistaken for orphans.

Explicitly not included

  • No V1 orchestration runtime, V1 activity fold, or compatibility data source.
  • No second Agents panel or alternate roster.
  • No new targeted subagent-stop command or backend-only provider-control path.
  • No replacement for background liveness or workflow script inspection already present in the V2 base.

Current stack

  • Base: 2d623ac6b41ad1ed25c0473474699866083e0b65 (t3code/codex-turn-mapping)
  • Head: b2bae43908a497a7ef2989bf665555be40d2a716
  • 60 commits, 89 files, +7,407 / -2,436
  • Rebased onto the current V2 base; no merge conflicts remain locally

Verification

  • 171/171 changed UI and client-runtime tests pass
  • 179/179 focused Orchestrator V2 and provider-adapter tests pass
  • 205/205 conflict-focused adapter and timeline tests pass after rebasing
  • Targeted formatting, lint, diff whitespace, web typecheck, client-runtime typecheck, and contracts typecheck pass
  • Chrome: live and completed native-agent batches, Agents-panel open/close, keyboard focus ring and pointer affordance, responsive 390×844 layout, no horizontal overflow, no top-level child-thread leakage, and hidden-child deep-link redirect
  • Clean reload and redirect flows produce no browser console errors
  • Server typecheck still hits the same unrelated ProviderSessionManager.test.ts:836 error on the untouched V2 base

Provider limitation: if a provider reports a nested agent only inside its parent's final text and emits no lifecycle frame, T3 shows that summary but does not invent a nested agent row.

Built with Codex (GPT-5.6 Sol) through T3 Code.

Note

Add subagent and workflow observability to OrchestrationV2 with activation tracking and timeline CTA rows

  • Introduces subagent_activation.updated domain and provider events, persisted in a new orchestration_v2_projection_subagent_activations table (migration 050) and returned as part of thread projections.
  • Extends the OrchestrationV2Subagent schema with kind, role, usage, currentActivationId, activationCount, recentActivity, and workflow/workflowMembership fields; older rows decode with populated defaults.
  • All provider adapters (Claude, Codex, Cursor, OpenCode, ACP) now emit activation events, accumulate usage across activations, preserve first completion timestamps, and expose an idle post-completion status.
  • The WebSocket event stream for a thread now tracks the full subagent tree (including nested children) by walking childThreadId links and merging descendant projections via mergeSubagentTreeProjection/routeSubagentTreeEvent.
  • Internal subagent threads (lineage subagent or forked from a node) are now filtered out of shell snapshots, sidebar, thread lists, and hooks via isOrchestrationV2InternalSubagentThread; deep-links redirect to the parent thread on web and mobile.
  • The messages timeline collapses per-subagent items into a single CTA row per workflow or per-run batch via collapseSubagentTimelineEntries, showing aggregated counts, token totals, and an open-agents call-to-action.
  • Adds a listAllThreadRefs WebSocket RPC that returns all thread references including hidden subagents, used for accurate worktree cleanup.
  • Risk: projection schema version bumps from 2 to 3; any cached projections at version 2 will be treated as stale and rebuilt.

Macroscope summarized 513b603.


Note

High Risk
Large orchestration and multi-adapter changes affect run attribution, settlement, and thread visibility; regressions could mis-route events, leak hidden threads, or mishandle resumed subagents after restart.

Overview
Adds per-activation subagent lifecycle across V2 provider adapters (Claude, Codex, Cursor, ACP, etc.): subagents gain kind/role/usage/workflow fields, emit subagent_activation.updated, and use shared observability helpers. Adapters handle reuse after idle or restart by seeding existingSubagents, rebinding runs, guarding registry races (Claude), mapping Codex collab states to idle for reusable agents, and folding token usage into activations.

Mobile hides internal subagent threads via isOrchestrationV2InternalSubagentThread (list + deep-link redirect to parent), delays “thread unavailable” to avoid flashing over redirects, drops stale subagent edges from the relationships banner, and tracks subagentActivations in the environment cache.

Server adds listAllThreadRefs auth scope and a Codex replay scenario for subagent reuse after idle. Nested subagent lifecycle can be routed to the root thread projection (SubagentTreeProjection) so the parent fleet stays complete without copying child transcripts.

Reviewed by Cursor Bugbot for commit 513b603. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: da663ebe-aab3-4311-a64f-f8287cdf1b2b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch subagent-obs/05-thread-visibility

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Jul 27, 2026
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from 940e8fb to c3a50f8 Compare July 27, 2026 20:20
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from d2ef6c7 to adb1e9b Compare July 27, 2026 20:20
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from c3a50f8 to 8c6a796 Compare July 27, 2026 20:52
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from adb1e9b to 39912b0 Compare July 27, 2026 20:52
Comment thread apps/web/src/components/chat/ThreadRelationshipsControl.tsx
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from 8c6a796 to 103b937 Compare July 27, 2026 23:03
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 39912b0 to 6bc3a8d Compare July 27, 2026 23:03
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from 103b937 to 31adb36 Compare July 28, 2026 15:31
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 6bc3a8d to 05fe44c Compare July 28, 2026 15:31
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from 31adb36 to b270c47 Compare July 28, 2026 19:27
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 05fe44c to 7458298 Compare July 28, 2026 19:27
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from b270c47 to daf1a88 Compare July 28, 2026 19:46
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 7458298 to 4daeac4 Compare July 28, 2026 19:46
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from daf1a88 to 1dce464 Compare July 28, 2026 20:26
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 4daeac4 to 095cab7 Compare July 28, 2026 20:26
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from 1dce464 to 4c169ed Compare July 29, 2026 01:19
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 095cab7 to 1d37a6f Compare July 29, 2026 01:19
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from 4c169ed to 70d5111 Compare July 29, 2026 01:24
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 1d37a6f to 6cf67a2 Compare July 29, 2026 01:25
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from 70d5111 to c78d0ff Compare July 30, 2026 00:00
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 6cf67a2 to 5e3c535 Compare July 30, 2026 00:00
@shivamhwp
shivamhwp force-pushed the subagent-obs/04-agents-panel branch from c78d0ff to a997806 Compare July 30, 2026 00:45
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 5e3c535 to 2430736 Compare July 30, 2026 00:45
@shivamhwp
shivamhwp marked this pull request as ready for review July 30, 2026 20:04
Comment thread apps/web/src/state/entities.ts
shivamhwp and others added 24 commits August 18, 2026 01:17
…n the turn-start test

The rebase reconstruction of this file left each service module imported
twice — once as a namespace and once for its tag. At a service boundary
the namespace shape is the convention, so the duplicate named imports go
and their uses read through the module.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@shivamhwp
shivamhwp force-pushed the subagent-obs/05-thread-visibility branch from 0d57bbd to eb31f1a Compare August 17, 2026 19:49
providerDriver = support.subagent?.driver ?? item.driver;
providerInstanceId = support.subagent?.providerInstanceId ?? item.providerInstanceId;
model = support.subagent?.model ?? model;
} else if (item.type === "fork") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium threads/thread-work-log.tsx:139

Subagent activity rows render an enabled-looking but permanently disabled Open CTA, so users cannot navigate to the child thread. Because this branch was removed while threadActivity.ts still exposes subagent items to ThreadActivityThreadRow, targetThreadId stays null; restore the item.type === "subagent" handling using support.subagent?.childThreadId ?? item.childThreadId and its provider metadata.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/mobile/src/features/threads/thread-work-log.tsx around line 139:

Subagent activity rows render an enabled-looking but permanently disabled `Open` CTA, so users cannot navigate to the child thread. Because this branch was removed while `threadActivity.ts` still exposes subagent items to `ThreadActivityThreadRow`, `targetThreadId` stays `null`; restore the `item.type === "subagent"` handling using `support.subagent?.childThreadId ?? item.childThreadId` and its provider metadata.

? {}
: {
kind: "workflow",
roleFallback: "workflow-coordinator",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium Adapters/ClaudeAdapterV2.ts:4481

Workflow coordinators that were first seen as ordinary subagents remain labeled with their generic role in the Agents UI when subagent_type is absent. updateClaudeSubagentNode only rewrites task.role when input.agentType is defined, so this roleFallback is ignored during the later workflow upgrade; apply the fallback when updating an existing task as well.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts around line 4481:

Workflow coordinators that were first seen as ordinary subagents remain labeled with their generic role in the Agents UI when `subagent_type` is absent. `updateClaudeSubagentNode` only rewrites `task.role` when `input.agentType` is defined, so this `roleFallback` is ignored during the later workflow upgrade; apply the fallback when updating an existing task as well.

Effect.gen(function* () {
const existing = context.input.existingSubagents ?? [];
if (existing.length === 0) return;
yield* Ref.update(subagentThreads, (current) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium Adapters/CodexAdapterV2.ts:1621

Recovered child turn/started events are dropped when they arrive before seedExistingSubagents, so the child activation is never created or projected and reuse attribution is lost after startup. Seeding populates subagentThreads but does not drain pendingSubagentTurns; subsequent registration takes the existing !== undefined early return instead of replaying the queued turn. Drain and replay pending turns for recovered threads during seeding (or in the existing-thread path).

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts around line 1621:

Recovered child `turn/started` events are dropped when they arrive before `seedExistingSubagents`, so the child activation is never created or projected and reuse attribution is lost after startup. Seeding populates `subagentThreads` but does not drain `pendingSubagentTurns`; subsequent registration takes the `existing !== undefined` early return instead of replaying the queued turn. Drain and replay pending turns for recovered threads during seeding (or in the existing-thread path).

if (subagent === undefined || payload.tokenUsage.total.totalTokens <= 0) {
return;
}
if (subagent.task.currentActivationId !== null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium Adapters/CodexAdapterV2.ts:3568

A trailing thread/tokenUsage/updated frame updates subagentActivationsByNativeTurnId but leaves subagent.task.usage and aggregate agent totals stale, so the final token usage is omitted. The task update is gated by currentActivationId before the handler merges the trailing frame; emit the task update after merging the activation, regardless of whether the activation is still current.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts around line 3568:

A trailing `thread/tokenUsage/updated` frame updates `subagentActivationsByNativeTurnId` but leaves `subagent.task.usage` and aggregate agent totals stale, so the final token usage is omitted. The task update is gated by `currentActivationId` before the handler merges the trailing frame; emit the task update after merging the activation, regardless of whether the activation is still current.

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two small consistency findings in the changed web UI; everything else (the new Agents CTA row's cursor/focus contract, the lineage panel's disconnect fallback, the panel model rename) looks consistent with the surrounding component system.

Posted via Macroscope — UI Consistency


<ThreadLineageRowList hiddenCount={hiddenCount} onShowMore={showMore}>
{visibleRows.length === 0 ? (
<div className="flex h-9 items-center gap-2.5 rounded-lg px-2.5 text-[13px] text-muted-foreground">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new static row hand-rolls the thread-details panel's row geometry instead of reusing the named treatment the panel already exports (THREAD_DETAILS_PANEL_LOCKED_ROW_CLASS), and it drifts from it: no border border-transparent and the icon misses -mx-0.5, so the icon sits ~1px and the label ~3px right of every other row in the panel (the lineage link rows and the workspace/environment locked rows all include both). Consider reusing the shared row + icon classes and keeping only the muted informational color at the call site:

<div className={cn(THREAD_DETAILS_PANEL_LOCKED_ROW_CLASS, "flex items-center font-normal text-muted-foreground")}>
  <BotIcon aria-hidden className={THREAD_DETAILS_PANEL_ICON_CLASS} />
  <span>Agent session connected</span>
</div>

(THREAD_DETAILS_PANEL_LOCKED_ROW_CLASS and THREAD_DETAILS_PANEL_ICON_CLASS need adding to the existing ./threadDetailsPanelStyles import.)

Posted via Macroscope — UI Consistency

detail={detail}
badge={item.status}
threadId={item.childThreadId}
threadId={null}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

subagent is the only item type that passes expandedDetail, so with threadId={null} the disclosure variant of RelatedThreadCard can never render its open-thread button — but its summary still carries pr-11, which was there purely to reserve space for that absolutely positioned button. The result is ~44px of dead space on every expandable subagent row, with the chevron floating well short of the right edge and out of line with the collapsed variant's trailing badge. Consider making the reserve conditional on the action existing:

           <summary
             aria-label={`Show full result for ${props.title}`}
-            className="flex min-w-0 cursor-pointer list-none items-center gap-2 px-3 py-2 pr-11 text-left transition-colors hover:bg-muted/50 [&::-webkit-details-marker]:hidden"
+            className={`flex min-w-0 cursor-pointer list-none items-center gap-2 px-3 py-2 ${threadId === null ? "" : "pr-11"} text-left transition-colors hover:bg-muted/50 [&::-webkit-details-marker]:hidden`}
           >

Posted via Macroscope — UI Consistency

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant