Persist cost and token telemetry when Review agents are cancelled by GitHub Actions

Problem

PR #6908 improved Claude stream parsing for runs that reach an EOF without a terminal ResultEvent, but a real GitHub Actions cancellation still loses the platform-level run metrics.

This means a cancelled Review agent can consume Claude API credits while reporting no usable metrics.json, no agent span, and no dollar cost.

Evidence

PR fix(#6905): capture token telemetry on cancelled Claude runs #6908: fix(#6905): capture token telemetry on cancelled Claude runs #6908
Post-merge run: https://github.com/fullsend-ai/fullsend/actions/runs/33687475120
Cancelled Review job: https://github.com/fullsend-ai/fullsend/actions/runs/33687475120/job/100438321995
Uploaded artifact: https://github.com/fullsend-ai/fullsend/actions/runs/33687475120/artifacts/9868972772
The job checked out merge commit 8a593ad7b64875c0c4016f8f3bd151cc6c83df8a.
The Review agent ran for approximately 8m50s and was cancelled while the agent process was active.
The post-cancellation artifact upload succeeded.

The artifact contained:

Raw Claude output.jsonl usage records with no terminal result event.
10 distinct usage-bearing messages, deduplicated by message/request ID:
- input: 599
- output: 119
- cache creation: 148,943
- cache read: 583,298
run-telemetry.jsonl containing only the sandbox span; no agent span.
No metrics.json.
No total_cost_usd.

Expected behavior

When a Review agent is cancelled after making API calls:

Best-effort token usage is persisted to metrics.json.
The agent telemetry span is finalized with the cancellation outcome and token counters.
The status comment and artifact expose the partial usage.
If dollar cost cannot be obtained from the Claude stream, that limitation is explicit, but token usage must not be silently lost.

Likely failure mode

The current PR handles an in-process EOF/no-ResultEvent path through a deferred TokensEvent. GitHub Actions cancellation terminates the fullsend process before the normal metrics/agent-span finalization path completes, so the deferred parser callback cannot make the data durable.

Please verify the signal/cancellation lifecycle across the Review workflow, the fullsend action, ClaudeRuntime.Run, and the run finalization/telemetry writer.

Acceptance criteria

Add a regression test for cancellation of the full Review action/process after non-zero Claude usage has been observed.
Ensure partial RunMetrics are written on graceful cancellation and cancellation-related process termination wherever the runner can still execute cleanup.
Ensure agent telemetry is ended with a cancellation/error status and partial token attributes.
Preserve the existing successful-run behavior where terminal ResultEvent values remain authoritative.
Document the remaining dollar-cost limitation or add a separate cost-reconciliation mechanism.
Validate with a real cancelled Review run and confirm the artifact contains the expected partial metrics.

🤖 Finished Triage · ✅ Success · Started 10:17 PM UTC · Completed 10:22 PM UTC

Commit: 318e04c · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.77

Triage summary

This is the remaining gap after PR #6908 landed: the in-process EOF path now captures tokens via a deferred TokensEvent, but an external kill (GitHub Actions SIGTERM) terminates the process before that deferred callback can flush data.

Related issues:

Review agent metrics zeroed when agent process exits with error after writing valid output #6806 — metrics zeroed on error exit (similar symptom, different trigger: agent process exits with error vs. process killed externally)
Claude runtime parser does not populate ReasoningTokens from Anthropic stream events #6904 — ReasoningTokens not populated (orthogonal parser gap)
bug: Claude runtime loses all cost/token telemetry on cancelled runs #6905 (closed) — original bug that PR fix(#6905): capture token telemetry on cancelled Claude runs #6908 addressed; this issue covers the remaining cancellation-specific path

Key implementation consideration: The signal handler must coordinate with the existing ClaudeRuntime.Run context cancellation — the Go context should propagate the cancellation while the handler runs cleanup before os.Exit. The cleanup window is bounded by GitHub Actions' grace period after SIGTERM (typically ~7.5s before SIGKILL), so the flush path must be fast and not block on network I/O.

Proposed test case:

// TestCancellationPersistsPartialMetrics
// Start a Claude runtime with a mock stream emitting usage-bearing
// messages but no terminal ResultEvent. After N usage events,
// cancel the context (simulating SIGTERM).
//
// Assert:
// - metrics.json exists with non-zero InputTokens/OutputTokens
// - agent telemetry span has status=cancelled with token attributes
// - Subsequent successful run with ResultEvent still overwrites
//   partial values (existing behavior preserved)
// - Cancellation before any usage: metrics.json has zeros +
//   explicit cancellation marker
// - Metrics write is atomic (temp + rename) to avoid partial JSON

Labels: Confirmed defect in Review agent runner's signal/cancellation lifecycle affecting cost telemetry persistence; high priority due to silent API credit loss on every cancelled run.

Next steps:

/fs-code — agent creates a PR to implement this issue
/fs-code <your instruction> — agent implements with your specific guidance

🤖 Finished Code · ✅ Success · Started 10:23 PM UTC · Completed 10:47 PM UTC

Commit: 318e04c · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $7.78

test(#6938): increase code coverage for signal handling and cancellat…

test(#6938): cover the cancellation short-circuit and tighten the cle…

fix(fullsend-ai#6936): persist token telemetry when agents are cancelled

Merge pull request fullsend-ai#6938 from fullsend-ai/agent/6936-persi…

fix: three-layer signal fix for cancelled run telemetry (#6936)

fix: SIGINT-into-sandbox for cancelled run cost data (#6936)

fix: stop SIGINTing openshell relay in Cancel (#6936)

refactor: remove rejected rate-cache approach, keep only SIGINT-into-…

fix: address review-squad findings — docstring, diagnostics, docs (#6936

Resolution: Token counts are the right abstraction for cancelled-run telemetry

What's been fixed

PR #6938 (merged) captures accurate token counts on cancelled runs via the deferred TokensEvent mechanism. This works without any signal propagation — it captures cumulative counts from stream events received before interruption. Cancelled runs now report non-zero token usage (input, output, cache_read, cache_creation) in metrics.json, the agent span, and the status comment.

Why `total_cost_usd` remains zero on cancelled runs

We built and verified a mechanism to send SIGINT into the sandbox so Claude Code would flush its result event containing total_cost_usd (PR #7208). It worked (total_cost_usd: 0.7879, was 0), but we're dropping that approach for two reasons:

1. It bypasses OpenShell's supported stop lifecycle.

sandbox exec is fire-and-forget for termination — no PID, no exec ID, no signal or kill RPC (NVIDIA/OpenShell#3159, closed as "not planned"). The supported alternative (sandbox stop) doesn't help either:

Stop sends SIGKILL, not SIGINT — Claude Code never gets a chance to flush its result event
Stop takes 45s on v0.0.116 (the stable release fullsend pins) due to the CAP_KILL bug (NVIDIA/OpenShell#2855, fixed in #3036 but only in prerelease)
To capture USD via stop, we'd need: stop (45s) → start → exec to extract cost → stop again — prohibitively complex

2. USD from Claude Code is a list-price estimate, not actual billed cost.

total_cost_usd from Claude Code's result event is calculated from published per-token rates. It does not reflect enterprise/volume discounts, negotiated rates, or the actual amount on the customer's invoice.

Recommended approach for cost visibility

Token counts are the fundamental unit — cost is always derived from tokens. Users who need cost visibility can:

Send fullsend telemetry to any OTEL-supported backend
Compute costs from token counts using their actual contracted rates
This is more accurate than Claude Code's list-price estimate

Acceptance criteria status

From the original issue:

Partial RunMetrics written on cancellation — via #6938's deferred TokensEvent
Agent telemetry ended with cancellation status and token attributes — via #6938
Existing successful-run behavior preserved — ResultEvent values remain authoritative
Dollar-cost limitation documented — see PR #7208 comment
Regression test for cancellation — covered by #6938's test suite

docs(#7229): record OpenShell exec/stop signal contract in ADR 0030

Navigation Menu

Repository navigation

Persist cost and token telemetry when Review agents are cancelled by GitHub Actions #6936

Problem

Evidence

Expected behavior

Likely failure mode

Acceptance criteria

Triage summary

Resolution: Token counts are the right abstraction for cancelled-run telemetry

What's been fixed

Why `total_cost_usd` remains zero on cancelled runs

Recommended approach for cost visibility

Acceptance criteria status

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Persist cost and token telemetry when Review agents are cancelled by GitHub Actions #6936

Description

Problem

Evidence

Expected behavior

Likely failure mode

Acceptance criteria

Activity

fullsend-ai-triage commented on Sep 2, 2026

fullsend-ai-triage commented on Sep 2, 2026

Triage summary

fullsend-ai-coder commented on Sep 2, 2026

ascerra commented on Sep 11, 2026

Resolution: Token counts are the right abstraction for cancelled-run telemetry

What's been fixed

Why total_cost_usd remains zero on cancelled runs

Recommended approach for cost visibility

Acceptance criteria status

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Why `total_cost_usd` remains zero on cancelled runs