feat(evals): Trace the agent conversation to Langfuse - #1239
Merged
Conversation
RobertCrupa
force-pushed
the
feat/trace-workflow-evals
branch
from
August 12, 2026 10:32
45c5693 to
f396684
Compare
RobertCrupa
force-pushed
the
feat/trace-workflow-evals
branch
3 times, most recently
from
August 14, 2026 09:55
f223ec4 to
6233c7e
Compare
RobertCrupa
force-pushed
the
feat/trace-workflow-evals
branch
from
August 14, 2026 09:56
6233c7e to
c28040b
Compare
RobertCrupa
force-pushed
the
feat/trace-workflow-evals
branch
from
August 14, 2026 12:36
4a9afbd to
0b0037e
Compare
MQ37
approved these changes
Aug 18, 2026
MQ37
left a comment
MQ37
left a comment
Contributor
There was a problem hiding this comment.
LGTM.
🔴 emitObservations errors are caught by makeTask's generic try/catch and treated as harness failures, dropping an otherwise-passing item. Suggest wrapping the emit call in its own try/catch (log + continue), same pattern already used for flush/shutdown.
🟡 usageNode uses three non-null assertions (metrics.promptTokens! etc). Destructuring first would let TS narrow without !.
🟡 emitObservations's asType dispatch is a nested ternary. An if/else reads the same and matches the "no nested ternaries" convention.
RobertCrupa
added a commit
that referenced
this pull request
Aug 18, 2026
Stacked on #1224. First of two: this one makes the agent under test the real Claude Code harness. #1239 traces its conversation to Langfuse. ## Why We want to simulate a user but having the same environment as them during evals. This means using the same harness, system prompt and tools. ## What The hand-rolled OpenRouter loop and MCP client are gone. Each case now runs Claude Code headlessly through the Claude Agent SDK, with the `claude_code` system-prompt and tool presets, driving its own freshly spawned Apify MCP server. The judge, dataset, scores, and run gate are unchanged ## Note for reviewers The evaluator is run with `allowDangerouslySkipPermissions` (headless, never prompts; the SDK requires both) so run it with the MCP tools only to stop it from executing bash ## Verification - `--id search-google-maps`: 1/1 passed, and a direct probe confirmed the agent calls the MCP `search-actors` tool (thinking, tool call, answer). - `report-problem-on-tool-error`: injection confirmed. The agent receives the refusal with the real `report-problem` nudge, retries `call-actor`, then explains to the user instead of calling `report-problem`. The case fails, and that is a genuine eval signal rather than a harness bug. - `type-check`, `lint`, `test:unit` (1280 pass / 1 skip), `format`, `check:agents` all green.
RobertCrupa
force-pushed
the
feat/trace-workflow-evals
branch
from
August 18, 2026 13:00
40386ad to
daf763b
Compare
github-actions
Bot
added
the
tested
RobertCrupa
force-pushed
the
feat/trace-workflow-evals
branch
from
August 18, 2026 14:00
daf763b to
6a0c3de
Compare
jirispilka
approved these changes
Aug 19, 2026
jirispilka
left a comment
jirispilka
left a comment
Collaborator
There was a problem hiding this comment.
Nice!
Let's not changes this PR, I created a follow up issue here
An experiment item's trace held a single span, the Langfuse SDK's own
experiment-item-run: the agent runs inside the Claude Code subprocess, so none of
its work is instrumented for us and the conversation was invisible in the UI.
Emit the tree by hand instead, per item:
experiment-item-run Langfuse SDK, holds the scores
|- agent the prompt in, the final answer out
| |- <agent model> generation: the run's aggregate tokens and cost
| |- <tool name> one span per tool call: arguments in, result out
|- <judge model> generation, emitted by llm_client.ts
- langfuse_observations.ts builds the tree from the adapted SDK stream and emits
it. Building is pure and separate from emitting, so the payload shaping is
tested without an OpenTelemetry provider.
- Tokens and cost only roll up to a trace from a generation, and the SDK reports
usage once for the whole run rather than per turn, so one generation spanning
the run is the finest honest granularity.
- Tool spans are timed from when the SDK delivered the call and its result.
claude_agent.ts stamps each message as it arrives and adaptSdkConversation
pairs the stamps into ToolInvocation.startedAt/endedAt. The stream carries no
timestamps of its own, and the tree is built after the run, so without them
every tool span would collapse to the moment it was emitted.
- llm_client.ts wraps each call in a generation, which is what puts the judge
call, its verdict, and its tokens on the trace.
- Add @opentelemetry/api, the peer dependency of @langfuse/tracing, for the
SpanContext type.
Verified on --id search-google-maps: the five observations land with the right
nesting, and the metrics API reports 71.5k tokens and $0.039 on the agent
generation plus 798 tokens on the judge one.
The CLI serializes every assistant turn as one wire frame per content block, all sharing message.id, and the SDK never re-aggregates them. So narration accompanying a tool call became its own text-only turn with finalResponse set and leaked into the judged transcript, thinking-only frames created empty filler turns, and turn counts inflated. Fold consecutive assistant frames with the same message.id into one turn and one transcript entry, clearing finalResponse once a tool call joins the turn.
String.replace reads `$&`, `$'`, `` $` `` and `$$` in the replacement as
patterns, so a transcript containing a Bash `$'\n'` spliced the template
around itself, and a `$`-sequence in expectedOutput could destroy the
{{conversation}} placeholder. Substitute both values via a function.
yargs accepted 0, negatives and NaN. Langfuse batches items with `i += concurrency`, so 0 hung forever and NaN ran nothing while reporting every id as "never completed". Reject anything but a positive integer at parse time.
Merging the frames of one API turn overwrote finalResponse with the last text block instead of accumulating it. Since finalResponse was then set, the post-loop fallback also skipped appending the result text, so the judge saw a truncated answer.
Spans are emitted for every tool call, not just MCP ones: --mcp-tools-only defaults to false, so Claude Code's built-in tools get spans too.
failTools is a PreToolUse deny, so the agent sees a refused call, not an INTERNAL_ERROR tool result. The task output comment claimed no transcript is returned, but one is.
adaptSdkConversation throws on every result subtype except success and error_max_turns, so a traced run that did not complete always hit the turn limit and the fallback status message could never render. Gate the agent span's warning on hitMaxTurns alone and drop completed, which nothing else read.
The generation carrying the run's tokens and cost spanned the whole agent run, so it started before every tool call. Langfuse orders siblings by start time, which put it above the tool spans and made the trace read as though the model answered before it called anything. Window it to the last model turn instead. The adapter stamps when that turn opened (finalTurnStartedAt, from the arrival times it already collects) and falls back to the run start when the caller did not time the stream. Usage still covers the whole run, marked with usageScope on the span.
A failed tool result was stored as JSON.stringify(content), so the error arrived at Langfuse quoted and escaped and rendered as one \n-riddled line. Keep the text as the model saw it: string content as-is, text blocks joined, pretty JSON only for a non-text payload. resultBytes still measures the compact serialization the agent received. Also drop statusMessage from a failed tool span. Langfuse renders it in its own box above the output preview, which showed the same message twice; the ERROR level still flags the span.
The SDK reports usage once for the whole run, so a multi-turn run re-reads the cached system prompt and tool definitions every turn: a 5-turn case read as a ~130k prompt when the fresh input was a few hundred tokens. Report the components separately (input, cache_read_input_tokens, cache_creation_input_tokens) so the breakdown is visible in the trace. The total is unchanged, so the total_tokens score stays comparable.
emitObservations ran inside makeTask's generic try/catch, so a failed span export turned an otherwise-passing item into a harness error. Guard it on its own, log and continue, like the flush/shutdown path.
Destructure promptTokens and completionTokens so the hasTokens check narrows them.
emitObservations recursed into children before ending the parent, so a child that threw while building its span left the parent unended and skipped the remaining siblings. An unended span never reaches Langfuse.
A run that hit the turn limit ends on a tool-calling turn, so windowing the generation to the final turn started it before its own tool spans and the UI ordered it ahead of them. Hold the window past the last tool result.
A tool result of hundreds of KB made Langfuse reject the ingestion event, which dropped the whole span. Replace the payload above 128 KB with a note and flag it in metadata.
The catch only sees span construction; export failures surface in the span processor's batch flush.
RobertCrupa
force-pushed
the
feat/trace-workflow-evals
branch
from
August 20, 2026 14:23
b209a30 to
150239f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1238. Second of two: #1238 makes the agent under test the real Claude Code harness, this one puts its conversation on the Langfuse trace.
What
langfuse_observations.tsbuilds the tree from the adapted SDK stream and emits it.llm_client.tswraps each call in a generation, which is what puts the judge call, its verdict, and its tokens on the trace.ERRORlevel with the payload as the status, and a run that never reached a final answer is flaggedWARNING, so both are findable in the UI.@opentelemetry/api, the peer dependency of@langfuse/tracing, for theSpanContexttype.Verification
--id search-google-maps: 1/1 passed, the five observations land with the right nesting, andGET /api/public/v2/metricsreports 71.5k tokens and $0.039 on the agent generation plus 798 tokens on the judge one.type-check,lint,test:unit(1289 pass / 1 skip),format,check:agentsall green.