feat(evals): Run workflow evals on the Claude Agent SDK - #1238
Conversation
45c5693 to
581d7e9
Compare
RobertCrupa
changed the title
581d7e9 to
1a13562
Compare
1a13562 to
b145326
Compare
github-actions
Bot
added
t-ai
MQ37
left a comment
MQ37
left a comment
There was a problem hiding this comment.
Approving. One issue worth addressing (before or after merge):
sdk_conversation_adapter.ts only handles the error_max_turns result subtype from the SDK. If the run instead fails mid-conversation for a different reason (e.g. error_during_execution — an Anthropic API error, context overflow, etc.), that failure is silently dropped. The partial conversation gets sent to the LLM judge exactly like a normal run, so the judge scores it PASS/FAIL as if the agent just answered badly — it has no way to know the run was actually cut short by an error. That failure then shows up in the run's pass/fail results looking like a real eval failure, with no error logged anywhere.
Verified this in a sandbox: an API-error run and a max-turns run produce identical conversation.turns, and workflow_judge.ts never even reads completed/hitMaxTurns.
This contradicts the harness's own stated design: harness errors should fail the run, not be scored as a failing eval. Recommend throwing on any non-success, non-max-turns result subtype instead.
1e048c8 to
a442572
Compare
jirispilka
left a comment
jirispilka
left a comment
There was a problem hiding this comment.
It is mostly good but, I believe, you've chosen a wrong transformation, which makes the code more complicated than it should be.
I'm thinking aloud here, we plan to move these evals to Actor so I don't think it is worth to change it :). I hope that we will remove this sooner than later. I also hope that Arto is not going to build on this. So please perhaps just check what it would take to re do, if u agree.
My notes below:
I would rather change judge than introduce conversation adapter. You need to rebuild ConversationHistory .
formatConversationForJudge reads four things: userPrompt, turns[].toolCalls[].name/.arguments, and turns[].finalResponse. langfuse_experiment.ts reads conversation.totalTokens. Nothing else in the repo reads a conversation. The turn grouping the judge needs is one blank line, so a formatter over SDKMessage[] produces the same prompt with no intermediate object.
Pros
- No conversion on the judge path. The judge reads the SDK's own types. And these types are pretty much a standard for other provides too!
- Five types deleted:
ConversationHistory,ConversationTurn,McpToolCall,McpTool,McpToolResult. - The 4-clause terminal-turn guard disappears. "Is the final answer already in the stream" becomes one condition.
- What is left of the adapter drops from 223 lines to 101, and it stops being an adapter: usage totals, the transcript, and the error-subtype gate, which the stream genuinely does not give you for free. Across
evals/workflows, 732 lines becomes 609. - The judge prompt is unchanged, so this is a refactor and not a behavior change.
Cons
- It touches the judge and its test, which this PR set out not to touch.
- #1239 needs rework, more than a rebase. It picks up
toolInvocations,metrics, andclaudeCodeVersionalong with thereceivedAtthreading, andlangfuse_observations.tsreads the final answer from theresultmessage instead ofturns.at(-1)?.finalResponse.
Port the Claude Agent SDK harness from braintrust-evals (cd733eef) onto the Langfuse runner, dropping that commit's Braintrust migration. The agent under test is now the real Claude Code harness driving its own Apify MCP server, so a run exercises the server the way a Claude Code user does. The judge, dataset, scores, and run gate are unchanged. - The SDK spawns the MCP server per item (fresh, isolated); a per-case AbortController tears it down. systemPrompt and tools use the claude_code presets, permissionMode is bypassPermissions, settingSources is empty (no repo config inherited), cwd is a temp dir, tool timeout maps to the per-server timeout. --mcp-tools-only drops the built-in tools. - Register the server with alwaysLoad: true. Deferred behind tool search (the default with built-in tools on) the agent never sees the Apify tools and answers from memory or Bash instead. - failTools injection moves to a PreToolUse deny, the hook that survives bypassPermissions. Verified: the agent receives the synthetic INTERNAL_ERROR with the real report-problem nudge. - sdk_conversation_adapter.ts folds the SDK message stream back into ConversationHistory, so the judge view matches earlier experiments. Cached prompt tokens count toward total_tokens; the API reports them separately and a cached run would otherwise look nearly free. - --agent-model now takes Anthropic model IDs, default claude-haiku-4-5. ANTHROPIC_API_KEY is optional: without it the SDK uses the Claude Code CLI credentials. agentSdkVersion is recorded in the run metadata. - Strip a forwarded bare `--` before yargs. `pnpm run evals:workflow -- --id x` passed it through to tsx, where yargs read it as end-of-options and ignored every flag, silently running all 30 cases instead of one. - Delete conversation_executor.ts, mcp_client.ts, and the executor's test.
The adapter only recognized error_max_turns, so a run aborted mid-conversation (API error, context overflow, budget) was handed to the judge as a normal partial conversation and scored as a failing eval with no error logged. Throw instead, so it fails the run like other harness errors.
The adapter populated seven fields nothing downstream read: turnNumber, toolResults, completed, hitMaxTurns, totalTurns, promptTokens and completionTokens. The judge reads userPrompt, toolCalls and finalResponse; the experiment reads totalTokens. Tool results and the token split are already on ToolInvocation and ConversationMetrics. Dropping the per-turn toolResults removes the turnIndex threading from PendingToolUse, so a tool result no longer has to find its turn. The max-turns test now asserts what the judge sees (no final answer appended) instead of the dropped flags. The judge prompt is unchanged.
209eb72 to
a52b707
Compare
|
@jirispilka Thank you for the comments. I have simplified the data structure. When we do the Actor implementation, we will change the judge also. |
Stacked on #1238. Second of two: #1238 makes the agent under test the real Claude Code harness, this one puts its conversation on the Langfuse trace. ## What - `langfuse_observations.ts` builds the tree from the adapted SDK stream and emits it. - Tool spans are timed from when the SDK delivered the call and its result. - `llm_client.ts` wraps each call in a generation, which is what puts the judge call, its verdict, and its tokens on the trace. - A failed tool call is raised to `ERROR` level with the payload as the status, and a run that never reached a final answer is flagged `WARNING`, so both are findable in the UI. - The tree is emitted before the judge call, so a failing judge still leaves the conversation on the trace to debug. A crashed agent run leaves no spans. - Adds `@opentelemetry/api`, the peer dependency of `@langfuse/tracing`, for the `SpanContext` type. ## Verification - `--id search-google-maps`: 1/1 passed, the five observations land with the right nesting, and `GET /api/public/v2/metrics` reports 71.5k tokens and $0.039 on the agent generation plus 798 tokens on the judge one. - `type-check`, `lint`, `test:unit` (1289 pass / 1 skip), `format`, `check:agents` all green.
4 participants
Stacked on #1224. First of two: this one makes the agent under test the real Claude Code harness. #1239 traces its conversation to Langfuse.
Why
We want to simulate a user but having the same environment as them during evals. This means using the same harness, system prompt and tools.
What
The hand-rolled OpenRouter loop and MCP client are gone. Each case now runs Claude Code headlessly through the Claude Agent SDK, with the
claude_codesystem-prompt and tool presets, driving its own freshly spawned Apify MCP server.The judge, dataset, scores, and run gate are unchanged
Note for reviewers
The evaluator is run with
allowDangerouslySkipPermissions(headless, never prompts; the SDK requires both) so run it with the MCP tools only to stop it from executing bashVerification
--id search-google-maps: 1/1 passed, and a direct probe confirmed the agent calls the MCPsearch-actorstool (thinking, tool call, answer).report-problem-on-tool-error: injection confirmed. The agent receives the refusal with the realreport-problemnudge, retriescall-actor, then explains to the user instead of callingreport-problem. The case fails, and that is a genuine eval signal rather than a harness bug.type-check,lint,test:unit(1280 pass / 1 skip),format,check:agentsall green.