Skip to content

fix(AIC-3576): record deferred judge scores under the evaluated config - #190

Merged
jeffdupont merged 1 commit into
mainfrom
jdupont/aic-3576-deferred-judge-trackdata
Oct 21, 2026
Merged

fix(AIC-3576): record deferred judge scores under the evaluated config#190
jeffdupont merged 1 commit into
mainfrom
jdupont/aic-3576-deferred-judge-trackdata

Conversation

@jeffdupont

@jeffdupont jeffdupont commented Oct 20, 2026 •
edited by atlassian Bot
Loading

Copy link
Copy Markdown
Contributor

Fixes AIC-3576.

Why

When judges run in the background (skip_judges=True and then run_judge(task, handlers)), the returned track_data was built by spreading the judge's own call track_data over the parent's. A caller who then tracks it records the score under the judge config's configKey, variationKey, version and runId, not under the config that was evaluated. The inline run_judges path already tracks {**base_track_data, "judgeConfigKey": ...}, so the same judge scored differently depending on which path ran it.

This reverses the model-stamp stripping that 5d1e678 added (omit_model_stamps(task.parent_track_data)). That change assumed the result described the judge, so it hid the parent's modelKey/modelVersion. The identity is now the parent's, so the parent's stamps describe it correctly, which is what the inline path sends.

Fix

  • packages/client/src/launchdarkly_ai_server/judges.py: run_judge returns {**task.parent_track_data, "judgeConfigKey": task.config_key} and drops the omit_model_stamps import. omit_model_stamps stays exported (tests/test_public_surface.py).
  • packages/client/tests/test_judges.py: replaced 5d1e678's two stamp tests in TestRunJudgeTrackData with one test asserting the deferred track_data equals {**parent, "judgeConfigKey": ...} and equals the payload run_judges passes to track().

Testing

  • make test: 2543 passed, 14 skipped
  • make typecheck: no issues in 75 source files
  • make lint: all checks passed; make format-check: 161 files formatted
  • With the fix reverted, the new test fails: {'configKey': 'judge-key'} != {'configKey': 'main-flag'} (and runId, variationKey, version, model fields).

Related: launchdarkly/js-ai-sdk#148, launchdarkly/go-ai-sdk#13, spec https://github.com/launchdarkly/ai-sdks-monorepo/pull/74

🤖 Generated with Claude Code

@jeffdupont @claude
fix(AIC-3576): record deferred judge scores under the evaluated config
52dfc1c 
run_judge overlaid the judge's own track_data on the parent's, so a deferred
judge score was tracked under the judge config's configKey, variationKey,
version and runId instead of the config it evaluated. It now returns
{**parent_track_data, "judgeConfigKey": ...}, the same payload the inline
run_judges path tracks. This reverses 5d1e678's model-stamp stripping: the
identity is now the parent's, so the parent's modelKey/modelVersion describe
it correctly. omit_model_stamps stays exported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 20, 2026
Merged
Merged
@jeffdupont
jeffdupont requested a review from a team October 21, 2026 15:20
jeffdupont added a commit to launchdarkly/go-ai-sdk that referenced this pull request Oct 21, 2026
@jeffdupont @claude
fix(AIC-3576): record deferred judge scores under the evaluated config (
cd71325 
#13)

Fixes [AIC-3576](https://launchdarkly.atlassian.net/browse/AIC-3576).

## Why

When judges run in the background (`SkipJudges: true` and then
`RunJudge(ctx, task, handlers)`), the returned `TrackData` started from
the judge's own `executeAndTrack` result and copied over only the
parent's `GraphKey`. A caller who then tracks it records the score under
the judge config's `ConfigKey`, `VariationKey`, `Version`, `RunID` and
model stamps, not under the config that was evaluated. The inline path
already tracks `a.parentTD` plus `JudgeConfigKey`
(`aiserver/judges.go`), so the same judge scored differently depending
on which path ran it. JS and Python had the same bug and are fixed in
the PRs below.

## Fix

- `aiserver/judges.go`: `RunJudge` builds the result from
`task.ParentTrackData` and sets `JudgeConfigKey`. `GraphKey` is still
carried because it is part of the parent's trackData.
- `aiserver/judges_test.go`: new
`TestDeferredJudgeTrackDataMatchesInline`. It checks that the deferred
`TrackData` equals the parent invocation's `TrackData` plus
`JudgeConfigKey`, that its JSON equals the inline `quality` metric
payload (only the per-invocation `runId` differs), and that a parent
`GraphKey` survives.

## Testing

- `make test` (`go test -race ./...`): all 16 packages ok (557 tests
pass, 0 fail)
- `go vet ./...` / `make lint`: clean
- With the fix reverted, the new test fails: `deferred
map[configKey:judge … modelKey:judge-model … version:5] != inline
map[configKey:cfg … modelKey:parent-model … version:3]`

Related: launchdarkly/js-ai-sdk#148,
launchdarkly/python-ai-sdk#190, spec
launchdarkly/ai-sdks-monorepo#74

🤖 Generated with [Claude Code](https://claude.com/claude-code)


[AIC-3576]:
https://launchdarkly.atlassian.net/browse/AIC-3576?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@jeffdupont
jeffdupont merged commit 354d3f1 into main Oct 21, 2026
8 checks passed
@jeffdupont
jeffdupont deleted the jdupont/aic-3576-deferred-judge-trackdata branch October 21, 2026 20:12
jeffdupont added a commit to launchdarkly/js-ai-sdk that referenced this pull request Oct 21, 2026
@jeffdupont @claude
fix(AIC-3576): record deferred judge scores under the evaluated config (
d9fbee6 
#148)

Fixes [AIC-3576](https://launchdarkly.atlassian.net/browse/AIC-3576).

## Why

When judges run in the background (`skipJudges: true` and then
`runJudge(task, handlers)`), the returned `trackData` was built by
spreading the judge's own call `trackData` over the parent's. A caller
who then tracks it records the score under the judge config's
`configKey`, `variationKey`, `version` and `runId`, not under the config
that was evaluated. The inline `runJudges` path already tracks `{
...baseTrackData, judgeConfigKey }`, so the same judge scored
differently depending on which path ran it.

This reverses the model-stamp stripping that d374962 added
(`omitModelStamps(parentTrackData)`). That change assumed the result
described the judge, so it hid the parent's `modelKey`/`modelVersion`.
The identity is now the parent's, so the parent's stamps describe it
correctly, which is what the inline path sends.

## Fix

- `packages/client/src/judges.ts`: `runJudge` returns `trackData: {
...parentTrackData, judgeConfigKey: configKey }`. It drops the unused
`trackData` destructure and the `omitModelStamps` import.
`omitModelStamps` stays exported from the package root.
- `packages/client/src/__tests__/judges.test.ts`: replaced d374962's two
stamp tests with one test asserting the deferred `trackData` equals `{
...parentTrackData, judgeConfigKey }` and equals the payload `runJudges`
passes to `track()`.

## Testing

- `yarn typecheck`: pass
- `yarn code:check`: pass (237 files)
- `yarn test`: all workspaces pass (client: 1400 passed, 11 skipped)
- With the fix reverted, the new test fails: `expected { runId:
'judge-run', … } to deeply equal { runId: 'parent-run', … }` (configKey
`judge-key` vs `main-flag`).

Related: launchdarkly/python-ai-sdk#190,
launchdarkly/go-ai-sdk#13, spec
launchdarkly/ai-sdks-monorepo#74

🤖 Generated with [Claude Code](https://claude.com/claude-code)


[AIC-3576]:
https://launchdarkly.atlassian.net/browse/AIC-3576?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot mentioned this pull request Oct 21, 2026
Merged
jeffdupont pushed a commit that referenced this pull request Oct 23, 2026
@github-actions
chore: release main (#166)
662d3f2 
🤖 I have created a release *beep* *boop*
---


<details><summary>launchdarkly-ai-server: 0.4.0</summary>

##
[0.4.0](launchdarkly-ai-server-0.3.0...launchdarkly-ai-server-0.4.0)
(2026-10-23)


### ⚠ BREAKING CHANGES

* **evaluations:** keep the management API client internal
([#202](#202))
* **evaluations:** rename base uri to match spec
([#199](#199))
* **evaluations:** add test proving harness edge case matches spec
([#200](#200))
* **evaluations:** route handlers by provider and mode
* **evaluations:** mark evaluations from code as experimental
([#157](#157))

### Experimental

* **evaluations:** add test proving harness edge case matches spec
([#200](#200))
([fa7162e](fa7162e))
* **evaluations:** keep the management API client internal
([#202](#202))
([17270ed](17270ed))
* **evaluations:** mark evaluations from code as experimental
([#157](#157))
([1389765](1389765))
* **evaluations:** rename base uri to match spec
([#199](#199))
([7d93f9c](7d93f9c))
* **evaluations:** route handlers by provider and mode
([f62591f](f62591f))
* **evaluations:** Set JSON as standard for output to pass to judge
([#203](#203))
([ffc728b](ffc728b))
* **skills:** Correct the HTTP 403 advice now that server-side FDv2 is
available to every account
([#178](#178))
([b17e322](b17e322))


### Features

* **agui:** add a shared reader for handler streams
([#164](#164))
([fc08dd5](fc08dd5))
* **AIC-3534:** add a TypeSafe Jev judge handler
([#149](#149))
([1b41ea4](1b41ea4))
* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **langchain:** emit AG-UI text and tool events
([#163](#163))
([ab5e9e1](ab5e9e1))
* **openai:** emit AG-UI text and tool events
([#161](#161))
([2b1b2d4](2b1b2d4))


### Bug Fixes

* **AIC-3570:** re-resolve graph flags on every invoke and stream
([#187](#187))
([8fdc4d4](8fdc4d4))
* **AIC-3571:** visit every child before its parent in reverse_traverse
([#189](#189))
([87c73d9](87c73d9))
* **AIC-3572:** treat a variation without _ldMeta.enabled as disabled
([#184](#184))
([7a0d260](7a0d260))
* **AIC-3574:** build the LaunchDarkly client off the event loop
([#185](#185))
([bf03b6c](bf03b6c))
* **AIC-3576:** record deferred judge scores under the evaluated config
([#190](#190))
([354d3f1](354d3f1))
* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
* **AIC-3596:** name the [otel] extra in the missing-OTel warning
([#186](#186))
([90ce7bf](90ce7bf))
* **client:** match a handler's provides_for after normalizing its mode
([f62591f](f62591f))
* give native graph spans their LaunchDarkly identity
([#121](#121))
([e863b6d](e863b6d))
</details>

<details><summary>launchdarkly-ai-python: 0.3.0</summary>

##
[0.3.0](launchdarkly-ai-python-0.2.0...launchdarkly-ai-python-0.3.0)
(2026-10-23)


### ⚠ BREAKING CHANGES

* **evaluations:** mark evaluations from code as experimental
([#157](#157))

### Experimental

* **evaluations:** mark evaluations from code as experimental
([#157](#157))
([1389765](1389765))
</details>

<details><summary>launchdarkly-ai-claude-agents: 0.2.6</summary>

##
[0.2.6](launchdarkly-ai-claude-agents-0.2.5...launchdarkly-ai-claude-agents-0.2.6)
(2026-10-23)


### Features

* **claude:** emit AG-UI text and tool events
([#165](#165))
([3f3a0ce](3f3a0ce))
* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))


### Bug Fixes

* **AIC-3571:** visit every child before its parent in reverse_traverse
([#189](#189))
([87c73d9](87c73d9))
* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
* **claude-agents:** fail the run when the CLI reports an API error as
success
([#174](#174))
([3d1a768](3d1a768))
* give native graph spans their LaunchDarkly identity
([#121](#121))
([e863b6d](e863b6d))
</details>

<details><summary>launchdarkly-ai-claude-messages: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-claude-messages-0.2.4...launchdarkly-ai-claude-messages-0.2.5)
(2026-10-23)


### Features

* **claude:** emit AG-UI text and tool events
([#165](#165))
([3f3a0ce](3f3a0ce))
* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))


### Bug Fixes

* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
</details>

<details><summary>launchdarkly-ai-openai-agents: 0.2.6</summary>

##
[0.2.6](launchdarkly-ai-openai-agents-0.2.5...launchdarkly-ai-openai-agents-0.2.6)
(2026-10-23)


### Features

* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **openai:** emit AG-UI text and tool events
([#161](#161))
([2b1b2d4](2b1b2d4))


### Bug Fixes

* **AIC-3571:** visit every child before its parent in reverse_traverse
([#189](#189))
([87c73d9](87c73d9))
* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
* give native graph spans their LaunchDarkly identity
([#121](#121))
([e863b6d](e863b6d))
</details>

<details><summary>launchdarkly-ai-vercel-agents: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-vercel-agents-0.1.0...launchdarkly-ai-vercel-agents-0.1.1)
(2026-10-23)


### Features

* **AIC-3408:** add Vercel AI SDK messages and agents adapters
([#103](#103))
([64d679e](64d679e))
</details>

<details><summary>launchdarkly-ai-litellm-agents: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-litellm-agents-0.1.0...launchdarkly-ai-litellm-agents-0.1.1)
(2026-10-23)


### Features

* add LiteLLM messages and agents integrations
([#102](#102))
([f2e9886](f2e9886))
</details>

<details><summary>launchdarkly-ai-openai-messages: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-openai-messages-0.2.4...launchdarkly-ai-openai-messages-0.2.5)
(2026-10-23)


### Features

* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **openai:** emit AG-UI text and tool events
([#161](#161))
([2b1b2d4](2b1b2d4))


### Bug Fixes

* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
</details>

<details><summary>launchdarkly-ai-vercel-messages: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-vercel-messages-0.1.0...launchdarkly-ai-vercel-messages-0.1.1)
(2026-10-23)


### Features

* **AIC-3408:** add Vercel AI SDK messages and agents adapters
([#103](#103))
([64d679e](64d679e))
</details>

<details><summary>launchdarkly-ai-litellm-messages: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-litellm-messages-0.1.0...launchdarkly-ai-litellm-messages-0.1.1)
(2026-10-23)


### Features

* add LiteLLM messages and agents integrations
([#102](#102))
([f2e9886](f2e9886))
</details>

<details><summary>launchdarkly-ai-langchain-agents: 0.2.6</summary>

##
[0.2.6](launchdarkly-ai-langchain-agents-0.2.5...launchdarkly-ai-langchain-agents-0.2.6)
(2026-10-23)


### Features

* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **langchain:** emit AG-UI text and tool events
([#163](#163))
([ab5e9e1](ab5e9e1))


### Bug Fixes

* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
* give native graph spans their LaunchDarkly identity
([#121](#121))
([e863b6d](e863b6d))
</details>

<details><summary>launchdarkly-ai-langchain-messages: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-langchain-messages-0.2.4...launchdarkly-ai-langchain-messages-0.2.5)
(2026-10-23)


### Features

* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **langchain:** emit AG-UI text and tool events
([#163](#163))
([ab5e9e1](ab5e9e1))


### Bug Fixes

* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
</details>

<details><summary>launchdarkly-ai-opencode-agents: 0.2.0</summary>

##
[0.2.0](launchdarkly-ai-opencode-agents-0.1.0...launchdarkly-ai-opencode-agents-0.2.0)
(2026-10-23)


### ⚠ BREAKING CHANGES

* **opencode:** start and stop opencode serve in the handler
([#198](#198))

### Features

* **opencode-agents:** add an OpenCode handler
([#159](#159))
([fc08e8c](fc08e8c))
* **opencode-agents:** emit the AG-UI text and tool subset
([#162](#162))
([7677621](7677621))
* **opencode-agents:** open one OpenCode session per call
([#179](#179))
([b225113](b225113))
* **opencode-agents:** render agent instruction templates
([#182](#182))
([83175aa](83175aa))
* **opencode:** start and stop opencode serve in the handler
([#198](#198))
([0748dcf](0748dcf))


### Bug Fixes

* **opencode-agents:** own the HTTP client
([#194](#194))
([ed2d419](ed2d419))
* **opencode-agents:** skip judges and record structured history
([#177](#177))
([6fadf1d](6fadf1d))
</details>

<details><summary>launchdarkly-ai-typesafe: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-typesafe-0.1.0...launchdarkly-ai-typesafe-0.1.1)
(2026-10-23)


### Features

* **AIC-3534:** add a TypeSafe Jev judge handler
([#149](#149))
([1b41ea4](1b41ea4))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewklatzke andrewklatzke andrewklatzke approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jeffdupont @andrewklatzke