Repository navigation
feat(client)!: add result variant and value to the feature_flag span event - #186
Conversation
extractVariation now returns the raw variation it already gets from client.variation, and invoke, stream, inline judges and graph nodes hand it to handlers as variables.__ldValue, next to __ld. It is kept out of TrackData on purpose: TrackData is the payload of every $ld:ai:* track event, and the value includes the prompt.
…vent AI Config monitoring only finds traces whose feature_flag event carries feature_flag.result.value, which the LaunchDarkly observability plugins always send. Write the variation key as feature_flag.result.variant (conditionally required by the OpenTelemetry feature-flag conventions) and the evaluated value as feature_flag.result.value, serialized with JSON.stringify exactly like TracingHook with includeValue. Both are omitted when there is nothing to write, for example on native-graph spans.
The raw variation holds the prompt, so writing it on every run's span event bypassed the content-capture opt-in. setLdSpanAttributes now takes a captureContent argument (default false). With capture on and a raw variation present, feature_flag.result.value is its JSON, as before. Otherwise it is the JSON-encoded variation key, and it is omitted when there is neither. Every handler passes the capture setting it already uses for content attributes, as does vercelEvaluate. Native graph and client graph spans carry no raw variation, so they keep the default.
Move the content-capture gate for feature_flag.result.value out of the handlers and into executeAndTrack/executeAndStream. The core now sets variables.__ldValue only when the handler that runs has captureContent on (ProviderHandler.captureContent, the same gate judge reasoning uses), so the raw config never reaches a capture-off handler. Config runs, judges and graph nodes all go through these two functions. setLdSpanAttributes goes back to its original signature, and the handler call sites and the API report match main again. The helper writes the raw variation's JSON when __ldValue is present, the JSON-encoded variation key otherwise, and nothing when there is neither.
jeffdupont
left a comment
jeffdupont
left a comment
There was a problem hiding this comment.
Looks good, and it matches the updated #113 spec. Gating on handler.captureContent in executeAndTrack / executeAndStream keeps the raw config away from capture-off handlers, and since createHandlerInternal (utils.ts:74) sets that property for every handler, it is a single check. The fallback also closes the vercelEvaluate / JudgeTask gap from the first version.
- One handler-level capture-on test. The four handler tests only assert the capture-off value (
JSON.stringify('variation-a')). §1.5 asks handler tests to cover capture on too. A single test that goes throughconfig().invokewithcreateXHandler({ captureContent: true })and assertsresult.value === JSON.stringify(rawVariation)would cover the wiring from start to finish. - Serialization question. gonfalon's flag Monitoring Traces tab groups chart series by the exact
feature_flag.result.valuestring, matched againstJSON.stringify(variation.value)(static/ld/components/feature-monitoring/hooks/useVariationChartConfig.ts:8,45-49). JS already matches with capture on. The capture-off"variation-a"fallback matches no variation, so if AI Config Monitoring uses that chart, those traces land in an unlabeled series. Do you know which UI readsresult.valuefor AI Configs? Not blocking.
Happy to approve once the capture-on test is in.
Run config().invoke with a handler built by createHandler with captureContent on and check that the root feature_flag event's feature_flag.result.value is the JSON of the raw variation extractVariation returned.
jeffdupont
left a comment
jeffdupont
left a comment
There was a problem hiding this comment.
bda9e7b adds the end-to-end capture-on test I asked for: it goes through config().invoke with capture on and asserts the raw variation as the value. Approving.
This matches launchdarkly/ai-sdks-monorepo#113. setLdSpanAttributes writes feature_flag.result.variant only for a non-empty variation key. It writes feature_flag.result.value as JSON.stringify of the raw variation, which is how the node-server-sdk-otel TracingHook writes it. The core sets __ldValue only when handler.captureContent is true. Otherwise the value is the JSON-encoded key, and it is left off when there is neither. The event is still written only by the shared helper on the root span. The tests cover capture on (client.test.ts), capture off and unset (tracking.test.ts, for track and stream), and the case with neither (utils.test.ts). The change is additive. extractVariation gains a return field, and no public type or API report changes, so API compatibility passes.
On parity with python-ai-sdk#240, the span output agrees, with two differences in how it gets there. First, JS gates capture in the core and Python gates it in the helper. Second, a judge run from a JudgeTask writes the raw value in Python but only the key here, because the JS JudgeTask path has no raw variation. Both stay within the spec.
….result.value A deferred judge (buildJudgeTasks, then runJudge in a worker) wrote the variation key as feature_flag.result.value even when the judge handler captures content, because the task had no raw variation to pass. JudgeTask now carries it as judgeValue, and runJudge hands it to the core as the evaluated value, which writes it only for a capture-on handler.
…tion captureContentFor returns a handler's own captureContent when it states one, otherwise the process-wide default (off). The evaluated-variation gate in executeAndTrack and executeAndStream now goes through it, so a later process-wide setting changes one place. Behavior is unchanged.
…on the graph root The launchdarkly.graph span wrote the variation key as feature_flag.result.value even when content capture is on, because the raw graph variation never reached it. buildGraph now returns the variation it fetched, and graph().stream (which invoke() walks) passes it as __ldValue when captureContentFor(undefined) is true. The root has no single handler, so it follows the process-wide default.
… ag/ff-result-variant
apucacao
changed the title
jeffdupont
left a comment
jeffdupont
left a comment
There was a problem hiding this comment.
Since my approval at bda9e7b: 8398f4e puts the judge variation on JudgeTask.judgeValue, and runJudge gates it with the judge's handler. That closes the deferred-judge gap I mentioned, and the structuredClone test is a good touch. 4ad0f34 gives the launchdarkly.graph root the graph variation under the process default. de89d4a routes both through captureContentFor. The main merge had no conflict resolutions.
About the red API compatibility runs: they come from 8398f4e. JudgeTask is public, so the new field changed ai-server.api.md, and the check counts any changed declaration as breaking. The rename to feat(client)! at 16:34 made it pass, and the PR is clean now. Adding an optional field doesn't break anyone, though. If you'd rather not have a BREAKING CHANGES entry in the changelog, the api-compatible label on a title without ! is the more accurate route. Your call.
One thing to settle with #190. I merged #190 onto this branch locally. It merges with no conflicts, but leaves two captureContentFor functions. The one in content.ts keeps the hard-coded false default, and tracking.ts and graph.ts still import it. The one in sdk-options.ts follows configure(), and judges.ts imports it. Handlers built with createHandler are fine, because #190 makes captureContent a getter that always returns a boolean. A plain function handler, and the graph root, would still send only the key while the default is on. A quick vitest probe on the merged tree confirms it for the plain function, and the full suite still passes, because graph.test.ts mocks content.js. Whichever of the two merges second should delete the content.ts version, point tracking.ts and graph.ts at sdk-options.ts, and use configure() in the graph test instead of the mock. A test with a plain function handler under the built-in default would keep it fixed.
Nit: executeAndTrack and executeAndStream spread the caller's variables before adding __ldValue. A capture-off handler therefore writes a caller-supplied variables.__ldValue. Overwriting or deleting it the way __ld is handled would close that.
executeAndTrack and executeAndStream spread the caller's variables before adding __ldValue, so a capture-off handler still received a caller-supplied __ldValue and wrote it as feature_flag.result.value. Both now build the handler's variables in one place that sets __ldValue only for a capture-on handler and removes it otherwise, as __ld is already overwritten.
… ag/ff-result-variant
#111 replaced createHandler's positional stream and captureContent arguments with an options object, and the positional form now throws. The two client tests this branch added still used it.
Summary
Runs made through the turnkey SDK don't show up on an AI Config's monitoring traces tab. Turnkey writes the
feature_flagspan event withoutfeature_flag.result.value. Since launchdarkly/observability#2553, a trace search for a flag key only scans hours where afeature_flagevent with a value was recorded, so turnkey runs are never found. The LaunchDarkly observability plugins always send the value (the SDK tracing hook withincludeValue: true).This adds two attributes to the event
setLdSpanAttributeswrites:feature_flag.result.variant: the variation key. The OpenTelemetry feature-flag conventions make it conditionally required when the provider has variants.feature_flag.result.value: with content capture on, the evaluated variation, serialized withJSON.stringifyexactly asTracingHookdoes. With capture off, the variation key, JSON-encoded like any flag value (for example"opus"), so the prompt is never sent unless capture is on.What changed
extractVariationreturns the raw variation it already receives fromclient.variation.invoke,stream, inline judges and graph nodes pass it to handlers asvariables.__ldValue, next to__ld.JudgeTaskhas a new optional field,judgeValue: the judge variation exactly as the LaunchDarkly SDK returned it.buildJudgeTasksfills it, andrunJudgepasses it on under the same capture check as the inline path. It is plain JSON, so a task still survives structured clone, and it adds no new kind of content, since the task already carriesjudgeConfigandactualOutput.launchdarkly.graphroot span gets the value too.graph().invoke()andgraph().stream()share one code path, which now passes the graph flag's raw variation to the root span. The root has no single handler, so it follows the process-wide content capture default, which is off today. With the default off, the root keeps the JSON-encoded variation key.TrackData.TrackDatais the payload of every$ld:ai:*track event, and the value includes the prompt.captureContenton.executeAndTrackandexecuteAndStreamset__ldValueonly when the handler'scaptureContentis true, the same check judge reasoning uses, so the raw config never reaches a capture-off handler. That check, and the graph root's, go through one function,captureContentFor, which returns a handler's owncaptureContentor the process-wide default when there is no handler or it states none. This uses the existingcaptureContenthandler option; there is no new setting.setLdSpanAttributeswrites the variant when the variation key is non-empty. For the value it writes the raw variation's JSON when__ldValueis present, otherwise the JSON-encoded variation key, and leaves it off when there is neither (native-graph spans).JudgeTask.judgeValuefield.setLdSpanAttributeskeeps its signature and the handlers are unchanged.vercelEvaluatehas no raw variation at hand, so its value is always the variation key.Testing
__ldValueonly to a capture-on handler, for bothexecuteAndTrackandexecuteAndStream; the value isJSON.stringifyof the raw variation when present, the JSON-encoded variation key otherwise, and absent with neither;__ldValuereaches the handler and stays out ofTrackDataand track payloads.JudgeTaskbuilt from a judge variation carries it asjudgeValuethrough structured clone, and itsrunJudgerun writes the raw variation asfeature_flag.result.valuewith capture on and the variation key with capture off. Thelaunchdarkly.graphroot writes the variation key with the default off and the raw graph variation with it on, for both invoke and stream.tsc --noEmitin every package,api:check,api:surfaceand biome.The cross-SDK spec change is in launchdarkly/ai-sdks-monorepo#113.
Note
Overview
Turnkey SDK runs were missing from AI Config monitoring traces because the root
feature_flagevent lackedfeature_flag.result.value, which observability now requires to correlate traces with a flag. This PR addsfeature_flag.result.variant(served variation key, omitted when empty) andfeature_flag.result.valueon the eventsetLdSpanAttributeswrites, aligned with LaunchDarkly OTelTracingHookserialization.extractVariationnow returns the raw SDKvalue, threaded throughinvoke/stream, graph nodes, and judges as optionalevaluatedValue. The core exposes it to handlers only asvariables.__ldValuewhencaptureContentFor(handler)is true (handlercaptureContentor process default); otherwise the event gets JSON-encoded variation key only. Raw variation never lands inTrackDataor track payloads. Deferred judges store it on optionalJudgeTask.judgeValue; thelaunchdarkly.graphroot uses the graph flag raw value only when the process-wide capture default is on.captureContentForcentralizes that gate. Handler package tests and AGENTS.md are updated; the only public API addition isJudgeTask.judgeValue.Reviewed by Cursor Bugbot for commit 869636a. Bugbot is set up for automated code reviews on this repo. Configure here.