Repository navigation
feat(evaluations)!: promote Evaluations from Code to core - #181
Merged
Merged
Conversation
Export the Evaluations from Code names from @launchdarkly/ai-server. Remove them from @launchdarkly/ai-server/experimental. Report the usage helper as client.initEvaluations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
aknight-ld
approved these changes
Oct 28, 2026
…ons-to-core # Conflicts: # packages/client/etc/ai-server-experimental.api.md
jeffdupont
approved these changes
Oct 28, 2026
10 tasks done
Merged
donei003
added a commit
to launchdarkly/python-ai-sdk
that referenced
this pull request
Oct 29, 2026
Promotes Evaluations from Code from `launchdarkly_ai_server.experimental.evaluations` to the package root. This follows the Agent Skills promotion in #210. > **Draft.** This PR releases together with launchdarkly/js-ai-sdk#181. A Jira ticket is not linked yet. ## Migration Import Evaluations from Code names from `launchdarkly_ai_server` instead of `launchdarkly_ai_server.experimental.evaluations`. The names, signatures and behavior do not change. ```python # before from launchdarkly_ai_server.experimental.evaluations import EvalTool, init_evaluations # after from launchdarkly_ai_server import EvalTool, init_evaluations ``` This is a breaking change (`!`). It bumps the minor version while the package is 0.x. ## What changed - **Root exports.** The 14 names that `experimental/evaluations.py` exported are now in `launchdarkly_ai_server.__all__`, with the same spelling. No name collides with a current root export. `launchdarkly-ai-python` gets the names through its existing `import *`. - **Experimental module removed.** `experimental/evaluations.py` is deleted. `experimental/__init__.py` stays as a docstring only, so a future experimental feature has a place to go. - **Usage ID.** `init_evaluations` now reports `client.initEvaluations`, not `experimental.initEvaluations`. Queries that count the old ID must also count the new one. - **API reports.** Regenerated with `make api-report`. The names are in `launchdarkly_ai_server.txt`, and `launchdarkly_ai_server.experimental.evaluations.txt` is deleted. - **Changelog.** The three `evaluations` rows are removed from `changelog-sections`, so `feat(evaluations):` lands under Features. The `AGENTS.md` scope examples now use a placeholder scope. - **Docs.** The client README, the `launchdarkly-ai-python` README, `packages/client/agents.md` and `AGENTS.md` no longer call Evaluations from Code experimental. The README heading anchor changed from `#run-an-evaluation-from-code-experimental` to `#run-an-evaluation-from-code`. ## Tests - `TestPackageExports` checks that every evaluations name is on the root and that `experimental.evaluations` no longer imports. - `test_public_surface.py` adds the 14 names to the root snapshot. - `test_reexports.py` checks that the barrel re-exports the names. - `test_core_does_not_import_experimental_code` now checks only `launchdarkly_ai_server.experimental`, because the root loads `evaluations`. A new test checks that `lifecycle` imports no evaluations module. - `test_sdk_usage.py` expects `client.initEvaluations`. ## Import time `import launchdarkly_ai_server` now loads the `evaluations` package. Measured with `python -X importtime`, it adds about 33 ms to a root import of about 130 ms. The largest part is `evaluations.api`. ## Checks - [x] `make lint`, `make format-check`, `make typecheck`, `make api-report-check`: pass - [x] `python -m scripts.api_breaking pr --base origin/main`: no breaking changes to the public API - [x] `make test`: 3640 passed, 18 skipped - [ ] Run one evaluation end to end against staging with the root import ## Conflicts The open evaluations PRs #228, #227, #143, #135 and #92 edit the same test files. Merge them first, or rebase them after this PR. ## Related PRs Releases together with: - launchdarkly/js-ai-sdk#181 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > **Evaluations from Code** moves from `launchdarkly_ai_server.experimental.evaluations` to the **`launchdarkly_ai_server` package root** (and through **`launchdarkly-ai-python`** re-exports). The experimental shim is **removed**; callers should import `init_evaluations`, `EvalTool`, `Judge`, `Scorer`, and the rest from the root instead. > > The **public API contract** shifts accordingly: generated **`launchdarkly_ai_server.txt`** now lists the evaluations surface, and **`launchdarkly_ai_server.experimental.evaluations.txt`** is deleted. **`init_evaluations`** usage telemetry changes from **`experimental.initEvaluations`** to **`client.initEvaluations`**. > > **Release/changelog policy** is updated: **`evaluations` is no longer an Experimental scope** in `release-please-config.json`, and **`AGENTS.md`** / READMEs describe Evaluations (and Skills) as core, with docs and examples using root imports and a non-experimental README anchor. > > Tests and snapshots (**`test_public_surface`**, **`TestPackageExports`**, **`test_reexports`**, lifecycle import guards) enforce the new export layout and that **`experimental.evaluations`** cannot be imported. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3ee3612. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
jeffdupont
added a commit
that referenced
this pull request
Oct 30, 2026
Steps 1 and 3 of [AIC-3673](https://launchdarkly.atlassian.net/browse/AIC-3673). The Python half is launchdarkly/python-ai-sdk#242. ## What this does - **`release-as: 1.0.0`** on all 14 packages in `release-please-config.json`. The `node-workspace` plugin updates the internal dependency ranges. - **Publish jobs for `litellm-agents` and `litellm-messages`.** They had none, so `@launchdarkly/ai-litellm-*-0.2.0` was tagged but never published. Both are also added to the manual-publish choices. `changelog-sections` is left alone: #181 (merged) removed the evaluations Experimental rows when it promoted Evaluations from Code to core. ##⚠️ npm setup: trusted publishers still to confirm for 5 of 14 packages The 26 Oct release run ([40393827352](https://github.com/launchdarkly/js-ai-sdk/actions/runs/40393827352)) failed 3 publish jobs, and the litellm packages have never had a job: | Package | On npm | Failure | |---|---|---| | `@launchdarkly/ai-typesafe` | 0.0.0 placeholder | `E404 … PUT @launchdarkly/ai-typesafe`: the package doesn't exist, and OIDC can't create it | | `@launchdarkly/ai-vercel-agents` | 0.0.0 placeholder | `E403 OIDC permission denied` | | `@launchdarkly/ai-vercel-messages` | 0.0.0 placeholder | `E403 OIDC permission denied` | | `@launchdarkly/ai-litellm-agents` | 0.0.0 placeholder | no publish job (added here) | | `@launchdarkly/ai-litellm-messages` | 0.0.0 placeholder | no publish job (added here) | Before the release PR merges, an npm org admin needs to: 1. ~~Publish a 0.0.0 placeholder for `ai-typesafe`, `ai-litellm-agents` and `ai-litellm-messages`.~~ Done: all three are on npm at 0.0.0 (checked 29 Oct). 2. Add a trusted publisher (`launchdarkly/js-ai-sdk`, workflow `release-please.yml`) to all five. Otherwise 1.0.0 ships 9 of 14 packages. ## Merge this last Merge this only after the items below are done. Anything merged before the release PR (#171) merges ships in 1.0.0, and a `!` change merged after 1.0 forces 2.0. - [x] #181 feat(evaluations)!: promote Evaluations from Code to core. After it, `@launchdarkly/ai-server/experimental` exports nothing. - [x] #180 fix(SEC-9507)!: dispatch tools only from the active variation's config.tools - [x] #183 fix(claude-agents)!: honour max_tokens and disable thinking by default - [x] #111 feat(AIC-3439)!: wildcard handlers accept only named providers (merged 29 Oct) - [x] npm placeholders (above) - [ ] Trusted publishers on all five packages (above; needs an npm org admin to confirm) - [ ] #192 fix(AIC-3592): make native LangGraph hand-offs complete (ready for review; spec launchdarkly/ai-sdks-monorepo#118) - [ ] #194 fix(AIC-3573): never send a NativeTool to the model as a function tool (ready for review; spec launchdarkly/ai-sdks-monorepo#116) - [ ] #190 feat(client)!: capture conversation content by default and add configure() (approved; breaking, so it must land before #171) - [ ] #196 test(AIC-3716): run the OpenAI native graph with a tool on the real SDK (test only; optional for the release) Not in 1.0: #187 (Shadow Requests, closed). 1.0 ships with nothing experimental. ## After merging 1. Confirm the regenerated #171 lists 1.0.0 for all 14 packages and its CI is green, then merge it. 2. Install each package from npm into an empty project and import its entry points, in both ESM and CJS (AIC-3673 step 5). 3. Open a PR removing `release-as` straight away. ## Checks - The workflow YAML parses: 14 per-package publish jobs plus release-please and the canary. The new jobs are copies of `release-typesafe`. - No local build was run: this is config only, made through the GitHub API. 🤖 Generated with [Claude Code](https://claude.com/claude-code) [AIC-3673]: https://launchdarkly.atlassian.net/browse/AIC-3673?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Prepares the monorepo for a coordinated **1.0.0** npm release by setting **`release-as: "1.0.0"`** on all 14 workspaces in `release-please-config.json` (the `node-workspace` plugin will align internal dependency ranges when the release PR lands). > > The **`release-please`** workflow now fully covers **`litellm-agents`** and **`litellm-messages`**: they appear in manual `workflow_dispatch` package choices, expose release-please job outputs, and get dedicated **`release-litellm-*`** publish jobs (build, attest, npm publish) matching the other packages—closing the gap where LiteLLM tags could be created without a publish step. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit af882d4. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: andrewklatzke <aklatzke@launchdarkly.com>
jeffdupont
pushed a commit
that referenced
this pull request
Oct 30, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>@launchdarkly/ai-claude-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-claude-agents-0.5.0...@launchdarkly/ai-claude-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them * **claude-agents:** honour the config's max_tokens and disable thinking unless the config asks ([#183](#183)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) * **skills:** give Claude Agents runs the skills attached to their variation ([#136](#136)) ([df16ba0](df16ba0)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) ([d3e7c5f](d3e7c5f)) ### Bug Fixes * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **claude-agents:** honour the config's max_tokens and disable thinking unless the config asks ([#183](#183)) ([f58ce21](f58ce21)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) * **SEC-9507:** dispatch litellm-messages tool calls only to tools offered for the request ([6926e8a](6926e8a)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them ([6926e8a](6926e8a)) * **SEC-9507:** pass handlers only the tools attached to the active variation ([6926e8a](6926e8a)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-claude-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-claude-messages-0.5.0...@launchdarkly/ai-claude-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-langchain-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-langchain-agents-0.5.0...@launchdarkly/ai-langchain-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) ([9371975](9371975)) * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3592:** make native LangGraph hand-offs complete ([#192](#192)) ([025f2ce](025f2ce)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-langchain-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-langchain-messages-0.5.0...@launchdarkly/ai-langchain-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ### Features * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) ([9371975](9371975)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ([15a0d37](15a0d37)) * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3726:** realign three usage IDs with the 1.0 factory names ([#197](#197)) ([fc83c7a](fc83c7a)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-litellm-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-litellm-agents-0.2.0...@launchdarkly/ai-litellm-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-litellm-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-litellm-messages-0.2.0...@launchdarkly/ai-litellm-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **SEC-9507:** enable Claude native tools only when the active variation attaches them * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) * **SEC-9507:** dispatch litellm-messages tool calls only to tools offered for the request ([6926e8a](6926e8a)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them ([6926e8a](6926e8a)) * **SEC-9507:** pass handlers only the tools attached to the active variation ([6926e8a](6926e8a)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-node: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-node-0.4.0...@launchdarkly/ai-node-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **evaluations:** promote Evaluations from Code to core ([#181](#181)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) ### Features * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ([15a0d37](15a0d37)) * **evaluations:** promote Evaluations from Code to core ([#181](#181)) ([676a2ed](676a2ed)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) ([d3e7c5f](d3e7c5f)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-openai-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-openai-agents-0.5.0...@launchdarkly/ai-openai-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **AIC-3687:** rename createOpenAIAgentHandler to createOpenAIAgentsHandler ([#174](#174)) ### Features * **AIC-3687:** rename createOpenAIAgentHandler to createOpenAIAgentsHandler ([#174](#174)) ([cfba4d2](cfba4d2)) * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3726:** realign three usage IDs with the 1.0 factory names ([#197](#197)) ([fc83c7a](fc83c7a)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-openai-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-openai-messages-0.5.0...@launchdarkly/ai-openai-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ([15a0d37](15a0d37)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **AIC-3726:** realign three usage IDs with the 1.0 factory names ([#197](#197)) ([fc83c7a](fc83c7a)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-otel: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-otel-0.3.0...@launchdarkly/ai-otel-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** spans carry launchdarkly.project_id instead of highlight.project_id ### Bug Fixes * **client:** stamp launchdarkly.project_id on the telemetry resource ([#176](#176)) ([93fb412](93fb412)) </details> <details><summary>@launchdarkly/ai-server: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-server-0.5.0...@launchdarkly/ai-server-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them * **evaluations:** export each offline row as one trace ([#175](#175)) * **evaluations:** promote Evaluations from Code to core ([#181](#181)) * **evaluations:** remove default pass rate threshold ([#184](#184)) * **client:** spans carry launchdarkly.project_id instead of highlight.project_id * **AIC-3687:** rename createOpenAIAgentHandler to createOpenAIAgentsHandler ([#174](#174)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) ([9371975](9371975)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ([15a0d37](15a0d37)) * **AIC-3687:** rename createOpenAIAgentHandler to createOpenAIAgentsHandler ([#174](#174)) ([cfba4d2](cfba4d2)) * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) * **client:** point judge spans at the run they graded ([#200](#200)) ([ba9c936](ba9c936)) * **evaluations:** export each offline row as one trace ([#175](#175)) ([05a7eb1](05a7eb1)) * **evaluations:** promote Evaluations from Code to core ([#181](#181)) ([676a2ed](676a2ed)) * **evaluations:** remove default pass rate threshold ([#184](#184)) ([f3a3e37](f3a3e37)) * **skills:** give Claude Agents runs the skills attached to their variation ([#136](#136)) ([df16ba0](df16ba0)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) ([d3e7c5f](d3e7c5f)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **AIC-3601:** stop multi-edge routing text from asking a router for a final answer ([#177](#177)) ([10cce54](10cce54)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) * **client:** stamp launchdarkly.project_id on the telemetry resource ([#176](#176)) ([93fb412](93fb412)) * **evaluations:** update stale test import ([#189](#189)) ([f4d8083](f4d8083)) * **evaluations:** warn once when a scorer returns an invalid score ([#199](#199)) ([f43c888](f43c888)) * **graph:** report a node's own judge scores on its node_done stream event ([#182](#182)) ([1230f02](1230f02)) * **SEC-9507:** dispatch litellm-messages tool calls only to tools offered for the request ([6926e8a](6926e8a)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them ([6926e8a](6926e8a)) * **SEC-9507:** pass handlers only the tools attached to the active variation ([6926e8a](6926e8a)) </details> <details><summary>@launchdarkly/ai-typesafe: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-typesafe-0.2.0...@launchdarkly/ai-typesafe-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-vercel-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-vercel-agents-0.3.0...@launchdarkly/ai-vercel-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-vercel-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-vercel-messages-0.3.0...@launchdarkly/ai-vercel-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > This **release-please** PR cuts **1.0.0** for every workspace package (`@launchdarkly/ai-server`, `@launchdarkly/ai-node`, provider adapters, `@launchdarkly/ai-otel`, `@launchdarkly/ai-typesafe`) by updating `.release-please-manifest.json`, each `package.json` / `src/version.ts`, and adding **1.0.0** changelog sections. Provider packages bump their `@launchdarkly/ai-server` dependency to **^1.0.0**. > > The release notes bundled here flag **breaking** upgrades integrators must handle: **`configure()`** with **conversation content on spans by default** (opt out via `captureContent: false`); **`createHandler`** now returns a **new wrapper** instead of mutating the passed function; renamed factories (**`*MessagesHandler`**, **`createOpenAIAgentsHandler`**); **`LaunchDarklySkillStore`** (was `FDv2SkillStore`); **Evaluations promoted to core** with offline export/trace and threshold behavior changes; telemetry **`launchdarkly.project_id`** (was `highlight.project_id`); richer **`feature_flag`** span events; and **SEC-9507** tool scoping (native tools and handlers only see tools on the active variation). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f87c9a6. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes Evaluations from Code from
@launchdarkly/ai-server/experimentalto the package root. This follows the Agent Skills promotion in #159.Migration
Change the import path. The names, signatures and behavior do not change.
Use
@launchdarkly/ai-nodein place of@launchdarkly/ai-node/experimentalin the same way.This is a breaking change (
!). It bumps the minor version while the package is 0.x.What changed
@launchdarkly/ai-serverroot with the same spelling. No name collides with a current root export.@launchdarkly/ai-nodegets them through its existingexport *.src/experimental.tskeeps its header and exports nothing. The./experimentalsubpath stays in both packages, so a future experimental feature has a place to go. An old import now gives a missing name, not a missing module.initEvaluationsnow reportsclient.initEvaluations, notexperimental.initEvaluations. Queries that count the old ID must also count the new one.api-surface/ai-server.txtas+and marked-inapi-surface/ai-server.experimental.txt. Both API Extractor reports are regenerated. The experimental report is now empty.ae-forgotten-exportwarnings, up from 7. The new ones are evaluations types that are not exported themselves, for exampleEvalToolImplementation,ScorerFnandTransport. The experimental report had the same warnings before, so nothing new is exposed.evaluationsrows are removed fromchangelog-sections, sofeat(evaluations):lands under Features. TheAGENTS.mdscope examples now use a placeholder scope.agents.mdfiles no longer call Evaluations from Code experimental. The generated export blocks inpackages/client/agents.mdare regenerated fromindex.tsandexperimental.ts. The README anchor changed from#evaluations-from-code-experimentalto#evaluations-from-code.Tests
/experimental.sdk-usage.test.tsexpectsclient.initEvaluations.initEvaluationsis on both roots and absent from both/experimentalsubpaths, at run time (CJS and ESM) and at type-check time.Checks
yarn code:check,yarn build,yarn typecheck: passyarn test: pass (client: 1929 passed, 11 skipped)yarn test:integration: 36 passedyarn api:surface,yarn test:api-surface,yarn test:api-breaking,yarn lint:pkg,yarn api:check: passnode scripts/api-breaking.mjs pr --base origin/main: no breaking changes to the public APIConflicts
The open PR #175 edits evaluations code. Merge it first, or rebase it after this PR.
Related PRs
Releases together with:
🤖 Generated with Claude Code
Note
Overview
Evaluations from Code is promoted from
@launchdarkly/ai-server/experimental(and@launchdarkly/ai-node/experimental) to the package root, mirroring the earlier Agent Skills promotion. Consumers should import names likeinitEvaluations,EvalTool,Judge, andScorerfrom@launchdarkly/ai-serveror@launchdarkly/ai-nodeinstead of the/experimentalsubpath; behavior and signatures are unchanged.The experimental barrel now exports nothing (
export {}), while the./experimentalsubpath remains for future features. Public API surface lists, API Extractor reports, andrelease-please-config.jsonare updated accordingly, andinitEvaluationsreports usage asclient.initEvaluationsrather thanexperimental.initEvaluations. Docs,AGENTS.md, and tests (including CommonJS/webpack integration fixtures) assert evaluations live on the root only.Reviewed by Cursor Bugbot for commit 591d138. Bugbot is set up for automated code reviews on this repo. Configure here.