Repository navigation
feat(AIC-3495): record which public helpers an application calls - #145
Merged
Merged
Conversation
… calls Send one $ld:ai:sdk:usage event per helper per client, and only for calls that come from outside the SDK. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0e76fa9. Configure here.
… a call-scope flag The ContextVar scope (within_sdk / call_within_sdk) decided who the caller was at run time, and got it wrong in both directions. Graph wrappers returned a public GraphInstance, so a later .invoke()/.stream() reported client.graph.* outside the scope. config().invoke wrapped the user's handler and tools in the scope, so helpers the application called from its own code were dropped, while config().stream had no scope and reported them: the same handler gave different results per path. Follow the spec (TESTING.md 3.27): each reporting helper has a non-reporting internal version, and SDK code calls that version. - Remove within_sdk, call_within_sdk and the ContextVar, and their exports. - Add _create_handler, _config, _graph (importable, not in __all__) and _build_judge_tasks. ConfigInstance/GraphInstance invoke and stream report, then delegate to non-reporting _invoke/_stream; _config/_graph return subclasses whose invoke/stream do not report. stream still reports at call time. - Package factories report, then call their own _ factory, which calls _create_handler. Wrappers and graph wrappers use _config/_graph and the _ factories. Public factories get their docstrings back. - Guard the registry with a lock. A helper is marked reported before delivery and the client is read under the lock, so a helper held while init runs is not lost and concurrent first calls send once. - Tests: graph wrapper invoke/stream and native adapter invoke emit only their own helper; helpers called from a user handler and tool report under both invoke and stream; core rows emit only their own helper; pending flush on SDK-key and already-initialized init; the test setter does not flush; payload-build failures are swallowed and not retried; tool-call and graph payloads carry no usage fields. test_client.py again requires generation events to exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ersion $ld:ai:sdk:usage now carries helperPackageName and helperPackageVersion next to the core aiSdkName/aiSdkVersion (TESTING.md 3.27, A.14). Handler packages are versioned separately from the core, so aiSdkVersion alone does not say which handler release ran a helper. $ld:ai:sdk:info already lists every package, but it sends them all on one shared context key, so it cannot be tied to the helpers an application calls. The usage event is the only place that link can live. report_usage takes optional package_name and package_version. Each handler package passes the same values it registers with sdk-info, from a new _version.py that holds PACKAGE_NAME and __version__ once, so __init__.py and the handler modules share them without a circular import. release-please now bumps _version.py. client.* helpers pass neither and default to the core identity. The dedupe key is still the helper alone, and held-until-init records keep their package so the flushed event is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…C-3495/sdk-usage
Conflict in packages/client/src/launchdarkly_ai_server/evaluations/module.py:
keep main's project_key check and api_key rename, with
report_usage("client.init_evaluations") still the first statement.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andrewklatzke
approved these changes
Oct 8, 2026
Merged
XieX
added a commit
that referenced
this pull request
Oct 9, 2026
Brings in #145 (helper usage), #150 (Experimental changelog heading) and #133 (strip tools from LangChain model constructor kwargs). lifecycle.py merged without a conflict and keeps both sides: the usage flush sits beside each flush_ai_sdk_info call, so on the BYOC path it runs only after telemetry setup succeeds and the client is adopted, and on the options path only after a successful setup. shutdown() runs _clear_experimental_state() before client teardown, then resets usage with sdk-info. Root surface: add main's report_usage to ROOT_SURFACE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andrewklatzke
pushed a commit
that referenced
this pull request
Oct 9, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>launchdarkly-ai-server: 0.3.0</summary> ## [0.3.0](launchdarkly-ai-server-0.2.4...launchdarkly-ai-server-0.3.0) (2026-10-09) ### ⚠ BREAKING CHANGES * **evaluations:** take tools as a list of EvalTool, with inline definitions ([#105](#105)) * **evaluations:** rename Tool to EvalTool, freeze it, and make tools.get async * **evaluations:** take tools as a list of Tool ### Experimental * **evaluations:** let tools= override a variation's tools ([90b293c](90b293c)) * **evaluations:** Model resolution for remote AI config fetch ([#147](#147)) ([0b4df05](0b4df05)) * **evaluations:** rename Tool to EvalTool, freeze it, and make tools.get async ([4b8a819](4b8a819)) * **evaluations:** take tools as a list of EvalTool, with inline definitions ([#105](#105)) ([cad1547](cad1547)) * **evaluations:** take tools as a list of Tool ([0a92b13](0a92b13)) * **skills:** add experimental Agent Skills under `launchdarkly_ai_server.experimental.skills` ([dfb519c](dfb519c)) ### Features * **AIC-3495:** record which public helpers an application calls ([#145](#145)) ([19d2c2b](19d2c2b)) * **client:** warn about unrecognized `init_client` options instead of ignoring them silently ([dfb519c](dfb519c)) ### Bug Fixes * **client:** `init_client` adopts a client only after telemetry setup succeeds, and closes a client it built when setup fails ([dfb519c](dfb519c)) * **client:** `shutdown()` releases the global OpenTelemetry tracer provider the SDK registered, so a later `init_client` exports spans again ([dfb519c](dfb519c)) ### Documentation * **evaluations:** call the credential an API key, not a token ([7a238e5](7a238e5)) </details> <details><summary>launchdarkly-ai-python: 0.2.0</summary> ## [0.2.0](launchdarkly-ai-python-0.1.8...launchdarkly-ai-python-0.2.0) (2026-10-09) ### ⚠ BREAKING CHANGES * **evaluations:** take tools as a list of EvalTool, with inline definitions ([#105](#105)) * **evaluations:** rename Tool to EvalTool, freeze it, and make tools.get async * **evaluations:** take tools as a list of Tool ### Experimental * **evaluations:** rename Tool to EvalTool, freeze it, and make tools.get async ([4b8a819](4b8a819)) * **evaluations:** take tools as a list of EvalTool, with inline definitions ([#105](#105)) ([cad1547](cad1547)) * **evaluations:** take tools as a list of Tool ([0a92b13](0a92b13)) * **skills:** add experimental Agent Skills under `launchdarkly_ai_server.experimental.skills` ([dfb519c](dfb519c)) ### Features * **client:** warn about unrecognized `init_client` options instead of ignoring them silently ([dfb519c](dfb519c)) ### Bug Fixes * **client:** `init_client` adopts a client only after telemetry setup succeeds, and closes a client it built when setup fails ([dfb519c](dfb519c)) * **client:** `shutdown()` releases the global OpenTelemetry tracer provider the SDK registered, so a later `init_client` exports spans again ([dfb519c](dfb519c)) </details> <details><summary>launchdarkly-ai-claude-agents: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-claude-agents-0.2.4...launchdarkly-ai-claude-agents-0.2.5) (2026-10-09) ### Features * **AIC-3495:** record which public helpers an application calls ([#145](#145)) ([19d2c2b](19d2c2b)) </details> <details><summary>launchdarkly-ai-claude-messages: 0.2.4</summary> ## [0.2.4](launchdarkly-ai-claude-messages-0.2.3...launchdarkly-ai-claude-messages-0.2.4) (2026-10-09) ### Features * **AIC-3495:** record which public helpers an application calls ([#145](#145)) ([19d2c2b](19d2c2b)) </details> <details><summary>launchdarkly-ai-openai-agents: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-openai-agents-0.2.4...launchdarkly-ai-openai-agents-0.2.5) (2026-10-09) ### Features * **AIC-3495:** record which public helpers an application calls ([#145](#145)) ([19d2c2b](19d2c2b)) ### Bug Fixes * **agents:** accept sync tool handlers on invoke and native graphs ([#134](#134)) ([4ab5020](4ab5020)) </details> <details><summary>launchdarkly-ai-openai-messages: 0.2.4</summary> ## [0.2.4](launchdarkly-ai-openai-messages-0.2.3...launchdarkly-ai-openai-messages-0.2.4) (2026-10-09) ### Features * **AIC-3495:** record which public helpers an application calls ([#145](#145)) ([19d2c2b](19d2c2b)) </details> <details><summary>launchdarkly-ai-langchain-agents: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-langchain-agents-0.2.4...launchdarkly-ai-langchain-agents-0.2.5) (2026-10-09) ### Features * **AIC-3495:** record which public helpers an application calls ([#145](#145)) ([19d2c2b](19d2c2b)) ### Bug Fixes * **agents:** accept sync tool handlers on invoke and native graphs ([#134](#134)) ([4ab5020](4ab5020)) * **AIC-3506:** strip tools from LangChain model constructor kwargs ([#133](#133)) ([77c4979](77c4979)) </details> <details><summary>launchdarkly-ai-langchain-messages: 0.2.4</summary> ## [0.2.4](launchdarkly-ai-langchain-messages-0.2.3...launchdarkly-ai-langchain-messages-0.2.4) (2026-10-09) ### Features * **AIC-3495:** record which public helpers an application calls ([#145](#145)) ([19d2c2b](19d2c2b)) ### Bug Fixes * **AIC-3506:** strip tools from LangChain model constructor kwargs ([#133](#133)) ([77c4979](77c4979)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > **Release-only PR** (Release Please): bumps package versions, `__version__` strings, `.release-please-manifest.json`, and `CHANGELOG.md` entries—no application source changes in the diff. > > **`launchdarkly-ai-server` 0.3.0** and **`launchdarkly-ai-python` 0.2.0** document **breaking evaluation tool APIs**: `Tool` → frozen **`EvalTool`**, tools passed as a **list** (with inline definitions), and **`tools.get` is async**. Also ships experimental **evaluations** work (remote AI config model resolution, `tools=` override) and **`launchdarkly_ai_server.experimental.skills`**, plus **`init_client`/`shutdown`** telemetry lifecycle fixes and warnings for unrecognized `init_client` options. > > **Provider packages** (0.2.4–0.2.5) mainly add **SDK usage telemetry** for which public helpers apps call (**AIC-3495**), with **LangChain** fixes (strip `tools` from model ctor kwargs; accept **sync** tool handlers on invoke/graphs) and the same sync-handler fix on **OpenAI agents**. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit c2a6a78. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
$ld:ai:sdk:usageevent per client, with the core package name, version, language, the same helper string the TypeScript SDK sends, and the helper's own package name and version (helperPackageName/helperPackageVersion, the same values the package registers with sdk-info; spec in ai-sdks-monorepo#45). Each handler package's name and version now live in its_version.py, and release-please'sextra-filespoint there. The same anonymous context as$ld:ai:sdk:infois used._create_handler,_config,_graph, ...; importable but not in__all__), rather than suppressing reports for the duration of a call. Python also reportsclient.init_evaluations.shutdown()clears the reported set. A throwntrackis swallowed and not retried.Fixes AIC-3495.
Test plan
trackdoes not fail the helper, and generation payloads are unchangedclient.graph.*config().invokeandconfig().streamPACKAGE_NAME/__version__, including when held until initMade with Cursor