Skip to content

feat(AIC-3495): record which public helpers an application calls - #145

Merged
jeffdupont merged 4 commits into
mainfrom
jdupont/AIC-3495/sdk-usage
Oct 8, 2026
Merged

jeffdupont merged 4 commits into
mainfrom
jdupont/AIC-3495/sdk-usage

Conversation

@jeffdupont

@jeffdupont jeffdupont commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Each public helper sends one $ld:ai:sdk:usage event per client, with the core package name, version, language, the same helper string the TypeScript SDK sends, and the helper's own package name and version (helperPackageName / helperPackageVersion, the same values the package registers with sdk-info; spec in ai-sdks-monorepo#45). Each handler package's name and version now live in its _version.py, and release-please's extra-files point there. The same anonymous context as $ld:ai:sdk:info is used.
  • A call made from inside the SDK does not send another event, so a wrapper reports only itself. SDK code calls non-reporting internal versions (_create_handler, _config, _graph, ...; importable but not in __all__), rather than suppressing reports for the duration of a call. Python also reports client.init_evaluations.
  • A helper called before a client exists is held and flushed on init. shutdown() clears the reported set. A thrown track is swallowed and not retried.

Fixes AIC-3495.

Test plan

  • Client usage tests: one event per helper, no repeat until shutdown, a call before init is sent once a client exists, a thrown track does not fail the helper, and generation payloads are unchanged
  • Each handler helper emits only its own string
  • Graph wrappers and native adapters: invoke / stream on the returned object emits only the wrapper's helper, not client.graph.*
  • A helper called from a user handler or tool reports the same way under config().invoke and config().stream
  • Each handler helper's event carries its own package's PACKAGE_NAME / __version__, including when held until init
  • Full suite: 1463 passed, 11 skipped. Ruff and mypy are clean

Made with Cursor

… calls

Send one $ld:ai:sdk:usage event per helper per client, and only for calls that come from outside the SDK.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0e76fa9. Configure here.

Comment thread packages/client/src/launchdarkly_ai_server/client.py Outdated
jeffdupont and others added 3 commits October 6, 2026 11:28
… a call-scope flag

The ContextVar scope (within_sdk / call_within_sdk) decided who the caller
was at run time, and got it wrong in both directions. Graph wrappers
returned a public GraphInstance, so a later .invoke()/.stream() reported
client.graph.* outside the scope. config().invoke wrapped the user's
handler and tools in the scope, so helpers the application called from
its own code were dropped, while config().stream had no scope and
reported them: the same handler gave different results per path.

Follow the spec (TESTING.md 3.27): each reporting helper has a
non-reporting internal version, and SDK code calls that version.

- Remove within_sdk, call_within_sdk and the ContextVar, and their exports.
- Add _create_handler, _config, _graph (importable, not in __all__) and
  _build_judge_tasks. ConfigInstance/GraphInstance invoke and stream report,
  then delegate to non-reporting _invoke/_stream; _config/_graph return
  subclasses whose invoke/stream do not report. stream still reports at
  call time.
- Package factories report, then call their own _ factory, which calls
  _create_handler. Wrappers and graph wrappers use _config/_graph and the
  _ factories. Public factories get their docstrings back.
- Guard the registry with a lock. A helper is marked reported before
  delivery and the client is read under the lock, so a helper held while
  init runs is not lost and concurrent first calls send once.
- Tests: graph wrapper invoke/stream and native adapter invoke emit only
  their own helper; helpers called from a user handler and tool report
  under both invoke and stream; core rows emit only their own helper;
  pending flush on SDK-key and already-initialized init; the test setter
  does not flush; payload-build failures are swallowed and not retried;
  tool-call and graph payloads carry no usage fields. test_client.py again
  requires generation events to exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ersion

$ld:ai:sdk:usage now carries helperPackageName and helperPackageVersion
next to the core aiSdkName/aiSdkVersion (TESTING.md 3.27, A.14).

Handler packages are versioned separately from the core, so aiSdkVersion
alone does not say which handler release ran a helper. $ld:ai:sdk:info
already lists every package, but it sends them all on one shared context
key, so it cannot be tied to the helpers an application calls. The usage
event is the only place that link can live.

report_usage takes optional package_name and package_version. Each handler
package passes the same values it registers with sdk-info, from a new
_version.py that holds PACKAGE_NAME and __version__ once, so __init__.py
and the handler modules share them without a circular import.
release-please now bumps _version.py. client.* helpers pass neither and
default to the core identity. The dedupe key is still the helper alone,
and held-until-init records keep their package so the flushed event is
complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…C-3495/sdk-usage

Conflict in packages/client/src/launchdarkly_ai_server/evaluations/module.py:
keep main's project_key check and api_key rename, with
report_usage("client.init_evaluations") still the first statement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeffdupont
jeffdupont merged commit 19d2c2b into main Oct 8, 2026
7 checks passed
@jeffdupont
jeffdupont deleted the jdupont/AIC-3495/sdk-usage branch October 8, 2026 21:33
@github-actions github-actions Bot mentioned this pull request Oct 8, 2026
XieX added a commit that referenced this pull request Oct 9, 2026
Brings in #145 (helper usage), #150 (Experimental changelog heading) and
#133 (strip tools from LangChain model constructor kwargs).

lifecycle.py merged without a conflict and keeps both sides: the usage
flush sits beside each flush_ai_sdk_info call, so on the BYOC path it runs
only after telemetry setup succeeds and the client is adopted, and on the
options path only after a successful setup. shutdown() runs
_clear_experimental_state() before client teardown, then resets usage
with sdk-info.

Root surface: add main's report_usage to ROOT_SURFACE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andrewklatzke pushed a commit that referenced this pull request Oct 9, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>launchdarkly-ai-server: 0.3.0</summary>

##
[0.3.0](launchdarkly-ai-server-0.2.4...launchdarkly-ai-server-0.3.0)
(2026-10-09)


### ⚠ BREAKING CHANGES

* **evaluations:** take tools as a list of EvalTool, with inline
definitions
([#105](#105))
* **evaluations:** rename Tool to EvalTool, freeze it, and make
tools.get async
* **evaluations:** take tools as a list of Tool

### Experimental

* **evaluations:** let tools= override a variation's tools
([90b293c](90b293c))
* **evaluations:** Model resolution for remote AI config fetch
([#147](#147))
([0b4df05](0b4df05))
* **evaluations:** rename Tool to EvalTool, freeze it, and make
tools.get async
([4b8a819](4b8a819))
* **evaluations:** take tools as a list of EvalTool, with inline
definitions
([#105](#105))
([cad1547](cad1547))
* **evaluations:** take tools as a list of Tool
([0a92b13](0a92b13))
* **skills:** add experimental Agent Skills under
`launchdarkly_ai_server.experimental.skills`
([dfb519c](dfb519c))


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))
* **client:** warn about unrecognized `init_client` options instead of
ignoring them silently
([dfb519c](dfb519c))


### Bug Fixes

* **client:** `init_client` adopts a client only after telemetry setup
succeeds, and closes a client it built when setup fails
([dfb519c](dfb519c))
* **client:** `shutdown()` releases the global OpenTelemetry tracer
provider the SDK registered, so a later `init_client` exports spans
again
([dfb519c](dfb519c))


### Documentation

* **evaluations:** call the credential an API key, not a token
([7a238e5](7a238e5))
</details>

<details><summary>launchdarkly-ai-python: 0.2.0</summary>

##
[0.2.0](launchdarkly-ai-python-0.1.8...launchdarkly-ai-python-0.2.0)
(2026-10-09)


### ⚠ BREAKING CHANGES

* **evaluations:** take tools as a list of EvalTool, with inline
definitions
([#105](#105))
* **evaluations:** rename Tool to EvalTool, freeze it, and make
tools.get async
* **evaluations:** take tools as a list of Tool

### Experimental

* **evaluations:** rename Tool to EvalTool, freeze it, and make
tools.get async
([4b8a819](4b8a819))
* **evaluations:** take tools as a list of EvalTool, with inline
definitions
([#105](#105))
([cad1547](cad1547))
* **evaluations:** take tools as a list of Tool
([0a92b13](0a92b13))
* **skills:** add experimental Agent Skills under
`launchdarkly_ai_server.experimental.skills`
([dfb519c](dfb519c))


### Features

* **client:** warn about unrecognized `init_client` options instead of
ignoring them silently
([dfb519c](dfb519c))


### Bug Fixes

* **client:** `init_client` adopts a client only after telemetry setup
succeeds, and closes a client it built when setup fails
([dfb519c](dfb519c))
* **client:** `shutdown()` releases the global OpenTelemetry tracer
provider the SDK registered, so a later `init_client` exports spans
again
([dfb519c](dfb519c))
</details>

<details><summary>launchdarkly-ai-claude-agents: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-claude-agents-0.2.4...launchdarkly-ai-claude-agents-0.2.5)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))
</details>

<details><summary>launchdarkly-ai-claude-messages: 0.2.4</summary>

##
[0.2.4](launchdarkly-ai-claude-messages-0.2.3...launchdarkly-ai-claude-messages-0.2.4)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))
</details>

<details><summary>launchdarkly-ai-openai-agents: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-openai-agents-0.2.4...launchdarkly-ai-openai-agents-0.2.5)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))


### Bug Fixes

* **agents:** accept sync tool handlers on invoke and native graphs
([#134](#134))
([4ab5020](4ab5020))
</details>

<details><summary>launchdarkly-ai-openai-messages: 0.2.4</summary>

##
[0.2.4](launchdarkly-ai-openai-messages-0.2.3...launchdarkly-ai-openai-messages-0.2.4)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))
</details>

<details><summary>launchdarkly-ai-langchain-agents: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-langchain-agents-0.2.4...launchdarkly-ai-langchain-agents-0.2.5)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))


### Bug Fixes

* **agents:** accept sync tool handlers on invoke and native graphs
([#134](#134))
([4ab5020](4ab5020))
* **AIC-3506:** strip tools from LangChain model constructor kwargs
([#133](#133))
([77c4979](77c4979))
</details>

<details><summary>launchdarkly-ai-langchain-messages: 0.2.4</summary>

##
[0.2.4](launchdarkly-ai-langchain-messages-0.2.3...launchdarkly-ai-langchain-messages-0.2.4)
(2026-10-09)


### Features

* **AIC-3495:** record which public helpers an application calls
([#145](#145))
([19d2c2b](19d2c2b))


### Bug Fixes

* **AIC-3506:** strip tools from LangChain model constructor kwargs
([#133](#133))
([77c4979](77c4979))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Release-only PR** (Release Please): bumps package versions,
`__version__` strings, `.release-please-manifest.json`, and
`CHANGELOG.md` entries—no application source changes in the diff.
> 
> **`launchdarkly-ai-server` 0.3.0** and **`launchdarkly-ai-python`
0.2.0** document **breaking evaluation tool APIs**: `Tool` → frozen
**`EvalTool`**, tools passed as a **list** (with inline definitions),
and **`tools.get` is async**. Also ships experimental **evaluations**
work (remote AI config model resolution, `tools=` override) and
**`launchdarkly_ai_server.experimental.skills`**, plus
**`init_client`/`shutdown`** telemetry lifecycle fixes and warnings for
unrecognized `init_client` options.
> 
> **Provider packages** (0.2.4–0.2.5) mainly add **SDK usage telemetry**
for which public helpers apps call (**AIC-3495**), with **LangChain**
fixes (strip `tools` from model ctor kwargs; accept **sync** tool
handlers on invoke/graphs) and the same sync-handler fix on **OpenAI
agents**.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
c2a6a78. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants