feat(AIC-3439): let wildcard handlers accept only named providers

[andrewklatzke]

Summary

• A wildcard handler can take a providers list, so a LangChain adapter used only for Bedrock accepts config.provider.name === "Bedrock" and no other provider.
• Exact [provider, mode] still wins. Among scoped wildcards that contain the name, the shortest list wins, then the earlier registration. An unscoped * is the fallback.
• Registry identity is providesFor plus the provider set, so two scoped wildcards of the same mode can be registered together. Judges and graph selection use the same order.

Test plan

• Vitest on the client, langchain-messages, and langchain-agents suites
• Biome and tsc on the touched packages
• Integration success and failure for agent, graph, streaming, graph-streaming, langchain, langchain-messages, langchain-agents, and native-graph-langchain
• TypeScript judge example still looks for examples/judge-worker.js while only judge-worker.ts is present, so that case does not start

Made with Cursor

[andrewklatzke]

Human message:

I think this is a "nice to have" not necessarily "must do it". I'm thinking of situations where, for example, a user might want to use the Google ADK and scope it only to gemini (create_google_adk_handler(providers=['Gemini'])). Another example might be folks wanting to use langchain but only for Bedrock so they don't have to deal with boto.

You can technically achieve this now by properly including the right subset of handlers, but this makes it more obvious/colocates it with the code if you want to scope them down.

tldr; if we think this is confusing or don't want it, we can close it

[jeffdupont]

Selection is right in config(), in graph node selection, and in the three judge entry points. The registry keeps scoped wildcards apart the way the PR body says. One judge path skips the allowlist (1 below), and the registry orders handlers differently from Python #136 (2). Those two need fixing before this merges.

At 6cd4ce42, build, tsc, Biome and Sherif are clean, and yarn test gives 1177 passed. I trial-merged it onto today's main (#73, #68 and the rest) and got no conflicts. The merged tree gives 1325 passed, with typecheck and Biome clean. The #73 parameter allowlists in langchain-messages and langchain-agents handler.ts merge without trouble next to the new providers option, and the factory signatures are unchanged.

I probed selectModeHandler, the registry and the judge functions directly:

• An exact handler beats a scoped wildcard, and a scoped wildcard beats an unscoped one, whatever the registration order.
• The shortest matching list wins. On equal lengths the earlier handler wins.
• Names match exactly, so bedrock doesn't match ['Bedrock']. Exact handlers follow the same rule (client.ts:39-44).
• [], [''] and [' '] throw at createHandler, and the message mentions providers.
• If a scoped wildcard doesn't list the provider, config() throws Handler for provider OpenAI not found. That's the same message as having no handler at all, and Python gives the same string.
• Two scoped wildcards of one mode both stay registered. Registering the same set again, in either order, warns and replaces. ['Bedrock', 'Bedrock'] counts as the same identity as ['Bedrock']. compose(a, b) lets b win on a matching set and keeps different sets.
• runJudges, buildJudgeTasks and runJudge all pick the shortest list. Each falls back to a scoped agent handler with collapseMessages: true, and each skips a judge whose provider isn't in any list.

I tried 24 mutations, and 20 of them fail a test. The 4 that get through are in the inline comments.

1. The graph-level judge ignores the allowlist. graph.ts:796-807 calls runJudges without handlers, so runJudges takes the judgeHandler = handler branch and never checks the provider (judges.ts:157-158). I ran graph().invoke() with one handler, ['*', 'messages'] scoped to ['Bedrock'], and a graphJudge whose provider is Anthropic. The Bedrock-only handler ran the Anthropic judge. Adding an Anthropic-scoped handler didn't help, because the root handler still ran it. The missing handlers argument predates this PR. Before, though, it only affected which of two acceptable handlers ran. Now it lets a scoped handler run a provider it was set up to refuse. Python passes handlers=resolved_handlers on the same call (graph.py:1016-1024 and :1238-1250). Please pass handlers: judgeHandlers and add a graphJudge test with a provider outside the list.
2. Re-registering moves a handler to the end, but Python replaces it in place. registry.ts:29 filters out the old entry and pushes the new one. Python's registry.py:74 assigns into the same index. Ties go to the earlier registration, so the two SDKs now pick different handlers. As a repro, register ['Bedrock', 'Gemini'], then ['Bedrock', 'Anthropic'], then ['Bedrock', 'Gemini'] again. JS ends up with [BA, BG], and a Bedrock config runs BA. Python keeps [BG, BA] and runs BG. compose shows the same split: JS gives [aU, bB, bA], Python [bB, aU, bA]. Before this PR it didn't matter, because only one wildcard per mode could be registered. Monorepo #39 doesn't say which is right. I'd go with in place, so a replacement takes the slot of what it replaces. That needs a line in #39 §3.6 and a test in both SDKs.
3. Repeated names count toward list length. The registry key collapses ['Bedrock', 'Bedrock'] to ['Bedrock'] (registry.ts:6), but bestWildcard compares allow.length (utils.ts:92). So ['Bedrock', 'Bedrock'] ties with ['Bedrock', 'Anthropic'] and loses on registration order, even though the registry treats it as the narrower list. Python does the same (utils.py:97). Deduping in createHandler fixes this, and then #39 can say "fewest distinct names".
4. createHandler keeps the caller's array. utils.ts:60 stores the array itself, not a copy. If the caller later pushes 'OpenAI' onto it, an OpenAI config routes to that handler, and its registry identity changes too. Python copies into a tuple. Object.freeze([...new Set(providers)]) would fix 3 and 4 together.

Questions and smaller items

• The createHandler signature freezes at 1.0. providers is the fifth positional argument, so a caller writes createHandler(pf, fn, undefined, undefined, ['Bedrock']), and #39's A.14 has to explain the undefineds. Python made it keyword-only. An options object ({ stream, captureContent, providers }) is harder to get wrong, but we can't switch to one after 1.0 without a major release. I'm fine either way, but let's decide now. The name and type match Python (providers, readonly string[], and Python's Sequence[str] stored as a tuple). The selection helpers aren't exported from index.ts, which is right.
• API report. main has no API Extractor setup yet, because it's on #71's branch. I trial-merged this PR into #71 (c27005cc). There were no conflicts, but api:check fails on two lines: createHandler(..., providers?: readonly string[]) and ProviderHandler.providers?. Whichever of the two lands second needs yarn api:update.
• Whitespace. [' Bedrock'] is accepted and never matches anything. Python's docstring says surrounding whitespace is rejected, but its code accepts it too. Either both should reject name !== name.trim(), or the claim should go.
• Case. Routing is exact, which fits exact handlers and #39. The LangChain handlers lower-case the provider themselves (langchain-messages handler.ts:70 and :198), though. So a config with provider.name = 'bedrock' would take the Bedrock model path, but it won't route to ['Bedrock']. That's fine if LaunchDarkly always serves the canonical spelling, which I haven't checked.
• Exact handlers that have a list. Routing ignores the list on an exact handler, but the registry key still includes it (registry.ts:5-6). So ['OpenAI', 'messages'] with a list and without one both stay registered, and the first one wins. #39 says two exact handlers collide. Python does the same thing. I'd leave providers out of the key when providesFor[0] !== '*'.
• Calling createHandler twice on one function. createHandler changes the function in place, so two calls with different lists return the same object, holding the last list. A later unscoped call on the same function keeps the old list, because utils.ts:60 only sets the property when one is passed. captureContent already works this way. The LangChain factories build a new closure each time, so they aren't affected. A JSDoc note is enough.
• The priority comment at judges.ts:132-138 doesn't mention scoped lists. Nit.

The judge example (from the PR body)

examples/judge.ts:56 resolves ./judge-worker.js, but only examples/judge-worker.ts exists. Both files date from the initial commit (977dba8), and this PR doesn't touch examples/, so the problem already exists on main. I didn't run it.

Parity with python-ai-sdk #136

On my probe cases, the two SDKs agree on selection order, ties, case, a missing provider (undefined / None), empty and blank lists, the error strings and set identity. They differ in five places:

• re-registration position (2)
• the graph-level judge (1)
• copying the caller's list (4)
• identity of ['a,b'] vs ['a', 'b']: JS joins with commas, so these collide and one replaces the other (registry.ts:6), while Python's frozenset keeps both
• a bare string or a non-string name: Python raises ValueError mentioning providers, while JS throws a TypeError (providers.some is not a function). That's acceptable, since the TS types already rule it out.

Python also routes judges in its evaluations runner. JS has no runner, and #39's A.14 says so.

[jeffdupont]

Node selection is right. The graphJudge call at :796-807 doesn't pass handlers, though, so runJudges runs the judge on the root handler without checking the provider (judges.ts:157-158). I set up a single ['*', 'messages'] handler scoped to ['Bedrock'] and an Anthropic graphJudge, and graph().invoke() called the scoped handler for the judge. Python passes handlers=resolved_handlers there. Could you pass handlers: judgeHandlers and add a test?

[jeffdupont]

This removes the old entry and appends the new one. Python's registry replaces in the same slot (registry.py:74). Since ties now go to the earlier registration, the SDKs pick differently. For example, register BG, then BA, then BG again: JS runs BA for Bedrock and Python runs BG. compose differs the same way. Can we pick one, in place I'd say, and write it into #39 §3.6?

[jeffdupont]

Two small identity things. (a) Joining with , makes ['a,b'] and ['a', 'b'] the same key, so one replaces the other. Python's frozenset keeps both. JSON.stringify of the sorted set avoids that. (b) The key includes providers on exact handlers too, so ['OpenAI', 'messages'] with a list and without one both stay registered, and the first one wins. #39 says two exact handlers collide. I'd only add the list when providesFor[0] === '*'.

[jeffdupont]

allow.length counts repeated names, but the registry treats ['Bedrock', 'Bedrock'] as ['Bedrock']. As a result ['Bedrock', 'Bedrock'] ties with ['Bedrock', 'Anthropic'] and loses on order. Deduping in createHandler fixes it. Python has the same issue (utils.py:97).

[jeffdupont]

This stores the caller's array. If the caller pushes 'OpenAI' onto it after creating the handler, OpenAI configs route here and the registry key changes. Python copies into a tuple. Object.freeze([...new Set(providers)])? Separately: since this function is changed in place, a later createHandler on the same function without providers keeps the old list. captureContent already behaves that way, so a JSDoc note is enough.

[jeffdupont]

[' Bedrock'] gets through and never matches. Python's docstring says surrounding whitespace is rejected, but neither SDK actually rejects it. Could both check name !== name.trim(), or should we drop the claim from Python and #39?

[jeffdupont]

This test only covers the OpenAI half of #39's "unscoped wildcard is the fallback" bullet. If I change bestWildcard to return unscoped ?? bestScoped?.handler, so unscoped beats scoped, all 445 client tests still pass. Could you add the Bedrock half: [scoped(['Bedrock']), unscoped] with a Bedrock variation should run the scoped handler?

[jeffdupont]

If I put back the old providesFor[0] === '*' coverage check here, the tests still pass. The error then becomes ... with mode messages not found, but the test at :393 only matches /OpenAI/. Could it assert the full Handler for provider OpenAI not found, since that's the string both SDKs give?

[jeffdupont]

No test covers parentCovers with a scoped parent. If I change it to providesFor[0] === '*', the suite passes. A test where the parent is a Bedrock-scoped wildcard, the judge is Anthropic, and no other handler applies would cover it.

[jeffdupont]

runJudge, the async task runner, isn't tested with a scoped handler. If I swap this back to the old any-* match, nothing fails. #39 §3.13 wants the allowlist honoured on the prepareAsyncJudge path too. My probe returns null here for an Anthropic task with only a Bedrock-scoped handler, so the behaviour is right and only the test is missing.