Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -392,7 +392,7 @@ Returns:
**Behavior when `.invoke()` is called:**

1. Fetches and validates the `AiConfigRep` variation from LaunchDarkly using `args.key` and the supplied `context`. Throws if the variation is disabled or invalid.
2. Selects the handler by matching on `[config.provider.name, normalized mode]`. Selection priority: (a) exact provider match, (b) wildcard `['*', mode]` fallback for multi-provider adapters (e.g. LangChain). Throws if no matching handler is found.
2. Selects the handler by matching on `[config.provider.name, normalized mode]`. Selection priority: (a) exact provider match, ignoring any `providers` list; (b) a wildcard `['*', mode]` whose `providers` list contains the name, shortest list first and earlier registration on a tie; (c) an unscoped wildcard `['*', mode]`. Throws if no matching handler is found.
3. Invokes the selected handler with the config, user input, tool handlers, variables, and history. The `context` passed to `.invoke()` is automatically merged into `variables` under the key `ldContext`, so templates can reference `{{ldContext.key}}`, `{{ldContext.email}}`, etc. Any caller-supplied `ldContext` variable is silently overwritten. If `history` is provided, it is passed to the handler as the 5th positional argument — messages-mode handlers splice it into the messages array; agent-mode handlers append it to the system prompt.
4. Emits LaunchDarkly telemetry events: duration (`$ld:ai:duration:total`), outcome (`$ld:ai:generation:success` / `$ld:ai:generation:error`), and token counts (`$ld:ai:tokens:*`).
5. If `judgeConfiguration` is present:
Expand Down Expand Up @@ -425,7 +425,7 @@ const registry = new Registry({
});
```

`.register(config)` can be called multiple times to add more handlers or tools. Duplicate `providesFor` keys or tool names produce a warning and the last registration wins.
`.register(config)` can be called multiple times to add more handlers or tools. Handler identity is `providesFor` plus the provider set: two wildcards of the same mode with different `providers` lists both stay registered, while the same set (order-independent; repeated names collapse) or two unscoped wildcards warn and the last registration wins. Duplicate tool names do the same.

`globalRegistry` is a pre-constructed singleton `Registry` instance.

Expand Down Expand Up @@ -476,7 +476,7 @@ A handler is a **callable that also carries metadata**. It must be both invokabl
providesFor: [providerName: string, mode: 'agent' | 'messages']
```

The `providesFor` tuple is how `config()` routes to the correct handler at runtime. The mode element must exactly match the normalized `meta.mode`. The provider element must either exactly match `config.provider.name` **or** be the wildcard `'*'`. A wildcard handler is chosen only when no handler with an exact provider name matches — it acts as a fallback for multi-provider adapters like LangChain. **Always attach `providesFor` using `createHandler` rather than direct property assignment.**
The `providesFor` tuple is how `config()` routes to the correct handler at runtime. The mode element must exactly match the normalized `meta.mode`. The provider element must either exactly match `config.provider.name` **or** be the wildcard `'*'`. A wildcard handler is chosen only when no handler with an exact provider name matches — it acts as a fallback for multi-provider adapters like LangChain. Pass `providers: ['Bedrock']` as the fifth argument to `createHandler` to limit which `config.provider.name` values that wildcard accepts; names must match exactly, and an empty list or a blank name is rejected. Omit `providers` to leave the property unset, which accepts every provider. **Always attach `providesFor` using `createHandler` rather than direct property assignment.**

### Factory Function

Expand Down
86 changes: 86 additions & 0 deletions packages/client/src/__tests__/client.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,92 @@ describe('config() — multi-handler routing', () => {
await expect(config({ key: 'flag', handler: [wildcardHandler] }).invoke('q', mockContext)).rejects.toThrow();
});

const scoped = (providers: string[], output: string): ProviderHandler => {
const handler: ProviderHandler = vi.fn().mockResolvedValue({ output, usage: {} });
handler.providesFor = ['*', 'messages'];
handler.providers = providers;
return handler;
};

it('selects a scoped wildcard when the provider is listed', async () => {
(extractVariation as ReturnType<typeof vi.fn>).mockResolvedValue({
config: { ...mockConfig, provider: { name: 'Bedrock' } },
meta: mockMeta,
});
const handler = scoped(['Bedrock'], 'from bedrock');
const result = await config({ key: 'flag', handler: [handler] }).invoke('q', mockContext);
expect(result.response).toBe('from bedrock');
});

it('rejects a scoped wildcard when the provider is not listed', async () => {
const handler = scoped(['Bedrock'], 'from bedrock');
await expect(config({ key: 'flag', handler: [handler] }).invoke('q', mockContext)).rejects.toThrow(/OpenAI/);
expect(handler).not.toHaveBeenCalled();
});

it('routes each provider to its own scoped wildcard', async () => {
(extractVariation as ReturnType<typeof vi.fn>).mockResolvedValue({
config: { ...mockConfig, provider: { name: 'Anthropic' } },
meta: mockMeta,
});
const bedrock = scoped(['Bedrock'], 'bedrock');
const anthropic = scoped(['Anthropic'], 'anthropic');
const result = await config({ key: 'flag', handler: [bedrock, anthropic] }).invoke('q', mockContext);
expect(result.response).toBe('anthropic');
expect(bedrock).not.toHaveBeenCalled();
});

it('prefers an exact provider handler over a scoped wildcard', async () => {
const wildcard = scoped(['OpenAI'], 'scoped');
const explicit = makeHandler('exact');
const result = await config({ key: 'flag', handler: [wildcard, explicit] }).invoke('q', mockContext);
expect(result.response).toBe('exact');
expect(wildcard).not.toHaveBeenCalled();
});

it('falls back to an unscoped wildcard when no scoped list matches', async () => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test only covers the OpenAI half of #39's "unscoped wildcard is the fallback" bullet. If I change bestWildcard to return unscoped ?? bestScoped?.handler, so unscoped beats scoped, all 445 client tests still pass. Could you add the Bedrock half: [scoped(['Bedrock']), unscoped] with a Bedrock variation should run the scoped handler?

const bedrock = scoped(['Bedrock'], 'bedrock');
const unscoped: ProviderHandler = vi.fn().mockResolvedValue({ output: 'any', usage: {} });
unscoped.providesFor = ['*', 'messages'];
const result = await config({ key: 'flag', handler: [bedrock, unscoped] }).invoke('q', mockContext);
expect(result.response).toBe('any');
expect(bedrock).not.toHaveBeenCalled();
});

it('prefers the shorter provider list when both match', async () => {
(extractVariation as ReturnType<typeof vi.fn>).mockResolvedValue({
config: { ...mockConfig, provider: { name: 'Bedrock' } },
meta: mockMeta,
});
const wide = scoped(['Bedrock', 'Anthropic'], 'wide');
const narrow = scoped(['Bedrock'], 'narrow');
const result = await config({ key: 'flag', handler: [wide, narrow] }).invoke('q', mockContext);
expect(result.response).toBe('narrow');
expect(wide).not.toHaveBeenCalled();
});

it('keeps the earlier handler when overlapping lists are the same length', async () => {
(extractVariation as ReturnType<typeof vi.fn>).mockResolvedValue({
config: { ...mockConfig, provider: { name: 'Bedrock' } },
meta: mockMeta,
});
const earlier = scoped(['Bedrock', 'Gemini'], 'earlier');
const later = scoped(['Bedrock', 'Anthropic'], 'later');
const result = await config({ key: 'flag', handler: [earlier, later] }).invoke('q', mockContext);
expect(result.response).toBe('earlier');
expect(later).not.toHaveBeenCalled();
});

it('a scoped wildcard does not match a different mode', async () => {
(extractVariation as ReturnType<typeof vi.fn>).mockResolvedValue({
config: { ...mockConfig, provider: { name: 'Bedrock' } },
meta: { ...mockMeta, mode: 'agent' as const },
});
const handler = scoped(['Bedrock'], 'bedrock');
await expect(config({ key: 'flag', handler: [handler] }).invoke('q', mockContext)).rejects.toThrow(/Bedrock/);
expect(handler).not.toHaveBeenCalled();
});

it('throws a mode-specific error when provider matches but mode does not', async () => {
const messagesHandler = makeHandler();
(extractVariation as ReturnType<typeof vi.fn>).mockResolvedValue({
Expand Down
9 changes: 9 additions & 0 deletions packages/client/src/__tests__/graph.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,15 @@ describe('graph().invoke()', () => {
await expect(graph('graph-flag', {}).invoke('hi', mockContext)).rejects.toThrow(/handlers/i);
});

it('does not run a scoped wildcard for a provider outside its list', async () => {
setupTwoNodeGraph();
const handler = makeHandler();
handler.providesFor = ['*', 'messages'];
handler.providers = ['Bedrock'];
await expect(graph('graph-flag', { handlers: [handler] }).invoke('hi', mockContext)).rejects.toThrow(/OpenAI/);
expect(handler).not.toHaveBeenCalled();
});

it('visits all nodes and accumulates usage', async () => {
setupTwoNodeGraph();
const handler = makeHandler();
Expand Down
27 changes: 27 additions & 0 deletions packages/client/src/__tests__/judges.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,33 @@ describe('runJudges', () => {
expect(callArgs.config).toBe(mockJudgeConfig);
});

it('skips a scoped wildcard when the judge provider is outside its list', async () => {
const scoped: ProviderHandler = vi.fn().mockResolvedValue({ output: '{"score":1,"reasoning":"ok"}', usage: {} });
scoped.providesFor = ['*', 'agent'];
scoped.providers = ['Bedrock'];
const parent = makeHandler();
mockExtractVariation.mockResolvedValue({
config: { ...mockJudgeConfig, provider: { name: 'Anthropic' } },
meta: mockJudgeMeta,
});
const config = {
model: { name: 'gpt-4o' },
provider: { name: 'OpenAI' },
instructions: 'Be helpful.',
judgeConfiguration: { judges: [{ key: 'judge-flag', samplingRate: 1 }] },
};
await runJudges({
config,
userContext: mockContext,
handler: parent,
handlers: [scoped],
userInput: 'hello',
llmResponse: 'world',
baseTrackData,
});
expect(mockExecuteAndTrack).not.toHaveBeenCalled();
});

it('skips a judge when no compatible handler is found (mismatched provider, no wildcard)', async () => {
const openaiHandler = makeHandler(); // ['OpenAI', 'messages']
mockExtractVariation.mockResolvedValue({
Expand Down
73 changes: 72 additions & 1 deletion packages/client/src/__tests__/registry.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@ import type { ProviderHandler } from '../types.js';

// ─── helpers ──────────────────────────────────────────────────────────────────

const makeHandler = (provider: string, mode: 'agent' | 'messages'): ProviderHandler => {
const makeHandler = (provider: string, mode: 'agent' | 'messages', providers?: readonly string[]): ProviderHandler => {
const h: ProviderHandler = async () => ({ output: 'ok' });
h.providesFor = [provider, mode];
if (providers) h.providers = providers;
return h;
};

Expand Down Expand Up @@ -47,6 +48,57 @@ describe('Registry', () => {
warn.mockRestore();
});

it('keeps scoped wildcards with different provider lists', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
const r = new Registry();
const bedrock = makeHandler('*', 'agent', ['Bedrock']);
const anthropic = makeHandler('*', 'agent', ['Anthropic']);
r.register({ handlers: [bedrock, anthropic] });
expect(r.handlers).toEqual([bedrock, anthropic]);
expect(warn).not.toHaveBeenCalled();
} finally {
warn.mockRestore();
}
});

it('keeps an unscoped wildcard beside a scoped one', () => {
const r = new Registry();
const unscoped = makeHandler('*', 'agent');
const scoped = makeHandler('*', 'agent', ['Bedrock']);
r.register({ handlers: [unscoped, scoped] });
expect(r.handlers).toEqual([unscoped, scoped]);
});

it('warns and replaces the same provider set regardless of order', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
const r = new Registry();
const first = makeHandler('*', 'agent', ['Anthropic', 'Bedrock']);
const second = makeHandler('*', 'agent', ['Bedrock', 'Anthropic']);
r.register({ handlers: [first] });
r.register({ handlers: [second] });
expect(r.handlers).toEqual([second]);
expect(warn).toHaveBeenCalledOnce();
} finally {
warn.mockRestore();
}
});

it('treats a repeated provider name as the same identity', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
const r = new Registry();
const first = makeHandler('*', 'agent', ['Bedrock']);
const second = makeHandler('*', 'agent', ['Bedrock', 'Bedrock']);
r.register({ handlers: [first, second] });
expect(r.handlers).toEqual([second]);
expect(warn).toHaveBeenCalledOnce();
} finally {
warn.mockRestore();
}
});

it('always appends handlers without providesFor (no dedup)', () => {
const r = new Registry();
const h: ProviderHandler = async () => ({ output: '' });
Expand Down Expand Up @@ -105,6 +157,25 @@ describe('compose', () => {
warn.mockRestore();
});

it('keeps scoped wildcards with different lists', () => {
const bedrock = makeHandler('*', 'agent', ['Bedrock']);
const anthropic = makeHandler('*', 'agent', ['Anthropic']);
const c = compose(new Registry({ handlers: [bedrock] }), new Registry({ handlers: [anthropic] }));
expect(c.handlers).toEqual([bedrock, anthropic]);
});

it('replaces the same provider set', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
const first = makeHandler('*', 'agent', ['Bedrock', 'Anthropic']);
const second = makeHandler('*', 'agent', ['Anthropic', 'Bedrock']);
const c = compose(new Registry({ handlers: [first] }), new Registry({ handlers: [second] }));
expect(c.handlers).toEqual([second]);
} finally {
warn.mockRestore();
}
});

it('b tool wins over a on the same tool key', () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const fnA = () => 'a';
Expand Down
24 changes: 24 additions & 0 deletions packages/client/src/__tests__/utils.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,30 @@ describe('createHandler', () => {
expect(handler.providesFor).toEqual(['MyProvider', 'agent']);
});

it('leaves providers unset when omitted', () => {
const handler = createHandler(['*', 'messages'], makeRawHandler() as any);
expect(handler.providers).toBeUndefined();
});

it('attaches a provider list', () => {
const handler = createHandler(['*', 'messages'], makeRawHandler() as any, undefined, false, [
'Bedrock',
'Anthropic',
]);
expect(handler.providesFor).toEqual(['*', 'messages']);
expect(handler.providers).toEqual(['Bedrock', 'Anthropic']);
});

it('rejects an empty provider list', () => {
expect(() => createHandler(['*', 'messages'], makeRawHandler() as any, undefined, false, [])).toThrow(/providers/i);
});

it('rejects a blank provider name', () => {
expect(() => createHandler(['*', 'messages'], makeRawHandler() as any, undefined, false, ['Bedrock', ''])).toThrow(
/providers/i,
);
});

it('returns the same function reference', () => {
const fn = makeRawHandler();
const handler = createHandler(['MyProvider', 'agent'], fn as any);
Expand Down
11 changes: 4 additions & 7 deletions packages/client/src/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import type {
StreamEvent,
VariationMeta,
} from './types.js';
import { normalizeMode, parseJSONWithPossibleFences } from './utils.js';
import { coversProviderName, normalizeMode, parseJSONWithPossibleFences, selectModeHandler } from './utils.js';

function resolveOutputFormatResponse(rawResponse: unknown, outputFormat: Record<string, unknown> | undefined): unknown {
if (!outputFormat) return rawResponse ?? '';
Expand Down Expand Up @@ -42,13 +42,10 @@ function selectHandler(
if (!provider) throw new Error('Provider not found');
const normalizedMode = normalizeMode(mode);

const exact = resolvedHandlers?.find((h) => h.providesFor?.[0] === provider && h.providesFor?.[1] === normalizedMode);
if (exact) return exact;
const chosen = resolvedHandlers ? selectModeHandler(resolvedHandlers, provider, normalizedMode) : undefined;
if (chosen) return chosen;

const wildcard = resolvedHandlers?.find((h) => h.providesFor?.[0] === '*' && h.providesFor?.[1] === normalizedMode);
if (wildcard) return wildcard;

if (!resolvedHandlers?.some((h) => h.providesFor?.[0] === provider || h.providesFor?.[0] === '*')) {
if (!resolvedHandlers?.some((handler) => coversProviderName(handler, provider))) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I put back the old providesFor[0] === '*' coverage check here, the tests still pass. The error then becomes ... with mode messages not found, but the test at :393 only matches /OpenAI/. Could it assert the full Handler for provider OpenAI not found, since that's the string both SDKs give?

throw new Error(`Handler for provider ${provider} not found`);
}
throw new Error(`Handler for provider ${provider} with mode ${normalizedMode} not found`);
Expand Down
17 changes: 10 additions & 7 deletions packages/client/src/graph.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import {
type TraverseVisitor,
type VariationMeta,
} from './types.js';
import { endSpanOnce, normalizeMode } from './utils.js';
import { endSpanOnce, normalizeMode, selectModeHandler } from './utils.js';

// Cycle protection: cap how many BFS layers a traversal will expand.
const MAX_TRAVERSAL_DEPTH = 100;
Expand Down Expand Up @@ -54,16 +54,19 @@ const selectHandler = (config: AiConfigRep, meta: VariationMeta, handlers: Provi

const mode = normalizeMode(meta.mode);

const exact = handlers.find((h) => h.providesFor?.[0] === provider && h.providesFor?.[1] === mode);
if (exact) return exact;

const wildcard = handlers.find((h) => h.providesFor?.[0] === '*' && h.providesFor?.[1] === mode);
if (wildcard) return wildcard;
const chosen = selectModeHandler(handlers, provider, mode);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Node selection is right. The graphJudge call at :796-807 doesn't pass handlers, though, so runJudges runs the judge on the root handler without checking the provider (judges.ts:157-158). I set up a single ['*', 'messages'] handler scoped to ['Bedrock'] and an Anthropic graphJudge, and graph().invoke() called the scoped handler for the judge. Python passes handlers=resolved_handlers there. Could you pass handlers: judgeHandlers and add a test?

if (chosen) return chosen;

const byProvider = handlers.find((h) => h.providesFor?.[0] === provider);
if (byProvider) return byProvider;

if (handlers.length === 1) return handlers[0];
if (handlers.length === 1) {
const only = handlers[0];
if (only.providers !== undefined && !only.providers.includes(provider)) {
throw new Error(`Handler for provider ${provider} not found`);
}
return only;
}

throw new Error(`Handler for provider ${provider} not found`);
};
Expand Down
Loading
Loading