feat: add --model-provider flag for keyless Bedrock LLM judge evaluators#1828
Open
stone-coding wants to merge 12 commits into
Open
feat: add --model-provider flag for keyless Bedrock LLM judge evaluators#1828stone-coding wants to merge 12 commits into
stone-coding wants to merge 12 commits into
Conversation
…Eval + Autoevals)
…support - DeepEval template: add /tmp workaround + DEEPEVAL_TELEMETRY_OPT_OUT for Lambda - Autoevals template: fix scorer= → metric= to match SDK API - Rename --from-3p-library → --3p-library - Add --param key=value (repeatable) for metric constructor kwargs - Add --parameters-file for JSON file of kwargs - Add parseParamFlags() utility + unit tests - --param and --parameters-file are mutually exclusive
- Add e2e-tests/third-party-eval-lifecycle.test.ts: full lifecycle test for DeepEval + Autoevals 3P evaluators (create → add → deploy → invoke → run eval) - Fix autoevals template: params go to AutoevalsAdapter (not metric constructor) - Add openai>=1.0.0 to autoevals pyproject.toml (required by LLM-based scorers)
Add validation guard so that explicitly passing --type llm-as-a-judge with --3p-library fails with a clear error message instead of silently proceeding into an invalid state.
Adds --model-provider <openai|bedrock> flag to `add evaluator` command.
When bedrock is selected:
- DeepEval: uses AmazonBedrockModel (native Bedrock via aiobotocore)
- Autoevals: uses LiteLLMClient routing to Bedrock Converse API
Templates are conditional via Handlebars {{#if ModelProviderBedrock}}.
Dependencies are dynamic: aiobotocore for DeepEval, litellm for Autoevals
(only included when bedrock is selected).
Default remains openai for backward compatibility.
Tested end-to-end on Lambda:
- DeepEval + Bedrock: 1.0, Pass (AnswerRelevancy, Claude Haiku)
- Autoevals + Bedrock: 0.6, Pass (Factuality, Claude Sonnet 4)
- No OpenAI API key, IAM-only auth
There was a problem hiding this comment.
Pull request overview
This PR extends the CLI’s third-party evaluator scaffolding to support Bedrock-backed “LLM-as-judge” execution via a new --model-provider <openai|bedrock> flag on agentcore add evaluator, generating provider-specific Python Lambda handlers and template dependencies.
Changes:
- Add third-party evaluator library registry + CLI flags (
--3p-library,--metric,--param/--parameters-file,--model-provider,--memory) and render Bedrock-aware templates. - Introduce new evaluator template assets for DeepEval and Autoevals (handler, pyproject, execution-role policy) with Handlebars conditionals for Bedrock paths.
- Extend evaluator managed code config schema/type to support
memorySizeMb.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/schema/schemas/primitives/evaluator.ts | Adds optional memorySizeMb to managed evaluator code config schema. |
| src/schema/llm-compacted/agentcore.ts | Mirrors memorySizeMb in the compacted schema interface. |
| src/cli/templates/EvaluatorRenderer.ts | Adds a renderer and template data shape for third-party evaluator templates. |
| src/cli/primitives/EvaluatorPrimitive.ts | Implements third-party evaluator registry/flag parsing and Bedrock/OpenAI template rendering. |
| src/cli/primitives/tests/EvaluatorPrimitive.test.ts | Adds unit tests covering third-party render context and param parsing helpers. |
| src/assets/evaluators/deepeval-lambda/pyproject.toml | New DeepEval Lambda pyproject with conditional Bedrock dependency. |
| src/assets/evaluators/deepeval-lambda/lambda_function.py | New DeepEval Lambda handler template with conditional Bedrock judge model wiring. |
| src/assets/evaluators/deepeval-lambda/execution-role-policy.json | New execution role policy template including Bedrock invoke permission. |
| src/assets/evaluators/autoevals-lambda/pyproject.toml | New Autoevals Lambda pyproject with conditional Bedrock (litellm) vs OpenAI deps. |
| src/assets/evaluators/autoevals-lambda/lambda_function.py | New Autoevals Lambda handler template with conditional Bedrock judge wiring. |
| src/assets/evaluators/autoevals-lambda/execution-role-policy.json | New execution role policy template including Bedrock invoke permission. |
| src/assets/tests/snapshots/assets.snapshot.test.ts.snap | Updates asset snapshot listing to include new evaluator template files. |
| e2e-tests/third-party-eval-lifecycle.test.ts | Adds an E2E lifecycle test for third-party evaluators (skipped unless AWS-capable). |
| .gitignore | Ignores a new E2E output directory. |
Comments suppressed due to low confidence (1)
src/assets/evaluators/autoevals-lambda/lambda_function.py:34
- When
EvaluatorParamsis empty (default when no --param/--parameters-file is provided), this rendersAutoEvalsAdapter(..., ), which is invalid Python syntax due to the dangling comma. Remove the unconditional comma so the no-params case renders valid code.
adapter = AutoEvalsAdapter(metric={{ EvaluatorClass }}(), {{{ EvaluatorParams }}})
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| REGION = os.environ.get("AWS_REGION", "us-west-2") | ||
|
|
||
| model = AmazonBedrockModel(model=MODEL_ID, region=REGION) | ||
| adapter = DeepEvalAdapter(metric={{ EvaluatorClass }}(model=model, {{{ EvaluatorParams }}})) |
| JUDGE_MODEL = os.environ.get("BEDROCK_MODEL_ID", "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0") | ||
| init(client=LiteLLMClient(), default_model=JUDGE_MODEL) | ||
|
|
||
| adapter = AutoEvalsAdapter(metric={{ EvaluatorClass }}(client=LiteLLMClient(), model=JUDGE_MODEL), {{{ EvaluatorParams }}}) |
Comment on lines
+147
to
+152
| export function jsonToKwargs(json: string): string { | ||
| const obj = JSON.parse(json) as Record<string, unknown>; | ||
| return Object.entries(obj) | ||
| .map(([key, value]) => `${key}=${jsonToPythonValue(value)}`) | ||
| .join(', '); | ||
| } |
Comment on lines
+154
to
+172
| export function parseParamFlags(params: string[]): string { | ||
| return params | ||
| .map(param => { | ||
| const eqIndex = param.indexOf('='); | ||
| if (eqIndex === -1) { | ||
| throw new Error(`"${param}" is not in key=value format`); | ||
| } | ||
| const key = param.slice(0, eqIndex); | ||
| const rawValue = param.slice(eqIndex + 1); | ||
| let value: unknown; | ||
| try { | ||
| value = JSON.parse(rawValue); | ||
| } catch { | ||
| value = rawValue; | ||
| } | ||
| return `${key}=${jsonToPythonValue(value)}`; | ||
| }) | ||
| .join(', '); | ||
| } |
Comment on lines
+451
to
+456
| if (cliOptions.memory) { | ||
| const memVal = parseInt(cliOptions.memory, 10); | ||
| if (isNaN(memVal) || memVal < 128 || memVal > 10240) { | ||
| fail('--memory must be an integer between 128 and 10240'); | ||
| } | ||
| } |
stone-coding
changed the title
chianhw
reviewed
Jul 24, 2026
| # Autoevals grades via an OpenAI-compatible client. LiteLLMClient routes to Bedrock; | ||
| # litellm auto-routes Anthropic Claude models through the Converse API. Cross-region | ||
| # inference profiles (us.*/eu.*/apac.*) are required for on-demand invocation. | ||
| JUDGE_MODEL = os.environ.get("BEDROCK_MODEL_ID", "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0") |
There was a problem hiding this comment.
Do we need to use bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0 as the default model? This model is not available in all aws regions. Can we instead add a validation check to ensure --model is provided when the user selects Bedrock as the model provider?
chianhw
reviewed
Jul 24, 2026
| from bedrock_agentcore.evaluation.custom_code_based_evaluators.third_party.deepeval import DeepEvalAdapter | ||
|
|
||
| {{#if ModelProviderBedrock}} | ||
| MODEL_ID = os.environ.get("BEDROCK_MODEL_ID", "anthropic.claude-3-haiku-20240307-v1:0") |
There was a problem hiding this comment.
we can modify and reuse the existing cli input option('--model <model>', '[LLM] Bedrock model ID for LLM-as-a-Judge').
Where possible, avoid relying on environment variables. The goal of the intern project is to reduce the customer effort required to set up environment configuration as much as possible.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
src/cli/primitives/EvaluatorPrimitive.ts:154
jsonToKwargsassumes the parsed JSON is an object and emits keys verbatim as Python kwarg names. If the JSON is an array/null/non-object or contains keys that are not valid Python identifiers, the generated handler will be invalid Python. Validate the top-level type and kwarg names and fail with a clear error.
export function jsonToKwargs(json: string): string {
const obj = JSON.parse(json) as Record<string, unknown>;
return Object.entries(obj)
.map(([key, value]) => `${key}=${jsonToPythonValue(value)}`)
.join(', ');
}
src/cli/primitives/EvaluatorPrimitive.ts:173
parseParamFlagsemits the left-hand side ofkey=valuedirectly as a Python kwarg name. If the key contains spaces/dashes/etc, the generated handler will not compile. Validate the key as a Python identifier and surface a user-friendly error.
export function parseParamFlags(params: string[]): string {
return params
.map(param => {
const eqIndex = param.indexOf('=');
if (eqIndex === -1) {
throw new Error(`"${param}" is not in key=value format`);
}
const key = param.slice(0, eqIndex);
const rawValue = param.slice(eqIndex + 1);
let value: unknown;
try {
value = JSON.parse(rawValue);
} catch {
value = rawValue;
}
return `${key}=${jsonToPythonValue(value)}`;
})
.join(', ');
Comment on lines
+91
to
+93
| function isSupportedLibrary(value: string): value is ThirdPartyLibrary { | ||
| return value in THIRD_PARTY_EVALUATOR_LIBRARIES; | ||
| } |
Comment on lines
+17
to
+19
| init(client=LiteLLMClient(), default_model="{{ Model }}") | ||
|
|
||
| adapter = AutoEvalsAdapter(metric={{ EvaluatorClass }}(client=LiteLLMClient(), model="{{ Model }}"), {{{ EvaluatorParams }}}) |
Comment on lines
+453
to
+455
| if (cliOptions.modelProvider === 'bedrock' && !cliOptions.model) { | ||
| fail('--model is required when using --model-provider bedrock'); | ||
| } |
…faults Per Irene's review feedback: - Remove hardcoded default model IDs (not available in all regions) - Require --model flag when --model-provider bedrock is selected - Bake model ID into generated template at scaffold time (no env vars) - Reuse existing --model CLI flag for Bedrock model selection
stone-coding
force-pushed
the
feature/bedrock-model-templates
branch
from
July 24, 2026 21:18
2cc3c4e to
1a860d9
Compare
- Reuse single LiteLLMClient instance (avoid duplicate instantiation) - Fix dangling comma when EvaluatorParams is empty (invalid Python) - Validate Python identifier format in jsonToKwargs and parseParamFlags - Add --timeout validation (1-300 range check)
E2E test now uses Bedrock as the LLM judge so it runs without manual OPENAI_API_KEY setup. Adds --model-provider bedrock and --model flags to both evaluator add commands.
chianhw
reviewed
Jul 24, 2026
| .option('--parameters-file <path>', '[3P library] JSON file of metric constructor kwargs') | ||
| .option( | ||
| '--model-provider <provider>', | ||
| `[3P library] LLM judge provider: ${MODEL_PROVIDERS.join(', ')} (default: openai)` |
chianhw
reviewed
Jul 24, 2026
| const warnings = getWarningsForMetric(libraryConfig, thirdParty.metricClass); | ||
| for (const warning of warnings) { | ||
| console.warn(`\n ${warning}`); | ||
| } |
There was a problem hiding this comment.
Can we add a warning message when the user specifies --model-provider openai, reminding them that manual setup (e.g., API key configuration) is required?
Per Irene's review: - Default model provider is now bedrock (was openai) - --model is required by default when using --3p-library - Warning printed when user explicitly selects openai: reminds them to set OPENAI_API_KEY on the Lambda
stone-coding
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
--model-provider <openai|bedrock>flag to theadd evaluatorcommand for third-party evaluationlibraries. When
bedrockis selected, the CLI generates Lambda handlers that use Amazon Bedrock as the LLMjudge instead of OpenAI — fully keyless, IAM-only auth (SigV4 via the Lambda execution role; the existing
bedrock:InvokeModelpolicy suffices).DeepEval (bedrock): Uses
AmazonBedrockModelfromdeepeval.modelsfor native Bedrock access. Judge modelfrom
BEDROCK_MODEL_IDenv var (defaultanthropic.claude-3-haiku-20240307-v1:0).Autoevals (bedrock): Uses
LiteLLMClientfromautoevals.litellmrouting to the Bedrock Converse API.Judge model from
BEDROCK_MODEL_IDenv var (defaultbedrock/us.anthropic.claude-sonnet-4-20250514-v1:0, across-region inference profile).
Default remains
openai— the default path generates byte-identical output to before this change; nobreaking change. Templates use Handlebars
{{#if ModelProviderBedrock}}conditionals. Dependencies aredynamic:
aiobotocore>=2.13.0(DeepEval) /litellm>=1.60,<1.85(Autoevals) only when bedrock is selected;openai>=1.0.0only on the Autoevals default path.Usage:
E2E tested on Lambda (us-west-2), scaffolded with the built CLI and deployed via agentcore deploy:
rationale
dependency install, e.g. pip --platform manylinux2014_aarch64 --only-binary=:all:)
Related Issue
Depends on #1779 (third-party eval library support). Runtime E2E requires the SDK third_party adapters from
bedrock-agentcore-sdk-python PR #568 (not yet on PyPI; tested against the locally built wheel).
Documentation PR
N/A — flag documented via --help text.
Type of Change
Testing
How have you tested the change?
Additionally:
pass py_compile
Checklist
needed
---By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this
contribution, under the terms of your choice.