Bedrock: Converse / ConverseStream API with InvokeModel fallback (#272, part 1/2) - #283
Open
dougcain wants to merge 4 commits into
Open
Bedrock: Converse / ConverseStream API with InvokeModel fallback (#272, part 1/2)#283dougcain wants to merge 4 commits into
dougcain wants to merge 4 commits into
Conversation
This was referenced Sep 7, 2026
Open
Closed
dougcain
force-pushed
the
feature/bedrock-converse-core-272
branch
from
September 7, 2026 15:30
251b112 to
03ca907
Compare
…r toolArgs, streaming-resume guard (ortus-boxlang#248) Cross-provider half of ortus-boxlang#248, independent of the Cohere and Bedrock streaming tool loops that build on it: - BaseService::resolveToolArgs / sameToolArgs: single resolver for middleware-edited tool arguments (normalized slot first, native-slot divergence detected, non-struct edits fall back to native args; canonical-JSON comparison because struct equals() is false for JSON null). Claude and OpenAI route both passes through it; OpenAI previously never read ctx.toolArgs at all. - HumanInTheLoopMiddleware edit decisions write both ctx.toolArgs and the native slot so an edit survives an earlier middleware rewrite. IAiMiddleware afterToolBatch doc: toolArgs frozen after beforeToolCall. - Resume ledger entries carry the resolved toolArgs (Claude, OpenAI); resume executes them instead of re-deriving from the raw tool call. Legacy ledgers without the field still work. - IAiService.supportsStreamingToolResume (structural default in BaseService, Gemini false); AiAgent.resumeStream throws StreamingResumeUnsupported instead of silently replaying the LLM. resume()/resumeStream() validate decisions before clearing the checkpoint. - ClosureTool.doInvoke shallow-copies args and injects _chatRequest only when declared, so the caller's struct is not mutated into message history. Tests: ClaudeTest beforeToolCall arg-rewrite cases, AiAgentResumeStreamGuardTest, SuspendResumeIntegrationTest ledger round trips (Claude, OpenAI, legacy fallback).
…us-boxlang#248) CohereService.chatStream() forwarded only parsedChunk.text and hardcoded toolCalls: [], so tools bound to a streamed Cohere call were sent to the model and then silently ignored. The stream now accumulates Cohere v1 tool events (tool-calls-chunk deltas, authoritative tool-calls-generation, stream-end fallback) into the same normalized tool calls the blocking path produces and runs the full tool loop: beforeToolCall/wrapToolCall/afterToolCall, the decide-then-execute split, afterToolBatch/defer with a combined suspension carrying assistantMessage + resumeLedger, and a tool_results follow-up turn honouring interactionCount. chatStream() reads _resumeContext and finishes a suspended batch via resumeToolBatchStream() with no LLM replay. SSE transport goes through wrapLLMCall (stream: true, emitSSEChunk). Fixes on the blocking path found while wiring the stream: only the trailing tool exchange is folded into tool_results; truncated streamed arguments abort instead of running with defaults; follow-ups that ask for more tools keep the exchange; non-single returnFormat resume no longer hangs; nameless tool calls are kept; unbound tool names are reported to the model; Pass 2 uses BaseService::resolveToolArgs and the ledger carries resolved toolArgs. Builds on feature/tool-arg-resolution-248 (shared resolveToolArgs / ledger / resume guard). Tests: CohereStreamingToolsTest, SuspendResumeIntegrationTest Cohere stream + ledger cases, AiAgentResumeStreamGuardTest cohere support assertion.
…del families (ortus-boxlang#248) BedrockService.chatStream() hardcoded toolCalls: [] and its stream transform dropped tool_use blocks, so streamed Bedrock calls never ran a tool. The event-stream reader now accumulates tool_use / input_json_delta blocks into normalized tool calls and runs the same tool loop as chat(): decide-then-execute split, afterToolBatch/defer with a combined suspension carrying assistantMessage + resumeLedger, resumeToolBatchStream() to finish a suspended batch without replaying the LLM, and follow-up turns honouring interactionCount. Structured output works on the streaming path via the forced-tool pattern. Model families: OpenAI-shaped Bedrock families (gpt-oss, DeepSeek, Qwen, Jamba, ...) get tools and response_format; Cohere Command R/R+ on Bedrock gets a real request transform; Nova/Titan/ Llama/Mistral families are detected; cache_control survives on Claude-on-Bedrock system prompts; stream exceptions surface as errors instead of a truncated answer. Pass 2 routes through BaseService::resolveToolArgs and the ledger carries resolved toolArgs. Builds on feature/cohere-streaming-tools-248 (which builds on the shared feature/tool-arg-resolution-248). Tree after this commit equals the former ortus-boxlang#277 head. Tests: BedrockServiceTest (families, stream tool loop, structured output), SuspendResumeIntegrationTest Bedrock stream suspend/resume/reject/cancel/edit cases.
…rtus-boxlang#272) BedrockService hand-built each vendor's native InvokeModel body. Bedrock's model-agnostic Converse / ConverseStream operations are now the default: one request transform serves every family (typed content blocks incl. image/document/toolUse/toolResult/reasoningContent/ cachePoint, system[], inferenceConfig, toolConfig + toolChoice, guardrailConfig, performanceConfig, additionalModelRequestFields), one response transform maps output/usage/ stopReason onto the OpenAI envelope, and the ConverseStream event reader feeds the same streaming tool loop as InvokeModel. Structured output is the forced structured_output tool for all families. API selection: providerOptions.bedrockApi > BOXLANG_MODULES_BXAI_BEDROCK_API > application scope > service setting > "converse". Automatic one-shot fallback to InvokeModel for text-completion-only models, rawBody requests, and Converse ValidationException "unsupported" answers. countTokens() exposes Bedrock CountTokens. PromptSecurity reads/writes the Converse output.message.content[] shape so OutputGuard sees the answer on the default path. Second half of this feature (fallback sticking, review fixes, countTokens body selection, live matrix) follows in feature/bedrock-converse-hardening-272. Builds on feature/bedrock-streaming-tools-248. Tests: BedrockServiceTest Converse transforms/stream/fallback, PromptSecurityRedactTest, OutputGuardMiddlewareTest Converse case, SuspendResumeIntegrationTest pins bedrockApi:"invoke" on the InvokeModel cases.
dougcain
force-pushed
the
feature/bedrock-converse-core-272
branch
from
September 10, 2026 19:54
03ca907 to
3d20569
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 1 of 2 of the former #278. Closes #272 together with #284. Stacked on #282 → #281 → #280; review only the top commit. This is the original Converse commit (
c2d8de4) as reviewed, before the hardening round.BedrockServicespeaks Bedrock's model-agnostic Converse / ConverseStream API by default, with InvokeModel as a selectable and automatically-applied fallback.API selection
providerOptions.bedrockApi:converse(default) |invoke; module settingbedrockApi; envBOXLANG_MODULES_BXAI_BEDROCK_API; precedence per-request > env > application scope > module.cohere.command-text-*,command-light-text-*,ai21.j2-*),providerOptions.rawBody, or a ConverseValidationExceptionnaming the model/operation as unsupported. Logged once per model id.Request (
transformRequestForConverse)Content blocks
text/image/document/video/toolUse/toolResult/reasoningContent/cachePoint;system[];inferenceConfig;toolConfig+toolChoice; unknown params →additionalModelRequestFields;guardrailConfig/performanceConfigfirst-class. Same-role turns merged; idempotent on already-Converse-shaped messages.Response / stream
transformResponseFromConverse: allstopReasonvalues mapped; cache-inclusive usage;metrics/tracekept.:event-typeheader into the existingstreamStateaccumulator → streaming tool loop → batch suspend /resumeToolBatchStream→ streaming structured output. ForcedtoolChoice.toolstructured output for every family.countTokens( chatRequest )→/model/{id}/count-tokens.Also
PromptSecurity::getResponseText/setResponseTextread the Converseoutput.message.content[]shape soOutputGuardMiddlewareredacts on the default Bedrock path (was a silent no-op).bedrockApi: "invoke". Readme Bedrock section.Tests
BedrockServiceTest 93 → ~135 (Converse request/response/stream/fallback), PromptSecurityRedactTest +5, OutputGuardMiddlewareTest +1. Full suite: failure set identical to
development(live-key / DB-backed tests only).Size after review fixes: main +2256/−267 in 2 files, tests +2489.
Review fixes folded in (xhigh code review of the stack, 2026-09-07)
Commit
3d20569. Each item also has a changelog bullet linking this PR.:error-messageheaderguardContentcounts as Converse dialecttool_choice: "none"is honoured on the Converse APIresponse_formaton a Converse call now warns instead of vanishingbytesvalue