Bedrock Converse: fallback sticking, countTokens, review fixes, live matrix (#272, part 2/2) - #284
Open
dougcain wants to merge 7 commits into
Open
Conversation
This was referenced Sep 7, 2026
Closed
…r toolArgs, streaming-resume guard (ortus-boxlang#248) Cross-provider half of ortus-boxlang#248, independent of the Cohere and Bedrock streaming tool loops that build on it: - BaseService::resolveToolArgs / sameToolArgs: single resolver for middleware-edited tool arguments (normalized slot first, native-slot divergence detected, non-struct edits fall back to native args; canonical-JSON comparison because struct equals() is false for JSON null). Claude and OpenAI route both passes through it; OpenAI previously never read ctx.toolArgs at all. - HumanInTheLoopMiddleware edit decisions write both ctx.toolArgs and the native slot so an edit survives an earlier middleware rewrite. IAiMiddleware afterToolBatch doc: toolArgs frozen after beforeToolCall. - Resume ledger entries carry the resolved toolArgs (Claude, OpenAI); resume executes them instead of re-deriving from the raw tool call. Legacy ledgers without the field still work. - IAiService.supportsStreamingToolResume (structural default in BaseService, Gemini false); AiAgent.resumeStream throws StreamingResumeUnsupported instead of silently replaying the LLM. resume()/resumeStream() validate decisions before clearing the checkpoint. - ClosureTool.doInvoke shallow-copies args and injects _chatRequest only when declared, so the caller's struct is not mutated into message history. Tests: ClaudeTest beforeToolCall arg-rewrite cases, AiAgentResumeStreamGuardTest, SuspendResumeIntegrationTest ledger round trips (Claude, OpenAI, legacy fallback).
…us-boxlang#248) CohereService.chatStream() forwarded only parsedChunk.text and hardcoded toolCalls: [], so tools bound to a streamed Cohere call were sent to the model and then silently ignored. The stream now accumulates Cohere v1 tool events (tool-calls-chunk deltas, authoritative tool-calls-generation, stream-end fallback) into the same normalized tool calls the blocking path produces and runs the full tool loop: beforeToolCall/wrapToolCall/afterToolCall, the decide-then-execute split, afterToolBatch/defer with a combined suspension carrying assistantMessage + resumeLedger, and a tool_results follow-up turn honouring interactionCount. chatStream() reads _resumeContext and finishes a suspended batch via resumeToolBatchStream() with no LLM replay. SSE transport goes through wrapLLMCall (stream: true, emitSSEChunk). Fixes on the blocking path found while wiring the stream: only the trailing tool exchange is folded into tool_results; truncated streamed arguments abort instead of running with defaults; follow-ups that ask for more tools keep the exchange; non-single returnFormat resume no longer hangs; nameless tool calls are kept; unbound tool names are reported to the model; Pass 2 uses BaseService::resolveToolArgs and the ledger carries resolved toolArgs. Builds on feature/tool-arg-resolution-248 (shared resolveToolArgs / ledger / resume guard). Tests: CohereStreamingToolsTest, SuspendResumeIntegrationTest Cohere stream + ledger cases, AiAgentResumeStreamGuardTest cohere support assertion.
…del families (ortus-boxlang#248) BedrockService.chatStream() hardcoded toolCalls: [] and its stream transform dropped tool_use blocks, so streamed Bedrock calls never ran a tool. The event-stream reader now accumulates tool_use / input_json_delta blocks into normalized tool calls and runs the same tool loop as chat(): decide-then-execute split, afterToolBatch/defer with a combined suspension carrying assistantMessage + resumeLedger, resumeToolBatchStream() to finish a suspended batch without replaying the LLM, and follow-up turns honouring interactionCount. Structured output works on the streaming path via the forced-tool pattern. Model families: OpenAI-shaped Bedrock families (gpt-oss, DeepSeek, Qwen, Jamba, ...) get tools and response_format; Cohere Command R/R+ on Bedrock gets a real request transform; Nova/Titan/ Llama/Mistral families are detected; cache_control survives on Claude-on-Bedrock system prompts; stream exceptions surface as errors instead of a truncated answer. Pass 2 routes through BaseService::resolveToolArgs and the ledger carries resolved toolArgs. Builds on feature/cohere-streaming-tools-248 (which builds on the shared feature/tool-arg-resolution-248). Tree after this commit equals the former ortus-boxlang#277 head. Tests: BedrockServiceTest (families, stream tool loop, structured output), SuspendResumeIntegrationTest Bedrock stream suspend/resume/reject/cancel/edit cases.
…rtus-boxlang#272) BedrockService hand-built each vendor's native InvokeModel body. Bedrock's model-agnostic Converse / ConverseStream operations are now the default: one request transform serves every family (typed content blocks incl. image/document/toolUse/toolResult/reasoningContent/ cachePoint, system[], inferenceConfig, toolConfig + toolChoice, guardrailConfig, performanceConfig, additionalModelRequestFields), one response transform maps output/usage/ stopReason onto the OpenAI envelope, and the ConverseStream event reader feeds the same streaming tool loop as InvokeModel. Structured output is the forced structured_output tool for all families. API selection: providerOptions.bedrockApi > BOXLANG_MODULES_BXAI_BEDROCK_API > application scope > service setting > "converse". Automatic one-shot fallback to InvokeModel for text-completion-only models, rawBody requests, and Converse ValidationException "unsupported" answers. countTokens() exposes Bedrock CountTokens. PromptSecurity reads/writes the Converse output.message.content[] shape so OutputGuard sees the answer on the default path. Second half of this feature (fallback sticking, review fixes, countTokens body selection, live matrix) follows in feature/bedrock-converse-hardening-272. Builds on feature/bedrock-streaming-tools-248. Tests: BedrockServiceTest Converse transforms/stream/fallback, PromptSecurityRedactTest, OutputGuardMiddlewareTest Converse case, SuspendResumeIntegrationTest pins bedrockApi:"invoke" on the InvokeModel cases.
… review fixes, live matrix (ortus-boxlang#272) Hardening of the Converse path from the code review and live runs against eu-west-2: - Converse -> InvokeModel fallback sticks per request (stickBedrockApi/stuckBedrockApi) so the tool loop and follow-up turns stay on one API; resolveBedrockApi(honourRuntimeDecision) lets countTokens count whichever body the request would actually send instead of a stale decision. Fallback refuses when the conversation is already in Converse dialect or mid-stream; the model-id regex no longer trips on inference profiles; error headers are parsed for the ValidationException detection. - Suspended batches resume in the dialect they were built in (bedrockApi/toolDialect carried in the suspension payload; ResumeLedgerMismatch otherwise). Streamed Claude/OpenAI assistant turns keep text and thinking blocks alongside tool blocks. - Request transform coercions: scalar image_url, array system, numeric max_tokens, data-URI mime types, finishReason vocabulary, tool status sniffing, max_tokens vs max_completion_tokens for OpenAI-shaped families. Guardrail/performance options accepted under either the camelCase or the X-Amzn-Bedrock-* header spelling. String event-stream replay no longer depends on key order. - Middleware parity: wrapLLMCall receives the same context as beforeLLMCall and the body is serialized inside the transport closure so in-place wrap edits reach the wire. Caller stream-callback exceptions are announced via onAIError and propagate instead of being swallowed. Tool loops route through BaseService::resolveToolArgs; AiChatRequest.runtimeOptions carries the per-request API decision. - PromptSecurity getResponseText/setResponseText share readTextBlocks/writeTextBlocks across the Converse and Claude shapes (all text blocks, not just the first). - Live matrix: BedrockConverseLiveTest (9 cases; Claude/Nova/gpt-oss over Converse+stream, fallback, countTokens) behind a new `testBedrockLive` gradle task, excluded from `test`. BedrockReviewRegressionTest pins the review findings with an HttpServer stub. Builds on feature/bedrock-converse-core-272. Tree after this commit equals the former ortus-boxlang#278 head.
dougcain
force-pushed
the
feature/bedrock-converse-hardening-272
branch
from
September 7, 2026 15:30
a371ab8 to
761cb9d
Compare
…e context Two BedrockConverseLiveTest cases covering 64af32c against eu-west-2: an object tool_choice maps to Converse toolChoice.tool (and "none" omits toolConfig), and the streaming beforeLLMCall context carries modelId + operation "converse-stream". Live matrix now 11 cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 2 of 2 of the former #278. Closes #272 with #283. Stacked on #283; review only the top commit. Descends from the former #278 head (
1871695) plus the folded review fixes and two follow-up commits; re-verified live 11/11 onf5459e4. #279 (stream-callback propagation) is rebased on top of this.Fallback and API decision
stickBedrockApi/stuckBedrockApi) inAiChatRequest.runtimeOptions, never written into caller-ownedproviderOptions. Refused after the tool loop has started, after a stream has delivered a chunk, and for feature-level 400s (toolChoice,toolResult, images…) where AWS's message is surfaced verbatim; throughput / profile / auth / throttling never trigger it. Model-id regex no longer trips on inference profiles; error headers parsed.countTokens()counts whichever body the request would actually send (resolveBedrockApi( honourRuntimeDecision=false )) instead of a previous call's stick; 400 hint about the bare-Anthropic-id restriction.bedrockApi/toolDialectin the suspension payload,ResumeLedgerMismatchotherwise).Review fixes (xhigh pass, 15 findings)
image_url, arraysystem, numericmax_tokens, data-URI mime types,finishReasonvocabulary, tool status sniffing,max_tokensvsmax_completion_tokensfor OpenAI-shaped families. Guardrail / performance options accepted under camelCase orX-Amzn-Bedrock-*header spelling. String event-stream replay independent of key order. Streamed Claude/OpenAI assistant turns keep text and thinking blocks alongside tool blocks.wrapLLMCallreceives the same context asbeforeLLMCall; body serialized inside the transport closure so in-place edits reach the wire; caller stream-callback exceptions announced ononAIErrorand propagated. Tool loops route throughBaseService::resolveToolArgs.PromptSecurityreadTextBlocks/writeTextBlocksshared by the Converse and Claude shapes (all text blocks, not only the first).Live matrix
BedrockConverseLiveTest(11 cases) behind a new./gradlew testBedrockLivetask, excluded fromtest;BedrockLiveTestBaseshared withBedrockTest;BedrockReviewRegressionTest(5, HttpServer stub) pins the review findings.Tests
BedrockServiceTest → 163, BedrockReviewRegressionTest 5, SuspendResumeIntegrationTest +3 Converse variants, BedrockTest 3. Live (eu-west-2, 2026-09-07, on the top of the stack):
testBedrockLive11/11 — Claude haiku 4.5 chat/stream/tool loop/thinking/invoke opt-out, Nova micro, Mistral 7B, gpt-oss stream + forced-toolChoicestructured output, CountTokens, plus (re-run onf5459e4) an objecttool_choicemapping to ConversetoolChoice.toolwith"none"omittingtoolConfig, and the streamingbeforeLLMCallcontext carryingmodelId+operation: "converse-stream". Note observed live: ConversetoolChoice.toolmakes the tool required, not exclusive — Claude may call sibling tools too and re-sends the forced call every turn, so setmaxInteractionswhen forcing.Size after review fixes: main +1296/−428 in 4 files, tests +3249.
Known leftovers (unchanged from #278)
Tool exceptions still propagate out of
chat()(module-wide);serviceTier/outputConfig.textFormatnot wired;ApplyGuardrailout of scope; structured output on Converse not visible toOutputGuardMiddleware; a real ConverseValidationExceptionfor an unsupported model was never observed live, soCONVERSE_UNSUPPORTED_PATTERNis inferred.Review fixes folded in (xhigh code review of the stack, 2026-09-07)
Commits
761cb9d(review fixes),64af32c(objecttool_choiceguard,modelId/operationin the stream middleware context),f5459e4(two live cases). Each item also has a changelog bullet linking this PR.bedrockHeadersshorthand goes through the Converse guardrail filtercountTokens()documents the scope of its count