Add AWS Bedrock as LLM provider

[atbrace]

Summary

Add AWS Bedrock as a third LLM provider option, enabling users to access Claude, Llama, Mistral, and other foundation models through their existing AWS credentials. This broadens model access and simplifies auth for users already running CDK labs.

Motivation

• Users already configure AWS credentials for CDK experiments — Bedrock piggybacks on those
• Bedrock provides access to multiple model families (Claude, Llama, Mistral, DeepSeek, Qwen, Nova, etc.) through a single unified API
• Enables in-app model switching since the Bedrock Converse API is consistent across text models
• Enterprise users may prefer/require Bedrock for compliance, VPC endpoints, or billing consolidation

Scope

• Implement BedrockProvider conforming to the existing LLMProvider interface
• Support streaming (SSE) and non-streaming chat via Converse / ConverseStream APIs
• Support tool/function calling with parity to Claude and Gemini providers
• Reuse existing AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION env vars
• Add BEDROCK_MODEL env var for model selection
• Investigate in-app model switching across Bedrock-hosted models

---

Research Findings

Methodology: All findings verified against live AWS documentation (April 2026). Sources linked at end of each section.

1. The Converse API — Unified Multi-Model Chat

Bedrock's Converse API (Converse and ConverseStream) is a unified chat interface that normalizes message formats across all model providers. This is the correct API to use — not the legacy per-model InvokeModel API.

SDK: @aws-sdk/client-bedrock-runtime v3.1024.0+

Key classes: BedrockRuntimeClient, ConverseCommand, ConverseStreamCommand

Other useful commands in the SDK: CountTokensCommand (token counting for quota awareness), ApplyGuardrailCommand, async invoke commands, bidirectional streaming.

IAM permissions: bedrock:InvokeModel (for Converse), bedrock:InvokeModelWithResponseStream (for ConverseStream). There are NO separate bedrock:Converse / bedrock:ConverseStream IAM actions — Converse shares permissions with InvokeModel. Verified via IAM Service Authorization Reference.

Request Structure

const response = await client.send(new ConverseCommand({
  modelId: "us.anthropic.claude-sonnet-4-6",        // URI path param
  messages: [
    { role: "user", content: [{ text: "Hello" }] }  // content is ContentBlock[]
  ],
  system: [{ text: "You are an AWS tutor." }],       // SystemContentBlock[]
  inferenceConfig: {
    maxTokens: 2048,           // required for most models
    temperature: 0.7,          // 0-1
    topP: 0.9,                 // 0-1
    stopSequences: [],         // max 2500 items
  },
  toolConfig: { tools: [...], toolChoice: { auto: {} } },
  additionalModelRequestFields: { top_k: 200 },     // model-specific params
  // Newer fields (not needed for MVP):
  performanceConfig: { latency: "standard" },        // "optimized" | "standard"
  outputConfig: { textFormat: { type, structure } },
  serviceTier: { ... },
  requestMetadata: { ... },                          // max 16 key-value pairs
}));

Key design points:

• Full conversation history passed as messages array each call (stateless, like our existing providers)
• System prompts are a separate system field — array of SystemContentBlock (text, guardContent, or cachePoint)
• Content blocks are discriminated unions: { text: "..." } not plain strings
• Model-specific parameters (e.g., Claude's top_k) go in additionalModelRequestFields
• inferenceConfig has exactly 4 fields — anything else must go in additionalModelRequestFields

ContentBlock Types (12 total)

Type	Description	Relevant for MVP?
text	Text content	Yes
toolUse	Model requesting tool execution	Yes
toolResult	Tool execution result	Yes
image	Image input	No
document	Document input	No
video	Video input	No
audio	Audio input (mp3, wav, etc.)	No
guardContent	Guardrail content	No
reasoningContent	Extended thinking (Claude 3.7+)	Future
citationsContent	Citation blocks	No
cachePoint	Prompt caching marker (TTL: 5m or 1h)	Future
searchResult	Natural citations (Claude only)	No

Response Structure

response.output.message.content  // ContentBlock[] — text, toolUse, reasoningContent, etc.
response.stopReason              // StopReason (see below)
response.usage                   // TokenUsage (see below)
response.metrics                 // { latencyMs: number }
response.trace                   // ConverseTrace (optional)
response.performanceConfig       // echo back
response.serviceTier             // echo back

StopReason Values (9 total)

Value	Meaning
end_turn	Natural completion
tool_use	Model wants to call a tool
max_tokens	Hit token limit
stop_sequence	Hit stop sequence
guardrail_intervened	Guardrail blocked response
content_filtered	Content policy filtered (separate from guardrails)
malformed_model_output	Model produced unparseable output
malformed_tool_use	Model produced invalid tool call
model_context_window_exceeded	Input exceeded model's context

TokenUsage Structure

{
  inputTokens: number,
  outputTokens: number,
  totalTokens: number,
  cacheReadInputTokens?: number,   // tokens served from cache (free from quota)
  cacheWriteInputTokens?: number,  // tokens written to cache
  cacheDetails?: [{                // per-cache-point breakdown
    inputTokens: number,
    ttl: "5m" | "1h"
  }]
}

Sources: Converse API Reference, ContentBlock types

---

2. Streaming via ConverseStream

Request body is identical to Converse (plus optional guardrailConfig.streamProcessingMode for streaming guardrails). Response is an async iterable of typed events:

Data Events (6 types, emitted in order)

Event	Purpose	Key Fields
messageStart	Stream begins	role: "assistant"
contentBlockStart	New content block	contentBlockIndex, start (union: toolUse, image, toolResult)
contentBlockDelta	Incremental content	contentBlockIndex, delta (union: see below)
contentBlockStop	Block complete	contentBlockIndex
messageStop	Message complete	stopReason, additionalModelResponseFields?
metadata	Final stats	usage, metrics, performanceConfig, serviceTier, trace

ContentBlockDelta Types (6 union members)

Delta Type	Description
text	Text string fragment
toolUse	{ input: "json fragment" } — must accumulate
reasoningContent	Extended thinking delta
toolResult	Tool result delta (array)
image	Image delta
citation	Citation delta

Error Events (can occur mid-stream)

Event	HTTP
internalServerException	500
modelStreamErrorException	424 (has originalStatusCode and originalMessage)
serviceUnavailableException	503
throttlingException	429
validationException	400

Tool Use in Streams — Complete Flow

let currentToolUseId = "";
let currentToolName = "";
let toolInputJson = "";
let fullText = "";

for await (const event of response.stream) {
  if (event.contentBlockDelta?.delta?.text) {
    fullText += event.contentBlockDelta.delta.text;
    // yield { type: 'text_delta', delta: event.contentBlockDelta.delta.text }
  }
  if (event.contentBlockStart?.start?.toolUse) {
    currentToolUseId = event.contentBlockStart.start.toolUse.toolUseId;
    currentToolName = event.contentBlockStart.start.toolUse.name;  // note: toolName in some SDK versions
    toolInputJson = "";
  }
  if (event.contentBlockDelta?.delta?.toolUse) {
    toolInputJson += event.contentBlockDelta.delta.toolUse.input;
  }
  if (event.contentBlockStop && currentToolName) {
    const parsedInput = JSON.parse(toolInputJson);
    // yield { type: 'tool_calls', calls: [{ id: currentToolUseId, name: currentToolName, arguments: parsedInput }] }
    currentToolName = "";
  }
  if (event.messageStop) {
    // yield { type: 'done', fullText }
  }
}

Critical: Tool use input arrives as JSON string fragments that must be concatenated, then JSON.parse() after contentBlockStop. No stream resume capability — on mid-stream error, discard partial output and retry the entire request.

Sources: ConverseStream API Reference, ConverseStream examples

---

3. Tool/Function Calling

Tool Definition

The tools array accepts 3 union types: toolSpec (standard), cachePoint (for caching), and systemTool (model-provider-defined tools).

toolConfig: {
  tools: [{
    toolSpec: {
      name: "get_study_progress",
      description: "Returns mastery scores and weak areas",
      inputSchema: {
        json: {  // JSON Schema wrapped in { json: ... }
          type: "object",
          properties: { examId: { type: "string" } },
          required: ["examId"]
        }
      }
    }
  }],
  toolChoice: { auto: {} }  // or { any: {} } or { tool: { name: "..." } }
}

Tool Choice Options (3 — unchanged)

Option	Behavior	Model Support
auto	Model decides whether to use tools	All tool-capable models
any	Model MUST call at least one tool	All tool-capable models
tool	Model MUST call the named tool	Claude 3+ and Amazon Nova only

Tool Results — Sent Back as User Messages

// 1. Add assistant's response (with toolUse blocks) to history
messages.push(assistantMessage);

// 2. Add tool results as a user message
messages.push({
  role: "user",
  content: [{
    toolResult: {
      toolUseId: "tooluse_01Hrsf3...",  // must match model's toolUseId
      content: [{ json: resultData }],   // or [{ text: "..." }]
      status: "success"                  // or "error" (optional)
    }
  }]
});

// 3. Call Converse again with updated messages

Model Support for Tool Calling (April 2026)

Full support (tool use + streaming tool use):

• All Claude 4.x models (Opus 4.6, Sonnet 4.6, Haiku 4.5, etc.)
• All Claude 3.x models (3.5 Sonnet, 3.5 Haiku, 3 Opus, 3 Sonnet, 3 Haiku)
• Amazon Nova (Premier, Pro, Lite, Micro)
• Cohere Command R/R+
• AI21 Jamba 1.5

Tool use only (no streaming tool use):

• Meta Llama 4 (Maverick, Scout)
• Meta Llama 3.1/3.2
• Mistral Large 3 / Large 2
• Qwen 3 models

No tool support:

• Claude 2.x legacy, DeepSeek R1/V3, Titan Text, Google Gemma 3

Implication: Since our tutor uses 7 tools, models without tool support cannot serve as tutor providers. We should gate model selection to tool-capable models only.

Sources: Tool Use with Converse, Tool Use Examples

---

4. Current Model Catalog (April 2026)

Claude Models on Bedrock

Model	Bedrock Model ID	Context	Max Output	~Price (Input/Output per MTok)
Claude Opus 4.6	anthropic.claude-opus-4-6-v1	1M	128K	~$5 / $25
Claude Sonnet 4.6	anthropic.claude-sonnet-4-6	1M	64K	~$3 / $15
Claude Haiku 4.5	anthropic.claude-haiku-4-5-20251001-v1:0	200K	64K	~$1 / $5
Claude Opus 4.5	anthropic.claude-opus-4-5-20251101-v1:0	200K	64K	~$5 / $25
Claude Sonnet 4.5	anthropic.claude-sonnet-4-5-20250929-v1:0	200K	64K	~$3 / $15
Claude Sonnet 4	anthropic.claude-sonnet-4-20250514-v1:0	200K	64K	~$3 / $15
Claude Opus 4	anthropic.claude-opus-4-20250514-v1:0	200K	32K	~$15 / $75
Claude Opus 4.1	anthropic.claude-opus-4-1-20250805-v1:0	200K	32K	~$15 / $75
Claude 3.5 Haiku	anthropic.claude-3-5-haiku-20241022-v1:0	200K	64K	~$0.80 / $4
Claude 3 Haiku	anthropic.claude-3-haiku-20240307-v1:0	—	—	Retiring April 19, 2026

Critical: Opus 4.6 has no in-region availability — only accessible via cross-region inference (us., eu., global. prefixes). Sonnet 4.6 is in-region only in eu-west-2.

Other Notable Models

Model	Bedrock Model ID	Tool Use	Streaming	Context
Llama 4 Maverick	meta.llama4-maverick-17b-instruct-v1:0	Yes	Yes	128K+
Llama 4 Scout	meta.llama4-scout-17b-instruct-v1:0	Yes	Yes	128K+
Mistral Large 3	mistral.mistral-large-3-675b-instruct	Yes	Partial	128K
DeepSeek R1	deepseek.r1-v1:0	No	Yes	128K
DeepSeek V3	deepseek.v3-v1:0	No	Yes	128K
Qwen 3 235B	qwen.qwen3-235b-a22b-2507-v1:0	Yes	Partial	128K
Amazon Nova Pro	Nova family	Yes	Yes	300K
Google Gemma 3 27B	google.gemma-3-27b-it	No	Yes	128K

Regional Availability

• us-east-1 (N. Virginia) — broadest catalog
• us-west-2 (Oregon) — second broadest
• Cross-region inference (us.*, eu.*, global.*) required for newest Claude models
• EU and AP regions lag for newer non-Claude models

Sources: Supported models, Converse API supported models, model cards for Opus 4.6, Sonnet 4.6

---

5. Authentication & Credentials

The BedrockRuntimeClient uses the standard AWS credential resolution chain:

1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
2. Shared credentials file (~/.aws/credentials)
3. IAM role (EC2/ECS/Lambda)

Our existing CDK credentials work out of the box.

const client = new BedrockRuntimeClient({
  region: process.env.AWS_REGION || "us-east-1",
  maxAttempts: 1,  // disable SDK retry, use our own withRetry()
});

Required IAM policy:

{
  "Effect": "Allow",
  "Action": [
    "bedrock:InvokeModel",
    "bedrock:InvokeModelWithResponseStream"
  ],
  "Resource": [
    "arn:aws:bedrock:*::foundation-model/*",
    "arn:aws:bedrock:*:*:inference-profile/*"
  ]
}

The inference-profile/* resource is needed for cross-region inference (required for Opus 4.6, Sonnet 4.6).

Model access opt-in required: Models must be explicitly enabled in the Bedrock console (Model Access page). Claude models get instant approval; Llama/Mistral require EULA acceptance (still instant after).

Sources: IAM Actions for Bedrock

---

6. Error Handling & Retry

Error Classification — Converse API

Exception	HTTP	Retryable	Notes
ThrottlingException	429	Yes	RPM or TPM quota exceeded
ModelTimeoutException	408	Yes	Model processing timeout
ModelNotReadyException	429	Yes	Custom/imported model restoring; SDK auto-retries 5x
ModelErrorException	424	Maybe	Model inference error; check details
ServiceUnavailableException	503	Yes	Transient service issue
InternalServerException	500	Yes	Generic server failure
ValidationException	400	No	Malformed request
AccessDeniedException	403	No	Missing permissions or model not enabled
ResourceNotFoundException	404	No	Model doesn't exist in region

ConverseStream adds: ModelStreamErrorException (424) — mid-stream error with originalStatusCode and originalMessage.

Note: ServiceQuotaExceededException is NOT thrown by Converse/ConverseStream (only by legacy InvokeModel). The general troubleshooting page uses display names (e.g., "InternalFailure") that differ from SDK class names (e.g., InternalServerException) — use SDK class names in code.

SDK Built-in Retry

The AWS SDK v3 defaults to standard retry mode with 3 max attempts (exponential backoff + jitter, MAX_BACKOFF = 20s). Set maxAttempts: 1 to disable SDK retry and use our existing withRetry() for consistent behavior across providers.

Throttling — Critical Details

• Quotas are per-model, per-region, per-account on a rolling window (not averaged)
• Reference defaults (vary by account): Claude Sonnet ~100-500 RPM, Haiku ~2,000 RPM, Llama 3.3 70B ~400 RPM
• No Retry-After header — backoff must be self-calculated
• AWS recommends backoff up to 60 seconds for RPM throttling (rolling minute window)
• Quotas adjustable via Service Quotas console

⚠️ Token Burndown — Important for Quota Management

Bedrock reserves quota immediately: input_tokens + max_tokens. After completion, actual consumption is calculated and the difference refunded.

Critical for Claude 3.7+ models: Output tokens have a 5x burndown rate — 1 output token costs 5 TPM quota tokens. This means:

• Setting maxTokens: 4096 reserves 4096 × 5 = 20,480 TPM quota tokens for output alone
• Keep maxTokens as low as practical (our tutor rarely needs >2048)
• CacheReadInputTokens are free from quota (relevant if we implement prompt caching)

Streaming Errors

• Pre-stream: standard HTTP exceptions (ThrottlingException, etc.)
• Mid-stream: typed error events with originalStatusCode
• No stream resume — on error, discard partial output and retry entire request

Guardrails

• Opt-in only — no content filtering by default (model's own refusals still apply)
• Intervention appears as stopReason: "guardrail_intervened" or "content_filtered" (HTTP 200, not an error)
• Not relevant for our study app unless explicitly configured

Sources: Converse API errors, Troubleshooting, Token burndown, Quotas, SDK retry

---

7. Mapping to Our LLMProvider Interface

Our Interface	Bedrock Equivalent	Complexity
chat(messages, options)	ConverseCommand	Low — direct mapping
continueWithToolResults(...)	Append toolUse + toolResult to messages, call ConverseCommand again	Low
chatStream(messages, options)	ConverseStreamCommand + async iterate	Medium — accumulate tool input fragments
continueWithToolResultsStream(...)	Same as chatStream with tool history	Medium
LLMMessage → Bedrock Message	Wrap content: string → content: [{ text: string }]	Low
LLMTool → Bedrock Tool	Wrap parameters → { toolSpec: { inputSchema: { json: ... } } }	Low
LLMToolCall ← Bedrock ToolUseBlock	Map toolUseId → id, input → arguments	Low
LLMToolResult → Bedrock ToolResultBlock	Wrap in { toolResult: { toolUseId, content: [{ json }] } }	Low
LLMError	Map SDK exception classes → { provider: "bedrock", statusCode, isRetryable }	Low
LLMStreamChunk	Map deltas → text_delta, accumulated tool JSON → tool_calls, messageStop → done	Medium

Overall: The Converse API maps cleanly to our abstraction. Bedrock provides toolUseId natively (unlike Gemini where we generate deterministic hashes). No changes to types.ts, tools.ts, tool-handlers.ts, retry.ts, or prompts.ts should be needed.

Future consideration: reasoningContent blocks (extended thinking) include a signature field that must be preserved across multi-turn conversations — similar to Gemini's thoughtSignature. Not needed for MVP but worth noting for future support.

---

8. In-App Model Switching Feasibility

The Converse API is consistent across all model families — switching models is just changing the modelId parameter. This makes in-app model switching highly feasible:

• All tool-capable models use identical request/response format
• Tool definitions don't need per-model adaptation
• System prompts handled uniformly
• CountTokensCommand could enable quota-aware model selection

Caveats:

• Tool calling reliability varies (Claude is excellent; Llama can be inconsistent)
• Models without tool support (DeepSeek, Gemma, Titan) can't use the tutor
• Cross-region model IDs use different prefixes (us., eu., global.)
• Could call ListFoundationModels API to discover which models user has enabled

---

9. Implementation Plan (Draft)

New files:

• src/lib/llm/providers/bedrock.ts — BedrockProvider implementation (~300-400 lines)

Modified files:

• src/lib/llm/provider.ts — add 'bedrock' to ProviderName, add factory case
• src/app/api/tutor/provider/route.ts — add Bedrock model display names
• .env.example — add BEDROCK_MODEL documentation

New dependency:

• @aws-sdk/client-bedrock-runtime (v3.1024.0+)

Environment variables:

• LLM_PROVIDER=bedrock
• BEDROCK_MODEL=us.anthropic.claude-sonnet-4-6 (default — cross-region Sonnet 4.6)
• Reuses existing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION

---

10. Open Questions

1. Should we gate model selection to tool-capable models only? Non-tool models can't use the tutor's 7 tools.
2. In-app switching UI — should this be a Bedrock-only feature or extend to all providers?
3. Model access detection — should we call ListFoundationModels to discover which models the user has enabled?
4. Cross-region inference — Opus 4.6/Sonnet 4.6 require it. Default to us.* prefix or make configurable?
5. Cost display — should the model selector show per-request cost estimates?
6. Prompt caching — Converse API now supports cachePoint blocks with TTL. Worth implementing for our tutor's repeated system prompt?
7. Extended thinking — Claude 4.x supports reasoningContent with signature preservation. Support in MVP or defer?

---

🤖 Generated with Claude Code

[claude]

Issue Enrichment Report

Type

Feature — adds a new LLM provider. No type label is present; recommend adding feature.

Scope & Alignment

In scope. The project already abstracts LLM providers behind an LLMProvider interface supporting Claude and Gemini. Bedrock is a natural third option that fits the local-first, single-user design (AWS credentials live in the user's environment), stays within the Next.js/TypeScript stack, and targets AWS-focused users who already configure CDK credentials.

Related Issues

• Duplicates: none found
• Overlaps with: Distinguish quota exhaustion from rate limiting in 429 handling #41 — both touch the LLM error-handling layer; Distinguish quota exhaustion from rate limiting in 429 handling #41 distinguishes quota exhaustion from rate-limiting in 429 responses. Bedrock uses typed exception classes (ThrottlingException vs ServiceQuotaExceededException) instead of HTTP status codes — the Bedrock provider should apply the same classification logic once Distinguish quota exhaustion from rate limiting in 429 handling #41 lands.
• Blocked by: Distinguish quota exhaustion from rate limiting in 429 handling #41 — recommended to resolve the retry/error classification pattern before adding a third provider that needs to implement it.
• Blocks: none

Acceptance Criteria

The issue contains a detailed scope list and open questions, but no objectively testable acceptance criteria. Proposed:

• LLM_PROVIDER=bedrock starts the app without errors when valid AWS credentials and BEDROCK_MODEL are set
• Non-streaming chat (ConverseCommand) returns a correct assistant reply with tool calls handled
• Streaming chat (ConverseStreamCommand) yields incremental text deltas; tool input JSON fragments are accumulated and emitted correctly as a complete tool_call after contentBlockStop
• All 7 tutor tools function correctly end-to-end with a Bedrock Claude model
• ThrottlingException is mapped to a retryable LLMError; AccessDeniedException and ValidationException are mapped to non-retryable errors
• getProvider() throws a clear error if AWS_REGION is missing when LLM_PROVIDER=bedrock
• /api/tutor/provider returns correct provider: "bedrock" and a human-readable displayName for the configured model
• pnpm test passes including a new bedrock.test.ts with mocked BedrockRuntimeClient

Code Surface Area

Files likely touched:

• src/lib/llm/providers/bedrock.ts (new) — BedrockProvider implementation (~300-400 lines, similar to claude.ts)
• src/lib/llm/providers/__tests__/bedrock.test.ts (new) — unit tests with mocked SDK
• src/lib/llm/types.ts — extend ProviderName union to include 'bedrock'
• src/lib/llm/provider.ts — add bedrock entry to providers map + env-var validation for AWS_REGION
• src/app/api/tutor/provider/route.ts — add Bedrock model IDs to MODEL_DISPLAY_NAMES map and a getCurrentModelId() branch for 'bedrock'
• .env.example / install.sh — document BEDROCK_MODEL and add Bedrock as a third provider option in the setup wizard

Effort estimate: large — comparable to the Gemini provider (473 lines + tests). The main complexity is streaming tool-input JSON accumulation, typed exception mapping, and a new @aws-sdk/client-bedrock-runtime dependency to wire in.

Research Findings

The issue body already contains thorough research (Converse API, streaming events, tool-calling matrix, error taxonomy, IAM policy). Key implementation notes for the developer:

• Follow claude.ts as the closer analog — Bedrock's Converse API maps more cleanly to the LLMProvider interface than Gemini (no thought-signature handling; toolUseId is provided natively rather than requiring deterministic generation).
• Streaming tool calls require concatenating contentBlockDelta.delta.toolUse.input fragments across events and JSON.parse()-ing after contentBlockStop. This is the main non-obvious piece.
• Disable SDK retry (maxAttempts: 1) and rely on the existing withRetry() utility for consistency across providers.
• Open question Add Claude Code GitHub Workflow #1 is load-bearing: models without tool support (e.g., Llama 2, Mistral 7B) cannot drive the tutor's 7 tools — gate BEDROCK_MODEL validation to the tool-capable subset or emit a clear startup error.
• In-app model switching (open question /study/domain-1-organizational-complexity page fails to load with error #2) is feasible since the Converse API is uniform, but should be tracked as a separate issue rather than bundled here.

Priority Assessment

No priority label is set (issue carries roadmap + idea). Given that #41 (P1-high bug) should land first to establish the error-classification pattern the Bedrock provider must implement, this is appropriately a later item. Recommend adding P3-low or leaving unprioritized until #41 is resolved.

---

Automated enrichment by Claude — verify findings before implementation