diff --git a/.agents/skills/agenthub-dev/SKILL.md b/.agents/skills/agenthub-dev/SKILL.md new file mode 100644 index 00000000..97b04500 --- /dev/null +++ b/.agents/skills/agenthub-dev/SKILL.md @@ -0,0 +1,64 @@ +--- +name: agenthub-dev +description: Fixed workflow for developing AgentHub itself — adding or updating model support. Use when asked to support a new model or protocol version in this repository, sync llmsdk_docs, or implement a provider client. Covers doc syncing, live API capture, paired Python/TypeScript implementation, and model-scoped e2e testing. +--- + +# AgentHub Development Workflow + +Adding or updating model support follows the stages below, in order. Where a stage says **stop and ask**, pause and ask the user; do not continue until the issue is resolved, and never fill the gap yourself. + +## Directory map + +``` +llmsdk_docs// Official docs snapshot, one folder per model generation (README.md + docs/) +api_captures// Git-ignored raw API captures: request payloads + stream events +src_py/agenthub// Python client, one folder per wire protocol +src_py/agenthub/auto_client.py Routes model names to protocol clients by explicit version +src_ts/src// TypeScript client, mirrors the Python folder +src_ts/src/autoClient.ts TypeScript routing, mirrors auto_client.py +src_py/tests/test_client.py Parameterized e2e tests (env-gated AVAILABLE_MODELS) +src_ts/tests/client.test.ts Same for TypeScript +changelog/ One detail file per CHANGELOG.md entry +CHANGELOG.md Brief one-line entries linking into changelog/ +``` + +## Stage 1 — Sync official docs into `llmsdk_docs/` + +- Sources must be the model vendor's official documentation site (e.g. docs.anthropic.com, platform.openai.com, ai.google.dev). Never use third-party mirrors, blog posts, or model memory. +- Save the snapshot under `llmsdk_docs//` following the existing folder conventions, and list the folder in `llmsdk_docs/README.md`. Running this workflow is the explicit request that the repository rule against editing `llmsdk_docs/` asks for. +- When the fetched docs differ from an existing snapshot, the new official docs win: update the old files in place. +- The snapshot must be complete enough to implement from: request/response schemas, streaming event sequence, thinking output, tool calling, usage fields, and error responses. +- **Stop and ask** if the official URL is uncertain or a page cannot be fetched. The user can paste the content manually. + +## Stage 2 — Capture a live API exchange into `api_captures/` + +- Gate: the provider's API key environment variable must be set and usable. Use the same environment variables and base URLs as `src_py/tests/test_client.py` (`AVAILABLE_MODELS` gating and `_create_client`). **Stop and ask** the user to supply the key if it is missing; the workflow must not continue without it. +- Using the provider's official SDK, or raw HTTP exactly as documented, run one streaming tool-call request with thinking enabled, then send the tool result back so the capture also shows how assistant turns are re-sent. +- Save the complete exchange unmodified under `api_captures//` (git-ignored), e.g. `round1.request.json` plus `round1.stream.jsonl` with every raw stream event in order. Never save credentials. +- **Stop and ask** on any API error (invalid key, insufficient quota, rate limit). Do not mock the response or continue from docs alone. +- The capture is the primary implementation reference and outranks the docs: where they disagree, implement what the API actually returned. + +## Stage 3 — Implement the Python and TypeScript clients + +- One folder per wire protocol, named after the newest model generation that uses it. Diff the new protocol (capture + docs) against the closest existing folder: + - Any difference between generations, even a single key name, means a separate folder per generation (e.g. `claude4_6/` vs `claude5/`). + - Only an identical wire protocol may share a folder; name it after the newest generation (rename and reroute if needed). This is how `claude5/` serves Claude 4.7, 4.8, and 5. +- `auto_client.py` / `autoClient.ts` route model names by explicit version matching only, never a bare substring like `"claude" in model`. +- Conversion must be bijective: a wire message converted to `UniMessage`/`UniEvent` and back must reproduce the original exactly, including `fidelity` payloads (thinking signatures, phase labels, reasoning field names) and tool-call IDs. Verify against the captured exchange. +- `UniConfig` keys rarely map one-to-one onto provider config keys. **Stop and ask**: list every non-obvious mapping and confirm it with the user before coding. Never decide silently. +- Implement Python and TypeScript together with identical behavior. + +## Stage 4 — Verify + +- Register the model in the env-gated `AVAILABLE_MODELS` lists of both test files with correct capability flags. Do not add model-specific test functions or files. +- Static checks: `make lint` in `src_py/`; `npm run lint` and `npm run build` in `src_ts/`. +- Run only the new model's e2e tests; the full suites are slow and spend real API quota: + - `cd src_py && uv run pytest -vvv tests/test_client.py -k ""` + - `cd src_ts && npm run test -- -t ""` +- Leave unrelated tests to CI. + +## Record and ship + +- Write `changelog/YYYY-MM-DD-.md` with the specifics: protocol differences found, config mapping decisions, notable capture findings. +- Add one brief line at the top of `CHANGELOG.md` linking to that file. The root file keeps a single line per change. +- Commit on a feature branch and open a PR with `gh pr create --base dev`; direct pushes to `dev` are rejected. diff --git a/.claude b/.claude new file mode 120000 index 00000000..c0ca4685 --- /dev/null +++ b/.claude @@ -0,0 +1 @@ +.agents \ No newline at end of file diff --git a/.gitignore b/.gitignore index 77c60b1a..836ee3af 100644 --- a/.gitignore +++ b/.gitignore @@ -355,6 +355,9 @@ vite.config.js.timestamp-* vite.config.ts.timestamp-* .vite/ +# Raw LLM API captures used as implementation reference (see .agents/skills/agenthub-dev) +api_captures/ + # Conversation monitor cache cache/ generated_image* diff --git a/CHANGELOG.md b/CHANGELOG.md index d0f113a0..e6404698 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,14 @@ # Changelog -Here, we record the addition and removal times of models, major functional updates, bug fixes, and release times of key versions. +Here, we record the addition and removal times of models, major functional updates, bug fixes, and release times of key versions. Each entry keeps one brief line; the full details of a change live in a file under [changelog/](changelog/). + +- [2026-07-20] Release version 0.4.0. Content items now carry an opaque `fidelity` payload that absorbs the former `signature`/`phase` fields (breaking), and OpenAI-compatible clients use it to replay thinking through exactly the reasoning field the upstream produced. ([details](changelog/2026-07-20-reasoning-field-fidelity.md)) + +- [2026-07-17] Add the `agenthub-dev` skill that fixes the model-support development workflow, and the `changelog/` details directory. ([details](changelog/2026-07-17-agenthub-dev-skill.md)) + +- [2026-07-14] Raise `EmptyResponseError` when a model completes a response with thinking output only, since sending it back would fail with a 400 error. It and `ToolCallArgumentParseError` now inherit the new `AgentHubError` base class. + +- [2026-06-10] Support Claude 5 models. - [2026-06-01] Release version 0.3.3. Support OpenAI-compatible embedding input format. diff --git a/README.md b/README.md index 4c9091f9..f0a98103 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ https://github.com/user-attachments/assets/c49a21a1-5bf9-4768-a76d-f73c9a03ca87 | Model Name | Vendor | Example Model ID | Input Modalities | Output Modalities | | -------------- | ----------------------------------- | ---------------------- | ---------------- | ------------------------------ | | Gemini 3-3.5 | Official/Google Vertex AI | `gemini-3.5-flash` | Text, Image | Text, Image, Speech, Embedding | -| Claude 4.6-4.8 | Official/Amazon Bedrock/UModelVerse | `claude-opus-4-8` | Text, Image | Text | +| Claude 4.6-5 | Official/Amazon Bedrock/UModelVerse | `claude-opus-4-8` | Text, Image | Text | | GPT-5.4/5.5 | Official/UModelVerse | `gpt-5.5` | Text, Image | Text, Embedding | | Kimi-K2.5/K2.6 | Official/OpenRouter/SiliconFlow | `kimi-k2.6` | Text, Image | Text | | DeepSeek V4 | Official/OpenRouter/SiliconFlow | `deepseek-v4-pro` | Text | Text | @@ -433,7 +433,7 @@ Example UniMessage: {"type": "text", "text": "How are you doing?"}, {"type": "image_url", "image_url": "https://example.com/image.jpg"}, {"type": "inline_data", "mime_type": "image/jpeg", "data": "base64-encoded-image"}, - {"type": "thinking", "thinking": "I am thinking.", "signature": "0x123456"}, + {"type": "thinking", "thinking": "I am thinking.", "fidelity": {"signature": "0x123456"}}, {"type": "inline_thinking", "mime_type": "image/jpeg", "data": "base64-encoded-image"}, {"type": "tool_call", "name": "math", "arguments": {"expression": "2 + 3"}, "tool_call_id": "123"}, {"type": "tool_result", "text": "2 + 3 = 5", "images": [], "tool_call_id": "123"} diff --git a/changelog/2026-07-17-agenthub-dev-skill.md b/changelog/2026-07-17-agenthub-dev-skill.md new file mode 100644 index 00000000..9fb45a37 --- /dev/null +++ b/changelog/2026-07-17-agenthub-dev-skill.md @@ -0,0 +1,6 @@ +# Add the agenthub-dev skill and the changelog details directory + +- Added `.agents/skills/agenthub-dev/SKILL.md`, fixing the model-support development workflow: sync official docs into `llmsdk_docs/`, capture a live streaming tool-call exchange with thinking into the git-ignored `api_captures/`, implement paired Python/TypeScript protocol clients with bijective message conversion, and verify with model-scoped e2e tests only. +- The workflow makes four situations hard stops that require asking the user: unclear or unfetchable official docs, a missing provider API key, any live API request error, and non-obvious `UniConfig` key mappings. +- Added `api_captures/` to `.gitignore` as the home for raw API captures; where docs and captures disagree, the capture wins. +- Added the `changelog/` directory: each `CHANGELOG.md` entry keeps one brief line and links to a detail file here (see `changelog/README.md`). diff --git a/changelog/2026-07-20-reasoning-field-fidelity.md b/changelog/2026-07-20-reasoning-field-fidelity.md new file mode 100644 index 00000000..6c31be14 --- /dev/null +++ b/changelog/2026-07-20-reasoning-field-fidelity.md @@ -0,0 +1,39 @@ +# Add the `fidelity` field and replay the exact reasoning field the upstream produced + +## The bug + +OpenAI Chat Completions-compatible servers spell the streamed thinking field differently — vLLM & SiliconFlow use `reasoning_content` while OpenRouter uses `reasoning` — and when sending assistant history back, the `openai`, `glm5_1`, and `kimi_k2_6` clients always set **both** fields on the message. Strict upstreams reject the spelling they did not emit (e.g. a server that returned `reasoning_content` refuses a request containing `reasoning`), breaking multi-turn conversations. + +## The `fidelity` field + +Fixing this needs a place to record which wire field carried the thinking. Rather than overloading `signature`, content items now carry a single dedicated field: + +- `fidelity` (`dict[str, Any]` / `Record`, optional) — an arbitrary JSON-style object of wire-level data a client records to reproduce the original message on replay. Opaque to consumers: pass it back unchanged. + +It replaces and absorbs the former item-level `signature` and `phase` fields: + +| Client | Old | New | +| --- | --- | --- | +| `claude5` / `claude4_6` | `signature: ` on thinking (also holds redacted-thinking data) | `fidelity: {"signature": }` | +| `gemini3` | `signature: ` on text / thinking / inline / tool_call items (key present even when `None`) | `fidelity: {"signature": }`, omitted entirely when absent | +| `gpt5_5` | `signature: json.dumps({"id": ..., "encrypted_content": ...})` on thinking; `phase:

` on text | `fidelity: {"id": ..., "encrypted_content": ...}` (no more JSON-in-a-string); `fidelity: {"phase":

}` | +| `openai` / `glm5_1` / `kimi_k2_6` / `deepseek_v4` | nothing recorded; both reasoning spellings sent back | `fidelity: {"reasoning_field": "reasoning_content" \| "reasoning"}` per thinking delta | + +## The reasoning-field fix + +On receive, the OpenAI-compatible clients record the wire field name that carried each thinking delta. On send, the message conversion replays the thinking through exactly that field. Fallbacks keep the old maximum-compatibility behavior: thinking without a recorded `reasoning_field` (hand-written histories, foreign-protocol fidelity), mixed fields within one message, and the ambiguous case where one chunk carries both spellings (such deltas record no fidelity) all still send both fields. + +## Concatenation rules + +`concat_uni_events_to_uni_message` / `concatUniEventsToUniMessage` now key on `fidelity`: + +- text: a phase change starts a new item (same-phase and phaseless deltas merge, per the GPT-5.5 `phase` guide); an incoming fidelity payload (e.g. a signature) merges into the open item's fidelity and finishes it. +- thinking: an incoming fidelity payload finishes the open item (Claude signature deltas, GPT-5.5 reasoning markers), and a run of deltas carrying **equal** fidelity concatenates into one item (the OpenAI-compatible per-delta `reasoning_field` tags). + +## Breaking change + +Histories recorded by earlier versions carry `signature` / `phase` at the top level of content items; clients no longer read those fields. To replay an old history, move each item's `signature`/`phase` into `fidelity` (`{"signature": ...}` for Claude/Gemini, `{"id": ..., "encrypted_content": ...}` parsed from the GPT-5.5 JSON string, `{"phase": ...}` for GPT-5.5 text). + +## Tests + +Offline fake-stream suites `src_py/tests/test_reasoning_fidelity.py` and `src_ts/tests/reasoning-fidelity.test.ts` cover both reasoning field spellings, the ambiguous both-fields case, and the no-fidelity fallback across the `openai`, `glm5_1`, and `kimi_k2_6` clients. diff --git a/changelog/README.md b/changelog/README.md new file mode 100644 index 00000000..19e37f94 --- /dev/null +++ b/changelog/README.md @@ -0,0 +1,7 @@ +# Changelog Details + +One file per entry in [../CHANGELOG.md](../CHANGELOG.md). The root file keeps a single brief line per change; the full story lives here. + +- File name: `YYYY-MM-DD-short-slug.md`, matching the entry date. +- Content: what changed and why, affected modules, and decisions worth keeping (protocol differences, config mappings, migration notes). +- Link the root entry to its file: `- [YYYY-MM-DD] Brief description. ([details](changelog/YYYY-MM-DD-short-slug.md))` diff --git a/llmsdk_docs/claude5/README.md b/llmsdk_docs/claude5/README.md new file mode 100644 index 00000000..73f7668a --- /dev/null +++ b/llmsdk_docs/claude5/README.md @@ -0,0 +1,10 @@ +# Claude 5 SDK Documentation + +This directory contains documentation for using Anthropic's Claude 5 API. + +## Documentation + +The `docs/` folder contains detailed guides on various Claude 5 features: + +- [introducing-claude-fable-5-and-claude-mythos-5.md](./docs/introducing-claude-fable-5-and-claude-mythos-5.md) - What's new in Claude 5 +- [migration-guide.md](./docs/migration-guide.md) - Migration guide from Claude 4.8 to Claude 5 diff --git a/llmsdk_docs/claude5/docs/introducing-claude-fable-5-and-claude-mythos-5.md b/llmsdk_docs/claude5/docs/introducing-claude-fable-5-and-claude-mythos-5.md new file mode 100644 index 00000000..59eae0c9 --- /dev/null +++ b/llmsdk_docs/claude5/docs/introducing-claude-fable-5-and-claude-mythos-5.md @@ -0,0 +1,109 @@ +# Introducing Claude Fable 5 and Claude Mythos 5 + +Claude Fable 5 and Claude Mythos 5 capabilities, API changes, and availability. + +--- + +Claude Fable 5 is Anthropic's most capable widely released model, built for the most demanding reasoning and long-horizon agentic work. Claude Mythos 5 shares the same capabilities without the safety classifiers and is available only in limited release through [Project Glasswing](https://anthropic.com/glasswing). + +## Models + +| Model | API model ID | Description | +|:------|:-------------|:------------| +| Claude Fable 5 | `claude-fable-5` | Anthropic's most capable widely released model, for the most demanding reasoning and long-horizon agentic work | +| Claude Mythos 5 | `claude-mythos-5` | Shares Claude Fable 5's capabilities without the safety classifiers. Available through Project Glasswing. Successor to Claude Mythos Preview. | + +Claude Fable 5 and Claude Mythos 5 support a [1M token context window](/docs/en/build-with-claude/context-windows) by default and up to 128k output tokens per request. + +Claude Fable 5 and Claude Mythos 5 are priced at $10 per million input tokens and $50 per million output tokens. For specs across all current models, see the [models overview](/docs/en/about-claude/models/overview). + +## Refusals, fallback, and billing on Claude Fable 5 + +Claude Fable 5 includes safety classifiers that can decline certain requests. The sections below summarize what that means for your integration; each links to the full guide. + +### Refusals + +When Claude Fable 5 declines a request, the Messages API returns `stop_reason: "refusal"` as a successful HTTP 200 response, not an error. The response also reports which classifier declined the request. See [Refusals and fallback](/docs/en/build-with-claude/refusals-and-fallback) for response shapes and handling guidance. + +### Fallback + +A request that Claude Fable 5 refuses can usually be served by another Claude model. Pass the `fallbacks` parameter to have the API retry for you (in beta on the Claude API and Claude Platform on AWS), or use the SDK middleware (TypeScript, Python, Go, Java, and C#) to retry from the client on any platform. See [Server-side fallback](/docs/en/build-with-claude/refusals-and-fallback#server-side-fallback), [client-side fallback](/docs/en/build-with-claude/refusals-and-fallback#client-side-fallback), and [SDK middleware](/docs/en/cli-sdks-libraries/middleware). + +### Billing + +You are not billed for a request that is refused before any output is generated. When you retry on another model, fallback credit refunds the prompt-cache cost of switching. See [Fallback credit](/docs/en/build-with-claude/fallback-credit). + +## Availability + +Claude Fable 5 and Claude Mythos 5 both become available on June 9, 2026: + +- **Claude Fable 5** is generally available on the Claude API, [Claude Platform on AWS](/docs/en/build-with-claude/claude-platform-on-aws), [Amazon Bedrock](/docs/en/build-with-claude/claude-in-amazon-bedrock), [Vertex AI](/docs/en/build-with-claude/claude-on-vertex-ai), and [Microsoft Foundry](/docs/en/build-with-claude/claude-in-microsoft-foundry). +- **Claude Mythos 5** is not generally available: it is offered in limited availability to approved customers in [Project Glasswing](https://anthropic.com/glasswing). For access, contact your Anthropic, AWS, or Google Cloud account team. Customers without access to Claude Mythos 5 can use Claude Fable 5, the generally available Mythos-class model. + +Claude Fable 5 and Claude Mythos 5 are designated [Covered Models](https://support.claude.com/en/articles/15425695), which carry 30-day data retention and are not available under zero data retention. See [Model-specific data retention requirements](/docs/en/manage-claude/api-and-data-retention#model-specific-data-retention-requirements). + +## Working with Claude Fable 5 and Claude Mythos 5 + +### Prompting + +Claude Fable 5 responds to the same prompting techniques as other Claude models, with a few differences in how to structure long-context prompts and reasoning instructions. See [Prompting Claude Fable 5](/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5). + +## Messages API on Claude Fable 5 and Claude Mythos 5 + + +The behaviors in this section are specific to Claude Fable 5 and Claude Mythos 5. The Messages API is unchanged for Opus, Sonnet, and Haiku models. + + +### Adaptive thinking is always on + +[Adaptive thinking](/docs/en/build-with-claude/adaptive-thinking) is the only thinking mode on Claude Fable 5 and Claude Mythos 5. It applies whenever the `thinking` parameter is unset, and `thinking: {"type": "disabled"}` is not supported. Use the [effort parameter](/docs/en/build-with-claude/effort) to control thinking depth. + +### Raw thinking content is never returned + +The raw chain of thought is never returned on Claude Fable 5 and Claude Mythos 5. `thinking.display` defaults to `"omitted"`, which returns thinking blocks with an empty `thinking` field; set `display: "summarized"` to receive readable summarized thinking. Pass thinking blocks back unchanged in multi-turn conversations on the same model. See [thinking output on Claude Fable 5 and Claude Mythos 5](/docs/en/build-with-claude/adaptive-thinking#thinking-output-on-claude-fable-5-and-claude-mythos-5) for cross-model handling. + +## Supported features + +At launch, Claude Fable 5 and Claude Mythos 5 support: + +- [Effort](/docs/en/build-with-claude/effort) +- [Task budgets](/docs/en/build-with-claude/task-budgets) (beta: set the `task-budgets-2026-03-13` header) +- The [memory tool](/docs/en/agents-and-tools/tool-use/memory-tool) +- Tool result clearing through [context editing](/docs/en/build-with-claude/context-editing) (beta: set the `context-management-2025-06-27` header) +- [Compaction](/docs/en/build-with-claude/compaction) +- [Vision](/docs/en/build-with-claude/vision) + +## Migrating from earlier models + +If you're migrating from Claude Mythos Preview, see [Migrating from Claude Mythos Preview to Claude Mythos 5](/docs/en/about-claude/models/migration-guide#migrating-from-claude-mythos-preview) for step-by-step instructions. + +If you're migrating from Claude Opus 4.8, see [Migrating from Claude Opus 4.8 to Claude Fable 5](/docs/en/about-claude/models/migration-guide#migrating-from-claude-opus-48). + +## Next steps + + + + Step-by-step upgrade instructions from Claude Opus 4.8 and Claude Mythos Preview. + + + Specs and comparison for all current Claude models. + + + The only thinking mode on Claude Fable 5 and Claude Mythos 5. + + + How Claude Fable 5 declines requests, and how to retry on another model. + + + Avoid paying the prompt-cache cost twice on a retry. + + + A worked end-to-end example of refusal handling, fallback, and billing. + + + Control thinking depth and cost on Claude Fable 5 and Claude Mythos 5. + + + Fable-specific prompting techniques. + + diff --git a/llmsdk_docs/claude5/docs/migration-guide.md b/llmsdk_docs/claude5/docs/migration-guide.md new file mode 100644 index 00000000..e75fe0c3 --- /dev/null +++ b/llmsdk_docs/claude5/docs/migration-guide.md @@ -0,0 +1,1741 @@ +# Migration guide + +Guide for migrating to the latest Claude models from previous Claude versions + +--- + + +This guide covers migrating [Messages API](/docs/en/build-with-claude/working-with-messages) code. If you use [Claude Managed Agents](/docs/en/managed-agents/overview), no changes beyond updating the model name are required. + + + + **Automate your migration with the Claude API skill.** In Claude Code, run `/claude-api migrate` to invoke the bundled [Claude API skill](/docs/en/agents-and-tools/agent-skills/claude-api-skill#migrating-to-a-newer-claude-model). It works for any target model on this page: + + ```text + /claude-api migrate this project to claude-opus-4-8 + ``` + + The skill applies the model ID swap and, as needed, breaking parameter changes, prefill replacement, and effort calibration for your target model across your codebase, then produces a checklist of items to verify manually. It asks you to confirm the migration scope (entire working directory, a subdirectory, or a specific file list) before editing any files. The skill also detects Amazon Bedrock, Vertex AI, Claude Platform on AWS, and Microsoft Foundry clients and adjusts model ID formats and feature changes for each platform. + + +## Migrating from Claude Mythos Preview to Claude Mythos 5 \{#migrating-from-claude-mythos-preview} + +[Claude Mythos 5](https://anthropic.com/glasswing) is the access-gated successor to [Claude Mythos Preview](https://anthropic.com/glasswing), the invitation-only research preview. For general availability, see Claude Fable 5. Migration is mostly drop-in: Claude Mythos 5 uses the same [Messages API](/docs/en/build-with-claude/working-with-messages) and the same [tool use](/docs/en/agents-and-tools/tool-use/overview) patterns as Claude Mythos Preview. The key changes are the features that are no longer available (listed in the next section) and thinking output. Token counts are roughly unchanged: Claude Mythos 5 uses the same tokenizer as Claude Mythos Preview. For the Claude Mythos Preview retirement timeline, see [Model deprecations](/docs/en/about-claude/model-deprecations). + +### Update your model name + +```python +model = "claude-mythos-preview" # Before +model = "claude-mythos-5" # After +``` + +### Features not available on Claude Mythos 5 + +1. **Extended thinking and thinking token budgets:** Manual extended thinking (`thinking: {type: "enabled", budget_tokens: N}`) is not supported on `claude-mythos-5` and returns a 400 error. [Adaptive thinking](/docs/en/build-with-claude/adaptive-thinking) is always on: the model determines when and how much to think on each request, and no `thinking` configuration is required. `thinking: {type: "disabled"}` returns an error. `budget_tokens` has no direct replacement: thinking is adaptive, and the [effort parameter](/docs/en/build-with-claude/effort) is a separate output-level control, not a thinking budget. + + Before (Claude Mythos Preview): + + ```python + client.messages.create( + model="claude-mythos-preview", + max_tokens=16000, + thinking={"type": "enabled", "budget_tokens": 10000}, + messages=[{"role": "user", "content": "..."}], + ) + ``` + + After (Claude Mythos 5): + + + ```python nocheck + client.messages.create( + model="claude-mythos-5", + max_tokens=16000, + messages=[{"role": "user", "content": "..."}], + ) + ``` + +2. **Assistant prefill:** Prefilling the assistant message is not supported on `claude-mythos-5` and returns a 400 error, the same as on Claude Mythos Preview. Use system prompt instructions instead. + +3. **Thinking output:** On `claude-mythos-5`, the raw chain of thought is never returned, but thinking blocks still carry readable summarized text when `thinking.display` is set to `summarized`. Pass thinking blocks back unchanged when continuing a conversation on the same model. See [Thinking output on Claude Fable 5 and Claude Mythos 5](/docs/en/build-with-claude/adaptive-thinking#thinking-output-on-claude-fable-5-and-claude-mythos-5). + +### Token counting and billing + +`claude-mythos-5` uses the same tokenizer as `claude-mythos-preview` (the tokenizer introduced with Claude Opus 4.7). Token counts are roughly unchanged when migrating from `claude-mythos-preview`. The same content can tokenize to roughly 30% more tokens compared with models before Claude Opus 4.7, varying by content and workload shape. + +[`/v1/messages/count_tokens`](/docs/en/build-with-claude/token-counting) returns roughly unchanged values for `claude-mythos-5` compared with `claude-mythos-preview`. Re-baseline cost and latency on your own workloads. + +### Migration checklist + +- [ ] Update the model name from `claude-mythos-preview` to `claude-mythos-5`. +- [ ] Remove manual extended thinking configuration (`thinking: {type: "enabled", budget_tokens: N}`). Adaptive thinking is always on, and no `thinking` field is required. +- [ ] Remove any `thinking: {type: "disabled"}` configuration. Disabling thinking returns an error on `claude-mythos-5`. +- [ ] Remove `budget_tokens`. It has no direct replacement: thinking is adaptive, and the `effort` parameter is a separate output-level control, not a thinking budget. +- [ ] Verify any code that parses the `thinking` field treats it as display text only and passes thinking blocks back unchanged when continuing on the same model. `thinking.display` defaults to `"omitted"` on `claude-mythos-5`, the same as on Claude Mythos Preview; set `display: "summarized"` to receive readable summaries. See [Thinking output on Claude Fable 5 and Claude Mythos 5](/docs/en/build-with-claude/adaptive-thinking#thinking-output-on-claude-fable-5-and-claude-mythos-5). +- [ ] If you replay conversation history on another model, strip `thinking` and `redacted_thinking` blocks from prior assistant turns first. Thinking blocks from `claude-mythos-5` are tied to the model that produced them; models other than Claude Fable 5 and Claude Mythos 5 silently ignore them; stripping keeps cross-model requests minimal and uniform. +- [ ] Re-baseline token counts and costs on your own workloads. Token counts are roughly unchanged when migrating from `claude-mythos-preview`. + +## Migrating from Claude Opus 4.8 to Claude Fable 5 \{#migrating-from-claude-opus-48} + +[Claude Fable 5](/docs/en/about-claude/models/introducing-claude-fable-5-and-claude-mythos-5) is Anthropic's most capable widely released model, generally available on the Claude API, [Claude Platform on AWS](/docs/en/build-with-claude/claude-platform-on-aws), [Amazon Bedrock](/docs/en/build-with-claude/claude-in-amazon-bedrock), [Vertex AI](/docs/en/build-with-claude/claude-on-vertex-ai), and [Microsoft Foundry](/docs/en/build-with-claude/claude-in-microsoft-foundry). + +Migration is mostly drop-in: Claude Fable 5 uses the same [Messages API](/docs/en/build-with-claude/working-with-messages) and the same [tool use](/docs/en/agents-and-tools/tool-use/overview) patterns as Claude Opus 4.8, and supports the same [1M token context window](/docs/en/build-with-claude/context-windows) by default and [128k max output tokens](/docs/en/about-claude/models/overview). The key changes are always-on [adaptive thinking](/docs/en/build-with-claude/adaptive-thinking), thinking output, safety classifier refusals, and pricing. Token counts are roughly unchanged: Claude Fable 5 uses the same tokenizer as Claude Opus 4.8. + + +Claude Fable 5 is priced at $10 per million input tokens and $50 per million output tokens, compared with $5 and $25 for Claude Opus 4.8. See [Claude pricing](/docs/en/about-claude/pricing) for details. + + + +If your code is on Claude Opus 4.7 or earlier, first apply [Migrating from Claude Opus 4.7 to Claude Opus 4.8](#migrating-from-claude-opus-47) and, for models earlier than Claude Opus 4.7, the [Claude Opus 4.7 migration steps](#migrating-to-claude-opus-4-7). Those sections cover breaking changes (sampling parameters rejected, manual extended thinking rejected, prefill removed, new tokenizer) that this section does not repeat. + + +### Update your model name + +```python +model = "claude-opus-4-8" # Before +model = "claude-fable-5" # After +``` + +### What changed + +The items in this section describe the API and behavior differences worth checking after you swap the model ID. + +1. **Adaptive thinking is always on:** [Adaptive thinking](/docs/en/build-with-claude/adaptive-thinking) is the only thinking mode on `claude-fable-5`: the model determines when and how much to think on each request, and no `thinking` configuration is required. `thinking: {type: "disabled"}` returns an error. On Claude Opus 4.8, requests without a `thinking` field run without thinking; on `claude-fable-5`, those requests run with adaptive thinking. `max_tokens` remains a hard limit on total output, thinking plus response text, so revisit it for workloads that ran without thinking on Claude Opus 4.8. See [Cost control](/docs/en/build-with-claude/adaptive-thinking#cost-control). Use the [effort parameter](/docs/en/build-with-claude/effort) to control thinking depth. + + Before (Claude Opus 4.8): + + ```python + client.messages.create( + model="claude-opus-4-8", + max_tokens=16000, + thinking={"type": "adaptive"}, + output_config={"effort": "high"}, + messages=[{"role": "user", "content": "..."}], + ) + ``` + + After (Claude Fable 5): + + ```python + client.messages.create( + model="claude-fable-5", + max_tokens=16000, + output_config={"effort": "high"}, + messages=[{"role": "user", "content": "..."}], + ) + ``` + +2. **Extended thinking and thinking budgets (unchanged):** Manual extended thinking (`thinking: {type: "enabled", budget_tokens: N}`) is not supported on `claude-fable-5` and returns a 400 error, the same as on Claude Opus 4.8. `budget_tokens` has no direct replacement: thinking is adaptive, and the [effort parameter](/docs/en/build-with-claude/effort) is a separate output-level control, not a thinking budget. + +3. **Assistant prefill (unchanged):** Prefilling the assistant message is not supported on `claude-fable-5` and returns a 400 error, the same as on Claude Opus 4.8. Use system prompt instructions instead. + +4. **Thinking output:** On `claude-fable-5`, the raw chain of thought is never returned, but thinking blocks still carry readable summarized text when `thinking.display` is set to `summarized`. Pass thinking blocks back unchanged when continuing a conversation on the same model. See [Thinking output on Claude Fable 5 and Claude Mythos 5](/docs/en/build-with-claude/adaptive-thinking#thinking-output-on-claude-fable-5-and-claude-mythos-5). + +5. **Safety classifiers and the `refusal` stop reason:** `claude-fable-5` runs safety classifiers on requests and during response generation. When a classifier declines a request, the Messages API returns `stop_reason: "refusal"`, and the `stop_details.category` field reports which classifier fired (`"cyber"`, `"bio"`, `"reasoning_extraction"`, or `null` when the refusal maps to no named category). A refused response is a successful HTTP 200 response, not an error, and you are not billed for the input tokens of a request refused before any output is generated. When a classifier fires mid-stream, the input and already-streamed output are billed; discard the partial output. To re-run refused requests on another model automatically, pass the opt-in `fallbacks` parameter (in beta on the Claude API and Claude Platform on AWS; not available on the Message Batches API or on Amazon Bedrock, Vertex AI, and Microsoft Foundry; on those three platforms run the retry client-side or use the SDK refusal-fallback middleware). See [Handling stop reasons](/docs/en/build-with-claude/refusals-and-fallback). + +6. **Start at `high` effort:** The [effort parameter](/docs/en/build-with-claude/effort) default remains `high`. On Claude Opus 4.8, the recommendation for coding and high-autonomy work is to set `xhigh` explicitly. On `claude-fable-5`, use `high` as the default for most tasks and reserve `xhigh` for the most capability-sensitive workloads: lower effort settings on `claude-fable-5` still perform well and often exceed `xhigh` performance on prior models. Reduce effort if a task completes but takes longer than necessary. See [Prompting Claude Fable 5](/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5#consider-all-effort-levels). + +7. **Lower prompt caching minimum:** The minimum cacheable prompt length on `claude-fable-5` is 512 tokens, lower than the 1,024 tokens on Claude Opus 4.8. Prompts that were too short to cache on Claude Opus 4.8 can now create cache entries, with no code changes required. On Amazon Bedrock, the minimum for `claude-fable-5` is 1,024 tokens. See [Prompt caching](/docs/en/build-with-claude/prompt-caching#cache-limitations) for per-model minimums. + +### Migration checklist + +- [ ] Update the model name from `claude-opus-4-8` to `claude-fable-5`. +- [ ] Remove any `thinking: {type: "disabled"}` configuration. Disabling thinking returns an error on `claude-fable-5`, and requests without a `thinking` field run with adaptive thinking. +- [ ] If you removed manual extended thinking and assistant prefills during earlier migrations, no action is needed: both remain unsupported on `claude-fable-5`. +- [ ] Verify any code that parses the `thinking` field treats it as display text only and passes thinking blocks back unchanged when continuing on the same model. `thinking.display` defaults to `"omitted"` on `claude-fable-5`, the same as on Claude Opus 4.8; set `display: "summarized"` to receive readable summaries. See [Thinking output on Claude Fable 5 and Claude Mythos 5](/docs/en/build-with-claude/adaptive-thinking#thinking-output-on-claude-fable-5-and-claude-mythos-5). +- [ ] If you replay conversation history on another model, strip `thinking` and `redacted_thinking` blocks from prior assistant turns first. Thinking blocks from `claude-fable-5` are tied to the model that produced them; models other than Claude Fable 5 and Claude Mythos 5 silently ignore them; stripping keeps cross-model requests minimal and uniform. The exception is redeeming a [fallback credit](/docs/en/build-with-claude/fallback-credit), which requires the request body echoed under that feature's exact rules. +- [ ] Handle `stop_reason: "refusal"` and read the `stop_details.category` field. To re-run refused requests on another model automatically, consider the opt-in `fallbacks` parameter (beta). See [Handling stop reasons](/docs/en/build-with-claude/refusals-and-fallback). +- [ ] Re-evaluate your `effort` setting. Start at `high` for most tasks, including workloads that ran at `xhigh` on Claude Opus 4.8. +- [ ] Re-baseline cost and latency on your own workloads. Token counts are roughly unchanged when migrating from `claude-opus-4-8`; per-token pricing differs. + +## Migrating from Claude Opus 4.7 to Claude Opus 4.8 \{#migrating-from-claude-opus-47} + +Claude Opus 4.8 is Anthropic's most capable Opus-tier model. It builds on Claude Opus 4.7. + +Claude Opus 4.8 should have strong out-of-the-box performance on existing Claude Opus 4.7 prompts and evals. There are no breaking API changes for code already running on Claude Opus 4.7. It supports the same set of features as Claude Opus 4.7, including the [1M token context window](/docs/en/build-with-claude/context-windows), [128k max output tokens](/docs/en/about-claude/models/overview), [adaptive thinking](/docs/en/build-with-claude/adaptive-thinking), [prompt caching](/docs/en/build-with-claude/prompt-caching), [batch processing](/docs/en/build-with-claude/batch-processing), the [Files API](/docs/en/build-with-claude/files), [PDF support](/docs/en/build-with-claude/pdf-support), [vision](/docs/en/build-with-claude/vision), and the full set of server-side and client-side [tools](/docs/en/agents-and-tools/tool-use/overview). It also adds [mid-conversation system messages](/docs/en/about-claude/models/whats-new-claude-4-8#mid-conversation-system-messages) and publicly documents [refusal stop details](/docs/en/about-claude/models/whats-new-claude-4-8#refusal-stop-details). + + +If your code is on Claude Opus 4.6 or earlier, also apply the [Claude Opus 4.7 migration steps](#migrating-to-claude-opus-4-7) below before upgrading to Claude Opus 4.8. Those steps include breaking changes (sampling parameters rejected, manual extended thinking rejected, new tokenizer) that the 4.8 upgrade alone does not cover. + + + +On Microsoft Foundry, Claude Opus 4.8 has a 200k-token context window at launch. The 1M context window applies on the Claude API, Amazon Bedrock, and Vertex AI. See [Claude in Microsoft Foundry](/docs/en/build-with-claude/claude-in-microsoft-foundry). + + +### Update your model name + +```python +# Opus migration +model = "claude-opus-4-7" # Before +model = "claude-opus-4-8" # After +``` + +### What changed + +These are not breaking changes. Code that runs on Claude Opus 4.7 continues to work unchanged on Claude Opus 4.8. The items below describe behavior differences worth checking after you swap the model ID. + +1. **Sampling parameters (unchanged):** Setting `temperature`, `top_p`, or `top_k` to a non-default value returns a 400 error on Claude Opus 4.8, the same as on Claude Opus 4.7. The SDK request types still define these fields for compatibility with earlier models, so code that sets them type-checks, but the API rejects the request server-side. If you removed these parameters when migrating to Opus 4.7, no further changes are needed. + +2. **Effort default is `high`:** The [effort parameter](/docs/en/build-with-claude/effort) default on Claude Opus 4.8 is `high` across all surfaces, including Claude Code and the Messages API. If you already set effort explicitly, your setting is unchanged. For coding and high-autonomy work, set `xhigh` explicitly. Re-evaluate your effort setting against your latency and cost budget. + +3. **1M context window is the default:** Claude Opus 4.8 serves the full 1M token [context window](/docs/en/build-with-claude/context-windows) by default with no beta header and no long-context premium. If your client passes a context-window beta header for compatibility with older models, you can remove it on Claude Opus 4.8. + +4. **Mid-conversation system messages:** Claude Opus 4.8 accepts `role: "system"` messages immediately after a user turn in the `messages` array (subject to [placement rules](/docs/en/build-with-claude/mid-conversation-system-messages#limitations)). Use the top-level `system` field for instructions that apply from the start. Earlier models, including Claude Opus 4.7, reject `role: "system"` in `messages` with a 400 error. If you maintain code paths that rebuild the full message history to update instructions, you can simplify them and preserve [prompt cache](/docs/en/build-with-claude/prompt-caching) hits on earlier turns. + +5. **Refusal stop details:** The `stop_details` object on refusal responses (available since Claude Opus 4.7) is now publicly documented. When the model declines a request, it identifies the category of refusal, in addition to the existing `refusal` stop reason. No beta header is required, and there is no opt-out. See [Handling stop reasons](/docs/en/build-with-claude/handling-stop-reasons). + +6. **Lower prompt caching minimum:** The minimum cacheable prompt length on Claude Opus 4.8 is 1,024 tokens, lower than on Claude Opus 4.7. Prompts that were too short to cache on Claude Opus 4.7 can now create cache entries, with no code changes required. See [Prompt caching](/docs/en/build-with-claude/prompt-caching#cache-limitations) for per-model minimums. + +7. **Effort levels recalibrated:** The token allocation behind each effort level changes on Claude Opus 4.8 compared to Claude Opus 4.7: `medium` allows somewhat more thinking, `high` somewhat less, and `xhigh` substantially more. If you tuned an effort level against Claude Opus 4.7 cost or latency, re-baseline at the same level before adjusting it. See [Effort](/docs/en/build-with-claude/effort). + +### Migration checklist + +- [ ] Update model name from `claude-opus-4-7` to `claude-opus-4-8` (or update aliases). +- [ ] If you removed sampling parameters during the Opus 4.7 migration, no action is needed. If you re-added them with a 400-retry path, remove that retry path. +- [ ] Re-evaluate your `effort` setting. The default is `high` across all surfaces; for coding and high-autonomy work, set `xhigh` explicitly. +- [ ] Remove any context-window beta header. The 1M context window is the default on the Claude API, Amazon Bedrock, and Vertex AI (200k on Microsoft Foundry). +- [ ] If you rebuild conversation history to update instructions, consider switching to a mid-conversation system message to preserve prompt cache hits. +- [ ] Verify your stop-reason handling reads `stop_details` on refusals (available since Claude Opus 4.7; now publicly documented). +- [ ] Re-baseline cost and latency at your chosen effort level. + +## Migrating to Claude Opus 4.7 + +Claude Opus 4.7 is highly autonomous and performs exceptionally well on long-horizon agentic work, knowledge work, vision tasks, and memory tasks. + +Claude Opus 4.7 should have strong out-of-the-box performance on existing Claude Opus 4.6 prompts and evals at the same `$5 / $25` per MTok pricing, but there are a handful of behavioral and API changes worth knowing about as you migrate. It supports the same set of features as Claude Opus 4.6, including: + +- [1M token context window](/docs/en/build-with-claude/context-windows) at standard API pricing with no long-context premium +- [128k max output tokens](/docs/en/about-claude/models/overview) +- [Adaptive thinking](/docs/en/build-with-claude/adaptive-thinking) +- [Prompt caching](/docs/en/build-with-claude/prompt-caching) +- [Batch processing](/docs/en/build-with-claude/batch-processing) +- [Files API](/docs/en/build-with-claude/files) +- [PDF support](/docs/en/build-with-claude/pdf-support) +- [Vision](/docs/en/build-with-claude/vision) +- The full set of server-side and client-side [tools](/docs/en/agents-and-tools/tool-use/overview) ([bash](/docs/en/agents-and-tools/tool-use/bash-tool), [code execution](/docs/en/agents-and-tools/tool-use/code-execution-tool), [computer use](/docs/en/agents-and-tools/tool-use/computer-use-tool), [text editor](/docs/en/agents-and-tools/tool-use/text-editor-tool), [web search](/docs/en/agents-and-tools/tool-use/web-search-tool), [web fetch](/docs/en/agents-and-tools/tool-use/web-fetch-tool), [MCP connector](/docs/en/agents-and-tools/mcp-connector), [memory](/docs/en/agents-and-tools/tool-use/memory-tool)) + +### Update your model name + +```python +# Opus migration +model = "claude-opus-4-6" # Before +model = "claude-opus-4-7" # After +``` + +### Breaking changes + +1. **Extended thinking removed:** `thinking: {type: "enabled", budget_tokens: N}` is no longer supported on Claude Opus 4.7 or later models and returns a 400 error. Switch to [adaptive thinking](/docs/en/build-with-claude/adaptive-thinking) (`thinking: {type: "adaptive"}`) and use the [effort parameter](/docs/en/build-with-claude/effort) to control thinking depth. Adaptive thinking is **off by default** on Claude Opus 4.7: requests with no `thinking` field run without thinking, matching Opus 4.6 behavior. Set `thinking: {type: "adaptive"}` explicitly to enable it. + + Before (Claude Opus 4.6): + + ```python + client.messages.create( + model="claude-opus-4-6", + max_tokens=16000, + thinking={"type": "enabled", "budget_tokens": 10000}, + messages=[{"role": "user", "content": "..."}], + ) + ``` + + After (Claude Opus 4.7): + + ```python + client.messages.create( + model="claude-opus-4-7", + max_tokens=16000, + thinking={"type": "adaptive"}, + output_config={"effort": "high"}, # or "max", "xhigh", "medium", "low" + messages=[{"role": "user", "content": "..."}], + ) + ``` + + Adaptive thinking is steerable through prompting. For guidance on tuning when the model over- or under-thinks, see [Calibrating effort and thinking depth](/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-4-8#calibrating-effort-and-thinking-depth). + +2. **Sampling parameters removed:** Setting `temperature`, `top_p`, or `top_k` to any non-default value on Claude Opus 4.7 returns a 400 error. The safest migration path is to omit these parameters entirely from request payloads. Prompting is the recommended way to guide model behavior on Claude Opus 4.7. If you were using `temperature = 0` for determinism, note that it never guaranteed identical outputs on prior models. + +3. **Thinking content omitted by default:** Thinking blocks still appear in the response stream on Claude Opus 4.7, but their `thinking` field is empty unless you explicitly opt in. This is a silent change from Claude Opus 4.6, where the default was to return summarized thinking text. To restore summarized thinking content on Claude Opus 4.7, set `thinking.display` to `"summarized"`: + + ```python + thinking = { + "type": "adaptive", + "display": "summarized", + } + ``` + + The default is `"omitted"` on Claude Opus 4.7. If your product streams reasoning to users, the new default appears as a long pause before output begins; set `display: "summarized"` to restore visible progress during thinking. See [Extended thinking](/docs/en/build-with-claude/extended-thinking#controlling-thinking-display) for details. + +4. **Updated token counting:** Claude Opus 4.7 uses a new tokenizer, contributing to its improved performance on a wide range of tasks. The new tokenizer may use roughly 1x to 1.35x as many tokens when processing text compared to previous models (up to ~35% more, varying by content). + + [`/v1/messages/count_tokens`](/docs/en/build-with-claude/token-counting) will return a different number of tokens for Claude Opus 4.7 than it did for Claude Opus 4.6. Token efficiency can vary by workload shape. + + Prompting interventions, `task_budget`, and `effort` can help control costs and ensure appropriate token usage. These controls may trade off model intelligence. Update your `max_tokens` parameters to give additional headroom, including compaction triggers. Claude Opus 4.7 provides a 1M context window at standard API pricing with no long-context premium. + +5. **Prefill removal (carried over from Opus 4.6):** Prefilling assistant messages returns a 400 error on Claude Opus 4.7. Use [structured outputs](/docs/en/build-with-claude/structured-outputs), system prompt instructions, or `output_config.format` instead. + +### Choosing an effort level + +The [effort parameter](/docs/en/build-with-claude/effort) allows you to tune Claude's intelligence vs. token spend, trading off capability for faster speed and lower costs. Start with the new `xhigh` effort level for coding and agentic use cases, and use a minimum of `high` effort for most intelligence-sensitive use cases. Experiment with other effort levels to further tune token usage and intelligence: + +- **`max`:** Max effort can deliver performance gains in some use cases, but may show diminishing returns from increased token usage. This setting can also sometimes be prone to overthinking. Test max effort for intelligence-demanding tasks. +- **`xhigh` (new):** Extra high effort is the best setting for most coding and agentic use cases. +- **`high`:** This setting balances token usage and intelligence. For most intelligence-sensitive use cases, use a minimum of `high` effort. +- **`medium`:** Good for cost-sensitive use cases that need to reduce token usage while trading off intelligence. +- **`low`:** Reserve for short, scoped tasks and latency-sensitive workloads that are not intelligence-sensitive. + +Effort is more important for this model than for any prior Opus. Experiment with it actively when you upgrade. + +### Behavior changes + +Claude Opus 4.7 has several behavioral differences from Claude Opus 4.6 that are not API breaking changes but may require prompt updates or scaffolding removal. + +1. **Response length varies by use case:** Claude Opus 4.7 calibrates response length to how complex it judges the task to be, rather than defaulting to a fixed verbosity. This usually means shorter answers on simple lookups and much longer ones on open-ended analysis. + + If your product depends on a certain style or verbosity of output, you may need to tune your prompts. For example, to decrease verbosity, add: "Provide concise, focused responses. Skip non-essential context, and keep examples minimal." If you see specific kinds of over-explaining, add targeted instructions in your prompt to prevent them. + + Positive examples showing how Claude can communicate with the appropriate level of concision tend to be more effective than negative examples or instructions that tell the model what not to do. + +2. **More literal instruction following:** Claude Opus 4.7 interprets prompts more literally and explicitly than Claude Opus 4.6, particularly at lower effort levels. It will not silently generalize an instruction from one item to another, and it will not infer requests you didn't make. The upside of this literalism is precision and less thrash. It generally performs better for API use cases with carefully tuned prompts, structured extraction, and pipelines where you want predictable behavior. A prompt and harness review may be especially helpful for migration to Claude Opus 4.7. + +3. **More direct tone:** As with any new model, prose style on long-form writing may shift. Claude Opus 4.7 is more direct and opinionated, with less validation-forward phrasing and fewer emoji than Claude Opus 4.6's warmer style. If your product relies on a specific voice, re-evaluate style prompts against the new baseline. + +4. **Built-in progress updates in agentic traces:** Claude Opus 4.7 provides more regular, higher-quality updates to the user throughout long agentic traces. If you've added scaffolding to force interim status messages ("After every 3 tool calls, summarize progress"), try removing it. If you find that the length or contents of Claude Opus 4.7's user-facing updates are not well-calibrated to your use case, explicitly describe what these updates should look like in the prompt and provide examples. + +5. **Fewer subagents spawned by default:** Claude Opus 4.7 tends to spawn fewer subagents by default. However, this behavior is steerable through prompting; give Claude Opus 4.7 explicit guidance around when subagents are desirable. + +6. **Stricter effort calibration:** Meaningfully changing from Claude Opus 4.6, Claude Opus 4.7 respects [effort levels](/docs/en/build-with-claude/effort) strictly, especially at the low end. At `low` and `medium`, the model scopes its work to what was asked rather than going above and beyond. + + This is good for latency and cost, but on moderately complex tasks running at `low` effort there is some risk of under-thinking. If you observe shallow reasoning on complex problems, raise effort to `high` or `xhigh` rather than prompting around it. + + If you need to keep effort at `low` for latency, add targeted guidance: "This task involves multi-step reasoning. Think carefully through the problem before responding." See [Recommended effort levels for Claude Opus 4.7](/docs/en/build-with-claude/effort#recommended-effort-levels-for-claude-opus-4-7). + +7. **Fewer tool calls by default:** Claude Opus 4.7 has a tendency to use tools less often than Claude Opus 4.6 and to use reasoning more. This produces better results in most cases. + + To increase tool usage, raise the effort setting. `high` or `xhigh` effort settings show substantially more tool usage in agentic search and coding. You can also adjust your prompt to explicitly instruct the model about when and how to properly use its tools. + +8. **Real-time cybersecurity safeguards:** Newly added in Claude Opus 4.7, requests that involve prohibited or high-risk topics may lead to refusals. For legitimate security work such as penetration testing, vulnerability research, or red-teaming, apply to the [Cyber Verification Program](https://claude.com/form/cyber-use-case) to request reduced restrictions. See [Safeguards, warnings, and appeals](https://support.claude.com/en/articles/8241253-safeguards-warnings-and-appeals) for background. + +9. **High-resolution image support:** Claude Opus 4.7 is the first Claude model with high-resolution image support. Maximum image resolution is 2576 pixels on the long edge, up from 1568 pixels on prior models. This unlocks gains on vision-heavy workloads and is particularly valuable for computer use, screenshot understanding, and document analysis. + + High-resolution support is automatic and requires no beta header or client-side opt-in. Two things to plan for: + + - Full-resolution images can use up to approximately 3x more image tokens than on prior models (up to 4,784 tokens per image, compared to the previous cap of roughly 1,600 tokens per image). Re-budget `max_tokens` and cost expectations for image-heavy workloads, or downsample before sending if you do not need the additional fidelity. + - Pointing and bounding-box coordinates returned by the model are 1\:1 with actual image pixels on Claude Opus 4.7, so no scale-factor conversion is required. + + See [High-resolution image support on Claude Opus 4.7](/docs/en/build-with-claude/vision#high-resolution-image-support-on-claude-opus-4-7) for details. + +### Recommended changes + +These are not required but will improve your experience: + +1. **Re-evaluate `max_tokens`:** Because the same text produces a higher token count on Claude Opus 4.7, update your `max_tokens` parameters to give additional headroom, including compaction triggers. Prompting interventions, [`task_budget`](/docs/en/build-with-claude/task-budgets), and [`effort`](/docs/en/build-with-claude/effort) can help control costs and ensure appropriate token usage. + +2. **Audit token-count expectations:** Any code path that estimates tokens client-side or assumes a fixed token-to-character ratio should be re-tested against Claude Opus 4.7. Use the [Token counting endpoint](/docs/en/build-with-claude/token-counting) to verify. + +3. **Adopt [task budgets](/docs/en/build-with-claude/task-budgets) (beta):** Claude Opus 4.7 introduces task budgets. These budgets let you inform Claude how many tokens it has for a full agentic loop, including thinking, tool calls, tool results, and final output. The model sees a running countdown and uses it to prioritize work and finish the task gracefully as the budget is consumed. To use, set the beta header `task-budgets-2026-03-13` and add the following to your output config: + + ```python + output_config = { + "effort": "high", + "task_budget": {"type": "tokens", "total": 128000}, + } + ``` + + You may need to experiment with different task budgets for your use case. If the model is given a task budget that is too restrictive, it may complete the task less thoroughly, referencing its budget as the constraint. + + For open-ended agentic tasks where quality matters more than speed, do not set a task budget. Reserve task budgets for workloads where you need the model to scope its work to a token allowance. The minimum value for a task budget is 20k tokens. + + A task budget is not a hard cap; it's a suggestion that the model is aware of. It differs from `max_tokens`: + + - **`task_budget`:** an advisory cap across the full agentic loop. The model sees it and uses it to pace itself. + - **`max_tokens`:** a hard per-request ceiling on generated tokens. It is not passed to the model, so the model is not aware of it. + + Use `task_budget` when you want the model to self-moderate, and `max_tokens` as a hard ceiling to cap usage. + +4. **Set a large `max_tokens` at `max` or `xhigh` effort:** If you are running Claude Opus 4.7 at `max` or `xhigh` effort, set a large max output token budget so the model has room to think and act across its subagents and tool calls. Start at 64k tokens and tune from there. + +5. **Downsample images if high resolution is unnecessary:** Claude Opus 4.7 supports images up to 2576px / 3.75MP. High-res images use more tokens. If the additional image fidelity is unnecessary, downsample images before sending to Claude to avoid token-usage increases. See [Images and vision](/docs/en/build-with-claude/vision). + +### Migration checklist + +- [ ] Update model name from `claude-opus-4-6` to `claude-opus-4-7` (or update aliases). +- [ ] Remove `temperature`, `top_p`, and `top_k` from request payloads. +- [ ] Replace `thinking: {type: "enabled", budget_tokens: N}` with `thinking: {type: "adaptive"}` plus the [effort parameter](/docs/en/build-with-claude/effort). +- [ ] Remove any assistant-message prefills. +- [ ] If your UI displays thinking content, explicitly opt in to thinking summarization. +- [ ] Re-benchmark end-to-end cost and latency under the updated tokenization. +- [ ] Re-tune `max_tokens` to account for the updated tokenization. +- [ ] Re-test any client-side token-count estimations. +- [ ] If your application sends images, re-budget for [high-resolution image support](/docs/en/build-with-claude/vision#high-resolution-image-support-on-claude-opus-4-7) (up to approximately 3x more image tokens per full-resolution image). Downsample before sending if you do not need the additional fidelity. +- [ ] If you consume pointing or bounding-box coordinates from the model, remove any scale-factor conversion; coordinates are 1\:1 with actual image pixels on Claude Opus 4.7. +- [ ] Review prompts for the behavior changes above (response length, literalism, tone, progress updates, subagents, effort calibration, tool triggering, cyber safeguards, high-resolution image handling). +- [ ] Re-baseline response length with existing length-control prompts removed, then tune explicitly. +- [ ] If using `xhigh` or `max` effort, raise `max_tokens` to at least 64k as a starting point. +- [ ] Consider adopting task budgets (beta) for agentic workflows. +- [ ] If your product does legitimate security work, apply to the [Cyber Verification Program](https://claude.com/form/cyber-use-case) for access to lower restrictions on cyber content. + +## Migrating to Claude Opus 4.7 from Opus 4.5 or earlier + +If you are migrating from Claude Opus 4.5, Opus 4.1 (deprecated), or an earlier model directly to Claude Opus 4.7, apply **all of the [Opus 4.7 changes above](#migrating-to-claude-opus-4-7)** plus the cumulative changes in this section that took effect between Opus 4.5 and Opus 4.7. If you are migrating from Opus 4.6, you only need the [Opus 4.7 section above](#migrating-to-claude-opus-4-7). + +### Update your model name + +```python +# Opus migration +model = "claude-opus-4-5" # Before +model = "claude-opus-4-7" # After +``` + +### Breaking changes + +1. **Prefill removal** is covered in the [Opus 4.7 breaking changes](#breaking-changes) above. + +2. **Tool parameter quoting:** Claude Opus 4.6 and later models may produce slightly different JSON string escaping in tool call arguments (e.g., different handling of Unicode escapes or forward slash escaping). If you parse tool call `input` as a raw string rather than using a JSON parser, verify your parsing logic. Standard JSON parsers (like `json.loads()` or `JSON.parse()`) handle these differences automatically. + +### Recommended changes + +These changes improve your experience on Opus 4.7. Items marked **(required on Opus 4.7)** were optional recommendations when Opus 4.6 launched but are now mandatory; the rest remain recommended. + +1. **Migrate to adaptive thinking (required on Opus 4.7):** `thinking: {type: "enabled", budget_tokens: N}` returns a 400 error on Claude Opus 4.7. Switch to `thinking: {type: "adaptive"}` and use the [effort parameter](/docs/en/build-with-claude/effort) to control thinking depth. See [Adaptive thinking](/docs/en/build-with-claude/adaptive-thinking). + + + ```bash cURL + curl -sS https://api.anthropic.com/v1/messages \ + -H "content-type: application/json" \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H "anthropic-version: 2023-06-01" \ + -d '{ + "model": "claude-opus-4-7", + "max_tokens": 16000, + "thinking": {"type": "adaptive"}, + "output_config": {"effort": "high"}, + "messages": [{"role": "user", "content": "Your prompt here"}] + }' + ``` + + ```python Before hidelines={1..3} + import anthropic + + client = anthropic.Anthropic() + response = client.beta.messages.create( + model="claude-opus-4-5", + max_tokens=16000, + thinking={"type": "enabled", "budget_tokens": 32000}, + betas=["interleaved-thinking-2025-05-14"], + messages=[{"role": "user", "content": "Your prompt here"}], + ) + ``` + + ```python After + response = client.messages.create( + model="claude-opus-4-7", + max_tokens=16000, + thinking={"type": "adaptive"}, + output_config={"effort": "high"}, + messages=[{"role": "user", "content": "Your prompt here"}], + ) + ``` + + ```bash CLI + ant messages create <<'YAML' + model: claude-opus-4-7 + max_tokens: 16000 + thinking: + type: adaptive + output_config: + effort: high + messages: + - role: user + content: Your prompt here + YAML + ``` + + ```typescript TypeScript hidelines={1..2} + import Anthropic from "@anthropic-ai/sdk"; + + const client = new Anthropic(); + + const response = await client.messages.create({ + model: "claude-opus-4-7", + max_tokens: 16000, + thinking: { type: "adaptive" }, + output_config: { effort: "high" }, + messages: [{ role: "user", content: "Your prompt here" }] + }); + ``` + + ```csharp C# + using Anthropic; + using Anthropic.Models.Messages; + + AnthropicClient client = new(); + + var parameters = new MessageCreateParams + { + Model = Model.ClaudeOpus4_7, + MaxTokens = 16000, + Thinking = new ThinkingConfigAdaptive(), + OutputConfig = new OutputConfig { Effort = Effort.High }, + Messages = [new() { Role = Role.User, Content = "Your prompt here" }] + }; + + var response = await client.Messages.Create(parameters); + Console.WriteLine(response); + ``` + + ```go Go hidelines={1..11,-1} + package main + + import ( + "context" + "fmt" + "log" + + "github.com/anthropics/anthropic-sdk-go" + ) + + func main() { + client := anthropic.NewClient() + + response, err := client.Messages.New(context.TODO(), anthropic.MessageNewParams{ + Model: anthropic.ModelClaudeOpus4_7, + MaxTokens: 16000, + Thinking: anthropic.ThinkingConfigParamUnion{ + OfAdaptive: &anthropic.ThinkingConfigAdaptiveParam{}, + }, + OutputConfig: anthropic.OutputConfigParam{ + Effort: anthropic.OutputConfigEffortHigh, + }, + Messages: []anthropic.MessageParam{ + anthropic.NewUserMessage(anthropic.NewTextBlock("Your prompt here")), + }, + }) + if err != nil { + log.Fatal(err) + } + fmt.Println(response) + } + ``` + + ```java Java hidelines={1..5,8..10,-2..} + import com.anthropic.client.AnthropicClient; + import com.anthropic.client.okhttp.AnthropicOkHttpClient; + import com.anthropic.models.messages.MessageCreateParams; + import com.anthropic.models.messages.Message; + import com.anthropic.models.messages.Model; + import com.anthropic.models.messages.OutputConfig; + import com.anthropic.models.messages.ThinkingConfigAdaptive; + + public class AdaptiveThinkingExample { + public static void main(String[] args) { + AnthropicClient client = AnthropicOkHttpClient.fromEnv(); + + MessageCreateParams params = MessageCreateParams.builder() + .model(Model.CLAUDE_OPUS_4_7) + .maxTokens(16000L) + .thinking(ThinkingConfigAdaptive.builder().build()) + .outputConfig(OutputConfig.builder() + .effort(OutputConfig.Effort.HIGH) + .build()) + .addUserMessage("Your prompt here") + .build(); + + Message response = client.messages().create(params); + System.out.println(response); + } + } + ``` + + ```php PHP hidelines={1..4} + messages->create( + maxTokens: 16000, + messages: [['role' => 'user', 'content' => 'Your prompt here']], + model: 'claude-opus-4-7', + thinking: ['type' => 'adaptive'], + outputConfig: ['effort' => 'high'], + ); + ``` + + ```ruby Ruby hidelines={1..2} + require "anthropic" + + client = Anthropic::Client.new + + response = client.messages.create( + model: "claude-opus-4-7", + max_tokens: 16000, + thinking: { type: "adaptive" }, + output_config: { effort: "high" }, + messages: [{ role: "user", content: "Your prompt here" }] + ) + ``` + + + Note that the migration also moves from `client.beta.messages.create` to `client.messages.create`. Adaptive thinking and effort are GA features and do not require the beta SDK namespace or any beta headers. + +2. **Remove effort beta header:** The effort parameter is now GA. Remove `betas=["effort-2025-11-24"]` from your requests. + +3. **Remove fine-grained tool streaming beta header:** Fine-grained tool streaming is now GA. Remove `betas=["fine-grained-tool-streaming-2025-05-14"]` from your requests. + +4. **Remove interleaved thinking beta header:** Adaptive thinking automatically enables interleaved thinking on Claude Opus 4.7, Opus 4.6, and Sonnet 4.6. Remove `betas=["interleaved-thinking-2025-05-14"]` from your requests. The header is still functional on Sonnet 4.6 with manual extended thinking, but manual mode is deprecated. + +5. **Migrate to output_config.format:** If using structured outputs, update `output_format={...}` to `output_config={"format": {...}}`. The old parameter remains functional but is deprecated and will be removed in a future model release. + +### Migrating from Claude 4.1 or earlier + +If you're migrating from Opus 4.1 (deprecated), Sonnet 4 (deprecated), or earlier models directly to Claude Opus 4.7, apply the Claude Opus 4.7 changes at the top of this guide and the cumulative changes above plus the additional changes in this section. + +```python +# From Opus 4.1 +model = "claude-opus-4-1-20250805" # Before +model = "claude-opus-4-7" # After + +# From Sonnet 4 +model = "claude-sonnet-4-20250514" # Before +model = "claude-opus-4-7" # After + +# From Sonnet 3.7 +model = "claude-3-7-sonnet-20250219" # Before +model = "claude-opus-4-7" # After +``` + +#### Additional breaking changes + +1. **Remove sampling parameters** + + + This is a breaking change when migrating from Claude 3.x models. + + + Starting with Claude Opus 4.7, setting `temperature`, `top_p`, or `top_k` to any non-default value will return a 400 error. The safest migration path is to omit these parameters entirely from requests, and to use prompting to guide the model's behavior. If you were using `temperature = 0` for determinism, note that it never guaranteed identical outputs. + + + ```python Python nocheck + # Before - This will error in Claude 4+ models + response = client.messages.create( + model="claude-3-7-sonnet-20250219", + temperature=0.7, + top_p=0.9, # Non-default sampling params return 400 on Opus 4.7 + # ... + ) + + # After + response = client.messages.create( + model="claude-opus-4-7", + # ... + ) + ``` + +2. **Update tool versions** + + + This is a breaking change when migrating from Claude 3.x models. + + + Update to the latest tool versions. Remove any code using the `undo_edit` command. + + ```python + # Before + tools = [{"type": "text_editor_20250124", "name": "str_replace_editor"}] + + # After + tools = [{"type": "text_editor_20250728", "name": "str_replace_based_edit_tool"}] + ``` + + - **Text editor:** Use `text_editor_20250728` and `str_replace_based_edit_tool`. See [Text editor tool documentation](/docs/en/agents-and-tools/tool-use/text-editor-tool) for details. + - **Code execution:** Upgrade to `code_execution_20250825`. See [Code execution tool documentation](/docs/en/agents-and-tools/tool-use/code-execution-tool#upgrade-to-latest-tool-version) for migration instructions. + +3. **Handle the `refusal` stop reason** + + Update your application to [handle `refusal` stop reasons](/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals): + + + ```python Python nocheck + response = client.messages.create(...) + + if response.stop_reason == "refusal": + # Handle refusal appropriately + pass + ``` + +4. **Handle the `model_context_window_exceeded` stop reason** + + Claude 4.5+ models return a `model_context_window_exceeded` stop reason when generation stops due to hitting the context window limit, rather than the requested `max_tokens` limit. Update your application to handle this new stop reason: + + + ```python Python nocheck + response = client.messages.create(...) + + if response.stop_reason == "model_context_window_exceeded": + # Handle context window limit appropriately + pass + ``` + +5. **Verify tool parameter handling (trailing newlines)** + + Claude 4.5+ models preserve trailing newlines in tool call string parameters that were previously stripped. If your tools rely on exact string matching against tool call parameters, verify your logic handles trailing newlines correctly. + +6. **Update your prompts for behavioral changes** + + Claude 4+ models have a more concise, direct communication style and require explicit direction. Review [prompting best practices](/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) for optimization guidance. + +#### Additional recommended changes + +- **Remove legacy beta headers:** Remove `token-efficient-tools-2025-02-19` and `output-128k-2025-02-19`. All Claude 4+ models have built-in token-efficient tool use and these headers have no effect. + +### Migration checklist (from Opus 4.5 or earlier) + +- [ ] Update model ID to `claude-opus-4-7` +- [ ] Apply all [Opus 4.7 breaking changes](#migrating-to-claude-opus-4-7) (extended thinking removed, sampling parameters removed, thinking display omitted by default, updated tokenization) +- [ ] **BREAKING:** Remove assistant message prefills (returns 400 error); use structured outputs or `output_config.format` instead +- [ ] **BREAKING on Opus 4.7:** Replace `thinking: {type: "enabled", budget_tokens: N}` with `thinking: {type: "adaptive"}` plus the [effort parameter](/docs/en/build-with-claude/effort) (returns 400 on Opus 4.7) +- [ ] Verify tool call JSON parsing uses a standard JSON parser +- [ ] Remove `effort-2025-11-24` beta header (effort is now GA) +- [ ] Remove `fine-grained-tool-streaming-2025-05-14` beta header +- [ ] Remove `interleaved-thinking-2025-05-14` beta header (adaptive thinking enables interleaved thinking automatically) +- [ ] Migrate `output_format` to `output_config.format` (if applicable) +- [ ] If migrating from Claude 4.1 or earlier: remove `temperature`, `top_p`, and `top_k` (non-default values return 400 on Opus 4.7) +- [ ] If migrating from Claude 4.1 or earlier: update tool versions (`text_editor_20250728`, `code_execution_20250825`) +- [ ] If migrating from Claude 4.1 or earlier: handle `refusal` stop reason +- [ ] If migrating from Claude 4.1 or earlier: handle `model_context_window_exceeded` stop reason +- [ ] If migrating from Claude 4.1 or earlier: verify tool string parameter handling for trailing newlines +- [ ] If migrating from Claude 4.1 or earlier: remove legacy beta headers (`token-efficient-tools-2025-02-19`, `output-128k-2025-02-19`) +- [ ] Review and update prompts following [prompting best practices](/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) +- [ ] Test in development environment before production deployment + +--- + +## Migrating to Claude Sonnet 4.6 + +Claude Sonnet 4.6 combines strong intelligence with fast performance, featuring improved agentic search capabilities and free code execution when used with web search or web fetch. It is ideal for everyday coding, analysis, and content tasks. + +For a complete overview of capabilities, see the [models overview](/docs/en/about-claude/models/overview). + + +Sonnet 4.6 pricing is $3 per million input tokens, $15 per million output tokens. See [Claude pricing](/docs/en/about-claude/pricing) for details. + + +**Update your model name:** + +```python +# From Sonnet 4.5 +model = "claude-sonnet-4-5" # Before +model = "claude-sonnet-4-6" # After + +# From Sonnet 4 +model = "claude-sonnet-4-20250514" # Before +model = "claude-sonnet-4-6" # After +``` + +### Breaking changes + +#### When migrating from Sonnet 4.5 + +1. **Prefilling assistant messages is no longer supported** + + + This is a breaking change when migrating from Sonnet 4.5 or earlier. + + + Prefilling assistant messages returns a `400` error on Sonnet 4.6. Use [structured outputs](/docs/en/build-with-claude/structured-outputs), system prompt instructions, or `output_config.format` instead. + + **Common prefill use cases and migrations:** + + - **Controlling output formatting** (forcing JSON/YAML output): Use [structured outputs](/docs/en/build-with-claude/structured-outputs) or tools with enum fields for classification tasks. + + - **Eliminating preambles** (removing "Here is..." phrases): Add direct instructions in the system prompt: "Respond directly without preamble. Do not start with phrases like 'Here is...', 'Based on...', etc." + + - **Avoiding bad refusals:** Claude is much better at appropriate refusals now. Clear prompting in the user message without prefill should be sufficient. + + - **Continuations** (resuming interrupted responses): Move the continuation to the user message: "Your previous response was interrupted and ended with `[previous_response]`. Continue from where you left off." + + - **Context hydration / role consistency** (refreshing context in long conversations): Inject what were previously prefilled-assistant reminders into the user turn instead. + +2. **Tool parameter JSON escaping may differ** + + + This is a breaking change when migrating from Sonnet 4.5 or earlier. + + + JSON string escaping in tool parameters may differ from previous models. Standard JSON parsers handle this automatically, but custom string-based parsing may need updates. + +#### When migrating from Claude 3.x + +3. **Update sampling parameters** + + + This is a breaking change when migrating from Claude 3.x models. + + + Use only `temperature` OR `top_p`, not both. + +4. **Update tool versions** + + + This is a breaking change when migrating from Claude 3.x models. + + + Update to the latest tool versions (`text_editor_20250728`, `code_execution_20250825`). Remove any code using the `undo_edit` command. + +5. **Handle the `refusal` stop reason** + + Update your application to [handle `refusal` stop reasons](/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals). + +6. **Update your prompts for behavioral changes** + + Claude 4 models have a more concise, direct communication style. Review [prompting best practices](/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) for optimization guidance. + +### Recommended changes + +1. **Remove `fine-grained-tool-streaming-2025-05-14` beta header:** Fine-grained tool streaming is now GA on Sonnet 4.6 and no longer requires a beta header. +2. **Migrate `output_format` to `output_config.format`:** The `output_format` parameter is deprecated. Use `output_config.format` instead. + +### Migrating from Sonnet 4.5 + +Consider migrating from Sonnet 4.5 to Sonnet 4.6, which delivers more intelligence at the same price point. + + +Sonnet 4.6 defaults to an effort level of `high`, in contrast to Sonnet 4.5 which had no effort parameter. Consider adjusting the effort parameter as you migrate from Sonnet 4.5 to Sonnet 4.6. If not explicitly set, you may experience higher latency with the default effort level. + + +#### If you're not using extended thinking + +If you're not using extended thinking on Sonnet 4.5, you can continue without it on Sonnet 4.6. You should explicitly set effort to the level appropriate for your use case. At `low` effort with thinking disabled, you can expect similar or better performance relative to Sonnet 4.5 with no extended thinking. + + +```bash cURL +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data \ +'{ + "model": "claude-sonnet-4-6", + "max_tokens": 8192, + "output_config": { + "effort": "low" + }, + "messages": [ + { + "role": "user", + "content": "Your prompt here" + } + ] +}' +``` + +```bash CLI +ant messages create <<'YAML' +model: claude-sonnet-4-6 +max_tokens: 8192 +output_config: + effort: low +messages: + - role: user + content: Your prompt here +YAML +``` + +```python Python +response = client.messages.create( + model="claude-sonnet-4-6", + max_tokens=8192, + output_config={"effort": "low"}, + messages=[{"role": "user", "content": "Your prompt here"}], +) +``` + +```typescript TypeScript +const response = await client.messages.create({ + model: "claude-sonnet-4-6", + max_tokens: 8192, + output_config: { effort: "low" }, + messages: [{ role: "user", content: "Your prompt here" }] +}); +``` + +```csharp C# +using Anthropic; +using Anthropic.Models.Messages; + +AnthropicClient client = new(); + +var parameters = new MessageCreateParams +{ + Model = Model.ClaudeSonnet4_6, + MaxTokens = 8192, + OutputConfig = new OutputConfig + { + Effort = Effort.Low + }, + Messages = [new() { Role = Role.User, Content = "Your prompt here" }] +}; +var message = await client.Messages.Create(parameters); +Console.WriteLine(message); +``` + +```go Go hidelines={1..11,-1} +package main + +import ( + "context" + "fmt" + "log" + + "github.com/anthropics/anthropic-sdk-go" +) + +func main() { + client := anthropic.NewClient() + + response, err := client.Messages.New(context.TODO(), anthropic.MessageNewParams{ + Model: anthropic.Model("claude-sonnet-4-6"), + MaxTokens: 8192, + OutputConfig: anthropic.OutputConfigParam{ + Effort: anthropic.OutputConfigEffortLow, + }, + Messages: []anthropic.MessageParam{ + anthropic.NewUserMessage(anthropic.NewTextBlock("Your prompt here")), + }, + }) + if err != nil { + log.Fatal(err) + } + fmt.Println(response.Content[0].Text) +} +``` + +```java Java hidelines={1..5,7..9,-2..} +import com.anthropic.client.AnthropicClient; +import com.anthropic.client.okhttp.AnthropicOkHttpClient; +import com.anthropic.models.messages.MessageCreateParams; +import com.anthropic.models.messages.Message; +import com.anthropic.models.messages.Model; +import com.anthropic.models.messages.OutputConfig; + +public class Main { + public static void main(String[] args) { + AnthropicClient client = AnthropicOkHttpClient.fromEnv(); + + MessageCreateParams params = MessageCreateParams.builder() + .model(Model.CLAUDE_SONNET_4_6) + .maxTokens(8192L) + .outputConfig(OutputConfig.builder() + .effort(OutputConfig.Effort.LOW) + .build()) + .addUserMessage("Your prompt here") + .build(); + + Message response = client.messages().create(params); + response.content().stream() + .flatMap(block -> block.text().stream()) + .forEach(textBlock -> System.out.println(textBlock.text())); + } +} +``` + +```php PHP hidelines={1..4} +messages->create( + maxTokens: 8192, + messages: [['role' => 'user', 'content' => 'Your prompt here']], + model: 'claude-sonnet-4-6', + outputConfig: ['effort' => 'low'], +); +echo $message->content[0]->text; +``` + +```ruby Ruby hidelines={1..2} +require "anthropic" + +client = Anthropic::Client.new + +message = client.messages.create( + model: "claude-sonnet-4-6", + max_tokens: 8192, + output_config: { + effort: "low" + }, + messages: [ + { role: "user", content: "Your prompt here" } + ] +) +puts message.content.first.text +``` + + +#### If you're using extended thinking + +If you're using extended thinking with `budget_tokens` on Sonnet 4.5, it is still functional on Sonnet 4.6 but is deprecated. Migrate to [adaptive thinking](/docs/en/build-with-claude/adaptive-thinking) with the [effort parameter](/docs/en/build-with-claude/effort). + +##### Migrating to adaptive thinking + +[Adaptive thinking](/docs/en/build-with-claude/adaptive-thinking) is the recommended replacement for `budget_tokens` on Sonnet 4.6. It is particularly well suited to the following workload patterns: + +- **Autonomous multi-step agents:** coding agents that turn requirements into working software, data analysis pipelines, and bug finding where the model runs independently across many steps. Adaptive thinking lets the model calibrate its reasoning per step, staying on path over longer trajectories. For these workloads, start at `high` effort. If latency or token usage is a concern, scale down to `medium`. +- **Computer use agents:** Sonnet 4.6 achieved best-in-class accuracy on computer use evaluations using adaptive mode. +- **Bimodal workloads:** a mix of easy and hard tasks where adaptive skips thinking on simple queries and reasons deeply on complex ones. + +When using adaptive thinking, evaluate `medium` and `high` effort on your tasks. The right level depends on your workload's tradeoff between quality, latency, and token usage. + + +```bash cURL +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "content-type: application/json" \ + --data \ +'{ + "model": "claude-sonnet-4-6", + "max_tokens": 64000, + "thinking": { + "type": "adaptive" + }, + "output_config": { + "effort": "medium" + }, + "messages": [ + { + "role": "user", + "content": "Your prompt here" + } + ] +}' +``` + +```bash CLI nocheck +ant messages create <<'YAML' +model: claude-sonnet-4-6 +max_tokens: 64000 +thinking: + type: adaptive +output_config: + effort: medium +messages: + - role: user + content: Your prompt here +YAML +``` + +```python Python nocheck +response = client.messages.create( + model="claude-sonnet-4-6", + max_tokens=64000, + thinking={"type": "adaptive"}, + output_config={"effort": "medium"}, + messages=[{"role": "user", "content": "Your prompt here"}], +) +``` + +```typescript TypeScript nocheck +const response = await client.messages.create({ + model: "claude-sonnet-4-6", + max_tokens: 64000, + thinking: { type: "adaptive" }, + output_config: { effort: "medium" }, + messages: [{ role: "user", content: "Your prompt here" }] +}); +``` + +```csharp C# nocheck +using Anthropic; +using Anthropic.Models.Messages; + +AnthropicClient client = new(); + +var parameters = new MessageCreateParams +{ + Model = Model.ClaudeSonnet4_6, + MaxTokens = 64000, + Thinking = new ThinkingConfigAdaptive(), + OutputConfig = new OutputConfig { Effort = Effort.Medium }, + Messages = [new() { Role = Role.User, Content = "Your prompt here" }] +}; + +var message = await client.Messages.Create(parameters); +Console.WriteLine(message); +``` + +```go Go nocheck hidelines={1..11,-1} +package main + +import ( + "context" + "fmt" + "log" + + "github.com/anthropics/anthropic-sdk-go" +) + +func main() { + client := anthropic.NewClient() + + response, err := client.Messages.New(context.TODO(), anthropic.MessageNewParams{ + Model: "claude-sonnet-4-6", + MaxTokens: 64000, + Thinking: anthropic.ThinkingConfigParamUnion{ + OfAdaptive: &anthropic.ThinkingConfigAdaptiveParam{}, + }, + OutputConfig: anthropic.OutputConfigParam{ + Effort: anthropic.OutputConfigEffortMedium, + }, + Messages: []anthropic.MessageParam{ + anthropic.NewUserMessage(anthropic.NewTextBlock("Your prompt here")), + }, + }) + if err != nil { + log.Fatal(err) + } + fmt.Println(response) +} +``` + +```java Java nocheck hidelines={1..5,8..10,-2..} +import com.anthropic.client.AnthropicClient; +import com.anthropic.client.okhttp.AnthropicOkHttpClient; +import com.anthropic.models.messages.MessageCreateParams; +import com.anthropic.models.messages.Message; +import com.anthropic.models.messages.Model; +import com.anthropic.models.messages.OutputConfig; +import com.anthropic.models.messages.ThinkingConfigAdaptive; + +public class Main { + public static void main(String[] args) { + AnthropicClient client = AnthropicOkHttpClient.fromEnv(); + + MessageCreateParams params = MessageCreateParams.builder() + .model(Model.CLAUDE_SONNET_4_6) + .maxTokens(64000L) + .thinking(ThinkingConfigAdaptive.builder().build()) + .outputConfig(OutputConfig.builder() + .effort(OutputConfig.Effort.MEDIUM) + .build()) + .addUserMessage("Your prompt here") + .build(); + + Message response = client.messages().create(params); + System.out.println(response); + } +} +``` + +```php PHP hidelines={1..4} nocheck +messages->create( + maxTokens: 64000, + messages: [['role' => 'user', 'content' => 'Your prompt here']], + model: 'claude-sonnet-4-6', + thinking: ['type' => 'adaptive'], + outputConfig: ['effort' => 'medium'], +); + +echo array_find($message->content, fn($block) => $block->type === 'text')->text; +``` + +```ruby Ruby nocheck hidelines={1..2} +require "anthropic" + +client = Anthropic::Client.new + +message = client.messages.create( + model: "claude-sonnet-4-6", + max_tokens: 64000, + thinking: { + type: "adaptive" + }, + output_config: { + effort: "medium" + }, + messages: [ + { role: "user", content: "Your prompt here" } + ] +) +puts message.content.find { |block| block.type == :text }.text +``` + + + +If you see inconsistent behavior or quality regressions with adaptive thinking, try lowering the [effort](/docs/en/build-with-claude/effort) setting or using `max_tokens` as a hard limit first. Extended thinking with `budget_tokens` is still functional on Sonnet 4.6 but is deprecated and no longer recommended. + + +##### Keeping budget_tokens during migration + +If you need to keep `budget_tokens` temporarily while migrating, a budget around 16k tokens provides headroom for harder problems without risk of runaway token usage. This configuration is deprecated and will be removed in a future model release. + +###### Coding and agentic use cases + +For agentic coding, frontend design, tool-heavy workflows, and complex enterprise workflows, start with `medium` effort. If you find latency is too high, consider reducing effort to `low`. If you need higher intelligence, consider increasing effort to `high` or migrating to Opus 4.7. + + +```bash cURL +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "anthropic-beta: interleaved-thinking-2025-05-14" \ + --header "content-type: application/json" \ + --data \ +'{ + "model": "claude-sonnet-4-6", + "max_tokens": 16384, + "thinking": { + "type": "enabled", + "budget_tokens": 16384 + }, + "output_config": { + "effort": "medium" + }, + "messages": [ + { + "role": "user", + "content": "Your prompt here" + } + ] +}' +``` + +```bash CLI +ant beta:messages create --beta interleaved-thinking-2025-05-14 <<'YAML' +model: claude-sonnet-4-6 +max_tokens: 16384 +thinking: + type: enabled + budget_tokens: 16384 +output_config: + effort: medium +messages: + - role: user + content: Your prompt here +YAML +``` + +```python Python +response = client.beta.messages.create( + model="claude-sonnet-4-6", + max_tokens=16384, + thinking={"type": "enabled", "budget_tokens": 16384}, + output_config={"effort": "medium"}, + betas=["interleaved-thinking-2025-05-14"], + messages=[{"role": "user", "content": "Your prompt here"}], +) +``` + +```typescript TypeScript +const response = await client.beta.messages.create({ + model: "claude-sonnet-4-6", + max_tokens: 16384, + thinking: { type: "enabled", budget_tokens: 16384 }, + output_config: { effort: "medium" }, + betas: ["interleaved-thinking-2025-05-14"], + messages: [{ role: "user", content: "Your prompt here" }] +}); +``` + +```csharp C# +using Anthropic; +using Anthropic.Models.Beta; +using Anthropic.Models.Beta.Messages; + +AnthropicClient client = new(); + +var parameters = new MessageCreateParams +{ + Model = "claude-sonnet-4-6", + MaxTokens = 16384, + Thinking = new BetaThinkingConfigEnabled { BudgetTokens = 16384 }, + OutputConfig = new BetaOutputConfig + { + Effort = Effort.Medium + }, + Betas = [AnthropicBeta.InterleavedThinking2025_05_14], + Messages = [new() { Role = Role.User, Content = "Your prompt here" }] +}; + +var message = await client.Beta.Messages.Create(parameters); +Console.WriteLine(message); +``` + +```go Go hidelines={1..11,-1} +package main + +import ( + "context" + "fmt" + "log" + + "github.com/anthropics/anthropic-sdk-go" +) + +func main() { + client := anthropic.NewClient() + + response, err := client.Beta.Messages.New(context.TODO(), anthropic.BetaMessageNewParams{ + Model: "claude-sonnet-4-6", + MaxTokens: 16384, + Thinking: anthropic.BetaThinkingConfigParamOfEnabled(16384), + OutputConfig: anthropic.BetaOutputConfigParam{ + Effort: anthropic.BetaOutputConfigEffortMedium, + }, + Messages: []anthropic.BetaMessageParam{ + anthropic.NewBetaUserMessage(anthropic.NewBetaTextBlock("Your prompt here")), + }, + Betas: []anthropic.AnthropicBeta{anthropic.AnthropicBetaInterleavedThinking2025_05_14}, + }) + if err != nil { + log.Fatal(err) + } + fmt.Println(response) +} +``` + +```java Java hidelines={1..6,9..11,-2..} +import com.anthropic.client.AnthropicClient; +import com.anthropic.client.okhttp.AnthropicOkHttpClient; +import com.anthropic.models.beta.messages.MessageCreateParams; +import com.anthropic.models.beta.messages.BetaMessage; +import com.anthropic.models.messages.Model; +import com.anthropic.models.beta.AnthropicBeta; +import com.anthropic.models.beta.messages.BetaThinkingConfigEnabled; +import com.anthropic.models.beta.messages.BetaOutputConfig; + +public class Main { + public static void main(String[] args) { + AnthropicClient client = AnthropicOkHttpClient.fromEnv(); + + MessageCreateParams params = MessageCreateParams.builder() + .model(Model.CLAUDE_SONNET_4_6) + .maxTokens(16384L) + .thinking(BetaThinkingConfigEnabled.builder() + .budgetTokens(16384L) + .build()) + .outputConfig(BetaOutputConfig.builder() + .effort(BetaOutputConfig.Effort.MEDIUM) + .build()) + .addBeta(AnthropicBeta.INTERLEAVED_THINKING_2025_05_14) + .addUserMessage("Your prompt here") + .build(); + + BetaMessage response = client.beta().messages().create(params); + System.out.println(response); + } +} +``` + +```php PHP hidelines={1..4} +beta->messages->create( + maxTokens: 16384, + messages: [['role' => 'user', 'content' => 'Your prompt here']], + model: 'claude-sonnet-4-6', + thinking: ['type' => 'enabled', 'budget_tokens' => 16384], + outputConfig: ['effort' => 'medium'], + betas: ['interleaved-thinking-2025-05-14'], +); + +echo array_find($message->content, fn($block) => $block->type === 'text')->text; +``` + +```ruby Ruby hidelines={1..2} +require "anthropic" + +client = Anthropic::Client.new + +message = client.beta.messages.create( + model: "claude-sonnet-4-6", + max_tokens: 16384, + thinking: { + type: "enabled", + budget_tokens: 16384 + }, + output_config: { + effort: "medium" + }, + betas: ["interleaved-thinking-2025-05-14"], + messages: [ + { role: "user", content: "Your prompt here" } + ] +) +puts message.content.find { |block| block.type == :text }.text +``` + + +###### Chat and non-coding use cases + +For chat, content generation, search, classification, and other non-coding tasks, start with `low` effort with extended thinking. If you need more depth, increase effort to `medium`. + + +```bash cURL +curl https://api.anthropic.com/v1/messages \ + --header "x-api-key: $ANTHROPIC_API_KEY" \ + --header "anthropic-version: 2023-06-01" \ + --header "anthropic-beta: interleaved-thinking-2025-05-14" \ + --header "content-type: application/json" \ + --data \ +'{ + "model": "claude-sonnet-4-6", + "max_tokens": 8192, + "thinking": { + "type": "enabled", + "budget_tokens": 16384 + }, + "output_config": { + "effort": "low" + }, + "messages": [ + { + "role": "user", + "content": "Your prompt here" + } + ] +}' +``` + +```bash CLI +ant beta:messages create --beta interleaved-thinking-2025-05-14 <<'YAML' +model: claude-sonnet-4-6 +max_tokens: 8192 +thinking: + type: enabled + budget_tokens: 16384 +output_config: + effort: low +messages: + - role: user + content: Your prompt here +YAML +``` + +```python Python +response = client.beta.messages.create( + model="claude-sonnet-4-6", + max_tokens=8192, + thinking={"type": "enabled", "budget_tokens": 16384}, + output_config={"effort": "low"}, + betas=["interleaved-thinking-2025-05-14"], + messages=[{"role": "user", "content": "Your prompt here"}], +) +``` + +```typescript TypeScript +const response = await client.beta.messages.create({ + model: "claude-sonnet-4-6", + max_tokens: 8192, + thinking: { type: "enabled", budget_tokens: 16384 }, + output_config: { effort: "low" }, + betas: ["interleaved-thinking-2025-05-14"], + messages: [{ role: "user", content: "Your prompt here" }] +}); +``` + +```csharp C# +using Anthropic; +using Anthropic.Models.Beta; +using Anthropic.Models.Beta.Messages; + +AnthropicClient client = new(); + +var parameters = new MessageCreateParams +{ + Model = "claude-sonnet-4-6", + MaxTokens = 8192, + Thinking = new BetaThinkingConfigEnabled { BudgetTokens = 16384 }, + OutputConfig = new BetaOutputConfig + { + Effort = Effort.Low + }, + Betas = [AnthropicBeta.InterleavedThinking2025_05_14], + Messages = [new() { Role = Role.User, Content = "Your prompt here" }] +}; + +var message = await client.Beta.Messages.Create(parameters); +Console.WriteLine(message); +``` + +```go Go hidelines={1..11,-1} +package main + +import ( + "context" + "fmt" + "log" + + "github.com/anthropics/anthropic-sdk-go" +) + +func main() { + client := anthropic.NewClient() + + response, err := client.Beta.Messages.New(context.TODO(), anthropic.BetaMessageNewParams{ + Model: "claude-sonnet-4-6", + MaxTokens: 8192, + Thinking: anthropic.BetaThinkingConfigParamOfEnabled(16384), + OutputConfig: anthropic.BetaOutputConfigParam{ + Effort: anthropic.BetaOutputConfigEffortLow, + }, + Messages: []anthropic.BetaMessageParam{ + anthropic.NewBetaUserMessage(anthropic.NewBetaTextBlock("Your prompt here")), + }, + Betas: []anthropic.AnthropicBeta{anthropic.AnthropicBetaInterleavedThinking2025_05_14}, + }) + if err != nil { + log.Fatal(err) + } + fmt.Println(response) +} +``` + +```java Java hidelines={1..6,9..11,-2..} +import com.anthropic.client.AnthropicClient; +import com.anthropic.client.okhttp.AnthropicOkHttpClient; +import com.anthropic.models.beta.messages.MessageCreateParams; +import com.anthropic.models.beta.messages.BetaMessage; +import com.anthropic.models.messages.Model; +import com.anthropic.models.beta.AnthropicBeta; +import com.anthropic.models.beta.messages.BetaThinkingConfigEnabled; +import com.anthropic.models.beta.messages.BetaOutputConfig; + +public class Main { + public static void main(String[] args) { + AnthropicClient client = AnthropicOkHttpClient.fromEnv(); + + MessageCreateParams params = MessageCreateParams.builder() + .model(Model.CLAUDE_SONNET_4_6) + .maxTokens(8192L) + .thinking(BetaThinkingConfigEnabled.builder() + .budgetTokens(16384L) + .build()) + .outputConfig(BetaOutputConfig.builder() + .effort(BetaOutputConfig.Effort.LOW) + .build()) + .addBeta(AnthropicBeta.INTERLEAVED_THINKING_2025_05_14) + .addUserMessage("Your prompt here") + .build(); + + BetaMessage response = client.beta().messages().create(params); + System.out.println(response); + } +} +``` + +```php PHP hidelines={1..4} +beta->messages->create( + maxTokens: 8192, + messages: [['role' => 'user', 'content' => 'Your prompt here']], + model: 'claude-sonnet-4-6', + thinking: ['type' => 'enabled', 'budget_tokens' => 16384], + outputConfig: ['effort' => 'low'], + betas: ['interleaved-thinking-2025-05-14'], +); + +echo array_find($message->content, fn($block) => $block->type === 'text')->text; +``` + +```ruby Ruby hidelines={1..2} +require "anthropic" + +client = Anthropic::Client.new + +message = client.beta.messages.create( + model: "claude-sonnet-4-6", + max_tokens: 8192, + thinking: { + type: "enabled", + budget_tokens: 16384 + }, + output_config: { + effort: "low" + }, + betas: ["interleaved-thinking-2025-05-14"], + messages: [ + { role: "user", content: "Your prompt here" } + ] +) +puts message.content.find { |block| block.type == :text }.text +``` + + +### Sonnet 4.6 migration checklist + +- [ ] Update model ID to `claude-sonnet-4-6` +- [ ] **BREAKING:** Remove assistant message prefilling; use structured outputs or `output_config.format` instead +- [ ] **BREAKING:** Verify tool parameter JSON parsing handles escaping differences +- [ ] **BREAKING:** Update tool versions to latest (`text_editor_20250728`, `code_execution_20250825`); legacy versions are not supported (if migrating from 3.x) +- [ ] **BREAKING:** Remove any code using the `undo_edit` command (if applicable) +- [ ] **BREAKING:** Update sampling parameters to use only `temperature` OR `top_p`, not both (if migrating from 3.x) +- [ ] Handle new `refusal` stop reason in your application +- [ ] Remove `fine-grained-tool-streaming-2025-05-14` beta header (now GA) +- [ ] Migrate `output_format` to `output_config.format` +- [ ] Review and update prompts following [prompting best practices](/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) +- [ ] **Recommended:** Migrate from `thinking: {type: "enabled", budget_tokens: N}` to `thinking: {type: "adaptive"}` with the [effort parameter](/docs/en/build-with-claude/effort) (`budget_tokens` is deprecated and will be removed in a future release) +- [ ] Test in development environment before production deployment + +--- + +## Migrating to Claude Sonnet 4.5 + +Claude Sonnet 4.5 combines strong intelligence with fast performance, making it ideal for everyday coding, analysis, and content tasks. + +For a complete overview of capabilities, see the [models overview](/docs/en/about-claude/models/overview). + + +Sonnet 4.5 pricing is $3 per million input tokens, $15 per million output tokens. See [Claude pricing](/docs/en/about-claude/pricing) for details. + + +**Update your model name:** + +```python +# From Sonnet 4 +model = "claude-sonnet-4-20250514" # Before +model = "claude-sonnet-4-5-20250929" # After + +# From Sonnet 3.7 +model = "claude-3-7-sonnet-20250219" # Before +model = "claude-sonnet-4-5-20250929" # After +``` + +### Breaking changes + +These breaking changes apply when migrating from Claude 3.x Sonnet models. + +1. **Update sampling parameters** + + + This is a breaking change when migrating from Claude 3.x models. + + + Use only `temperature` OR `top_p`, not both. + +2. **Update tool versions** + + + This is a breaking change when migrating from Claude 3.x models. + + + Update to the latest tool versions (`text_editor_20250728`, `code_execution_20250825`). Remove any code using the `undo_edit` command. + +3. **Handle the `refusal` stop reason** + + Update your application to [handle `refusal` stop reasons](/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals). + +4. **Update your prompts for behavioral changes** + + Claude 4 models have a more concise, direct communication style. Review [prompting best practices](/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) for optimization guidance. + +### Sonnet 4.5 migration checklist + +- [ ] Update model ID to `claude-sonnet-4-5-20250929` +- [ ] **BREAKING:** Update tool versions to latest (`text_editor_20250728`, `code_execution_20250825`); legacy versions are not supported (if migrating from 3.x) +- [ ] **BREAKING:** Remove any code using the `undo_edit` command (if applicable) +- [ ] **BREAKING:** Update sampling parameters to use only `temperature` OR `top_p`, not both (if migrating from 3.x) +- [ ] Handle new `refusal` stop reason in your application +- [ ] Review and update prompts following [prompting best practices](/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) +- [ ] Consider enabling extended thinking for complex reasoning tasks +- [ ] Test in development environment before production deployment + +--- + +## Migrating to Claude Haiku 4.5 + +Claude Haiku 4.5 is the fastest and most intelligent Haiku model with near-frontier performance, delivering premium model quality for interactive applications and high-volume processing. + +For a complete overview of capabilities, see the [models overview](/docs/en/about-claude/models/overview). + + +Haiku 4.5 pricing is $1 per million input tokens, $5 per million output tokens. See [Claude pricing](/docs/en/about-claude/pricing) for details. + + +**Update your model name:** + +```python +# From Haiku 3.5 +model = "claude-3-5-haiku-20241022" # Before +model = "claude-haiku-4-5-20251001" # After +``` + +**Review new rate limits:** Haiku 4.5 has separate rate limits from Haiku 3.5. See [Rate limits documentation](/docs/en/api/rate-limits) for details. + + +For significant performance improvements on coding and reasoning tasks, consider enabling extended thinking with `thinking: {type: "enabled", budget_tokens: N}`. + + + +Extended thinking impacts [prompt caching](/docs/en/build-with-claude/prompt-caching#caching-with-thinking-blocks) efficiency. + +Extended thinking is deprecated in Claude 4.6 models and removed in Claude Opus 4.7. If using newer models, use [adaptive thinking](/docs/en/build-with-claude/adaptive-thinking) instead. + + +**Explore new capabilities:** See the [models overview](/docs/en/about-claude/models/overview) for details on context awareness, increased output capacity (64k tokens), higher intelligence, and improved speed. + +### Breaking changes + +These breaking changes apply when migrating from Claude 3.x Haiku models. + +1. **Update sampling parameters** + + + This is a breaking change when migrating from Claude 3.x models. + + + Use only `temperature` OR `top_p`, not both. + +2. **Update tool versions** + + + This is a breaking change when migrating from Claude 3.x models. + + + Update to the latest tool versions (`text_editor_20250728`, `code_execution_20250825`). Remove any code using the `undo_edit` command. + +3. **Handle the `refusal` stop reason** + + Update your application to [handle `refusal` stop reasons](/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals). + +4. **Update your prompts for behavioral changes** + + Claude 4 models have a more concise, direct communication style. Review [prompting best practices](/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) for optimization guidance. + +### Haiku 4.5 migration checklist + +- [ ] Update model ID to `claude-haiku-4-5-20251001` +- [ ] **BREAKING:** Update tool versions to latest (`text_editor_20250728`, `code_execution_20250825`); legacy versions are not supported +- [ ] **BREAKING:** Remove any code using the `undo_edit` command (if applicable) +- [ ] **BREAKING:** Update sampling parameters to use only `temperature` OR `top_p`, not both +- [ ] Handle new `refusal` stop reason in your application +- [ ] Review and adjust for new rate limits (separate from Haiku 3.5) +- [ ] Review and update prompts following [prompting best practices](/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) +- [ ] Consider enabling extended thinking for complex reasoning tasks +- [ ] Test in development environment before production deployment + +--- + +## Get help + +- Check the [API documentation](/docs/en/api/overview) for detailed specifications +- Review [model capabilities](/docs/en/about-claude/models/overview) for performance comparisons +- Review [API release notes](/docs/en/release-notes/api) for API updates +- Contact support if you encounter any issues during migration diff --git a/llmsdk_docs/gpt5_5/README.md b/llmsdk_docs/gpt5_5/README.md index 1b9bba9d..34e01deb 100644 --- a/llmsdk_docs/gpt5_5/README.md +++ b/llmsdk_docs/gpt5_5/README.md @@ -15,6 +15,7 @@ The `docs/` folder contains detailed guides on various GPT-5.5 features: - [images-vision.md](./docs/images-vision.md) - Image and vision capabilities - [latest-model.md](./docs/latest-model.md) - Latest model features and updates - [migrate-to-responses.md](./docs/migrate-to-responses.md) - Migration guide to Responses API +- [reasoning.md](./docs/reasoning.md) - Reasoning effort, summaries, encrypted reasoning, and the `phase` parameter - [text.md](./docs/text.md) - Text generation and completion ## Examples diff --git a/llmsdk_docs/gpt5_5/docs/reasoning.md b/llmsdk_docs/gpt5_5/docs/reasoning.md new file mode 100644 index 00000000..7df1750b --- /dev/null +++ b/llmsdk_docs/gpt5_5/docs/reasoning.md @@ -0,0 +1,58 @@ +# Reasoning models + +Source: https://developers.openai.com/api/docs/guides/reasoning (Responses API; applies to GPT-5.6 variants, GPT-5.5, and GPT-5.4) + +Page outline: Get started with reasoning / Reasoning effort / Reasoning mode / How reasoning works / Managing the context window / Controlling costs / Allocating space for reasoning / Handling incomplete responses / Preserve reasoning across calls / Continue reasoning with stored responses / Preserve reasoning without stored responses / Reasoning summaries / `phase` parameter / Advice on prompting / Use case examples + +## Reasoning effort + +The `reasoning.effort` parameter guides how much the model thinks: `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, or `max`. Lower values prioritize speed and cost; higher values enable more thorough analysis. Defaults vary by model; GPT-5.5 defaults to `medium`. + +## Reasoning summaries + +Models can emit summaries of their internal reasoning when `reasoning.summary` is set (`concise`, `detailed`, or `auto`). Summaries appear in the `summary` array of reasoning output items and require explicit opt-in. + +## Preserve reasoning without stored responses + +When `store` is `false`, reasoning items include an `encrypted_content` property by default. Pass these encrypted tokens back on later calls to preserve reasoning context without storing responses server-side. + +## `phase` parameter + +> For long-running or tool-heavy flows with GPT-5.5 and GPT-5.4 in the Responses API, use the assistant message `phase` field to avoid early stopping. + +- Use `phase: "commentary"` for intermediate assistant updates, such as preambles before tool calls. +- Use `phase: "final_answer"` for the completed answer. +- These are the only two `phase` values. +- Don't add `phase` to user messages. +- `phase` is optional at the API level, but OpenAI recommends using it. When replaying assistant history manually, preserve each original `phase` value. Missing or dropped `phase` can cause preambles to be treated as final answers in those workflows. +- Using `previous_response_id` is usually the simplest path because prior assistant state is preserved. + +Official example (input array with `phase` on assistant messages): + +```python +from openai import OpenAI + +client = OpenAI() + +response = client.responses.create( + model="gpt-5.6", + input=[ + { + "role": "assistant", + "phase": "commentary", + "content": "I'll inspect the logs and then summarize root cause and remediation.", + }, + { + "role": "assistant", + "phase": "final_answer", + "content": "Root cause: cache invalidation race.", + }, + { + "role": "user", + "content": "Great—now give me a rollout-safe fix plan.", + }, + ], +) + +print(response.output_text) +``` diff --git a/skills/agenthub-python/SKILL.md b/skills/agenthub-python/SKILL.md index bc18c0e8..8b6fa6c5 100644 --- a/skills/agenthub-python/SKILL.md +++ b/skills/agenthub-python/SKILL.md @@ -15,205 +15,7 @@ uv add agenthub-python pip install agenthub-python ``` -## Model Selection - -Use exact model IDs. If a model ID is not listed, ask the user to confirm the exact ID before using it. - -| Family | Provider | Model IDs | API Key | Base URL | -| --- | --- | --- | --- | --- | -| Gemini 3 | Official / Vertex AI | `gemini-3.1-pro-preview`, `gemini-3.5-flash`, `gemini-3.1-flash-lite` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | -| Gemini 3 Image | Official / Vertex AI | `gemini-3.1-flash-image-preview`, `gemini-3-pro-image-preview` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | -| Gemini 3 TTS | Official / Vertex AI | `gemini-3.1-flash-tts-preview` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | -| Gemini Embedding | Official / Vertex AI | `gemini-embedding-2` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | -| Claude 4.6 | Official / ModelVerse | `claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.6 | Bedrock | `global.anthropic.claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.7 | Official / ModelVerse | `claude-opus-4-7` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.7 | Bedrock | `global.anthropic.claude-opus-4-7` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.8 | Official / ModelVerse | `claude-opus-4-8` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.8 | Bedrock | `global.anthropic.claude-opus-4-8` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| GPT 5.4 | Official / ModelVerse | `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | -| GPT 5.5 | Official / ModelVerse | `gpt-5.5` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | -| OpenAI Embedding | Official | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | -| Kimi-K2.6 | Official | `kimi-k2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | -| Kimi-K2.6 | OpenRouter | `moonshotai/kimi-k2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | -| Kimi-K2.6 | SiliconFlow | `Pro/moonshotai/Kimi-K2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | -| DeepSeek V4 | Official | `deepseek-v4-pro`, `deepseek-v4-flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | -| DeepSeek V4 | OpenRouter | `deepseek/deepseek-v4-pro`, `deepseek/deepseek-v4-flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | -| DeepSeek V4 | SiliconFlow | `deepseek-ai/DeepSeek-V4-Pro`, `deepseek-ai/DeepSeek-V4-Flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | -| GLM-5.1 | Official | `glm-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | -| GLM-5.1 | OpenRouter | `z-ai/glm-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | -| GLM-5.1 | SiliconFlow | `Pro/zai-org/GLM-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | - -Common gateway base URLs: - -- OpenRouter: `https://openrouter.ai/api/v1` -- SiliconFlow: `https://api.siliconflow.cn/v1` -- ModelVerse: `https://api.modelverse.cn/v1` (`https://api.modelverse.cn/` for Claude) -- vLLM: `http://127.0.0.1:8000/v1/` - -For models accessed through OpenAI-compatible APIs (e.g., Qwen series models via SiliconFlow or OpenRouter), pass `client_type="openai"` (`client_type="openai-embedding"` for embedding endpoints), and set `OPENAI_API_KEY` and `OPENAI_BASE_URL`: - -```python -client = AutoLLMClient(model="Qwen/Qwen3-Embedding-0.6B", client_type="openai-embedding") -``` - -## Data Models - -AgentHub uses `UniConfig`, `UniMessage`, and `UniEvent` to represent request options, conversation history, and streamed outputs across providers. - -### UniConfig - -`UniConfig` is the request config for `streaming_response` and `streaming_response_stateful`. All fields are optional. - -```python -config = { - "max_tokens": 1024, - "temperature": 1.0, - "tools": [{ - "name": "get_weather", - "description": "Get weather.", - "parameters": { - "type": "object", - "properties": {"location": {"type": "string", "description": "City name."}}, - "required": ["location"], - }, - }], - "tool_choice": "auto", - "thinking_summary": True, - "thinking_level": "high", - "system_prompt": "You are helpful.", - "prompt_caching": "enable", - "image_config": {"aspect_ratio": "4:3", "image_size": "1K"}, - "tts_config": [{"voice": "Kore"}], - "embedding_config": {"dimensions": 768}, - "trace_id": "agent1/conversation_001", -} -``` - -Fields: - -- `max_tokens` (`int`): Output token limit. -- `temperature` (`float`): Sampling temperature; support varies by model. -- `tools` (`list[ToolSchema]`): Tools with `name`, `description`, and optional JSON Schema `parameters`. -- `thinking_summary` (`bool`): Request a thinking summary when supported. -- `thinking_level` (`ThinkingLevel`): `none`, `low`, `medium`, `high`, or `xhigh`. -- `tool_choice` (`ToolChoice`): `auto`, `required`, `none`, or a list of tool names; support varies by model. -- `system_prompt` (`str`): System instruction text. -- `prompt_caching` (`PromptCaching`): `enable`, `disable`, or `enhance`. -- `image_config` (`ImageConfig`): `aspect_ratio` (`1:1`, `2:3`, `3:2`, `3:4`, `4:3`, `9:16`, `16:9`, `21:9`) and `image_size` (`1K`, `2K`). -- `tts_config` (`list[SpeakerConfig]`): Voice config; each item has `voice` and optional `speaker`. -- `embedding_config` (`EmbeddingConfig`): Embedding config, currently `dimensions`. -- `trace_id` (`str`): Stable ID for tracer output. - -### UniMessage - -`UniMessage` is the durable message shape used in history. - -```python -message = { - "role": "user", - "content_items": [ - {"type": "text", "text": "Hello", "phase": None, "signature": "sig"}, - {"type": "image_url", "image_url": "https://example.com/image.jpg"}, - {"type": "inline_data", "data": b"...", "mime_type": "image/png", "signature": "sig"}, - {"type": "thinking", "thinking": "Reasoning", "signature": "sig"}, - {"type": "inline_thinking", "data": b"...", "mime_type": "image/png", "signature": "sig"}, - {"type": "tool_call", "name": "get_weather", "arguments": {"location": "Paris"}, "tool_call_id": "call_1", "signature": "sig"}, - {"type": "tool_result", "text": "22 C", "tool_call_id": "call_1"}, - {"type": "embedding", "embedding": [0.1, 0.2]}, - ], -} -``` - -Fields: - -- `role` (`Role`): `user` or `assistant`. -- `content_items` (`list[ContentItem]`): Message payload. -- `usage_metadata` (`UsageMetadata | None`): Optional token counts on completed assistant messages. -- `finish_reason` (`FinishReason | None`): `stop`, `length`, `tool_call`, `unknown`, or `None`. -- `created_at` (`int`): Unix milliseconds. - -Content items: - -- `text`: Text chunk; `phase` marks sub-stage; `signature` verifies signed content. -- `image_url`: Image URL or data URI. -- `inline_data`: Inline media bytes with MIME type; may carry `signature`. -- `thinking`: Text reasoning content; may carry `signature`. -- `inline_thinking`: Binary reasoning artifact; may carry `signature`. -- `tool_call`: Complete model tool request with name, args, ID, and optional `signature`. -- `tool_result`: Tool output text for a `tool_call_id`; may include image URLs. -- `embedding`: Numeric embedding vector. - -Preserve `phase` and `signature`; never drop either field. - -### UniEvent - -`UniEvent` is the streamed output shape. Read token counts from `usage_metadata` here. - -```python -event = { - "role": "assistant", - "event_type": "delta", - "content_items": [ - {"type": "partial_tool_call", "name": "get_weather", "arguments": "{\"location\":\"Par", "tool_call_id": "call_1"} - ], - "usage_metadata": {"cached_tokens": 0, "prompt_tokens": 10, "thoughts_tokens": None, "response_tokens": 1}, - "finish_reason": None, - "created_at": 1694502400000, -} -``` - -Fields: - -- `role` (`Role`): `user` or `assistant`. -- `event_type` (`EventType`): `start`, `delta`, `stop`, or `unused`. -- `content_items` (`list[PartialContentItem]`): Stream payload; includes `ContentItem` plus `partial_tool_call`. -- `usage_metadata` (`UsageMetadata | None`): Token counts: `cached_tokens`, `prompt_tokens`, `thoughts_tokens`, `response_tokens`. - Token math: `input = cached_tokens + prompt_tokens`; `output = thoughts_tokens + response_tokens`; treat `None` as `0`. -- `finish_reason` (`FinishReason | None`): `stop`, `length`, `tool_call`, `unknown`, or `None`. -- `created_at` (`int`): Unix milliseconds. - -Event-only content item: - -- `partial_tool_call`: Streaming tool-call fragment with `name`, partial JSON `arguments`, and `tool_call_id`. - -## APIs - -`AutoLLMClient` exposes five basic APIs. Prefer the stateful stream for agent loops. - -Initialize `AutoLLMClient` in one of three common ways: - -```python -# Initialize with model name -client = AutoLLMClient(model="gpt-5.5") - -# Optionally specify API key (if not using environment variables) -client = AutoLLMClient( - model="gpt-5.5", - api_key="your-openai-api-key", - base_url="https://api.openai.com/v1", -) - -# Use OpenAI Chat Completions-compatible routing explicitly -client = AutoLLMClient(model="custom-model", client_type="openai") -``` - -```python -async def streaming_response(messages: list[UniMessage], config: UniConfig) -> AsyncIterator[UniEvent]: - """Stream one stateless response from a full message list.""" - -async def streaming_response_stateful(message: UniMessage, config: UniConfig) -> AsyncIterator[UniEvent]: - """Stream one stateful response and update client history.""" - -def get_history() -> list[UniMessage]: - """Return a copy of stateful history.""" - -def set_history(history: list[UniMessage]) -> None: - """Replace stateful history with a copy.""" - -def clear_history() -> None: - """Clear stateful history.""" -``` +For model IDs, API keys, and base URLs, see [Model selection](reference/models.md). ## Basic Usage @@ -228,6 +30,10 @@ def get_weather(location: str) -> str: return f"Temperature in {location}: 22 C" +# Map tool names to their implementations so calls can be dispatched by name. +TOOLS = {"get_weather": get_weather} + + async def main(): weather_function = { "name": "get_weather", @@ -247,7 +53,7 @@ async def main(): client = AutoLLMClient(model="gpt-5.5") config = {"tools": [weather_function]} - events = [] + tool_call = None async for event in client.streaming_response_stateful( message={ "role": "user", @@ -255,20 +61,13 @@ async def main(): }, config=config ): - events.append(event) - - tool_call = None - for event in events: for item in event["content_items"]: - if item["type"] == "tool_call": + if item["type"] == "tool_call": # collected as the stream arrives; no second pass tool_call = item - break - - if tool_call: - break if tool_call: - result = get_weather(**tool_call["arguments"]) + # Dispatch by tool name instead of hardcoding the function. + result = TOOLS[tool_call["name"]](**tool_call["arguments"]) async for event in client.streaming_response_stateful( message={ @@ -284,70 +83,30 @@ async def main(): config=config ): print(event) + # Streams the final answer token by token, then a stop event carrying usage: + # {'role': 'assistant', 'event_type': 'delta', 'content_items': [{'type': 'text', 'text': 'The'}], 'usage_metadata': None, 'finish_reason': None} + # {'role': 'assistant', 'event_type': 'delta', 'content_items': [{'type': 'text', 'text': ' weather'}], 'usage_metadata': None, 'finish_reason': None} + # {'role': 'assistant', 'event_type': 'delta', 'content_items': [{'type': 'text', 'text': ' is'}], 'usage_metadata': None, 'finish_reason': None} + # {'role': 'assistant', 'event_type': 'delta', 'content_items': [{'type': 'text', 'text': ' 22 C.'}], 'usage_metadata': None, 'finish_reason': None} + # {'role': 'assistant', 'event_type': 'stop', 'content_items': [], 'usage_metadata': {'cached_tokens': 0, 'prompt_tokens': 12, 'thoughts_tokens': 0, 'response_tokens': 8}, 'finish_reason': 'stop'} asyncio.run(main()) ``` -## Tracer - -Tracer saves trace files and serves a local UI for inspecting conversations. - -Set `trace_id` to save trace files: - -```python -from agenthub import AutoLLMClient - -client = AutoLLMClient(model="gpt-5.5") - -config = {"trace_id": "agent1/conversation_001"} - -async for event in client.streaming_response_stateful( - message={"role": "user", "content_items": [{"type": "text", "text": "Hello"}]}, - config=config -): - pass -``` - -Default cache dir: `cache`, or `AGENTHUB_CACHE_DIR`. For `trace_id="agent1/conversation_001"`, AgentHub writes: - -- `cache/agent1/conversation_001.json`: Structured trace data with the full history and config. -- `cache/agent1/conversation_001.txt`: Human-readable conversation transcript. - -Browse traces: - -```python -from agenthub.integration.tracer import Tracer - -Tracer().start_web_server(host="127.0.0.1", port=25750) -``` - -Or CLI: - -```bash -python -m agenthub.integration.tracer --cache_dir ./cache --host 127.0.0.1 --port 25750 -``` - -Open Tracer at `http://127.0.0.1:25750`. - -## Playground - -Playground starts a local chat UI for manual model checks. - -Start Playground for manual chat: - -```python -from agenthub.integration.playground import start_playground_server - -start_playground_server(host="127.0.0.1", port=25751) -``` - -Open Playground at `http://127.0.0.1:25751`. - ## Notes Agent loop rules: - Send every tool result with the exact `tool_call_id` from its originating `tool_call`. Do not invent, normalize, or reuse IDs across unrelated tool calls. -- Preserve `thinking` and `inline_thinking` items. Do not strip `phase` or `signature` fields. +- If streamed tool-call arguments cannot be parsed, AgentHub raises `ToolCallArgumentParseError`. Do not execute the tool from partial arguments; let the agent runtime retry or re-prompt the model. +- Preserve `thinking` and `inline_thinking` items. Do not strip or modify `fidelity` fields. +- Do not accumulate `usage_metadata` across events. Take the latest `usage_metadata` as the usage of the current request. - For embedding models, each `UniMessage` in the `messages` array produces **one embedding vector**. Within a single message, all items in `content_items` are aggregated into a single embedding. Set `embedding_config.dimensions` in the config to control vector size. + +## Reference + +- [Model selection](reference/models.md) — model IDs, API keys, base URLs, and OpenAI-compatible routing. +- [Data models](reference/data-models.md) — `UniConfig`, `UniMessage`, `UniEvent`, and the tool-call streaming protocol. +- [APIs](reference/api.md) — client initialization and method signatures. +- [Tracer & Playground](reference/integrations.md) — local tracing UI and the manual chat playground. diff --git a/skills/agenthub-python/reference/api.md b/skills/agenthub-python/reference/api.md new file mode 100644 index 00000000..f4196c35 --- /dev/null +++ b/skills/agenthub-python/reference/api.md @@ -0,0 +1,41 @@ +# APIs + +`AutoLLMClient` exposes five basic APIs. Prefer the stateful stream for agent loops. See [Basic Usage](../SKILL.md#basic-usage) for a full tool-use example. + +## Initialization + +Initialize `AutoLLMClient` in one of three common ways: + +```python +# Initialize with model name +client = AutoLLMClient(model="gpt-5.5") + +# Optionally specify API key (if not using environment variables) +client = AutoLLMClient( + model="gpt-5.5", + api_key="your-openai-api-key", + base_url="https://api.openai.com/v1", +) + +# Use OpenAI Chat Completions-compatible routing explicitly +client = AutoLLMClient(model="custom-model", client_type="openai") +``` + +## Method signatures + +```python +async def streaming_response(messages: list[UniMessage], config: UniConfig) -> AsyncIterator[UniEvent]: + """Stream one stateless response from a full message list.""" + +async def streaming_response_stateful(message: UniMessage, config: UniConfig) -> AsyncIterator[UniEvent]: + """Stream one stateful response and update client history.""" + +def get_history() -> list[UniMessage]: + """Return a copy of stateful history.""" + +def set_history(history: list[UniMessage]) -> None: + """Replace stateful history with a copy.""" + +def clear_history() -> None: + """Clear stateful history.""" +``` diff --git a/skills/agenthub-python/reference/data-models.md b/skills/agenthub-python/reference/data-models.md new file mode 100644 index 00000000..cf518185 --- /dev/null +++ b/skills/agenthub-python/reference/data-models.md @@ -0,0 +1,138 @@ +# Data Models + +AgentHub uses `UniConfig`, `UniMessage`, and `UniEvent` to represent request options, conversation history, and streamed outputs across providers. + +## UniConfig + +`UniConfig` is the request config for `streaming_response` and `streaming_response_stateful`. All fields are optional. + +```python +config = { + "max_tokens": 1024, + "temperature": 1.0, + "tools": [{ + "name": "get_weather", + "description": "Get weather.", + "parameters": { + "type": "object", + "properties": {"location": {"type": "string", "description": "City name."}}, + "required": ["location"], + }, + }], + "tool_choice": "auto", + "thinking_summary": True, + "thinking_level": "high", + "system_prompt": "You are helpful.", + "prompt_caching": "enable", + "image_config": {"aspect_ratio": "4:3", "image_size": "1K"}, + "tts_config": [{"voice": "Kore"}], + "embedding_config": {"dimensions": 768}, + "trace_id": "agent1/conversation_001", +} +``` + +Fields: + +- `max_tokens` (`int`): Output token limit. +- `temperature` (`float`): Sampling temperature; support varies by model. +- `tools` (`list[ToolSchema]`): Tools with `name`, `description`, and optional JSON Schema `parameters`. +- `thinking_summary` (`bool`): Request a thinking summary when supported. +- `thinking_level` (`ThinkingLevel`): `none`, `low`, `medium`, `high`, or `xhigh`. +- `tool_choice` (`ToolChoice`): `auto`, `required`, `none`, or a list of tool names; support varies by model. +- `system_prompt` (`str`): System instruction text. +- `prompt_caching` (`PromptCaching`): `enable`, `disable`, or `enhance`. +- `image_config` (`ImageConfig`): `aspect_ratio` (`1:1`, `2:3`, `3:2`, `3:4`, `4:3`, `9:16`, `16:9`, `21:9`) and `image_size` (`1K`, `2K`). +- `tts_config` (`list[SpeakerConfig]`): Voice config; each item has `voice` and optional `speaker`. +- `embedding_config` (`EmbeddingConfig`): Embedding config, currently `dimensions`. +- `trace_id` (`str`): Stable ID for tracer output. + +## UniMessage + +`UniMessage` is the durable message shape used in history. + +```python +message = { + "role": "user", + "content_items": [ + {"type": "text", "text": "Hello", "fidelity": {"phase": "commentary"}}, + {"type": "image_url", "image_url": "https://example.com/image.jpg"}, + {"type": "inline_data", "data": b"...", "mime_type": "image/png", "fidelity": {"signature": "sig"}}, + {"type": "thinking", "thinking": "Reasoning", "fidelity": {"signature": "sig"}}, + {"type": "inline_thinking", "data": b"...", "mime_type": "image/png", "fidelity": {"signature": "sig"}}, + {"type": "tool_call", "name": "get_weather", "arguments": {"location": "Paris"}, "tool_call_id": "call_1", "fidelity": {"signature": "sig"}}, + {"type": "tool_result", "text": "22 C", "tool_call_id": "call_1"}, + {"type": "embedding", "embedding": [0.1, 0.2]}, + ], +} +``` + +Fields: + +- `role` (`Role`): `user` or `assistant`. +- `content_items` (`list[ContentItem]`): Message payload. +- `usage_metadata` (`UsageMetadata | None`): Optional token counts on completed assistant messages. +- `finish_reason` (`FinishReason | None`): `stop`, `length`, `tool_call`, `unknown`, or `None`. +- `created_at` (`int`): Unix milliseconds. + +Content items: + +- `text`: Text chunk; may carry `fidelity`. +- `image_url`: Image URL or data URI. +- `inline_data`: Inline media bytes with MIME type; may carry `fidelity`. +- `thinking`: Text reasoning content; may carry `fidelity`. +- `inline_thinking`: Binary reasoning artifact; may carry `fidelity`. +- `tool_call`: Complete model tool request with name, args, ID, and optional `fidelity`. +- `tool_result`: Tool output text for a `tool_call_id`; may include image URLs. +- `embedding`: Numeric embedding vector. + +`fidelity` is an arbitrary JSON object of wire-level data the client recorded to reproduce the original message on replay — thinking signatures, phase labels, the upstream reasoning field name, and the like. It is opaque: pass it back unchanged, never modify or drop it. + +## UniEvent + +`UniEvent` is the streamed output shape. Read token counts from `usage_metadata` here. + +```python +event = { + "role": "assistant", + "event_type": "delta", + "content_items": [ + {"type": "partial_tool_call", "name": "get_weather", "arguments": "{\"location\":\"Par", "tool_call_id": "call_1"} + ], + "usage_metadata": {"cached_tokens": 0, "prompt_tokens": 10, "thoughts_tokens": None, "response_tokens": 1}, + "finish_reason": None, + "created_at": 1694502400000, +} +``` + +Fields: + +- `role` (`Role`): `user` or `assistant`. +- `event_type` (`EventType`): `start`, `delta`, `stop`, or `unused`. +- `content_items` (`list[PartialContentItem]`): Stream payload; includes `ContentItem` plus `partial_tool_call`. +- `usage_metadata` (`UsageMetadata | None`): Token counts: `cached_tokens`, `prompt_tokens`, `thoughts_tokens`, `response_tokens`. + Token math: `input = cached_tokens + prompt_tokens`; `output = thoughts_tokens + response_tokens`; treat `None` as `0`. +- `finish_reason` (`FinishReason | None`): `stop`, `length`, `tool_call`, `unknown`, or `None`. +- `created_at` (`int`): Unix milliseconds. + +Event-only content item: + +- `partial_tool_call`: Streaming tool-call fragment with `name`, partial JSON `arguments`, and `tool_call_id`. + +## Tool-Call Streaming Protocol + +Across providers a tool call streams as the same ordered sequence of events, so consumers handle every model the same way: + +1. **Announce (name + id first).** The first event for a tool call carries a `partial_tool_call` whose `name` and `tool_call_id` are non-empty and whose `arguments` is a JSON **string fragment** (often `""`). The tool's identity arrives no later than the first argument bytes. +2. **Argument deltas.** Zero or more `delta` events follow, each carrying a `partial_tool_call` whose `arguments` is the next fragment of the arguments JSON string (`name` and `tool_call_id` are empty `""`). Concatenate the fragments in order. +3. **Complete call (last).** One final event carries a complete `tool_call` item: `name`, `tool_call_id`, and `arguments` parsed into a dict. Read tool calls from these `tool_call` items; treat the `partial_tool_call` fragments as live progress only. + +The final `arguments` value must parse to a JSON object. If the streamed JSON is malformed, truncated, or parses to a non-object value such as an array, AgentHub raises `ToolCallArgumentParseError` instead of yielding a complete `tool_call`. The error carries `client`, `tool_name`, `tool_call_id`, `raw_arguments_length`, and `raw_arguments_preview` so the caller can log the bad model output and retry or re-prompt without executing a tool from partial arguments. + +For consecutive or parallel tool calls, each new call restarts at step 1 with its own `name` and `tool_call_id`, so one call's arguments never bleed into the next. Send each tool result back with the exact `tool_call_id` from its `tool_call`. + +## Errors + +Errors raised by AgentHub inherit `AgentHubError`, a `ValueError` subclass: + +- `ToolCallArgumentParseError` — streamed tool-call arguments were malformed or not a JSON object. It carries `client`, `tool_name`, `tool_call_id`, `raw_arguments_length`, and `raw_arguments_preview`. +- `EmptyResponseError` — the response finished with thinking content only, which fails with a 400 error when sent back on the next turn. It carries `client` and `finish_reason`. diff --git a/skills/agenthub-python/reference/integrations.md b/skills/agenthub-python/reference/integrations.md new file mode 100644 index 00000000..b9bbf682 --- /dev/null +++ b/skills/agenthub-python/reference/integrations.md @@ -0,0 +1,56 @@ +# Tracer & Playground + +## Tracer + +Tracer saves trace files and serves a local UI for inspecting conversations. + +Set `trace_id` to save trace files: + +```python +from agenthub import AutoLLMClient + +client = AutoLLMClient(model="gpt-5.5") + +config = {"trace_id": "agent1/conversation_001"} + +async for event in client.streaming_response_stateful( + message={"role": "user", "content_items": [{"type": "text", "text": "Hello"}]}, + config=config +): + pass +``` + +Default cache dir: `cache`, or `AGENTHUB_CACHE_DIR`. For `trace_id="agent1/conversation_001"`, AgentHub writes: + +- `cache/agent1/conversation_001.json`: Structured trace data with the full history and config. +- `cache/agent1/conversation_001.txt`: Human-readable conversation transcript. + +Browse traces: + +```python +from agenthub.integration.tracer import Tracer + +Tracer().start_web_server(host="127.0.0.1", port=25750) +``` + +Or CLI: + +```bash +python -m agenthub.integration.tracer --cache_dir ./cache --host 127.0.0.1 --port 25750 +``` + +Open Tracer at `http://127.0.0.1:25750`. + +## Playground + +Playground starts a local chat UI for manual model checks. + +Start Playground for manual chat: + +```python +from agenthub.integration.playground import start_playground_server + +start_playground_server(host="127.0.0.1", port=25751) +``` + +Open Playground at `http://127.0.0.1:25751`. diff --git a/skills/agenthub-python/reference/models.md b/skills/agenthub-python/reference/models.md new file mode 100644 index 00000000..560b78cd --- /dev/null +++ b/skills/agenthub-python/reference/models.md @@ -0,0 +1,43 @@ +# Model Selection + +Use exact model IDs. If a model ID is not listed, ask the user to confirm the exact ID before using it. + +| Family | Provider | Model IDs | API Key | Base URL | +| --- | --- | --- | --- | --- | +| Gemini 3 | Official / Vertex AI | `gemini-3.1-pro-preview`, `gemini-3.5-flash`, `gemini-3.1-flash-lite` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | +| Gemini 3 Image | Official / Vertex AI | `gemini-3.1-flash-image-preview`, `gemini-3-pro-image-preview` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | +| Gemini 3 TTS | Official / Vertex AI | `gemini-3.1-flash-tts-preview` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | +| Gemini Embedding | Official / Vertex AI | `gemini-embedding-2` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | +| Claude 4.6 | Official / ModelVerse | `claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.6 | Bedrock | `global.anthropic.claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.7 | Official / ModelVerse | `claude-opus-4-7` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.7 | Bedrock | `global.anthropic.claude-opus-4-7` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.8 | Official / ModelVerse | `claude-opus-4-8` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.8 | Bedrock | `global.anthropic.claude-opus-4-8` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 5 | Official / ModelVerse | `claude-fable-5` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 5 | Bedrock | `global.anthropic.claude-fable-5` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| GPT 5.4 | Official / ModelVerse | `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | +| GPT 5.5 | Official / ModelVerse | `gpt-5.5` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | +| OpenAI Embedding | Official | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | +| Kimi-K2.6 | Official | `kimi-k2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | +| Kimi-K2.6 | OpenRouter | `moonshotai/kimi-k2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | +| Kimi-K2.6 | SiliconFlow | `Pro/moonshotai/Kimi-K2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | +| DeepSeek V4 | Official | `deepseek-v4-pro`, `deepseek-v4-flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | +| DeepSeek V4 | OpenRouter | `deepseek/deepseek-v4-pro`, `deepseek/deepseek-v4-flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | +| DeepSeek V4 | SiliconFlow | `deepseek-ai/DeepSeek-V4-Pro`, `deepseek-ai/DeepSeek-V4-Flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | +| GLM-5.1 | Official | `glm-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | +| GLM-5.1 | OpenRouter | `z-ai/glm-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | +| GLM-5.1 | SiliconFlow | `Pro/zai-org/GLM-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | + +Common gateway base URLs: + +- OpenRouter: `https://openrouter.ai/api/v1` +- SiliconFlow: `https://api.siliconflow.cn/v1` +- ModelVerse: `https://api.modelverse.cn/v1` (`https://api.modelverse.cn/` for Claude) +- vLLM: `http://127.0.0.1:8000/v1/` + +For models accessed through OpenAI-compatible APIs (e.g., Qwen series models via SiliconFlow or OpenRouter), pass `client_type="openai"` (`client_type="openai-embedding"` for embedding endpoints), and set `OPENAI_API_KEY` and `OPENAI_BASE_URL`: + +```python +client = AutoLLMClient(model="Qwen/Qwen3-Embedding-0.6B", client_type="openai-embedding") +``` diff --git a/skills/agenthub-typescript/SKILL.md b/skills/agenthub-typescript/SKILL.md index 22e728bc..8fa879c9 100644 --- a/skills/agenthub-typescript/SKILL.md +++ b/skills/agenthub-typescript/SKILL.md @@ -13,208 +13,7 @@ AgentHub is a unified SDK for calling LLMs across providers with shared data mod npm install @prismshadow/agenthub ``` -## Model Selection - -Use exact model IDs. If a model ID is not listed, ask the user to confirm the exact ID before using it. - -| Family | Provider | Model IDs | API Key | Base URL | -| --- | --- | --- | --- | --- | -| Gemini 3 | Official / Vertex AI | `gemini-3.1-pro-preview`, `gemini-3.5-flash`, `gemini-3.1-flash-lite` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | -| Gemini 3 Image | Official / Vertex AI | `gemini-3.1-flash-image-preview`, `gemini-3-pro-image-preview` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | -| Gemini 3 TTS | Official / Vertex AI | `gemini-3.1-flash-tts-preview` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | -| Gemini Embedding | Official / Vertex AI | `gemini-embedding-2` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | -| Claude 4.6 | Official / ModelVerse | `claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.6 | Bedrock | `global.anthropic.claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.7 | Official / ModelVerse | `claude-opus-4-7` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.7 | Bedrock | `global.anthropic.claude-opus-4-7` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.8 | Official / ModelVerse | `claude-opus-4-8` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| Claude 4.8 | Bedrock | `global.anthropic.claude-opus-4-8` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | -| GPT 5.4 | Official / ModelVerse | `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | -| GPT 5.5 | Official / ModelVerse | `gpt-5.5` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | -| OpenAI Embedding | Official | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | -| Kimi-K2.6 | Official | `kimi-k2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | -| Kimi-K2.6 | OpenRouter | `moonshotai/kimi-k2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | -| Kimi-K2.6 | SiliconFlow | `Pro/moonshotai/Kimi-K2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | -| DeepSeek V4 | Official | `deepseek-v4-pro`, `deepseek-v4-flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | -| DeepSeek V4 | OpenRouter | `deepseek/deepseek-v4-pro`, `deepseek/deepseek-v4-flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | -| DeepSeek V4 | SiliconFlow | `deepseek-ai/DeepSeek-V4-Pro`, `deepseek-ai/DeepSeek-V4-Flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | -| GLM-5.1 | Official | `glm-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | -| GLM-5.1 | OpenRouter | `z-ai/glm-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | -| GLM-5.1 | SiliconFlow | `Pro/zai-org/GLM-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | - -Common gateway base URLs: - -- OpenRouter: `https://openrouter.ai/api/v1` -- SiliconFlow: `https://api.siliconflow.cn/v1` -- ModelVerse: `https://api.modelverse.cn/v1` (`https://api.modelverse.cn/` for Claude) -- vLLM: `http://127.0.0.1:8000/v1/` - -For models accessed through OpenAI-compatible APIs (e.g., Qwen series models via SiliconFlow or OpenRouter), pass `clientType: "openai"` (`clientType: "openai-embedding"` for embedding endpoints). These models use `OPENAI_API_KEY` and `OPENAI_BASE_URL`: - -```typescript -const client = new AutoLLMClient({ model: "Qwen/Qwen3-Embedding-0.6B", clientType: "openai-embedding" }); -``` - -## Data Models - -AgentHub uses `UniConfig`, `UniMessage`, and `UniEvent` to represent request options, conversation history, and streamed outputs across providers. - -### UniConfig - -`UniConfig` is the request config for `streamingResponse` and `streamingResponseStateful`. All fields are optional. - -```typescript -const config = { - max_tokens: 1024, - temperature: 1.0, - tools: [{ - name: "get_weather", - description: "Get weather.", - parameters: { - type: "object", - properties: { location: { type: "string", description: "City name." } }, - required: ["location"], - }, - }], - tool_choice: "auto", - thinking_summary: true, - thinking_level: ThinkingLevel.HIGH, - system_prompt: "You are helpful.", - prompt_caching: PromptCaching.ENABLE, - image_config: { aspect_ratio: "4:3", image_size: "1K" }, - tts_config: [{ voice: "Kore" }], - embedding_config: { dimensions: 768 }, - trace_id: "agent1/conversation_001", -}; -``` - -Fields: - -- `max_tokens` (`number`): Output token limit. -- `temperature` (`number`): Sampling temperature; support varies by model. -- `tools` (`ToolSchema[]`): Tools with `name`, `description`, and optional JSON Schema `parameters`. -- `thinking_summary` (`boolean`): Request a thinking summary when supported. -- `thinking_level` (`ThinkingLevel`): `NONE`, `LOW`, `MEDIUM`, `HIGH`, or `XHIGH`. -- `tool_choice` (`ToolChoice`): `auto`, `required`, `none`, or a list of tool names; support varies by model. -- `system_prompt` (`string`): System instruction text. -- `prompt_caching` (`PromptCaching`): `ENABLE`, `DISABLE`, or `ENHANCE`. -- `image_config` (`ImageConfig`): `aspect_ratio` (`1:1`, `2:3`, `3:2`, `3:4`, `4:3`, `9:16`, `16:9`, `21:9`) and `image_size` (`1K`, `2K`). -- `tts_config` (`SpeakerConfig[]`): Voice config; each item has `voice` and optional `speaker`. -- `embedding_config` (`EmbeddingConfig`): Embedding config, currently `dimensions`. -- `trace_id` (`string`): Stable ID for tracer output. - -### UniMessage - -`UniMessage` is the durable message shape used in history. - -```typescript -const message = { - role: "user", - content_items: [ - { type: "text", text: "Hello", phase: null, signature: "sig" }, - { type: "image_url", image_url: "https://example.com/image.jpg" }, - { type: "inline_data", data: Buffer.from("..."), mime_type: "image/png", signature: "sig" }, - { type: "thinking", thinking: "Reasoning", signature: "sig" }, - { type: "inline_thinking", data: Buffer.from("..."), mime_type: "image/png", signature: "sig" }, - { type: "tool_call", name: "get_weather", arguments: { location: "Paris" }, tool_call_id: "call_1", signature: "sig" }, - { type: "tool_result", text: "22 C", tool_call_id: "call_1" }, - { type: "embedding", embedding: [0.1, 0.2] }, - ], -}; -``` - -Fields: - -- `role` (`Role`): `user` or `assistant`. -- `content_items` (`ContentItem[]`): Message payload. -- `usage_metadata` (`UsageMetadata | null`): Optional token counts on completed assistant messages. -- `finish_reason` (`FinishReason | null`): `stop`, `length`, `tool_call`, `unknown`, or `null`. -- `created_at` (`number`): Unix milliseconds. - -Content items: - -- `text`: Text chunk; `phase` marks sub-stage; `signature` verifies signed content. -- `image_url`: Image URL or data URI. -- `inline_data`: Inline media bytes with MIME type; may carry `signature`. -- `thinking`: Text reasoning content; may carry `signature`. -- `inline_thinking`: Binary reasoning artifact; may carry `signature`. -- `tool_call`: Complete model tool request with name, args, ID, and optional `signature`. -- `tool_result`: Tool output text for a `tool_call_id`; may include image URLs. -- `embedding`: Numeric embedding vector. - -Preserve `phase` and `signature`; never drop either field. - -### UniEvent - -`UniEvent` is the streamed output shape. Read token counts from `usage_metadata` here. - -```typescript -const event = { - role: "assistant", - event_type: "delta", - content_items: [ - { type: "partial_tool_call", name: "get_weather", arguments: "{\"location\":\"Par", tool_call_id: "call_1" }, - ], - usage_metadata: { cached_tokens: 0, prompt_tokens: 10, thoughts_tokens: null, response_tokens: 1 }, - finish_reason: null, - created_at: 1694502400000, -}; -``` - -Fields: - -- `role` (`Role`): `user` or `assistant`. -- `event_type` (`EventType`): `start`, `delta`, `stop`, or `unused`. -- `content_items` (`PartialContentItem[]`): Stream payload; includes `ContentItem` plus `partial_tool_call`. -- `usage_metadata` (`UsageMetadata | null`): Token counts: `cached_tokens`, `prompt_tokens`, `thoughts_tokens`, `response_tokens`. - Token math: `input = cached_tokens + prompt_tokens`; `output = thoughts_tokens + response_tokens`; treat `null` as `0`. -- `finish_reason` (`FinishReason | null`): `stop`, `length`, `tool_call`, `unknown`, or `null`. -- `created_at` (`number`): Unix milliseconds. - -Event-only content item: - -- `partial_tool_call`: Streaming tool-call fragment with `name`, partial JSON `arguments`, and `tool_call_id`. - -## APIs - -`AutoLLMClient` exposes five basic APIs. Prefer the stateful stream for agent loops. - -Initialize `AutoLLMClient` in one of three common ways: - -```typescript -// Initialize with model name -const clientByModel = new AutoLLMClient({ model: "gpt-5.5" }); - -// Optionally specify API key (if not using environment variables) -const clientWithEndpoint = new AutoLLMClient({ - model: "gpt-5.5", - apiKey: "your-openai-api-key", - baseUrl: "https://api.openai.com/v1", -}); - -// Use OpenAI Chat Completions-compatible routing explicitly -const clientWithType = new AutoLLMClient({ - model: "custom-model", - clientType: "openai", -}); -``` - -```typescript -/** Stream one stateless response from a full message list. */ -streamingResponse(options: { messages: UniMessage[]; config: UniConfig }): AsyncGenerator; - -/** Stream one stateful response and update client history. */ -streamingResponseStateful(options: { message: UniMessage; config: UniConfig }): AsyncGenerator; - -/** Return a copy of stateful history. */ -getHistory(): UniMessage[]; - -/** Replace stateful history with a copy. */ -setHistory(history: UniMessage[]): void; - -/** Clear stateful history. */ -clearHistory(): void; -``` +For model IDs, API keys, and base URLs, see [Model selection](reference/models.md). ## Basic Usage @@ -227,6 +26,11 @@ function getWeather(location: string): string { return `Temperature in ${location}: 22 C`; } +// Map tool names to their implementations so calls can be dispatched by name. +const TOOLS: Record) => string> = { + get_weather: (args) => getWeather(args.location as string), +}; + async function main(): Promise { const weatherTool = { name: "get_weather", @@ -246,7 +50,7 @@ async function main(): Promise { const client = new AutoLLMClient({ model: "gpt-5.5" }); const config = { tools: [weatherTool] }; - const events = []; + let toolCall: { name: string; arguments: Record; tool_call_id: string } | null = null; for await (const event of client.streamingResponseStateful({ message: { role: "user", @@ -254,22 +58,16 @@ async function main(): Promise { }, config, })) { - events.push(event); - } - - let toolCall: { name: string; arguments: Record; tool_call_id: string } | null = null; - for (const event of events) { for (const item of event.content_items) { if (item.type === "tool_call") { - toolCall = item; - break; + toolCall = item; // collected as the stream arrives; no second pass } } - if (toolCall) break; } if (toolCall) { - const result = getWeather(toolCall.arguments.location as string); + // Dispatch by tool name instead of hardcoding the function. + const result = TOOLS[toolCall.name](toolCall.arguments); for await (const event of client.streamingResponseStateful({ message: { @@ -285,6 +83,12 @@ async function main(): Promise { config, })) { console.log(event); + // Streams the final answer token by token, then a stop event carrying usage: + // { role: 'assistant', event_type: 'delta', content_items: [ { type: 'text', text: 'The' } ], usage_metadata: null, finish_reason: null } + // { role: 'assistant', event_type: 'delta', content_items: [ { type: 'text', text: ' weather' } ], usage_metadata: null, finish_reason: null } + // { role: 'assistant', event_type: 'delta', content_items: [ { type: 'text', text: ' is' } ], usage_metadata: null, finish_reason: null } + // { role: 'assistant', event_type: 'delta', content_items: [ { type: 'text', text: ' 22 C.' } ], usage_metadata: null, finish_reason: null } + // { role: 'assistant', event_type: 'stop', content_items: [], usage_metadata: { cached_tokens: 0, prompt_tokens: 12, thoughts_tokens: 0, response_tokens: 8 }, finish_reason: 'stop' } } } } @@ -292,64 +96,19 @@ async function main(): Promise { void main(); ``` -## Tracer - -Tracer saves trace files and serves a local UI for inspecting conversations. - -Set `trace_id` to save trace files: - -```typescript -import { AutoLLMClient } from "@prismshadow/agenthub"; - -const client = new AutoLLMClient({ model: "gpt-5.5" }); - -const config = { trace_id: "agent1/conversation_001" }; - -for await (const event of client.streamingResponseStateful({ - message: { - role: "user", - content_items: [{ type: "text", text: "Hello" }], - }, - config, -})) { - console.log(event); -} -``` - -Default cache dir: `cache`, or `AGENTHUB_CACHE_DIR`. For `trace_id="agent1/conversation_001"`, AgentHub writes: - -- `cache/agent1/conversation_001.json`: Structured trace data with the full history and config. -- `cache/agent1/conversation_001.txt`: Human-readable conversation transcript. - -Browse traces: - -```typescript -import { Tracer } from "@prismshadow/agenthub/integration/tracer"; - -const tracer = new Tracer(); -tracer.startWebServer("127.0.0.1", 25750); -``` - -Open Tracer at `http://127.0.0.1:25750`. - -## Playground - -Playground starts a local chat UI for manual model checks. - -Start Playground for manual chat: - -```typescript -import { startPlaygroundServer } from "@prismshadow/agenthub/integration/playground"; - -startPlaygroundServer("127.0.0.1", 25751); -``` - -Open Playground at `http://127.0.0.1:25751`. - ## Notes Keep these points in mind for agent loops: - Send every tool result with the exact `tool_call_id` from its originating `tool_call`. Do not invent, normalize, or reuse IDs across unrelated tool calls. -- Preserve `thinking` and `inline_thinking` items. Do not strip `phase` or `signature` fields. +- If streamed tool-call arguments cannot be parsed, AgentHub raises `ToolCallArgumentParseError`. Do not execute the tool from partial arguments; let the agent runtime retry or re-prompt the model. +- Preserve `thinking` and `inline_thinking` items. Do not strip or modify `fidelity` fields. +- Do not accumulate `usage_metadata` across events. Take the latest `usage_metadata` as the usage of the current request. - For embedding models, each `UniMessage` in the `messages` array produces **one embedding vector**. Within a single message, all items in `content_items` are aggregated into a single embedding. Set `embedding_config.dimensions` in the config to control vector size. + +## Reference + +- [Model selection](reference/models.md) — model IDs, API keys, base URLs, and OpenAI-compatible routing. +- [Data models](reference/data-models.md) — `UniConfig`, `UniMessage`, `UniEvent`, and the tool-call streaming protocol. +- [APIs](reference/api.md) — client initialization and method signatures. +- [Tracer & Playground](reference/integrations.md) — local tracing UI and the manual chat playground. diff --git a/skills/agenthub-typescript/reference/api.md b/skills/agenthub-typescript/reference/api.md new file mode 100644 index 00000000..53aff981 --- /dev/null +++ b/skills/agenthub-typescript/reference/api.md @@ -0,0 +1,44 @@ +# APIs + +`AutoLLMClient` exposes five basic APIs. Prefer the stateful stream for agent loops. See [Basic Usage](../SKILL.md#basic-usage) for a full tool-use example. + +## Initialization + +Initialize `AutoLLMClient` in one of three common ways: + +```typescript +// Initialize with model name +const clientByModel = new AutoLLMClient({ model: "gpt-5.5" }); + +// Optionally specify API key (if not using environment variables) +const clientWithEndpoint = new AutoLLMClient({ + model: "gpt-5.5", + apiKey: "your-openai-api-key", + baseUrl: "https://api.openai.com/v1", +}); + +// Use OpenAI Chat Completions-compatible routing explicitly +const clientWithType = new AutoLLMClient({ + model: "custom-model", + clientType: "openai", +}); +``` + +## Method signatures + +```typescript +/** Stream one stateless response from a full message list. */ +streamingResponse(options: { messages: UniMessage[]; config: UniConfig }): AsyncGenerator; + +/** Stream one stateful response and update client history. */ +streamingResponseStateful(options: { message: UniMessage; config: UniConfig }): AsyncGenerator; + +/** Return a copy of stateful history. */ +getHistory(): UniMessage[]; + +/** Replace stateful history with a copy. */ +setHistory(history: UniMessage[]): void; + +/** Clear stateful history. */ +clearHistory(): void; +``` diff --git a/skills/agenthub-typescript/reference/data-models.md b/skills/agenthub-typescript/reference/data-models.md new file mode 100644 index 00000000..9ff5e4b7 --- /dev/null +++ b/skills/agenthub-typescript/reference/data-models.md @@ -0,0 +1,138 @@ +# Data Models + +AgentHub uses `UniConfig`, `UniMessage`, and `UniEvent` to represent request options, conversation history, and streamed outputs across providers. + +## UniConfig + +`UniConfig` is the request config for `streamingResponse` and `streamingResponseStateful`. All fields are optional. + +```typescript +const config = { + max_tokens: 1024, + temperature: 1.0, + tools: [{ + name: "get_weather", + description: "Get weather.", + parameters: { + type: "object", + properties: { location: { type: "string", description: "City name." } }, + required: ["location"], + }, + }], + tool_choice: "auto", + thinking_summary: true, + thinking_level: ThinkingLevel.HIGH, + system_prompt: "You are helpful.", + prompt_caching: PromptCaching.ENABLE, + image_config: { aspect_ratio: "4:3", image_size: "1K" }, + tts_config: [{ voice: "Kore" }], + embedding_config: { dimensions: 768 }, + trace_id: "agent1/conversation_001", +}; +``` + +Fields: + +- `max_tokens` (`number`): Output token limit. +- `temperature` (`number`): Sampling temperature; support varies by model. +- `tools` (`ToolSchema[]`): Tools with `name`, `description`, and optional JSON Schema `parameters`. +- `thinking_summary` (`boolean`): Request a thinking summary when supported. +- `thinking_level` (`ThinkingLevel`): `NONE`, `LOW`, `MEDIUM`, `HIGH`, or `XHIGH`. +- `tool_choice` (`ToolChoice`): `auto`, `required`, `none`, or a list of tool names; support varies by model. +- `system_prompt` (`string`): System instruction text. +- `prompt_caching` (`PromptCaching`): `ENABLE`, `DISABLE`, or `ENHANCE`. +- `image_config` (`ImageConfig`): `aspect_ratio` (`1:1`, `2:3`, `3:2`, `3:4`, `4:3`, `9:16`, `16:9`, `21:9`) and `image_size` (`1K`, `2K`). +- `tts_config` (`SpeakerConfig[]`): Voice config; each item has `voice` and optional `speaker`. +- `embedding_config` (`EmbeddingConfig`): Embedding config, currently `dimensions`. +- `trace_id` (`string`): Stable ID for tracer output. + +## UniMessage + +`UniMessage` is the durable message shape used in history. + +```typescript +const message = { + role: "user", + content_items: [ + { type: "text", text: "Hello", fidelity: { phase: "commentary" } }, + { type: "image_url", image_url: "https://example.com/image.jpg" }, + { type: "inline_data", data: Buffer.from("..."), mime_type: "image/png", fidelity: { signature: "sig" } }, + { type: "thinking", thinking: "Reasoning", fidelity: { signature: "sig" } }, + { type: "inline_thinking", data: Buffer.from("..."), mime_type: "image/png", fidelity: { signature: "sig" } }, + { type: "tool_call", name: "get_weather", arguments: { location: "Paris" }, tool_call_id: "call_1", fidelity: { signature: "sig" } }, + { type: "tool_result", text: "22 C", tool_call_id: "call_1" }, + { type: "embedding", embedding: [0.1, 0.2] }, + ], +}; +``` + +Fields: + +- `role` (`Role`): `user` or `assistant`. +- `content_items` (`ContentItem[]`): Message payload. +- `usage_metadata` (`UsageMetadata | null`): Optional token counts on completed assistant messages. +- `finish_reason` (`FinishReason | null`): `stop`, `length`, `tool_call`, `unknown`, or `null`. +- `created_at` (`number`): Unix milliseconds. + +Content items: + +- `text`: Text chunk; may carry `fidelity`. +- `image_url`: Image URL or data URI. +- `inline_data`: Inline media bytes with MIME type; may carry `fidelity`. +- `thinking`: Text reasoning content; may carry `fidelity`. +- `inline_thinking`: Binary reasoning artifact; may carry `fidelity`. +- `tool_call`: Complete model tool request with name, args, ID, and optional `fidelity`. +- `tool_result`: Tool output text for a `tool_call_id`; may include image URLs. +- `embedding`: Numeric embedding vector. + +`fidelity` is an arbitrary JSON object of wire-level data the client recorded to reproduce the original message on replay — thinking signatures, phase labels, the upstream reasoning field name, and the like. It is opaque: pass it back unchanged, never modify or drop it. + +## UniEvent + +`UniEvent` is the streamed output shape. Read token counts from `usage_metadata` here. + +```typescript +const event = { + role: "assistant", + event_type: "delta", + content_items: [ + { type: "partial_tool_call", name: "get_weather", arguments: "{\"location\":\"Par", tool_call_id: "call_1" }, + ], + usage_metadata: { cached_tokens: 0, prompt_tokens: 10, thoughts_tokens: null, response_tokens: 1 }, + finish_reason: null, + created_at: 1694502400000, +}; +``` + +Fields: + +- `role` (`Role`): `user` or `assistant`. +- `event_type` (`EventType`): `start`, `delta`, `stop`, or `unused`. +- `content_items` (`PartialContentItem[]`): Stream payload; includes `ContentItem` plus `partial_tool_call`. +- `usage_metadata` (`UsageMetadata | null`): Token counts: `cached_tokens`, `prompt_tokens`, `thoughts_tokens`, `response_tokens`. + Token math: `input = cached_tokens + prompt_tokens`; `output = thoughts_tokens + response_tokens`; treat `null` as `0`. +- `finish_reason` (`FinishReason | null`): `stop`, `length`, `tool_call`, `unknown`, or `null`. +- `created_at` (`number`): Unix milliseconds. + +Event-only content item: + +- `partial_tool_call`: Streaming tool-call fragment with `name`, partial JSON `arguments`, and `tool_call_id`. + +## Tool-Call Streaming Protocol + +Across providers a tool call streams as the same ordered sequence of events, so consumers handle every model the same way: + +1. **Announce (name + id first).** The first event for a tool call carries a `partial_tool_call` whose `name` and `tool_call_id` are non-empty and whose `arguments` is a JSON **string fragment** (often `""`). The tool's identity arrives no later than the first argument bytes. +2. **Argument deltas.** Zero or more `delta` events follow, each carrying a `partial_tool_call` whose `arguments` is the next fragment of the arguments JSON string (`name` and `tool_call_id` are empty `""`). Concatenate the fragments in order. +3. **Complete call (last).** One final event carries a complete `tool_call` item: `name`, `tool_call_id`, and `arguments` parsed into an object. Read tool calls from these `tool_call` items; treat the `partial_tool_call` fragments as live progress only. + +The final `arguments` value must parse to a JSON object. If the streamed JSON is malformed, truncated, or parses to a non-object value such as an array, AgentHub raises `ToolCallArgumentParseError` instead of yielding a complete `tool_call`. The error carries `client`, `toolName`, `toolCallId`, `rawArgumentsLength`, and `rawArgumentsPreview` so the caller can log the bad model output and retry or re-prompt without executing a tool from partial arguments. + +For consecutive or parallel tool calls, each new call restarts at step 1 with its own `name` and `tool_call_id`, so one call's arguments never bleed into the next. Send each tool result back with the exact `tool_call_id` from its `tool_call`. + +## Errors + +Errors thrown by AgentHub inherit `AgentHubError`, an `Error` subclass: + +- `ToolCallArgumentParseError` — streamed tool-call arguments were malformed or not a JSON object. It carries `client`, `toolName`, `toolCallId`, `rawArgumentsLength`, and `rawArgumentsPreview`. +- `EmptyResponseError` — the response finished with thinking content only, which fails with a 400 error when sent back on the next turn. It carries `client` and `finishReason`. diff --git a/skills/agenthub-typescript/reference/integrations.md b/skills/agenthub-typescript/reference/integrations.md new file mode 100644 index 00000000..ec1847d7 --- /dev/null +++ b/skills/agenthub-typescript/reference/integrations.md @@ -0,0 +1,55 @@ +# Tracer & Playground + +## Tracer + +Tracer saves trace files and serves a local UI for inspecting conversations. + +Set `trace_id` to save trace files: + +```typescript +import { AutoLLMClient } from "@prismshadow/agenthub"; + +const client = new AutoLLMClient({ model: "gpt-5.5" }); + +const config = { trace_id: "agent1/conversation_001" }; + +for await (const event of client.streamingResponseStateful({ + message: { + role: "user", + content_items: [{ type: "text", text: "Hello" }], + }, + config, +})) { + console.log(event); +} +``` + +Default cache dir: `cache`, or `AGENTHUB_CACHE_DIR`. For `trace_id="agent1/conversation_001"`, AgentHub writes: + +- `cache/agent1/conversation_001.json`: Structured trace data with the full history and config. +- `cache/agent1/conversation_001.txt`: Human-readable conversation transcript. + +Browse traces: + +```typescript +import { Tracer } from "@prismshadow/agenthub/integration/tracer"; + +const tracer = new Tracer(); +tracer.startWebServer("127.0.0.1", 25750); +``` + +Open Tracer at `http://127.0.0.1:25750`. + +## Playground + +Playground starts a local chat UI for manual model checks. + +Start Playground for manual chat: + +```typescript +import { startPlaygroundServer } from "@prismshadow/agenthub/integration/playground"; + +startPlaygroundServer("127.0.0.1", 25751); +``` + +Open Playground at `http://127.0.0.1:25751`. diff --git a/skills/agenthub-typescript/reference/models.md b/skills/agenthub-typescript/reference/models.md new file mode 100644 index 00000000..3727633a --- /dev/null +++ b/skills/agenthub-typescript/reference/models.md @@ -0,0 +1,43 @@ +# Model Selection + +Use exact model IDs. If a model ID is not listed, ask the user to confirm the exact ID before using it. + +| Family | Provider | Model IDs | API Key | Base URL | +| --- | --- | --- | --- | --- | +| Gemini 3 | Official / Vertex AI | `gemini-3.1-pro-preview`, `gemini-3.5-flash`, `gemini-3.1-flash-lite` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | +| Gemini 3 Image | Official / Vertex AI | `gemini-3.1-flash-image-preview`, `gemini-3-pro-image-preview` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | +| Gemini 3 TTS | Official / Vertex AI | `gemini-3.1-flash-tts-preview` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | +| Gemini Embedding | Official / Vertex AI | `gemini-embedding-2` | `GEMINI_API_KEY` | `GEMINI_BASE_URL` | +| Claude 4.6 | Official / ModelVerse | `claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.6 | Bedrock | `global.anthropic.claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.7 | Official / ModelVerse | `claude-opus-4-7` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.7 | Bedrock | `global.anthropic.claude-opus-4-7` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.8 | Official / ModelVerse | `claude-opus-4-8` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 4.8 | Bedrock | `global.anthropic.claude-opus-4-8` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 5 | Official / ModelVerse | `claude-fable-5` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| Claude 5 | Bedrock | `global.anthropic.claude-fable-5` | `ANTHROPIC_API_KEY` | `ANTHROPIC_BASE_URL` | +| GPT 5.4 | Official / ModelVerse | `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | +| GPT 5.5 | Official / ModelVerse | `gpt-5.5` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | +| OpenAI Embedding | Official | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | +| Kimi-K2.6 | Official | `kimi-k2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | +| Kimi-K2.6 | OpenRouter | `moonshotai/kimi-k2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | +| Kimi-K2.6 | SiliconFlow | `Pro/moonshotai/Kimi-K2.6` | `MOONSHOT_API_KEY` | `MOONSHOT_BASE_URL` | +| DeepSeek V4 | Official | `deepseek-v4-pro`, `deepseek-v4-flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | +| DeepSeek V4 | OpenRouter | `deepseek/deepseek-v4-pro`, `deepseek/deepseek-v4-flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | +| DeepSeek V4 | SiliconFlow | `deepseek-ai/DeepSeek-V4-Pro`, `deepseek-ai/DeepSeek-V4-Flash` | `DEEPSEEK_API_KEY` | `DEEPSEEK_BASE_URL` | +| GLM-5.1 | Official | `glm-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | +| GLM-5.1 | OpenRouter | `z-ai/glm-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | +| GLM-5.1 | SiliconFlow | `Pro/zai-org/GLM-5.1` | `ZAI_API_KEY` | `ZAI_BASE_URL` | + +Common gateway base URLs: + +- OpenRouter: `https://openrouter.ai/api/v1` +- SiliconFlow: `https://api.siliconflow.cn/v1` +- ModelVerse: `https://api.modelverse.cn/v1` (`https://api.modelverse.cn/` for Claude) +- vLLM: `http://127.0.0.1:8000/v1/` + +For models accessed through OpenAI-compatible APIs (e.g., Qwen series models via SiliconFlow or OpenRouter), pass `clientType: "openai"` (`clientType: "openai-embedding"` for embedding endpoints). These models use `OPENAI_API_KEY` and `OPENAI_BASE_URL`: + +```typescript +const client = new AutoLLMClient({ model: "Qwen/Qwen3-Embedding-0.6B", clientType: "openai-embedding" }); +``` diff --git a/src_py/agenthub/__init__.py b/src_py/agenthub/__init__.py index 468a0700..aa36ed07 100644 --- a/src_py/agenthub/__init__.py +++ b/src_py/agenthub/__init__.py @@ -13,7 +13,15 @@ # limitations under the License. from .auto_client import AutoLLMClient +from .errors import AgentHubError, EmptyResponseError, ToolCallArgumentParseError from .types import PromptCaching, ThinkingLevel -__all__ = ["AutoLLMClient", "PromptCaching", "ThinkingLevel"] +__all__ = [ + "AgentHubError", + "AutoLLMClient", + "EmptyResponseError", + "PromptCaching", + "ThinkingLevel", + "ToolCallArgumentParseError", +] diff --git a/src_py/agenthub/auto_client.py b/src_py/agenthub/auto_client.py index 0817aadf..eae05d53 100644 --- a/src_py/agenthub/auto_client.py +++ b/src_py/agenthub/auto_client.py @@ -53,10 +53,12 @@ def _create_client_for_model( from .gemini3 import Gemini3Client return Gemini3Client(model=model, api_key=api_key, base_url=base_url) - elif "claude" in client_type and ("4-7" in client_type or "4-8" in client_type): # e.g., claude-opus-4-7 - from .claude4_8 import Claude4_8Client + elif "claude" in client_type and ( + "4-7" in client_type or "4-8" in client_type or "-5" in client_type + ): # e.g., claude-opus-4-7 + from .claude5 import Claude5Client - return Claude4_8Client(model=model, api_key=api_key, base_url=base_url) + return Claude5Client(model=model, api_key=api_key, base_url=base_url) elif "claude" in client_type and "4-6" in client_type: # e.g., claude-sonnet-4-6 from .claude4_6 import Claude4_6Client @@ -81,14 +83,14 @@ def _create_client_for_model( from .openai_embedding import OpenaiEmbeddingClient return OpenaiEmbeddingClient(model=model, api_key=api_key, base_url=base_url) - elif "openai" in client_type: + elif "openai" in client_type and "embedding" not in client_type: from .openai import OpenaiClient return OpenaiClient(model=model, api_key=api_key, base_url=base_url) else: raise ValueError( f"{client_type} is not supported. " - "Supported client types: gemini-3, claude-4-8, claude-4-7, claude-4-6, gpt-5.5, gpt-5.4, glm-5.1, kimi-k2.6, kimi-k2.5, deepseek-v4, openai-embedding, openai." + "Supported client types: gemini-3, claude-5, claude-4-8, claude-4-7, claude-4-6, gpt-5.5, gpt-5.4, glm-5.1, kimi-k2.6, kimi-k2.5, deepseek-v4, openai-embedding, openai." ) def transform_uni_config_to_model_config(self, config: UniConfig) -> Any: diff --git a/src_py/agenthub/base_client.py b/src_py/agenthub/base_client.py index 92a73c6d..36488086 100644 --- a/src_py/agenthub/base_client.py +++ b/src_py/agenthub/base_client.py @@ -19,6 +19,7 @@ from typing import Any, AsyncIterator from .abort_signal import AbortSignal +from .errors import EmptyResponseError from .types import ( ContentItem, FinishReason, @@ -100,28 +101,40 @@ def concat_uni_events_to_uni_message(self, events: list[UniEvent]) -> UniMessage for event in events: # Merge content_items from all events for item in event["content_items"]: + last_fidelity = (content_items[-1].get("fidelity") or {}) if content_items else {} + item_fidelity = item.get("fidelity") or {} if item["type"] == "text": + # a delta announcing a different phase starts a new item; same-phase and + # phaseless deltas merge until a signature finishes the item if ( content_items and content_items[-1]["type"] == "text" - and content_items[-1].get("signature") is None # no signature yet - and item.get("phase") is None # no new phase + and last_fidelity.get("signature") is None # not finished by a signature yet + and ( + item_fidelity.get("phase") is None # phaseless deltas continue the item + or item_fidelity.get("phase") == last_fidelity.get("phase") # same phase merges + ) ): content_items[-1]["text"] += item["text"] - if "signature" in item: # finish the current item if signature is not None - content_items[-1]["signature"] = item["signature"] - elif item["text"] or item.get("phase") is not None: # text or new phase starts an item + if item_fidelity: # a signature finishes the current item + content_items[-1]["fidelity"] = {**last_fidelity, **item_fidelity} + elif item["text"] or item_fidelity.get("phase") is not None: # text or new phase starts an item content_items.append(item.copy()) elif item["type"] == "thinking": + # a new item starts only when the open item's fidelity is non-empty and + # differs from the incoming delta's; everything else merges into it if ( content_items and content_items[-1]["type"] == "thinking" - and content_items[-1].get("signature") is None # no signature yet + and ( + not last_fidelity # not finished by fidelity yet + or last_fidelity == item_fidelity # a run of equal fidelity is one item + ) ): content_items[-1]["thinking"] += item["thinking"] - if "signature" in item: # finish the current item if signature is not None - content_items[-1]["signature"] = item["signature"] - elif item["thinking"] or item.get("signature"): # omit empty thinking items + if item_fidelity: # fidelity finishes the current item + content_items[-1]["fidelity"] = item_fidelity + elif item["thinking"] or item_fidelity: # omit empty thinking items content_items.append(item.copy()) elif item["type"] == "partial_tool_call": # Skip partial_tool_call items - they should already be converted to tool_call @@ -244,6 +257,7 @@ def cancel_streaming_task(task: asyncio.Task[None]) -> None: await stream.aclose() self._validate_last_event(last_event) + self._validate_non_thinking_output(events) # Save history to file if trace_id is specified if config.get("trace_id") and events: @@ -312,6 +326,25 @@ def _validate_last_event(last_event: UniEvent | None) -> None: if last_event["finish_reason"] is None: raise ValueError(f"Last event must carry finish_reason, got: {last_event}") + def _validate_non_thinking_output(self, events: list[UniEvent]) -> None: + """Validate that the completed response carries content other than thinking. + + Replaying a thinking-only assistant message on the next turn fails with a 400 + error, so the response is rejected as soon as the stream completes. + + Args: + events: All events yielded by streaming_response + + Raises: + EmptyResponseError: If every content item in the response is thinking + """ + thinking_only = all( + item["type"] in ("thinking", "inline_thinking") for event in events for item in event["content_items"] + ) + if thinking_only: + finish_reason = events[-1]["finish_reason"] if events else None + raise EmptyResponseError(self.__class__.__name__, finish_reason) + def clear_history(self) -> None: """Clear the message history.""" self._history.clear() diff --git a/src_py/agenthub/claude4_6/client.py b/src_py/agenthub/claude4_6/client.py index 29f1d7fe..42469303 100644 --- a/src_py/agenthub/claude4_6/client.py +++ b/src_py/agenthub/claude4_6/client.py @@ -13,7 +13,6 @@ # limitations under the License. import base64 -import json import mimetypes import os import re @@ -24,6 +23,7 @@ from anthropic.types.beta import BetaMessageParam, BetaRawMessageStreamEvent from ..base_client import LLMClient +from ..errors import parse_tool_call_arguments from ..types import ( EventType, FinishReason, @@ -204,10 +204,14 @@ async def transform_uni_message_to_model_input(self, messages: list[UniMessage]) content_blocks.append(await self._convert_image_url_to_source(item["image_url"])) elif item["type"] == "thinking": if item["thinking"] == REDACTED_THINKING: - content_blocks.append({"type": "redacted_thinking", "data": item["signature"]}) + content_blocks.append({"type": "redacted_thinking", "data": item["fidelity"]["signature"]}) else: content_blocks.append( - {"type": "thinking", "thinking": item["thinking"], "signature": item["signature"]} + { + "type": "thinking", + "thinking": item["thinking"], + "signature": item["fidelity"]["signature"], + } ) elif item["type"] == "tool_call": content_blocks.append( @@ -263,7 +267,9 @@ def transform_model_output_to_uni_event(self, model_output: BetaRawMessageStream {"type": "partial_tool_call", "name": block.name, "arguments": "", "tool_call_id": block.id} ) elif block.type == "redacted_thinking": - content_items.append({"type": "thinking", "thinking": REDACTED_THINKING, "signature": block.data}) + content_items.append( + {"type": "thinking", "thinking": REDACTED_THINKING, "fidelity": {"signature": block.data}} + ) elif claude_event_type == "content_block_delta": event_type = "delta" @@ -277,7 +283,7 @@ def transform_model_output_to_uni_event(self, model_output: BetaRawMessageStream {"type": "partial_tool_call", "name": "", "arguments": delta.partial_json, "tool_call_id": ""} ) elif delta.type == "signature_delta": - content_items.append({"type": "thinking", "thinking": "", "signature": delta.signature}) + content_items.append({"type": "thinking", "thinking": "", "fidelity": {"signature": delta.signature}}) elif claude_event_type == "content_block_stop": event_type = "stop" @@ -403,7 +409,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"]), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], diff --git a/src_py/agenthub/claude4_8/__init__.py b/src_py/agenthub/claude5/__init__.py similarity index 90% rename from src_py/agenthub/claude4_8/__init__.py rename to src_py/agenthub/claude5/__init__.py index b9db80d5..9253dc6b 100644 --- a/src_py/agenthub/claude4_8/__init__.py +++ b/src_py/agenthub/claude5/__init__.py @@ -12,7 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -from .client import Claude4_8Client +from .client import Claude5Client -__all__ = ["Claude4_8Client"] +__all__ = ["Claude5Client"] diff --git a/src_py/agenthub/claude4_8/client.py b/src_py/agenthub/claude5/client.py similarity index 94% rename from src_py/agenthub/claude4_8/client.py rename to src_py/agenthub/claude5/client.py index eaf06a08..8ca199e1 100644 --- a/src_py/agenthub/claude4_8/client.py +++ b/src_py/agenthub/claude5/client.py @@ -13,7 +13,6 @@ # limitations under the License. import base64 -import json import mimetypes import os import re @@ -24,6 +23,7 @@ from anthropic.types.beta import BetaMessageParam, BetaRawMessageStreamEvent from ..base_client import LLMClient +from ..errors import parse_tool_call_arguments from ..types import ( EventType, FinishReason, @@ -41,11 +41,11 @@ REDACTED_THINKING = "_REDACTED_THINKING" -class Claude4_8Client(LLMClient): - """Claude 4.8-specific LLM client implementation.""" +class Claude5Client(LLMClient): + """Claude 5-specific LLM client implementation.""" def __init__(self, model: str, api_key: str | None = None, base_url: str | None = None): - """Initialize Claude 4.8 client with model and API key.""" + """Initialize Claude 5 client with model and API key.""" self._model = model api_key = api_key or os.getenv("ANTHROPIC_API_KEY") base_url = base_url or os.getenv("ANTHROPIC_BASE_URL") @@ -204,10 +204,14 @@ async def transform_uni_message_to_model_input(self, messages: list[UniMessage]) content_blocks.append(await self._convert_image_url_to_source(item["image_url"])) elif item["type"] == "thinking": if item["thinking"] == REDACTED_THINKING: - content_blocks.append({"type": "redacted_thinking", "data": item["signature"]}) + content_blocks.append({"type": "redacted_thinking", "data": item["fidelity"]["signature"]}) else: content_blocks.append( - {"type": "thinking", "thinking": item["thinking"], "signature": item["signature"]} + { + "type": "thinking", + "thinking": item["thinking"], + "signature": item["fidelity"]["signature"], + } ) elif item["type"] == "tool_call": content_blocks.append( @@ -263,7 +267,9 @@ def transform_model_output_to_uni_event(self, model_output: BetaRawMessageStream {"type": "partial_tool_call", "name": block.name, "arguments": "", "tool_call_id": block.id} ) elif block.type == "redacted_thinking": - content_items.append({"type": "thinking", "thinking": REDACTED_THINKING, "signature": block.data}) + content_items.append( + {"type": "thinking", "thinking": REDACTED_THINKING, "fidelity": {"signature": block.data}} + ) elif claude_event_type == "content_block_delta": event_type = "delta" @@ -277,7 +283,7 @@ def transform_model_output_to_uni_event(self, model_output: BetaRawMessageStream {"type": "partial_tool_call", "name": "", "arguments": delta.partial_json, "tool_call_id": ""} ) elif delta.type == "signature_delta": - content_items.append({"type": "thinking", "thinking": "", "signature": delta.signature}) + content_items.append({"type": "thinking", "thinking": "", "fidelity": {"signature": delta.signature}}) elif claude_event_type == "content_block_stop": event_type = "stop" @@ -403,7 +409,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"]), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], diff --git a/src_py/agenthub/deepseek_v4/client.py b/src_py/agenthub/deepseek_v4/client.py index dffb99fd..99e27168 100644 --- a/src_py/agenthub/deepseek_v4/client.py +++ b/src_py/agenthub/deepseek_v4/client.py @@ -20,6 +20,7 @@ from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam from ..base_client import LLMClient +from ..errors import parse_tool_call_arguments from ..types import ( EventType, FinishReason, @@ -200,7 +201,15 @@ def transform_model_output_to_uni_event(self, model_output: ChatCompletionChunk) if getattr(delta, "reasoning_content", None): event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning_content")}) + # record the wire field so a replay through another OpenAI-compatible + # client reproduces the exact field DeepSeek produced + content_items.append( + { + "type": "thinking", + "thinking": getattr(delta, "reasoning_content"), + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ) if delta.content: event_type = "delta" @@ -289,7 +298,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"] or "{}"), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], @@ -317,7 +331,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"] or "{}"), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], diff --git a/src_py/agenthub/errors.py b/src_py/agenthub/errors.py new file mode 100644 index 00000000..94e1a692 --- /dev/null +++ b/src_py/agenthub/errors.py @@ -0,0 +1,75 @@ +# Copyright 2025 Prism Shadow. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json +from typing import Any + + +def _preview_tool_call_arguments(raw: str) -> str: + max_length = 160 + if len(raw) <= max_length: + return raw + + edge_length = 72 + return f"{raw[:edge_length]}...[truncated]...{raw[-edge_length:]}" + + +class AgentHubError(ValueError): + """Base class for errors raised by AgentHub clients.""" + + +class EmptyResponseError(AgentHubError): + """Raised when a completed response carries no non-thinking content and no tool calls. + + Models occasionally finish a turn with thinking output only (reasoning models in + particular); replaying such an assistant message on the next turn fails with a 400 + error, so the response is rejected as soon as the stream completes. + """ + + def __init__(self, client: str, finish_reason: str | None) -> None: + self.client = client + self.finish_reason = finish_reason + super().__init__(f"{client} returned no content other than thinking (finish_reason={finish_reason!r}).") + + +class ToolCallArgumentParseError(AgentHubError): + def __init__(self, client: str, tool_name: str, tool_call_id: str, raw_arguments: str, reason: str) -> None: + self.client = client + self.tool_name = tool_name + self.tool_call_id = tool_call_id + self.raw_arguments_length = len(raw_arguments) + self.raw_arguments_preview = _preview_tool_call_arguments(raw_arguments) + super().__init__( + f'Invalid streamed tool call arguments from {client} for tool "{tool_name}" ' + f'(tool_call_id="{tool_call_id}", length={self.raw_arguments_length}, ' + f"preview={self.raw_arguments_preview!r}): {reason}" + ) + + +def parse_tool_call_arguments( + raw_arguments: str | None, + client: str, + tool_name: str, + tool_call_id: str, +) -> dict[str, Any]: + raw = raw_arguments or "{}" + try: + parsed = json.loads(raw) + except (TypeError, ValueError) as exc: + raise ToolCallArgumentParseError(client, tool_name, tool_call_id, raw, str(exc)) from exc + + if not isinstance(parsed, dict): + raise ToolCallArgumentParseError(client, tool_name, tool_call_id, raw, "Expected a JSON object.") + + return parsed diff --git a/src_py/agenthub/gemini3/client.py b/src_py/agenthub/gemini3/client.py index 7a8c07eb..bfaa87d5 100644 --- a/src_py/agenthub/gemini3/client.py +++ b/src_py/agenthub/gemini3/client.py @@ -26,7 +26,9 @@ from ..base_client import LLMClient from ..types import ( + ContentItem, EventType, + Fidelity, FinishReason, PartialContentItem, PromptCaching, @@ -188,6 +190,19 @@ def transform_uni_config_to_model_config(self, config: UniConfig) -> types.Gener return types.GenerateContentConfig(**config_params) if config_params else None + @staticmethod + def _part_fidelity(part: types.Part) -> dict[str, Fidelity]: + """Wrap a part's thought signature as a fidelity payload, or nothing when absent.""" + if part.thought_signature is None: + return {} + + return {"fidelity": {"signature": part.thought_signature}} + + @staticmethod + def _item_thought_signature(item: ContentItem) -> str | bytes | None: + """Read the thought signature recorded in an item's fidelity payload.""" + return (item.get("fidelity") or {}).get("signature") + async def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> list[types.Content]: """ Transform universal message format to Gemini's Content format. @@ -204,26 +219,34 @@ async def transform_uni_message_to_model_input(self, messages: list[UniMessage]) parts = [] for item in msg["content_items"]: if item["type"] == "text": - parts.append(types.Part(text=item["text"], thought_signature=item.get("signature"))) + parts.append(types.Part(text=item["text"], thought_signature=self._item_thought_signature(item))) elif item["type"] == "image_url": image_url = item["image_url"] image_data = await self._get_image_bytes_and_mime_type(image_url) parts.append(types.Part.from_bytes(**image_data)) elif item["type"] == "inline_data": inline_data = types.Blob(data=item["data"], mime_type=item["mime_type"]) - parts.append(types.Part(inline_data=inline_data, thought_signature=item.get("signature"))) + parts.append( + types.Part(inline_data=inline_data, thought_signature=self._item_thought_signature(item)) + ) elif item["type"] == "thinking": parts.append( - types.Part(text=item["thinking"], thought=True, thought_signature=item.get("signature")) + types.Part( + text=item["thinking"], thought=True, thought_signature=self._item_thought_signature(item) + ) ) elif item["type"] == "inline_thinking": inline_data = types.Blob(data=item["data"], mime_type=item["mime_type"]) parts.append( - types.Part(inline_data=inline_data, thought=True, thought_signature=item.get("signature")) + types.Part( + inline_data=inline_data, thought=True, thought_signature=self._item_thought_signature(item) + ) ) elif item["type"] == "tool_call": function_call = types.FunctionCall(name=item["name"], args=item["arguments"]) - parts.append(types.Part(function_call=function_call, thought_signature=item.get("signature"))) + parts.append( + types.Part(function_call=function_call, thought_signature=self._item_thought_signature(item)) + ) elif item["type"] == "tool_result": if "tool_call_id" not in item: raise ValueError("tool_call_id is required for tool result.") @@ -266,31 +289,30 @@ def transform_model_output_to_uni_event(self, model_output: types.GenerateConten usage_metadata: UsageMetadata | None = None finish_reason: FinishReason | None = None - if len(model_output.candidates) > 0: + if model_output.candidates: candidate = model_output.candidates[0] - for part in candidate.content.parts: + content = getattr(candidate, "content", None) + for part in getattr(content, "parts", None) or []: if part.function_call is not None: content_items.append( { "type": "tool_call", "name": part.function_call.name, - "arguments": part.function_call.args, + "arguments": part.function_call.args or {}, "tool_call_id": part.function_call.name, - "signature": part.thought_signature, + **self._part_fidelity(part), } ) elif part.thought: if part.text is not None: - content_items.append( - {"type": "thinking", "thinking": part.text, "signature": part.thought_signature} - ) + content_items.append({"type": "thinking", "thinking": part.text, **self._part_fidelity(part)}) elif part.inline_data is not None: content_items.append( { "type": "inline_thinking", "data": part.inline_data.data, "mime_type": part.inline_data.mime_type, - "signature": part.thought_signature, + **self._part_fidelity(part), } ) elif part.inline_data is not None: @@ -299,11 +321,11 @@ def transform_model_output_to_uni_event(self, model_output: types.GenerateConten "type": "inline_data", "data": part.inline_data.data, "mime_type": part.inline_data.mime_type, - "signature": part.thought_signature, + **self._part_fidelity(part), } ) elif part.text is not None: - content_items.append({"type": "text", "text": part.text, "signature": part.thought_signature}) + content_items.append({"type": "text", "text": part.text, **self._part_fidelity(part)}) else: raise ValueError(f"Unknown output: {part}") @@ -414,7 +436,7 @@ async def _streaming_response_internal( "name": item["name"], "arguments": json.dumps(item["arguments"], ensure_ascii=False), "tool_call_id": item["tool_call_id"], - "signature": item.get("signature"), + "fidelity": item.get("fidelity"), } ], "usage_metadata": None, diff --git a/src_py/agenthub/glm5_1/client.py b/src_py/agenthub/glm5_1/client.py index b8b274f0..c39e42ab 100644 --- a/src_py/agenthub/glm5_1/client.py +++ b/src_py/agenthub/glm5_1/client.py @@ -20,6 +20,7 @@ from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam from ..base_client import LLMClient +from ..errors import parse_tool_call_arguments from ..types import ( EventType, FinishReason, @@ -115,6 +116,7 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> li content_parts = [] # may be empty for tool results tool_calls = [] # may be empty for no tool calls thinking = "" + thinking_fields: set[str | None] = set() for item in msg["content_items"]: if item["type"] == "text": content_parts.append({"type": "text", "text": item["text"]}) @@ -122,6 +124,7 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> li raise ValueError("GLM-5 does not support image inputs.") elif item["type"] == "thinking": thinking += item["thinking"] + thinking_fields.add((item.get("fidelity") or {}).get("reasoning_field")) elif item["type"] == "tool_call": tool_calls.append( { @@ -159,8 +162,15 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> li message["tool_calls"] = tool_calls if thinking: - message["reasoning_content"] = thinking # vLLM & siliconflow compatibility - message["reasoning"] = thinking # openrouter compatibility + # send thinking back through the exact field the upstream produced (recorded + # in the item fidelity); servers may reject the spelling they did not emit + if thinking_fields == {"reasoning_content"}: + message["reasoning_content"] = thinking + elif thinking_fields == {"reasoning"}: + message["reasoning"] = thinking + else: + message["reasoning_content"] = thinking # vLLM & siliconflow compatibility + message["reasoning"] = thinking # openrouter compatibility # message may be empty for tool results if len(message.keys()) > 1: @@ -191,15 +201,29 @@ def transform_model_output_to_uni_event(self, model_output: ChatCompletionChunk) event_type = "delta" content_items.append({"type": "text", "text": delta.content}) - # vLLM & siliconflow compatibility - if getattr(delta, "reasoning_content", None): + # the thinking field name differs by server: vLLM & siliconflow use reasoning_content + # while openrouter uses reasoning; record the wire field that carried each delta + # so a replay can reproduce exactly the field the upstream produced + reasoning_content = getattr(delta, "reasoning_content", None) + reasoning = getattr(delta, "reasoning", None) + if reasoning_content and reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning_content")}) - - # openrouter compatibility - elif getattr(delta, "reasoning", None): + # ambiguous origin: record no fidelity so a replay sends both fields back + content_items.append({"type": "thinking", "thinking": reasoning_content}) + elif reasoning_content: + event_type = "delta" + content_items.append( + { + "type": "thinking", + "thinking": reasoning_content, + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ) + elif reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning")}) + content_items.append( + {"type": "thinking", "thinking": reasoning, "fidelity": {"reasoning_field": "reasoning"}} + ) if delta.tool_calls: event_type = "delta" @@ -307,7 +331,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"] or "{}"), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], @@ -335,7 +364,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"] or "{}"), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], diff --git a/src_py/agenthub/gpt5_5/client.py b/src_py/agenthub/gpt5_5/client.py index 933b08df..eafb2033 100644 --- a/src_py/agenthub/gpt5_5/client.py +++ b/src_py/agenthub/gpt5_5/client.py @@ -20,6 +20,7 @@ from openai.types.responses import ResponseInputParam, ResponseStreamEvent from ..base_client import LLMClient +from ..errors import parse_tool_call_arguments from ..types import ( EventType, FinishReason, @@ -119,12 +120,13 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> Re for item in msg["content_items"]: if item["type"] == "text": - if msg["role"] == "assistant" and item.get("phase"): # split different phases - if last_phase is not None and content_items: + phase = (item.get("fidelity") or {}).get("phase") + if msg["role"] == "assistant" and phase: # split different phases + if last_phase is not None and last_phase != phase and content_items: input_list.append({"role": msg["role"], "content": content_items, "phase": last_phase}) content_items = [] - last_phase = item["phase"] + last_phase = phase if msg["role"] == "user": content_items.append({"type": "input_text", "text": item["text"]}) @@ -133,15 +135,15 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> Re elif item["type"] == "image_url": content_items.append({"type": "input_image", "image_url": item["image_url"]}) elif item["type"] == "thinking": - signature = json.loads(item["signature"]) + fidelity = item["fidelity"] input_list.append( { "type": "reasoning", - "id": signature["id"], + "id": fidelity["id"], "summary": [{"type": "summary_text", "text": item["thinking"]}] if item["thinking"] else [], - "encrypted_content": signature["encrypted_content"], + "encrypted_content": fidelity["encrypted_content"], } ) elif item["type"] == "tool_call": @@ -216,16 +218,16 @@ def transform_model_output_to_uni_event(self, model_output: ResponseStreamEvent) # adding the following thinking item leads to 400 invalid request error, why? # elif model_output.item.type == "reasoning": # event_type = "delta" - # signature = { + # fidelity = { # "id": model_output.item.id, # "encrypted_content": model_output.item.encrypted_content, # } - # content_items.append({"type": "thinking", "thinking": "", "signature": json.dumps(signature)}) + # content_items.append({"type": "thinking", "thinking": "", "fidelity": fidelity}) elif model_output.item.type == "message": if hasattr(model_output.item, "phase"): event_type = "delta" content_items.append( - {"type": "text", "text": "", "phase": getattr(model_output.item, "phase", None)} + {"type": "text", "text": "", "fidelity": {"phase": getattr(model_output.item, "phase", None)}} ) else: event_type = "unused" @@ -236,11 +238,11 @@ def transform_model_output_to_uni_event(self, model_output: ResponseStreamEvent) # not sure about the signature of openai, need to check if model_output.item.type == "reasoning": event_type = "delta" - signature = { + fidelity = { "id": model_output.item.id, "encrypted_content": model_output.item.encrypted_content, } - content_items.append({"type": "thinking", "thinking": "", "signature": json.dumps(signature)}) + content_items.append({"type": "thinking", "thinking": "", "fidelity": fidelity}) else: event_type = "unused" @@ -341,7 +343,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"]), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], diff --git a/src_py/agenthub/kimi_k2_6/client.py b/src_py/agenthub/kimi_k2_6/client.py index 51115770..a3578147 100644 --- a/src_py/agenthub/kimi_k2_6/client.py +++ b/src_py/agenthub/kimi_k2_6/client.py @@ -23,6 +23,7 @@ from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam from ..base_client import LLMClient +from ..errors import parse_tool_call_arguments from ..types import ( EventType, FinishReason, @@ -143,6 +144,7 @@ async def transform_uni_message_to_model_input( content_parts = [] # may be empty for tool results tool_calls = [] # may be empty for no tool calls thinking = "" + thinking_fields: set[str | None] = set() for item in msg["content_items"]: if item["type"] == "text": content_parts.append({"type": "text", "text": item["text"]}) @@ -151,6 +153,7 @@ async def transform_uni_message_to_model_input( content_parts.append({"type": "image_url", "image_url": {"url": base64_image}}) elif item["type"] == "thinking": thinking += item["thinking"] + thinking_fields.add((item.get("fidelity") or {}).get("reasoning_field")) elif item["type"] == "tool_call": tool_calls.append( { @@ -196,8 +199,15 @@ async def transform_uni_message_to_model_input( message["tool_calls"] = tool_calls if thinking: - message["reasoning_content"] = thinking # vLLM & siliconflow compatibility - message["reasoning"] = thinking # openrouter compatibility + # send thinking back through the exact field the upstream produced (recorded + # in the item fidelity); servers may reject the spelling they did not emit + if thinking_fields == {"reasoning_content"}: + message["reasoning_content"] = thinking + elif thinking_fields == {"reasoning"}: + message["reasoning"] = thinking + else: + message["reasoning_content"] = thinking # vLLM & siliconflow compatibility + message["reasoning"] = thinking # openrouter compatibility # message may be empty for tool results if len(message.keys()) > 1: @@ -228,15 +238,29 @@ def transform_model_output_to_uni_event(self, model_output: ChatCompletionChunk) event_type = "delta" content_items.append({"type": "text", "text": delta.content}) - # vLLM & siliconflow compatibility - if getattr(delta, "reasoning_content", None): + # the thinking field name differs by server: vLLM & siliconflow use reasoning_content + # while openrouter uses reasoning; record the wire field that carried each delta + # so a replay can reproduce exactly the field the upstream produced + reasoning_content = getattr(delta, "reasoning_content", None) + reasoning = getattr(delta, "reasoning", None) + if reasoning_content and reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning_content")}) - - # openrouter compatibility - elif getattr(delta, "reasoning", None): + # ambiguous origin: record no fidelity so a replay sends both fields back + content_items.append({"type": "thinking", "thinking": reasoning_content}) + elif reasoning_content: + event_type = "delta" + content_items.append( + { + "type": "thinking", + "thinking": reasoning_content, + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ) + elif reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning")}) + content_items.append( + {"type": "thinking", "thinking": reasoning, "fidelity": {"reasoning_field": "reasoning"}} + ) if delta.tool_calls: event_type = "delta" @@ -341,7 +365,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"] or "{}"), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], @@ -369,7 +398,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"] or "{}"), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], diff --git a/src_py/agenthub/openai/client.py b/src_py/agenthub/openai/client.py index 6497c455..369bb52f 100644 --- a/src_py/agenthub/openai/client.py +++ b/src_py/agenthub/openai/client.py @@ -23,6 +23,7 @@ from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam from ..base_client import LLMClient +from ..errors import parse_tool_call_arguments from ..types import ( EventType, FinishReason, @@ -128,6 +129,7 @@ async def transform_uni_message_to_model_input( content_parts = [] # may be empty for tool results tool_calls = [] # may be empty for no tool calls thinking = "" + thinking_fields: set[str | None] = set() for item in msg["content_items"]: if item["type"] == "text": content_parts.append({"type": "text", "text": item["text"]}) @@ -136,6 +138,7 @@ async def transform_uni_message_to_model_input( content_parts.append({"type": "image_url", "image_url": {"url": base64_image}}) elif item["type"] == "thinking": thinking += item["thinking"] + thinking_fields.add((item.get("fidelity") or {}).get("reasoning_field")) elif item["type"] == "tool_call": tool_calls.append( { @@ -181,8 +184,15 @@ async def transform_uni_message_to_model_input( message["tool_calls"] = tool_calls if thinking: - message["reasoning_content"] = thinking # vLLM & siliconflow compatibility - message["reasoning"] = thinking # openrouter compatibility + # send thinking back through the exact field the upstream produced (recorded + # in the item fidelity); servers may reject the spelling they did not emit + if thinking_fields == {"reasoning_content"}: + message["reasoning_content"] = thinking + elif thinking_fields == {"reasoning"}: + message["reasoning"] = thinking + else: + message["reasoning_content"] = thinking # vLLM & siliconflow compatibility + message["reasoning"] = thinking # openrouter compatibility # message may be empty for tool results if len(message.keys()) > 1: @@ -213,15 +223,29 @@ def transform_model_output_to_uni_event(self, model_output: ChatCompletionChunk) event_type = "delta" content_items.append({"type": "text", "text": delta.content}) - # vLLM & siliconflow compatibility - if getattr(delta, "reasoning_content", None): + # the thinking field name differs by server: vLLM & siliconflow use reasoning_content + # while openrouter uses reasoning; record the wire field that carried each delta + # so a replay can reproduce exactly the field the upstream produced + reasoning_content = getattr(delta, "reasoning_content", None) + reasoning = getattr(delta, "reasoning", None) + if reasoning_content and reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning_content")}) - - # openrouter compatibility - elif getattr(delta, "reasoning", None): + # ambiguous origin: record no fidelity so a replay sends both fields back + content_items.append({"type": "thinking", "thinking": reasoning_content}) + elif reasoning_content: + event_type = "delta" + content_items.append( + { + "type": "thinking", + "thinking": reasoning_content, + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ) + elif reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning")}) + content_items.append( + {"type": "thinking", "thinking": reasoning, "fidelity": {"reasoning_field": "reasoning"}} + ) if delta.tool_calls: for tool_call in delta.tool_calls: @@ -325,7 +349,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"] or "{}"), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], @@ -353,7 +382,12 @@ async def _streaming_response_internal( { "type": "tool_call", "name": partial_tool_call["name"], - "arguments": json.loads(partial_tool_call["arguments"] or "{}"), + "arguments": parse_tool_call_arguments( + partial_tool_call["arguments"], + self.__class__.__name__, + partial_tool_call["name"], + partial_tool_call["tool_call_id"], + ), "tool_call_id": partial_tool_call["tool_call_id"], } ], diff --git a/src_py/agenthub/types.py b/src_py/agenthub/types.py index 2339ba91..cc7c36e3 100644 --- a/src_py/agenthub/types.py +++ b/src_py/agenthub/types.py @@ -42,12 +42,16 @@ class PromptCaching(StrEnum): AspectRatio = Literal["1:1", "2:3", "3:2", "3:4", "4:3", "9:16", "16:9", "21:9"] ImageSize = Literal["1K", "2K"] +# Arbitrary JSON-style payload of wire-fidelity data recorded by a client, such as +# thinking signatures, phase labels, or the upstream reasoning field name. Opaque to +# consumers: pass it back unchanged so a replay reproduces the original wire message. +Fidelity = dict[str, Any] + class TextContentItem(TypedDict): type: Literal["text"] text: str - phase: NotRequired[str | None] - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class ImageContentItem(TypedDict): @@ -59,20 +63,20 @@ class InlineDataContentItem(TypedDict): type: Literal["inline_data"] data: bytes mime_type: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class ThinkingContentItem(TypedDict): type: Literal["thinking"] thinking: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class InlineThinkingContentItem(TypedDict): type: Literal["inline_thinking"] data: bytes mime_type: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class ToolCallContentItem(TypedDict): @@ -80,7 +84,7 @@ class ToolCallContentItem(TypedDict): name: str arguments: dict[str, Any] tool_call_id: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class PartialToolCallContentItem(TypedDict): @@ -88,7 +92,7 @@ class PartialToolCallContentItem(TypedDict): name: str arguments: str tool_call_id: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class ToolResultContentItem(TypedDict): diff --git a/src_py/pyproject.toml b/src_py/pyproject.toml index 910b03a4..7697e7ce 100644 --- a/src_py/pyproject.toml +++ b/src_py/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "agenthub-python" -version = "0.3.3" +version = "0.4.0" description = "AgentHub is the LLM API Hub for the Agent era, built for high-precision autonomous agents." keywords = ["agent", "llm", "gemini", "claude", "gpt"] readme = "README.md" @@ -11,6 +11,11 @@ authors = [ {name = "PrismShadow"}, ] +[project.urls] +Homepage = "https://github.com/Prism-Shadow/agenthub" +Repository = "https://github.com/Prism-Shadow/agenthub" +Issues = "https://github.com/Prism-Shadow/agenthub/issues" + [project.optional-dependencies] dev = ["httpx[socks]", "pytest>=8.4.2", "pytest-asyncio>=0.23.0", "ruff>=0.14.3", "pillow>=10.0.0"] diff --git a/src_py/tests/test_empty_response.py b/src_py/tests/test_empty_response.py new file mode 100644 index 00000000..4a58b84b --- /dev/null +++ b/src_py/tests/test_empty_response.py @@ -0,0 +1,179 @@ +# Copyright 2025 Prism Shadow. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from collections.abc import AsyncIterator +from dataclasses import dataclass +from types import SimpleNamespace + +import pytest + +from agenthub import AgentHubError, AutoLLMClient, EmptyResponseError, ToolCallArgumentParseError + + +@dataclass +class ReasoningStreamCase: + expected_client: str + model: str + client_type: str + + +REASONING_STREAM_CASES = [ + ReasoningStreamCase( + expected_client="OpenaiClient", + model="gpt-5.5", + client_type="openai", + ), + ReasoningStreamCase( + expected_client="GLM5_1Client", + model="glm-5.1", + client_type="glm-5.1", + ), + ReasoningStreamCase( + expected_client="KimiK2_6Client", + model="kimi-k2.6", + client_type="kimi-k2.6", + ), + ReasoningStreamCase( + expected_client="DeepSeekV4Client", + model="deepseek-v4", + client_type="deepseek-v4", + ), +] + + +def _create_auto_client(case: ReasoningStreamCase) -> AutoLLMClient: + return AutoLLMClient(model=case.model, api_key="test-key", client_type=case.client_type) + + +async def _stream_from_chunks(chunks: list[object]) -> AsyncIterator[object]: + for chunk in chunks: + yield chunk + + +class _FakeOpenAICompatibleCompletions: + def __init__(self, chunks: list[object]) -> None: + self._chunks = chunks + + async def create(self, **_kwargs: object) -> AsyncIterator[object]: + return _stream_from_chunks(self._chunks) + + +class _FakeOpenAICompatibleClient: + def __init__(self, chunks: list[object]) -> None: + self.base_url = "https://api.test.invalid/v1" + self.chat = SimpleNamespace(completions=_FakeOpenAICompatibleCompletions(chunks)) + + +def _install_fake_openai_compatible_stream(client: AutoLLMClient, chunks: list[object]) -> None: + client._client._client = _FakeOpenAICompatibleClient(chunks) # noqa: SLF001 + + +def _delta_chunk(text: str | None = None, reasoning_content: str | None = None) -> object: + return SimpleNamespace( + choices=[ + SimpleNamespace( + delta=SimpleNamespace(content=text, tool_calls=None, reasoning_content=reasoning_content), + finish_reason=None, + ) + ], + usage=None, + ) + + +def _stop_chunk(finish_reason: str = "stop") -> object: + return SimpleNamespace( + choices=[SimpleNamespace(delta=SimpleNamespace(content=None, tool_calls=None), finish_reason=finish_reason)], + usage=SimpleNamespace( + prompt_tokens=1, + completion_tokens=1, + prompt_tokens_details=None, + completion_tokens_details=SimpleNamespace(reasoning_tokens=1), + prompt_cache_hit_tokens=0, + prompt_cache_miss_tokens=1, + ), + ) + + +MESSAGES = [{"role": "user", "content_items": [{"type": "text", "text": "Create a memo."}]}] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "case", + REASONING_STREAM_CASES, + ids=[case.client_type for case in REASONING_STREAM_CASES], +) +async def test_reasoning_clients_reject_thinking_only_response(case: ReasoningStreamCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _delta_chunk(reasoning_content="Let me think about the memo."), + _stop_chunk(finish_reason="stop"), + ], + ) + + with pytest.raises(EmptyResponseError) as exc_info: + async for _event in client.streaming_response(MESSAGES, {}): + pass + + empty_error = exc_info.value + assert empty_error.client == case.expected_client + assert empty_error.finish_reason == "stop" + assert "no content other than thinking" in str(empty_error) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "case", + REASONING_STREAM_CASES, + ids=[case.client_type for case in REASONING_STREAM_CASES], +) +async def test_reasoning_clients_reject_response_without_any_content(case: ReasoningStreamCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream(client, [_stop_chunk(finish_reason="length")]) + + with pytest.raises(EmptyResponseError) as exc_info: + async for _event in client.streaming_response(MESSAGES, {}): + pass + + assert exc_info.value.finish_reason == "length" + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "case", + REASONING_STREAM_CASES, + ids=[case.client_type for case in REASONING_STREAM_CASES], +) +async def test_reasoning_clients_accept_response_with_text_content(case: ReasoningStreamCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _delta_chunk(reasoning_content="Let me think about the memo."), + _delta_chunk(text="Here is the memo."), + _stop_chunk(finish_reason="stop"), + ], + ) + + events = [event async for event in client.streaming_response(MESSAGES, {})] + texts = [item["text"] for event in events for item in event["content_items"] if item["type"] == "text"] + assert texts == ["Here is the memo."] + + +def test_agenthub_error_hierarchy(): + assert issubclass(AgentHubError, ValueError) + assert issubclass(EmptyResponseError, AgentHubError) + assert issubclass(ToolCallArgumentParseError, AgentHubError) diff --git a/src_py/tests/test_reasoning_fidelity.py b/src_py/tests/test_reasoning_fidelity.py new file mode 100644 index 00000000..bae110ef --- /dev/null +++ b/src_py/tests/test_reasoning_fidelity.py @@ -0,0 +1,244 @@ +# Copyright 2025 Prism Shadow. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import inspect +from collections.abc import AsyncIterator +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any + +import pytest + +from agenthub import AutoLLMClient + + +@dataclass +class ReasoningReplayCase: + model: str + client_type: str + + +REASONING_REPLAY_CASES = [ + ReasoningReplayCase(model="gpt-5.5", client_type="openai"), + ReasoningReplayCase(model="glm-5.1", client_type="glm-5.1"), + ReasoningReplayCase(model="kimi-k2.6", client_type="kimi-k2.6"), +] + + +def _create_auto_client(case: ReasoningReplayCase) -> AutoLLMClient: + return AutoLLMClient(model=case.model, api_key="test-key", client_type=case.client_type) + + +async def _stream_from_chunks(chunks: list[object]) -> AsyncIterator[object]: + for chunk in chunks: + yield chunk + + +class _FakeOpenAICompatibleCompletions: + def __init__(self, chunks: list[object]) -> None: + self._chunks = chunks + + async def create(self, **_kwargs: object) -> AsyncIterator[object]: + return _stream_from_chunks(self._chunks) + + +class _FakeOpenAICompatibleClient: + def __init__(self, chunks: list[object]) -> None: + self.base_url = "https://api.test.invalid/v1" + self.chat = SimpleNamespace(completions=_FakeOpenAICompatibleCompletions(chunks)) + + +def _install_fake_openai_compatible_stream(client: AutoLLMClient, chunks: list[object]) -> None: + client._client._client = _FakeOpenAICompatibleClient(chunks) # noqa: SLF001 + + +def _delta_chunk(text: str | None = None, **reasoning_fields: str) -> object: + return SimpleNamespace( + choices=[ + SimpleNamespace( + delta=SimpleNamespace(content=text, tool_calls=None, **reasoning_fields), + finish_reason=None, + ) + ], + usage=None, + ) + + +def _stop_chunk(finish_reason: str = "stop") -> object: + return SimpleNamespace( + choices=[SimpleNamespace(delta=SimpleNamespace(content=None, tool_calls=None), finish_reason=finish_reason)], + usage=SimpleNamespace( + prompt_tokens=1, + completion_tokens=1, + prompt_tokens_details=None, + completion_tokens_details=SimpleNamespace(reasoning_tokens=1), + prompt_cache_hit_tokens=0, + prompt_cache_miss_tokens=1, + ), + ) + + +def _user_message() -> dict[str, Any]: + return {"role": "user", "content_items": [{"type": "text", "text": "Create a memo."}]} + + +async def _transform_history(client: AutoLLMClient, history: list[dict[str, Any]]) -> list[dict[str, Any]]: + model_input = client.transform_uni_message_to_model_input(history) + if inspect.isawaitable(model_input): + model_input = await model_input + + return model_input + + +async def _run_turn_and_replay(client: AutoLLMClient) -> tuple[dict[str, Any], dict[str, Any]]: + """Run one fake streamed turn, then rebuild the request payload from the stored history.""" + async for _event in client.streaming_response_stateful(_user_message(), {}): + pass + + history = client.get_history() + model_input = await _transform_history(client, history) + return history[-1], model_input[-1] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("case", REASONING_REPLAY_CASES, ids=[case.client_type for case in REASONING_REPLAY_CASES]) +async def test_replay_preserves_reasoning_content_field(case: ReasoningReplayCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _delta_chunk(reasoning_content="Let me think"), + _delta_chunk(reasoning_content=" about the memo."), + _delta_chunk(text="Here is the memo."), + _stop_chunk(), + ], + ) + + history_message, replayed_message = await _run_turn_and_replay(client) + thinking_items = [item for item in history_message["content_items"] if item["type"] == "thinking"] + assert thinking_items == [ + { + "type": "thinking", + "thinking": "Let me think about the memo.", + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ] + assert replayed_message["reasoning_content"] == "Let me think about the memo." + assert "reasoning" not in replayed_message + + +@pytest.mark.asyncio +@pytest.mark.parametrize("case", REASONING_REPLAY_CASES, ids=[case.client_type for case in REASONING_REPLAY_CASES]) +async def test_replay_preserves_reasoning_field(case: ReasoningReplayCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _delta_chunk(reasoning="Let me think"), + _delta_chunk(reasoning=" about the memo."), + _delta_chunk(text="Here is the memo."), + _stop_chunk(), + ], + ) + + history_message, replayed_message = await _run_turn_and_replay(client) + thinking_items = [item for item in history_message["content_items"] if item["type"] == "thinking"] + assert thinking_items == [ + { + "type": "thinking", + "thinking": "Let me think about the memo.", + "fidelity": {"reasoning_field": "reasoning"}, + } + ] + assert replayed_message["reasoning"] == "Let me think about the memo." + assert "reasoning_content" not in replayed_message + + +@pytest.mark.asyncio +@pytest.mark.parametrize("case", REASONING_REPLAY_CASES, ids=[case.client_type for case in REASONING_REPLAY_CASES]) +async def test_replay_keeps_both_fields_when_origin_is_ambiguous(case: ReasoningReplayCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _delta_chunk(reasoning_content="Let me think.", reasoning="Let me think."), + _delta_chunk(text="Here is the memo."), + _stop_chunk(), + ], + ) + + history_message, replayed_message = await _run_turn_and_replay(client) + thinking_items = [item for item in history_message["content_items"] if item["type"] == "thinking"] + assert thinking_items == [{"type": "thinking", "thinking": "Let me think."}] + assert replayed_message["reasoning_content"] == "Let me think." + assert replayed_message["reasoning"] == "Let me think." + + +@pytest.mark.asyncio +@pytest.mark.parametrize("case", REASONING_REPLAY_CASES, ids=[case.client_type for case in REASONING_REPLAY_CASES]) +async def test_replay_of_thinking_without_fidelity_sends_both_fields(case: ReasoningReplayCase): + client = _create_auto_client(case) + history = [ + _user_message(), + { + "role": "assistant", + "content_items": [ + {"type": "thinking", "thinking": "Let me think."}, + {"type": "text", "text": "Here is the memo."}, + ], + }, + ] + + model_input = await _transform_history(client, history) + replayed_message = model_input[-1] + assert replayed_message["reasoning_content"] == "Let me think." + assert replayed_message["reasoning"] == "Let me think." + + +def _text_delta_event(text: str, phase: str | None = None) -> dict[str, Any]: + item: dict[str, Any] = {"type": "text", "text": text} + if phase is not None: + item["fidelity"] = {"phase": phase} + + return { + "role": "assistant", + "event_type": "delta", + "content_items": [item], + "usage_metadata": None, + "finish_reason": None, + } + + +def test_concat_splits_text_items_only_on_phase_change(): + client = _create_auto_client(REASONING_REPLAY_CASES[0]) + message = client.concat_uni_events_to_uni_message( + [ + _text_delta_event("", phase="commentary"), + _text_delta_event("I'll inspect the logs."), + _text_delta_event("", phase="final_answer"), + _text_delta_event("Root cause:"), + _text_delta_event(" cache invalidation race."), + _text_delta_event("", phase="final_answer"), + _text_delta_event(" Remediation follows."), + ] + ) + + assert message["content_items"] == [ + {"type": "text", "text": "I'll inspect the logs.", "fidelity": {"phase": "commentary"}}, + { + "type": "text", + "text": "Root cause: cache invalidation race. Remediation follows.", + "fidelity": {"phase": "final_answer"}, + }, + ] diff --git a/src_py/tests/test_tool_call_arguments.py b/src_py/tests/test_tool_call_arguments.py new file mode 100644 index 00000000..e329a8ef --- /dev/null +++ b/src_py/tests/test_tool_call_arguments.py @@ -0,0 +1,213 @@ +# Copyright 2025 Prism Shadow. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from collections.abc import AsyncIterator +from dataclasses import dataclass +from types import SimpleNamespace + +import pytest + +from agenthub import AutoLLMClient, ToolCallArgumentParseError + + +@dataclass +class OpenAICompatibleToolStreamCase: + expected_client: str + model: str + client_type: str + + +OPENAI_COMPATIBLE_TOOL_STREAM_CASES = [ + OpenAICompatibleToolStreamCase( + expected_client="OpenaiClient", + model="gpt-5.5", + client_type="openai", + ), + OpenAICompatibleToolStreamCase( + expected_client="GLM5_1Client", + model="glm-5.1", + client_type="glm-5.1", + ), + OpenAICompatibleToolStreamCase( + expected_client="KimiK2_6Client", + model="kimi-k2.6", + client_type="kimi-k2.6", + ), + OpenAICompatibleToolStreamCase( + expected_client="DeepSeekV4Client", + model="deepseek-v4", + client_type="deepseek-v4", + ), +] + + +def _create_auto_client(case: OpenAICompatibleToolStreamCase) -> AutoLLMClient: + return AutoLLMClient(model=case.model, api_key="test-key", client_type=case.client_type) + + +async def _stream_from_chunks(chunks: list[object]) -> AsyncIterator[object]: + for chunk in chunks: + yield chunk + + +class _FakeOpenAICompatibleCompletions: + def __init__(self, chunks: list[object]) -> None: + self._chunks = chunks + + async def create(self, **_kwargs: object) -> AsyncIterator[object]: + return _stream_from_chunks(self._chunks) + + +class _FakeOpenAICompatibleClient: + def __init__(self, chunks: list[object]) -> None: + self.base_url = "https://api.test.invalid/v1" + self.chat = SimpleNamespace(completions=_FakeOpenAICompatibleCompletions(chunks)) + + +def _install_fake_openai_compatible_stream(client: AutoLLMClient, chunks: list[object]) -> None: + client._client._client = _FakeOpenAICompatibleClient(chunks) # noqa: SLF001 + + +def _tool_delta_chunk(tool_call_id: str, name: str, arguments: str) -> object: + return SimpleNamespace( + choices=[ + SimpleNamespace( + delta=SimpleNamespace( + content=None, + tool_calls=[ + SimpleNamespace( + id=tool_call_id, + function=SimpleNamespace(name=name, arguments=arguments), + ) + ], + ), + finish_reason=None, + ) + ], + usage=None, + ) + + +def _tool_stop_chunk() -> object: + return SimpleNamespace( + choices=[SimpleNamespace(delta=SimpleNamespace(content=None, tool_calls=None), finish_reason="tool_calls")], + usage=SimpleNamespace( + prompt_tokens=1, + completion_tokens=1, + prompt_tokens_details=None, + completion_tokens_details=SimpleNamespace(reasoning_tokens=0), + prompt_cache_hit_tokens=0, + prompt_cache_miss_tokens=1, + ), + ) + + +async def _capture_tool_argument_error(stream: AsyncIterator[object]) -> ToolCallArgumentParseError: + with pytest.raises(ToolCallArgumentParseError) as exc_info: + async for _event in stream: + pass + return exc_info.value + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "case", + OPENAI_COMPATIBLE_TOOL_STREAM_CASES, + ids=[case.client_type for case in OPENAI_COMPATIBLE_TOOL_STREAM_CASES], +) +async def test_openai_compatible_clients_combine_streamed_tool_call_arguments( + case: OpenAICompatibleToolStreamCase, +): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _tool_delta_chunk("call_ok", "exec_command", '{"cmd":'), + _tool_delta_chunk("", "", '"echo ok"}'), + _tool_stop_chunk(), + ], + ) + + messages = [{"role": "user", "content_items": [{"type": "text", "text": "Create a memo."}]}] + events = [event async for event in client.streaming_response(messages, {})] + tool_calls = [item for event in events for item in event["content_items"] if item["type"] == "tool_call"] + + assert tool_calls == [ + { + "type": "tool_call", + "name": "exec_command", + "arguments": {"cmd": "echo ok"}, + "tool_call_id": "call_ok", + } + ] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "case", + OPENAI_COMPATIBLE_TOOL_STREAM_CASES, + ids=[case.client_type for case in OPENAI_COMPATIBLE_TOOL_STREAM_CASES], +) +async def test_openai_compatible_clients_report_malformed_streamed_tool_call_arguments( + case: OpenAICompatibleToolStreamCase, +): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _tool_delta_chunk("call_bad", "exec_command", '{"cmd":"python create_docx.py'), + _tool_stop_chunk(), + ], + ) + + messages = [{"role": "user", "content_items": [{"type": "text", "text": "Create a memo."}]}] + parse_error = await _capture_tool_argument_error(client.streaming_response(messages, {})) + assert parse_error.client == case.expected_client + assert parse_error.tool_name == "exec_command" + assert parse_error.tool_call_id == "call_bad" + assert parse_error.raw_arguments_length > 0 + assert "create_docx.py" in parse_error.raw_arguments_preview + message = str(parse_error) + assert "exec_command" in message + assert "call_bad" in message + assert "length=" in message + assert "Unterminated string" in message + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "case", + OPENAI_COMPATIBLE_TOOL_STREAM_CASES, + ids=[case.client_type for case in OPENAI_COMPATIBLE_TOOL_STREAM_CASES], +) +async def test_openai_compatible_clients_report_non_object_streamed_tool_call_arguments( + case: OpenAICompatibleToolStreamCase, +): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _tool_delta_chunk("call_array", "exec_command", "[]"), + _tool_stop_chunk(), + ], + ) + + messages = [{"role": "user", "content_items": [{"type": "text", "text": "Create a memo."}]}] + parse_error = await _capture_tool_argument_error(client.streaming_response(messages, {})) + assert parse_error.client == case.expected_client + assert parse_error.tool_name == "exec_command" + assert parse_error.tool_call_id == "call_array" + assert parse_error.raw_arguments_length == 2 + assert parse_error.raw_arguments_preview == "[]" + assert "Expected a JSON object." in str(parse_error) diff --git a/src_ts/package-lock.json b/src_ts/package-lock.json index 00c57cb2..9107deaf 100644 --- a/src_ts/package-lock.json +++ b/src_ts/package-lock.json @@ -1,12 +1,12 @@ { "name": "@prismshadow/agenthub", - "version": "0.3.3", + "version": "0.4.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@prismshadow/agenthub", - "version": "0.3.3", + "version": "0.4.0", "license": "Apache-2.0", "dependencies": { "@anthropic-ai/bedrock-sdk": "^0.26.4", diff --git a/src_ts/package.json b/src_ts/package.json index 71812be1..92608086 100644 --- a/src_ts/package.json +++ b/src_ts/package.json @@ -1,6 +1,6 @@ { "name": "@prismshadow/agenthub", - "version": "0.3.3", + "version": "0.4.0", "description": "AgentHub is the LLM API Hub for the Agent era, built for high-precision autonomous agents.", "main": "dist/index.js", "types": "dist/index.d.ts", @@ -48,7 +48,7 @@ }, "repository": { "type": "git", - "url": "git+https://github.com/Prism-Shadow/AgentHub.git" + "url": "git+https://github.com/Prism-Shadow/agenthub.git" }, "keywords": [ "agent", @@ -60,7 +60,7 @@ "author": "PrismShadow", "license": "Apache-2.0", "bugs": { - "url": "https://github.com/Prism-Shadow/AgentHub/issues" + "url": "https://github.com/Prism-Shadow/agenthub/issues" }, - "homepage": "https://github.com/Prism-Shadow/AgentHub#readme" + "homepage": "https://github.com/Prism-Shadow/agenthub#readme" } diff --git a/src_ts/src/autoClient.ts b/src_ts/src/autoClient.ts index 5c506751..de319408 100644 --- a/src_ts/src/autoClient.ts +++ b/src_ts/src/autoClient.ts @@ -15,7 +15,7 @@ import { LLMClient } from "./baseClient"; import { Gemini3Client } from "./gemini3"; import { Claude4_6Client } from "./claude4_6"; -import { Claude4_8Client } from "./claude4_8"; +import { Claude5Client } from "./claude5"; import { GPT5_5Client } from "./gpt5_5"; import { GLM5_1Client } from "./glm5_1"; import { KimiK2_6Client } from "./kimi_k2_6"; @@ -82,9 +82,11 @@ export class AutoLLMClient extends LLMClient { return new Gemini3Client({ model, apiKey, baseUrl }); } else if ( clientType.includes("claude") && - (clientType.includes("4-7") || clientType.includes("4-8")) + (clientType.includes("4-7") || + clientType.includes("4-8") || + clientType.includes("-5")) ) { - return new Claude4_8Client({ model, apiKey, baseUrl }); + return new Claude5Client({ model, apiKey, baseUrl }); } else if (clientType.includes("claude") && clientType.includes("4-6")) { return new Claude4_6Client({ model, apiKey, baseUrl }); } else if ( @@ -106,12 +108,15 @@ export class AutoLLMClient extends LLMClient { clientType.includes("embedding") ) { return new OpenaiEmbeddingClient({ model, apiKey, baseUrl }); - } else if (clientType.includes("openai")) { + } else if ( + clientType.includes("openai") && + !clientType.includes("embedding") + ) { return new OpenaiClient({ model, apiKey, baseUrl }); } else { throw new Error( `${clientType} is not supported. ` + - "Supported client types: gemini-3, claude-4-8, claude-4-7, claude-4-6, gpt-5.5, gpt-5.4, glm-5.1, kimi-k2.6, kimi-k2.5, deepseek-v4, openai-embedding, openai.", + "Supported client types: gemini-3, claude-5, claude-4-8, claude-4-7, claude-4-6, gpt-5.5, gpt-5.4, glm-5.1, kimi-k2.6, kimi-k2.5, deepseek-v4, openai-embedding, openai.", ); } } diff --git a/src_ts/src/baseClient.ts b/src_ts/src/baseClient.ts index 4e97a77e..04eb70b4 100644 --- a/src_ts/src/baseClient.ts +++ b/src_ts/src/baseClient.ts @@ -12,7 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. +import { EmptyResponseError } from "./errors"; import { + Fidelity, FinishReason, ContentItem, UniConfig, @@ -21,6 +23,22 @@ import { UsageMetadata, } from "./types"; +/** + * Whether a content item carries a non-empty fidelity payload. + */ +function hasFidelity(fidelity?: Fidelity): boolean { + return fidelity != null && Object.keys(fidelity).length > 0; +} + +/** + * Compare two fidelity payloads by value. Fidelity dicts are built with a + * stable key order by each client, so JSON serialization is a faithful + * equality check. + */ +function fidelityEquals(a?: Fidelity, b?: Fidelity): boolean { + return JSON.stringify(a ?? {}) === JSON.stringify(b ?? {}); +} + /** * Abstract base class for LLM clients. * @@ -84,32 +102,42 @@ export abstract class LLMClient { for (const item of event.content_items) { if (item.type === "text") { const lastItem = contentItems[contentItems.length - 1]; + const itemFidelity = item.fidelity ?? {}; + // a delta announcing a different phase starts a new item; same-phase and + // phaseless deltas merge until a signature finishes the item if ( lastItem && lastItem.type === "text" && - lastItem.signature == null && // no signature yet - item.phase == null // no new phase + lastItem.fidelity?.signature == null && // not finished by a signature yet + (itemFidelity.phase == null || // phaseless deltas continue the item + itemFidelity.phase === lastItem.fidelity?.phase) // same phase merges ) { lastItem.text += item.text; - if (item.signature) { - lastItem.signature = item.signature; + if (hasFidelity(item.fidelity)) { + // a signature finishes the current item + lastItem.fidelity = { ...lastItem.fidelity, ...item.fidelity }; } - } else if (item.text || item.phase != null) { + } else if (item.text || itemFidelity.phase != null) { // text or new phase starts an item contentItems.push({ ...item }); } } else if (item.type === "thinking") { const lastItem = contentItems[contentItems.length - 1]; + // a new item starts only when the open item's fidelity is non-empty and + // differs from the incoming delta's; everything else merges into it if ( lastItem && lastItem.type === "thinking" && - lastItem.signature == null + (!hasFidelity(lastItem.fidelity) || // not finished by fidelity yet + // a run of equal fidelity is one item + fidelityEquals(lastItem.fidelity, item.fidelity)) ) { lastItem.thinking += item.thinking; - if (item.signature) { - lastItem.signature = item.signature; + if (hasFidelity(item.fidelity)) { + // fidelity finishes the current item + lastItem.fidelity = item.fidelity; } - } else if (item.thinking || item.signature) { + } else if (item.thinking || hasFidelity(item.fidelity)) { contentItems.push({ ...item }); } } else if (item.type === "partial_tool_call") { @@ -181,6 +209,7 @@ export abstract class LLMClient { yield event; } LLMClient._validateLastEvent(lastEvent); + this._validateNonThinkingOutput(events); // Save history to file if trace_id is specified if (config.trace_id && events.length > 0) { @@ -259,6 +288,31 @@ export abstract class LLMClient { } } + /** + * Validate that the completed response carries content other than thinking. + * + * Replaying a thinking-only assistant message on the next turn fails with a 400 + * error, so the response is rejected as soon as the stream completes. + * + * @param events - All events yielded by streamingResponse + * @throws EmptyResponseError if every content item in the response is thinking + */ + protected _validateNonThinkingOutput(events: UniEvent[]): void { + const thinkingOnly = events.every((event) => + event.content_items.every( + (item) => item.type === "thinking" || item.type === "inline_thinking", + ), + ); + if (thinkingOnly) { + const finishReason = + events.length > 0 ? events[events.length - 1].finish_reason : null; + throw new EmptyResponseError({ + client: this.constructor.name, + finishReason, + }); + } + } + /** * Clear the message history. */ diff --git a/src_ts/src/claude4_6/client.ts b/src_ts/src/claude4_6/client.ts index 71cd9432..97ddf7f5 100644 --- a/src_ts/src/claude4_6/client.ts +++ b/src_ts/src/claude4_6/client.ts @@ -20,6 +20,7 @@ import { } from "@anthropic-ai/sdk/resources/beta/messages"; import { Stream } from "@anthropic-ai/sdk/core/streaming"; import { LLMClient } from "../baseClient"; +import { parseToolCallArguments } from "../errors"; import { EventType, FinishReason, @@ -268,13 +269,13 @@ export class Claude4_6Client extends LLMClient { if (item.thinking === REDACTED_THINKING) { contentBlocks.push({ type: "redacted_thinking", - data: item.signature, + data: item.fidelity?.signature, }); } else { contentBlocks.push({ type: "thinking", thinking: item.thinking, - signature: item.signature, + signature: item.fidelity?.signature, }); } } else if (item.type === "tool_call") { @@ -345,7 +346,7 @@ export class Claude4_6Client extends LLMClient { contentItems.push({ type: "thinking", thinking: REDACTED_THINKING, - signature: block.data, + fidelity: { signature: block.data }, }); } } else if (claudeEventType === "content_block_delta") { @@ -366,7 +367,7 @@ export class Claude4_6Client extends LLMClient { contentItems.push({ type: "thinking", thinking: "", - signature: delta.signature, + fidelity: { signature: delta.signature }, }); } } else if (claudeEventType === "content_block_stop") { @@ -527,7 +528,12 @@ export class Claude4_6Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], diff --git a/src_ts/src/claude4_8/client.ts b/src_ts/src/claude5/client.ts similarity index 96% rename from src_ts/src/claude4_8/client.ts rename to src_ts/src/claude5/client.ts index a94738e4..af32daa5 100644 --- a/src_ts/src/claude4_8/client.ts +++ b/src_ts/src/claude5/client.ts @@ -20,6 +20,7 @@ import { } from "@anthropic-ai/sdk/resources/beta/messages"; import { Stream } from "@anthropic-ai/sdk/core/streaming"; import { LLMClient } from "../baseClient"; +import { parseToolCallArguments } from "../errors"; import { EventType, FinishReason, @@ -36,15 +37,15 @@ import { const REDACTED_THINKING = "_REDACTED_THINKING"; /** - * Claude 4.8-specific LLM client implementation. + * Claude 5-specific LLM client implementation. */ -export class Claude4_8Client extends LLMClient { +export class Claude5Client extends LLMClient { protected _model: string; private _client: Anthropic | AnthropicBedrock; private _use_bedrock: boolean; /** - * Initialize Claude 4.8 client with model and API key. + * Initialize Claude 5 client with model and API key. */ constructor(options: { model: string; @@ -270,13 +271,13 @@ export class Claude4_8Client extends LLMClient { if (item.thinking === REDACTED_THINKING) { contentBlocks.push({ type: "redacted_thinking", - data: item.signature, + data: item.fidelity?.signature, }); } else { contentBlocks.push({ type: "thinking", thinking: item.thinking, - signature: item.signature, + signature: item.fidelity?.signature, }); } } else if (item.type === "tool_call") { @@ -347,7 +348,7 @@ export class Claude4_8Client extends LLMClient { contentItems.push({ type: "thinking", thinking: REDACTED_THINKING, - signature: block.data, + fidelity: { signature: block.data }, }); } } else if (claudeEventType === "content_block_delta") { @@ -368,7 +369,7 @@ export class Claude4_8Client extends LLMClient { contentItems.push({ type: "thinking", thinking: "", - signature: delta.signature, + fidelity: { signature: delta.signature }, }); } } else if (claudeEventType === "content_block_stop") { @@ -529,7 +530,12 @@ export class Claude4_8Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], diff --git a/src_ts/src/claude4_8/index.ts b/src_ts/src/claude5/index.ts similarity index 93% rename from src_ts/src/claude4_8/index.ts rename to src_ts/src/claude5/index.ts index 9eb1ff3d..e19f2fd8 100644 --- a/src_ts/src/claude4_8/index.ts +++ b/src_ts/src/claude5/index.ts @@ -12,4 +12,4 @@ // See the License for the specific language governing permissions and // limitations under the License. -export { Claude4_8Client } from "./client"; +export { Claude5Client } from "./client"; diff --git a/src_ts/src/deepseek_v4/client.ts b/src_ts/src/deepseek_v4/client.ts index 26bce32a..da3fa343 100644 --- a/src_ts/src/deepseek_v4/client.ts +++ b/src_ts/src/deepseek_v4/client.ts @@ -19,6 +19,7 @@ import type { ChatCompletionCreateParamsStreaming, } from "openai/resources/chat/completions"; import { LLMClient } from "../baseClient"; +import { parseToolCallArguments } from "../errors"; import { EventType, FinishReason, @@ -229,10 +230,13 @@ export class DeepSeekV4Client extends LLMClient { // eslint-disable-next-line @typescript-eslint/no-explicit-any if ((delta as any)?.reasoning_content) { eventType = "delta"; + // record the wire field so a replay through another OpenAI-compatible + // client reproduces the exact field DeepSeek produced contentItems.push({ type: "thinking", // eslint-disable-next-line @typescript-eslint/no-explicit-any thinking: (delta as any).reasoning_content, + fidelity: { reasoning_field: "reasoning_content" }, }); } @@ -358,7 +362,12 @@ export class DeepSeekV4Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments || "{}"), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], @@ -384,7 +393,12 @@ export class DeepSeekV4Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments || "{}"), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], diff --git a/src_ts/src/errors.ts b/src_ts/src/errors.ts new file mode 100644 index 00000000..93045e9f --- /dev/null +++ b/src_ts/src/errors.ts @@ -0,0 +1,114 @@ +// Copyright 2025 Prism Shadow. and/or its affiliates +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +function previewToolCallArguments(raw: string): string { + const maxLength = 160; + if (raw.length <= maxLength) { + return raw; + } + const edgeLength = 72; + return `${raw.slice(0, edgeLength)}...[truncated]...${raw.slice(-edgeLength)}`; +} + +export class AgentHubError extends Error { + constructor(message: string) { + super(message); + this.name = "AgentHubError"; + } +} + +/** + * Raised when a completed response carries no non-thinking content and no tool calls. + * + * Models occasionally finish a turn with thinking output only (reasoning models in + * particular); replaying such an assistant message on the next turn fails with a 400 + * error, so the response is rejected as soon as the stream completes. + */ +export class EmptyResponseError extends AgentHubError { + readonly client: string; + readonly finishReason: string | null; + + constructor(args: { client: string; finishReason: string | null }) { + super( + `${args.client} returned no content other than thinking ` + + `(finish_reason=${JSON.stringify(args.finishReason)}).`, + ); + this.name = "EmptyResponseError"; + this.client = args.client; + this.finishReason = args.finishReason; + } +} + +export class ToolCallArgumentParseError extends AgentHubError { + readonly client: string; + readonly toolName: string; + readonly toolCallId: string; + readonly rawArgumentsLength: number; + readonly rawArgumentsPreview: string; + + constructor(args: { + client: string; + toolName: string; + toolCallId: string; + rawArguments: string; + reason: string; + }) { + const preview = previewToolCallArguments(args.rawArguments); + super( + `Invalid streamed tool call arguments from ${args.client} for tool "${args.toolName}" ` + + `(tool_call_id="${args.toolCallId}", length=${args.rawArguments.length}, ` + + `preview=${JSON.stringify(preview)}): ${args.reason}`, + ); + this.name = "ToolCallArgumentParseError"; + this.client = args.client; + this.toolName = args.toolName; + this.toolCallId = args.toolCallId; + this.rawArgumentsLength = args.rawArguments.length; + this.rawArgumentsPreview = preview; + } +} + +export function parseToolCallArguments( + rawArguments: string | undefined, + client: string, + toolName: string, + toolCallId: string, +): Record { + const raw = rawArguments || "{}"; + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (error) { + const reason = error instanceof Error ? error.message : String(error); + throw new ToolCallArgumentParseError({ + client, + toolName, + toolCallId, + rawArguments: raw, + reason, + }); + } + + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) { + throw new ToolCallArgumentParseError({ + client, + toolName, + toolCallId, + rawArguments: raw, + reason: "Expected a JSON object.", + }); + } + + return parsed as Record; +} diff --git a/src_ts/src/gemini3/client.ts b/src_ts/src/gemini3/client.ts index 5c2f2111..9be71ef9 100644 --- a/src_ts/src/gemini3/client.ts +++ b/src_ts/src/gemini3/client.ts @@ -40,6 +40,7 @@ import * as path from "path"; import { LLMClient } from "../baseClient"; import { EventType, + Fidelity, FinishReason, PartialContentItem, PromptCaching, @@ -51,6 +52,25 @@ import { UsageMetadata, } from "../types"; +/** + * Wrap a part's thought signature as a fidelity payload, or nothing when absent. + */ +function partFidelity(part: Part): { fidelity?: Fidelity } { + if (part.thoughtSignature == null) { + return {}; + } + return { fidelity: { signature: part.thoughtSignature } }; +} + +/** + * Read the thought signature recorded in an item's fidelity payload. + */ +function itemThoughtSignature(item: { + fidelity?: Fidelity; +}): string | undefined { + return item.fidelity?.signature; +} + /** * Gemini 3-specific LLM client implementation. */ @@ -312,7 +332,7 @@ export class Gemini3Client extends LLMClient { if (item.type === "text") { parts.push({ text: item.text, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "image_url") { const urlValue = item.image_url; @@ -332,13 +352,13 @@ export class Gemini3Client extends LLMClient { mimeType: item.mime_type, data: item.data.toString("base64"), }, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "thinking") { parts.push({ text: item.thinking, thought: true, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "inline_thinking") { parts.push({ @@ -347,7 +367,7 @@ export class Gemini3Client extends LLMClient { data: item.data.toString("base64"), }, thought: true, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "tool_call") { const functionCall: FunctionCall = { @@ -356,7 +376,7 @@ export class Gemini3Client extends LLMClient { }; parts.push({ functionCall: functionCall, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "tool_result") { if (!item.tool_call_id) { @@ -426,21 +446,21 @@ export class Gemini3Client extends LLMClient { name: part.functionCall.name || "", arguments: part.functionCall.args || {}, tool_call_id: part.functionCall.name || "", - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } else if (part.thought) { if (part.text !== undefined) { contentItems.push({ type: "thinking", thinking: part.text, - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } else if (part.inlineData) { contentItems.push({ type: "inline_thinking", data: Buffer.from(part.inlineData.data || "", "base64"), mime_type: part.inlineData.mimeType || "application/octet-stream", - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } } else if (part.inlineData) { @@ -448,13 +468,13 @@ export class Gemini3Client extends LLMClient { type: "inline_data", data: Buffer.from(part.inlineData.data || "", "base64"), mime_type: part.inlineData.mimeType || "application/octet-stream", - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } else if (part.text !== undefined) { contentItems.push({ type: "text", text: part.text, - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } else { throw new Error(`Unknown output: ${JSON.stringify(part)}`); @@ -596,7 +616,7 @@ export class Gemini3Client extends LLMClient { name: item.name, arguments: JSON.stringify(item.arguments), tool_call_id: item.tool_call_id, - signature: item.signature, + fidelity: item.fidelity, }, ], usage_metadata: null, diff --git a/src_ts/src/glm5_1/client.ts b/src_ts/src/glm5_1/client.ts index c0406eab..a38188bb 100644 --- a/src_ts/src/glm5_1/client.ts +++ b/src_ts/src/glm5_1/client.ts @@ -19,6 +19,7 @@ import type { ChatCompletionCreateParamsStreaming, } from "openai/resources/chat/completions"; import { LLMClient } from "../baseClient"; +import { parseToolCallArguments } from "../errors"; import { EventType, FinishReason, @@ -158,6 +159,7 @@ export class GLM5_1Client extends LLMClient { // eslint-disable-next-line @typescript-eslint/no-explicit-any const toolCalls: any[] = []; let thinking = ""; + const thinkingFields = new Set(); for (const item of msg.content_items) { if (item.type === "text") { @@ -166,6 +168,7 @@ export class GLM5_1Client extends LLMClient { throw new Error("GLM-5 does not support image inputs."); } else if (item.type === "thinking") { thinking += item.thinking; + thinkingFields.add(item.fidelity?.reasoning_field); } else if (item.type === "tool_call") { toolCalls.push({ id: item.tool_call_id, @@ -207,8 +210,22 @@ export class GLM5_1Client extends LLMClient { } if (thinking) { - message.reasoning_content = thinking; - message.reasoning = thinking; + // send thinking back through the exact field the upstream produced (recorded + // in the item fidelity); servers may reject the spelling they did not emit + if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning_content") + ) { + message.reasoning_content = thinking; + } else if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning") + ) { + message.reasoning = thinking; + } else { + message.reasoning_content = thinking; // vLLM & siliconflow compatibility + message.reasoning = thinking; // openrouter compatibility + } } if (Object.keys(message).length > 1) { @@ -237,24 +254,31 @@ export class GLM5_1Client extends LLMClient { contentItems.push({ type: "text", text: delta.content }); } - // vLLM & siliconflow compatibility + // the thinking field name differs by server: vLLM & siliconflow use + // reasoning_content while openrouter uses reasoning; record the wire + // field that carried each delta so a replay can reproduce exactly the + // field the upstream produced + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const reasoningContent = (delta as any)?.reasoning_content; // eslint-disable-next-line @typescript-eslint/no-explicit-any - if ((delta as any)?.reasoning_content) { + const reasoning = (delta as any)?.reasoning; + if (reasoningContent && reasoning) { + eventType = "delta"; + // ambiguous origin: record no fidelity so a replay sends both fields back + contentItems.push({ type: "thinking", thinking: reasoningContent }); + } else if (reasoningContent) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning_content, + thinking: reasoningContent, + fidelity: { reasoning_field: "reasoning_content" }, }); - } - // openrouter compatibility - // eslint-disable-next-line @typescript-eslint/no-explicit-any - else if ((delta as any)?.reasoning) { + } else if (reasoning) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning, + thinking: reasoning, + fidelity: { reasoning_field: "reasoning" }, }); } @@ -385,7 +409,12 @@ export class GLM5_1Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments || "{}"), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], @@ -414,7 +443,12 @@ export class GLM5_1Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments || "{}"), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], diff --git a/src_ts/src/gpt5_5/client.ts b/src_ts/src/gpt5_5/client.ts index dea53c10..64bd9e13 100644 --- a/src_ts/src/gpt5_5/client.ts +++ b/src_ts/src/gpt5_5/client.ts @@ -19,6 +19,7 @@ import type { ResponseCreateParamsStreaming, } from "openai/resources/responses/responses"; import { LLMClient } from "../baseClient"; +import { parseToolCallArguments } from "../errors"; import { EventType, FinishReason, @@ -155,9 +156,14 @@ export class GPT5_5Client extends LLMClient { for (const item of msg.content_items) { if (item.type === "text") { - if (msg.role === "assistant" && item.phase) { + const phase = item.fidelity?.phase; + if (msg.role === "assistant" && phase) { // split different phases - if (lastPhase !== null && contentItems.length > 0) { + if ( + lastPhase !== null && + lastPhase !== phase && + contentItems.length > 0 + ) { inputList.push({ role: msg.role, content: contentItems, @@ -165,7 +171,7 @@ export class GPT5_5Client extends LLMClient { }); contentItems = []; } - lastPhase = item.phase; + lastPhase = phase; } if (msg.role === "user") { contentItems.push({ type: "input_text", text: item.text }); @@ -178,15 +184,14 @@ export class GPT5_5Client extends LLMClient { image_url: item.image_url, }); } else if (item.type === "thinking") { - const signatureStr = item.signature || "{}"; - const signature = JSON.parse(signatureStr); + const fidelity = item.fidelity ?? {}; inputList.push({ type: "reasoning", - id: signature.id, + id: fidelity.id, summary: item.thinking ? [{ type: "summary_text", text: item.thinking }] : [], - encrypted_content: signature.encrypted_content, + encrypted_content: fidelity.encrypted_content, }); } else if (item.type === "tool_call") { inputList.push({ @@ -262,21 +267,21 @@ export class GPT5_5Client extends LLMClient { // adding the following thinking item leads to 400 invalid request error, why? // } else if (item.type === "reasoning") { // eventType = "delta"; - // const signature = { + // const fidelity = { // id: item.id, // encrypted_content: item.encrypted_content, // }; // contentItems.push({ // type: "thinking", // thinking: "", - // signature: JSON.stringify(signature), + // fidelity, // }); } else if (item.type === "message") { // eslint-disable-next-line @typescript-eslint/no-explicit-any const phase = (item as any).phase as string | undefined; if (phase != null) { eventType = "delta"; - contentItems.push({ type: "text", text: "", phase }); + contentItems.push({ type: "text", text: "", fidelity: { phase } }); } else { eventType = "unused"; } @@ -288,14 +293,14 @@ export class GPT5_5Client extends LLMClient { const item = modelOutput.item; if (item.type === "reasoning") { eventType = "delta"; - const signature = { + const fidelity = { id: item.id, encrypted_content: item.encrypted_content, }; contentItems.push({ type: "thinking", thinking: "", - signature: JSON.stringify(signature), + fidelity, }); } else { eventType = "unused"; @@ -420,7 +425,12 @@ export class GPT5_5Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], diff --git a/src_ts/src/index.ts b/src_ts/src/index.ts index f871eb27..30487697 100644 --- a/src_ts/src/index.ts +++ b/src_ts/src/index.ts @@ -13,4 +13,9 @@ // limitations under the License. export { AutoLLMClient } from "./autoClient"; +export { + AgentHubError, + EmptyResponseError, + ToolCallArgumentParseError, +} from "./errors"; export * from "./types"; diff --git a/src_ts/src/kimi_k2_6/client.ts b/src_ts/src/kimi_k2_6/client.ts index 766b58a9..e565ff3f 100644 --- a/src_ts/src/kimi_k2_6/client.ts +++ b/src_ts/src/kimi_k2_6/client.ts @@ -20,6 +20,7 @@ import type { ChatCompletionCreateParamsStreaming, } from "openai/resources/chat/completions"; import { LLMClient } from "../baseClient"; +import { parseToolCallArguments } from "../errors"; import { EventType, FinishReason, @@ -206,6 +207,7 @@ export class KimiK2_6Client extends LLMClient { // eslint-disable-next-line @typescript-eslint/no-explicit-any const toolCalls: any[] = []; let thinking = ""; + const thinkingFields = new Set(); for (const item of msg.content_items) { if (item.type === "text") { @@ -221,6 +223,7 @@ export class KimiK2_6Client extends LLMClient { }); } else if (item.type === "thinking") { thinking += item.thinking; + thinkingFields.add(item.fidelity?.reasoning_field); } else if (item.type === "tool_call") { toolCalls.push({ id: item.tool_call_id, @@ -282,8 +285,22 @@ export class KimiK2_6Client extends LLMClient { } if (thinking) { - message.reasoning_content = thinking; - message.reasoning = thinking; + // send thinking back through the exact field the upstream produced (recorded + // in the item fidelity); servers may reject the spelling they did not emit + if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning_content") + ) { + message.reasoning_content = thinking; + } else if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning") + ) { + message.reasoning = thinking; + } else { + message.reasoning_content = thinking; // vLLM & siliconflow compatibility + message.reasoning = thinking; // openrouter compatibility + } } if (Object.keys(message).length > 1) { @@ -312,24 +329,31 @@ export class KimiK2_6Client extends LLMClient { contentItems.push({ type: "text", text: delta.content }); } - // vLLM & siliconflow compatibility + // the thinking field name differs by server: vLLM & siliconflow use + // reasoning_content while openrouter uses reasoning; record the wire + // field that carried each delta so a replay can reproduce exactly the + // field the upstream produced // eslint-disable-next-line @typescript-eslint/no-explicit-any - if ((delta as any)?.reasoning_content) { + const reasoningContent = (delta as any)?.reasoning_content; + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const reasoning = (delta as any)?.reasoning; + if (reasoningContent && reasoning) { + eventType = "delta"; + // ambiguous origin: record no fidelity so a replay sends both fields back + contentItems.push({ type: "thinking", thinking: reasoningContent }); + } else if (reasoningContent) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning_content, + thinking: reasoningContent, + fidelity: { reasoning_field: "reasoning_content" }, }); - } - // openrouter compatibility - // eslint-disable-next-line @typescript-eslint/no-explicit-any - else if ((delta as any)?.reasoning) { + } else if (reasoning) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning, + thinking: reasoning, + fidelity: { reasoning_field: "reasoning" }, }); } @@ -460,7 +484,12 @@ export class KimiK2_6Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments || "{}"), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], @@ -489,7 +518,12 @@ export class KimiK2_6Client extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments || "{}"), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id || "", }, ], diff --git a/src_ts/src/openai/client.ts b/src_ts/src/openai/client.ts index a3a688f5..1011f0ec 100644 --- a/src_ts/src/openai/client.ts +++ b/src_ts/src/openai/client.ts @@ -20,6 +20,7 @@ import type { ChatCompletionCreateParamsStreaming, } from "openai/resources/chat/completions"; import { LLMClient } from "../baseClient"; +import { parseToolCallArguments } from "../errors"; import { EventType, FinishReason, @@ -178,6 +179,7 @@ export class OpenaiClient extends LLMClient { // eslint-disable-next-line @typescript-eslint/no-explicit-any const toolCalls: any[] = []; let thinking = ""; + const thinkingFields = new Set(); for (const item of msg.content_items) { if (item.type === "text") { @@ -193,6 +195,7 @@ export class OpenaiClient extends LLMClient { }); } else if (item.type === "thinking") { thinking += item.thinking; + thinkingFields.add(item.fidelity?.reasoning_field); } else if (item.type === "tool_call") { toolCalls.push({ id: item.tool_call_id, @@ -253,8 +256,22 @@ export class OpenaiClient extends LLMClient { } if (thinking) { - message.reasoning_content = thinking; - message.reasoning = thinking; + // send thinking back through the exact field the upstream produced (recorded + // in the item fidelity); servers may reject the spelling they did not emit + if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning_content") + ) { + message.reasoning_content = thinking; + } else if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning") + ) { + message.reasoning = thinking; + } else { + message.reasoning_content = thinking; // vLLM & siliconflow compatibility + message.reasoning = thinking; // openrouter compatibility + } } if (Object.keys(message).length > 1) { @@ -283,24 +300,31 @@ export class OpenaiClient extends LLMClient { contentItems.push({ type: "text", text: delta.content }); } - // vLLM & siliconflow compatibility + // the thinking field name differs by server: vLLM & siliconflow use + // reasoning_content while openrouter uses reasoning; record the wire + // field that carried each delta so a replay can reproduce exactly the + // field the upstream produced // eslint-disable-next-line @typescript-eslint/no-explicit-any - if ((delta as any)?.reasoning_content) { + const reasoningContent = (delta as any)?.reasoning_content; + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const reasoning = (delta as any)?.reasoning; + if (reasoningContent && reasoning) { + eventType = "delta"; + // ambiguous origin: record no fidelity so a replay sends both fields back + contentItems.push({ type: "thinking", thinking: reasoningContent }); + } else if (reasoningContent) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning_content, + thinking: reasoningContent, + fidelity: { reasoning_field: "reasoning_content" }, }); - } - // openrouter compatibility - // eslint-disable-next-line @typescript-eslint/no-explicit-any - else if ((delta as any)?.reasoning) { + } else if (reasoning) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning, + thinking: reasoning, + fidelity: { reasoning_field: "reasoning" }, }); } @@ -431,7 +455,12 @@ export class OpenaiClient extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments || "{}"), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id as string, }, ], @@ -461,7 +490,12 @@ export class OpenaiClient extends LLMClient { { type: "tool_call", name: partialToolCall.name, - arguments: JSON.parse(partialToolCall.arguments || "{}"), + arguments: parseToolCallArguments( + partialToolCall.arguments, + this.constructor.name, + partialToolCall.name || "", + partialToolCall.tool_call_id || "", + ), tool_call_id: partialToolCall.tool_call_id as string, }, ], diff --git a/src_ts/src/types.ts b/src_ts/src/types.ts index d08a65f1..2bbacac0 100644 --- a/src_ts/src/types.ts +++ b/src_ts/src/types.ts @@ -47,12 +47,19 @@ export type AspectRatio = | "21:9"; export type ImageSize = "1K" | "2K"; +/** + * Arbitrary JSON-style payload of wire-fidelity data recorded by a client, + * such as thinking signatures, phase labels, or the upstream reasoning field + * name. Opaque to consumers: pass it back unchanged so a replay reproduces + * the original wire message. + */ +// eslint-disable-next-line @typescript-eslint/no-explicit-any +export type Fidelity = Record; + export interface TextContentItem { type: "text"; text: string; - phase?: string | null; - // signature is always base64 encode string in typescript - signature?: string; + fidelity?: Fidelity; } export interface ImageContentItem { @@ -64,20 +71,20 @@ export interface InlineDataContentItem { type: "inline_data"; data: Buffer; mime_type: string; - signature?: string; + fidelity?: Fidelity; } export interface ThinkingContentItem { type: "thinking"; thinking: string; - signature?: string; + fidelity?: Fidelity; } export interface InlineThinkingContentItem { type: "inline_thinking"; data: Buffer; mime_type: string; - signature?: string; + fidelity?: Fidelity; } export interface ToolCallContentItem { @@ -86,7 +93,7 @@ export interface ToolCallContentItem { // eslint-disable-next-line @typescript-eslint/no-explicit-any arguments: Record; tool_call_id: string; - signature?: string; + fidelity?: Fidelity; } export interface PartialToolCallContentItem { @@ -94,7 +101,7 @@ export interface PartialToolCallContentItem { name: string; arguments: string; tool_call_id: string; - signature?: string; + fidelity?: Fidelity; } export interface ToolResultContentItem { diff --git a/src_ts/tests/empty-response.test.ts b/src_ts/tests/empty-response.test.ts new file mode 100644 index 00000000..f9d96d18 --- /dev/null +++ b/src_ts/tests/empty-response.test.ts @@ -0,0 +1,228 @@ +// Copyright 2025 Prism Shadow. and/or its affiliates +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { expect, describe, test } from "@jest/globals"; +import { + AgentHubError, + AutoLLMClient, + EmptyResponseError, + TextContentItem, + ToolCallArgumentParseError, + UniConfig, + UniEvent, + UniMessage, +} from "../src"; + +type FakeOpenAICompatibleClient = { + baseURL: string; + chat: { + completions: { + create: () => Promise>; + }; + }; +}; + +type ReasoningStreamClient = { + streamingResponse(options: { + messages: UniMessage[]; + config: UniConfig; + }): AsyncIterable; +}; + +interface ReasoningStreamCase { + expectedClient: string; + model: string; + clientType: string; +} + +const REASONING_STREAM_CASES: ReasoningStreamCase[] = [ + { + expectedClient: "OpenaiClient", + model: "gpt-5.5", + clientType: "openai", + }, + { + expectedClient: "GLM5_1Client", + model: "glm-5.1", + clientType: "glm-5.1", + }, + { + expectedClient: "KimiK2_6Client", + model: "kimi-k2.6", + clientType: "kimi-k2.6", + }, + { + expectedClient: "DeepSeekV4Client", + model: "deepseek-v4", + clientType: "deepseek-v4", + }, +]; + +const messages: UniMessage[] = [ + { + role: "user", + content_items: [{ type: "text", text: "Create a memo." }], + }, +]; + +function streamFromChunks(chunks: unknown[]): AsyncIterable { + return { + async *[Symbol.asyncIterator]() { + for (const chunk of chunks) { + yield chunk; + } + }, + }; +} + +function installFakeOpenAICompatibleStream( + client: ReasoningStreamClient, + chunks: unknown[], +): void { + const fakeClient: FakeOpenAICompatibleClient = { + baseURL: "https://api.test.invalid/v1", + chat: { + completions: { + create: async () => streamFromChunks(chunks), + }, + }, + }; + const routedClient = ( + client as unknown as { _client: { _client: FakeOpenAICompatibleClient } } + )._client; + routedClient._client = fakeClient; +} + +function createAutoClient(testCase: ReasoningStreamCase): AutoLLMClient { + return new AutoLLMClient({ + model: testCase.model, + apiKey: "test-key", + clientType: testCase.clientType, + }); +} + +function deltaChunk(delta: { + content?: string; + reasoning_content?: string; +}): unknown { + return { + choices: [{ delta, finish_reason: null }], + usage: null, + }; +} + +function stopChunk(finishReason: string): unknown { + return { + choices: [{ delta: {}, finish_reason: finishReason }], + usage: { + prompt_tokens: 1, + completion_tokens: 1, + completion_tokens_details: { reasoning_tokens: 1 }, + prompt_cache_hit_tokens: 0, + prompt_cache_miss_tokens: 1, + }, + }; +} + +async function collectEvents( + stream: AsyncIterable, +): Promise { + const events: UniEvent[] = []; + for await (const event of stream) { + events.push(event); + } + return events; +} + +async function captureStreamError( + stream: AsyncIterable, +): Promise { + let capturedError: unknown; + try { + await collectEvents(stream); + } catch (error) { + capturedError = error; + } + return capturedError; +} + +describe.each(REASONING_STREAM_CASES)( + "Reasoning output validation for $clientType", + (testCase) => { + test("rejects thinking-only responses", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + deltaChunk({ reasoning_content: "Let me think about the memo." }), + stopChunk("stop"), + ]); + + const capturedError = await captureStreamError( + client.streamingResponse({ messages, config: {} }), + ); + + expect(capturedError).toBeInstanceOf(EmptyResponseError); + const emptyError = capturedError as EmptyResponseError; + expect(emptyError.client).toBe(testCase.expectedClient); + expect(emptyError.finishReason).toBe("stop"); + expect(emptyError.message).toContain("no content other than thinking"); + }); + + test("rejects responses without any content", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [stopChunk("length")]); + + const capturedError = await captureStreamError( + client.streamingResponse({ messages, config: {} }), + ); + + expect(capturedError).toBeInstanceOf(EmptyResponseError); + expect((capturedError as EmptyResponseError).finishReason).toBe("length"); + }); + + test("accepts responses with text content", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + deltaChunk({ reasoning_content: "Let me think about the memo." }), + deltaChunk({ content: "Here is the memo." }), + stopChunk("stop"), + ]); + + const events = await collectEvents( + client.streamingResponse({ messages, config: {} }), + ); + const texts = events.flatMap((event) => + event.content_items + .filter((item): item is TextContentItem => item.type === "text") + .map((item) => item.text), + ); + expect(texts).toEqual(["Here is the memo."]); + }); + }, +); + +test("AgentHub errors share the AgentHubError base class", () => { + const emptyError = new EmptyResponseError({ + client: "OpenaiClient", + finishReason: "stop", + }); + expect(emptyError).toBeInstanceOf(AgentHubError); + const parseError = new ToolCallArgumentParseError({ + client: "OpenaiClient", + toolName: "exec_command", + toolCallId: "call_ok", + rawArguments: "[]", + reason: "Expected a JSON object.", + }); + expect(parseError).toBeInstanceOf(AgentHubError); +}); diff --git a/src_ts/tests/reasoning-fidelity.test.ts b/src_ts/tests/reasoning-fidelity.test.ts new file mode 100644 index 00000000..fb76c6eb --- /dev/null +++ b/src_ts/tests/reasoning-fidelity.test.ts @@ -0,0 +1,275 @@ +// Copyright 2025 Prism Shadow. and/or its affiliates +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { expect, describe, test } from "@jest/globals"; +import { AutoLLMClient, TextContentItem, UniEvent, UniMessage } from "../src"; + +type FakeOpenAICompatibleClient = { + baseURL: string; + chat: { + completions: { + create: () => Promise>; + }; + }; +}; + +interface ReasoningReplayCase { + model: string; + clientType: string; +} + +const REASONING_REPLAY_CASES: ReasoningReplayCase[] = [ + { model: "gpt-5.5", clientType: "openai" }, + { model: "glm-5.1", clientType: "glm-5.1" }, + { model: "kimi-k2.6", clientType: "kimi-k2.6" }, +]; + +function streamFromChunks(chunks: unknown[]): AsyncIterable { + return { + async *[Symbol.asyncIterator]() { + for (const chunk of chunks) { + yield chunk; + } + }, + }; +} + +function installFakeOpenAICompatibleStream( + client: AutoLLMClient, + chunks: unknown[], +): void { + const fakeClient: FakeOpenAICompatibleClient = { + baseURL: "https://api.test.invalid/v1", + chat: { + completions: { + create: async () => streamFromChunks(chunks), + }, + }, + }; + const routedClient = ( + client as unknown as { _client: { _client: FakeOpenAICompatibleClient } } + )._client; + routedClient._client = fakeClient; +} + +function createAutoClient(testCase: ReasoningReplayCase): AutoLLMClient { + return new AutoLLMClient({ + model: testCase.model, + apiKey: "test-key", + clientType: testCase.clientType, + }); +} + +function deltaChunk(delta: { + content?: string; + reasoning_content?: string; + reasoning?: string; +}): unknown { + return { + choices: [{ delta, finish_reason: null }], + usage: null, + }; +} + +function stopChunk(finishReason: string = "stop"): unknown { + return { + choices: [{ delta: {}, finish_reason: finishReason }], + usage: { + prompt_tokens: 1, + completion_tokens: 1, + completion_tokens_details: { reasoning_tokens: 1 }, + prompt_cache_hit_tokens: 0, + prompt_cache_miss_tokens: 1, + }, + }; +} + +function userMessage(): UniMessage { + return { + role: "user", + content_items: [{ type: "text", text: "Create a memo." }], + }; +} + +async function transformHistory( + client: AutoLLMClient, + history: UniMessage[], +): Promise[]> { + return (await client.transformUniMessageToModelInput(history)) as Record< + string, + unknown + >[]; +} + +async function runTurnAndReplay(client: AutoLLMClient): Promise<{ + historyMessage: UniMessage; + replayedMessage: Record; +}> { + const events: UniEvent[] = []; + for await (const event of client.streamingResponseStateful({ + message: userMessage(), + config: {}, + })) { + events.push(event); + } + + const history = client.getHistory(); + const modelInput = await transformHistory(client, history); + const historyMessage = history[history.length - 1]; + const replayedMessage = modelInput[modelInput.length - 1]; + if (!historyMessage || !replayedMessage) { + throw new Error("history or model input is empty"); + } + + return { historyMessage, replayedMessage }; +} + +function thinkingItems(message: UniMessage): unknown[] { + return message.content_items.filter((item) => item.type === "thinking"); +} + +describe.each(REASONING_REPLAY_CASES)( + "Reasoning field fidelity for $clientType", + (testCase) => { + test("replay preserves the reasoning_content field", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + deltaChunk({ reasoning_content: "Let me think" }), + deltaChunk({ reasoning_content: " about the memo." }), + deltaChunk({ content: "Here is the memo." }), + stopChunk(), + ]); + + const { historyMessage, replayedMessage } = + await runTurnAndReplay(client); + expect(thinkingItems(historyMessage)).toEqual([ + { + type: "thinking", + thinking: "Let me think about the memo.", + fidelity: { reasoning_field: "reasoning_content" }, + }, + ]); + expect(replayedMessage.reasoning_content).toBe( + "Let me think about the memo.", + ); + expect(replayedMessage).not.toHaveProperty("reasoning"); + }); + + test("replay preserves the reasoning field", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + deltaChunk({ reasoning: "Let me think" }), + deltaChunk({ reasoning: " about the memo." }), + deltaChunk({ content: "Here is the memo." }), + stopChunk(), + ]); + + const { historyMessage, replayedMessage } = + await runTurnAndReplay(client); + expect(thinkingItems(historyMessage)).toEqual([ + { + type: "thinking", + thinking: "Let me think about the memo.", + fidelity: { reasoning_field: "reasoning" }, + }, + ]); + expect(replayedMessage.reasoning).toBe("Let me think about the memo."); + expect(replayedMessage).not.toHaveProperty("reasoning_content"); + }); + + test("replay keeps both fields when the origin is ambiguous", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + deltaChunk({ + reasoning_content: "Let me think.", + reasoning: "Let me think.", + }), + deltaChunk({ content: "Here is the memo." }), + stopChunk(), + ]); + + const { historyMessage, replayedMessage } = + await runTurnAndReplay(client); + expect(thinkingItems(historyMessage)).toEqual([ + { type: "thinking", thinking: "Let me think." }, + ]); + expect(replayedMessage.reasoning_content).toBe("Let me think."); + expect(replayedMessage.reasoning).toBe("Let me think."); + }); + + test("replay of thinking without fidelity sends both fields", async () => { + const client = createAutoClient(testCase); + const history: UniMessage[] = [ + userMessage(), + { + role: "assistant", + content_items: [ + { type: "thinking", thinking: "Let me think." }, + { type: "text", text: "Here is the memo." }, + ], + }, + ]; + + const modelInput = await transformHistory(client, history); + const replayedMessage = modelInput[modelInput.length - 1]; + if (!replayedMessage) { + throw new Error("model input is empty"); + } + + expect(replayedMessage.reasoning_content).toBe("Let me think."); + expect(replayedMessage.reasoning).toBe("Let me think."); + }); + }, +); + +function textDeltaEvent(text: string, phase?: string): UniEvent { + const item: TextContentItem = { type: "text", text }; + if (phase !== undefined) { + item.fidelity = { phase }; + } + + return { + role: "assistant", + event_type: "delta", + content_items: [item], + usage_metadata: null, + finish_reason: null, + }; +} + +test("concatenation splits text items only on phase change", () => { + const client = createAutoClient(REASONING_REPLAY_CASES[0]); + const message = client.concatUniEventsToUniMessage([ + textDeltaEvent("", "commentary"), + textDeltaEvent("I'll inspect the logs."), + textDeltaEvent("", "final_answer"), + textDeltaEvent("Root cause:"), + textDeltaEvent(" cache invalidation race."), + textDeltaEvent("", "final_answer"), + textDeltaEvent(" Remediation follows."), + ]); + + expect(message.content_items).toEqual([ + { + type: "text", + text: "I'll inspect the logs.", + fidelity: { phase: "commentary" }, + }, + { + type: "text", + text: "Root cause: cache invalidation race. Remediation follows.", + fidelity: { phase: "final_answer" }, + }, + ]); +}); diff --git a/src_ts/tests/tool-call-arguments.test.ts b/src_ts/tests/tool-call-arguments.test.ts new file mode 100644 index 00000000..000c96ee --- /dev/null +++ b/src_ts/tests/tool-call-arguments.test.ts @@ -0,0 +1,247 @@ +// Copyright 2025 Prism Shadow. and/or its affiliates +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { expect, describe, test } from "@jest/globals"; +import { + AutoLLMClient, + ToolCallArgumentParseError, + ToolCallContentItem, + UniConfig, + UniEvent, + UniMessage, +} from "../src"; + +type FakeOpenAICompatibleClient = { + baseURL: string; + chat: { + completions: { + create: () => Promise>; + }; + }; +}; + +type OpenAICompatibleToolStreamClient = { + streamingResponse(options: { + messages: UniMessage[]; + config: UniConfig; + }): AsyncIterable; +}; + +interface OpenAICompatibleToolStreamCase { + expectedClient: string; + model: string; + clientType: string; +} + +const OPENAI_COMPATIBLE_TOOL_STREAM_CASES: OpenAICompatibleToolStreamCase[] = [ + { + expectedClient: "OpenaiClient", + model: "gpt-5.5", + clientType: "openai", + }, + { + expectedClient: "GLM5_1Client", + model: "glm-5.1", + clientType: "glm-5.1", + }, + { + expectedClient: "KimiK2_6Client", + model: "kimi-k2.6", + clientType: "kimi-k2.6", + }, + { + expectedClient: "DeepSeekV4Client", + model: "deepseek-v4", + clientType: "deepseek-v4", + }, +]; + +const messages: UniMessage[] = [ + { + role: "user", + content_items: [{ type: "text", text: "Create a memo." }], + }, +]; + +function streamFromChunks(chunks: unknown[]): AsyncIterable { + return { + async *[Symbol.asyncIterator]() { + for (const chunk of chunks) { + yield chunk; + } + }, + }; +} + +function installFakeOpenAICompatibleStream( + client: OpenAICompatibleToolStreamClient, + chunks: unknown[], +): void { + const fakeClient: FakeOpenAICompatibleClient = { + baseURL: "https://api.test.invalid/v1", + chat: { + completions: { + create: async () => streamFromChunks(chunks), + }, + }, + }; + const routedClient = ( + client as unknown as { _client: { _client: FakeOpenAICompatibleClient } } + )._client; + routedClient._client = fakeClient; +} + +function createAutoClient( + testCase: OpenAICompatibleToolStreamCase, +): AutoLLMClient { + return new AutoLLMClient({ + model: testCase.model, + apiKey: "test-key", + clientType: testCase.clientType, + }); +} + +function toolDeltaChunk( + toolCallId: string, + name: string, + args: string, +): unknown { + return { + choices: [ + { + delta: { + tool_calls: [ + { + id: toolCallId, + function: { name, arguments: args }, + }, + ], + }, + finish_reason: null, + }, + ], + usage: null, + }; +} + +function toolStopChunk(): unknown { + return { + choices: [{ delta: {}, finish_reason: "tool_calls" }], + usage: { + completion_tokens: 1, + completion_tokens_details: { reasoning_tokens: 0 }, + prompt_cache_hit_tokens: 0, + prompt_cache_miss_tokens: 1, + }, + }; +} + +async function collectEvents( + stream: AsyncIterable, +): Promise { + const events: UniEvent[] = []; + for await (const event of stream) { + events.push(event); + } + return events; +} + +async function captureStreamError( + stream: AsyncIterable, +): Promise { + let capturedError: unknown; + try { + await collectEvents(stream); + } catch (error) { + capturedError = error; + } + return capturedError; +} + +describe.each(OPENAI_COMPATIBLE_TOOL_STREAM_CASES)( + "OpenAI-compatible tool call streaming for $clientType", + (testCase) => { + test("combines valid streamed tool call arguments", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + toolDeltaChunk("call_ok", "exec_command", '{"cmd":'), + toolDeltaChunk("", "", '"echo ok"}'), + toolStopChunk(), + ]); + + const events = await collectEvents( + client.streamingResponse({ messages, config: {} }), + ); + const toolCalls = events.flatMap((event) => + event.content_items.filter( + (item): item is ToolCallContentItem => item.type === "tool_call", + ), + ); + + expect(toolCalls).toHaveLength(1); + expect(toolCalls[0]).toEqual({ + type: "tool_call", + name: "exec_command", + arguments: { cmd: "echo ok" }, + tool_call_id: "call_ok", + }); + }); + + test("reports malformed streamed tool call arguments with context", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + toolDeltaChunk( + "call_bad", + "exec_command", + '{"cmd":"python create_docx.py', + ), + toolStopChunk(), + ]); + + const capturedError = await captureStreamError( + client.streamingResponse({ messages, config: {} }), + ); + + expect(capturedError).toBeInstanceOf(ToolCallArgumentParseError); + const parseError = capturedError as ToolCallArgumentParseError; + expect(parseError.client).toBe(testCase.expectedClient); + expect(parseError.toolName).toBe("exec_command"); + expect(parseError.toolCallId).toBe("call_bad"); + expect(parseError.rawArgumentsLength).toBeGreaterThan(0); + expect(parseError.rawArgumentsPreview).toContain("create_docx.py"); + expect(parseError.message).toMatch(/Unterminated string/u); + }); + + test("reports non-object streamed tool call arguments with context", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + toolDeltaChunk("call_array", "exec_command", "[]"), + toolStopChunk(), + ]); + + const capturedError = await captureStreamError( + client.streamingResponse({ messages, config: {} }), + ); + + expect(capturedError).toBeInstanceOf(ToolCallArgumentParseError); + const parseError = capturedError as ToolCallArgumentParseError; + expect(parseError.client).toBe(testCase.expectedClient); + expect(parseError.toolName).toBe("exec_command"); + expect(parseError.toolCallId).toBe("call_array"); + expect(parseError.rawArgumentsLength).toBe(2); + expect(parseError.rawArgumentsPreview).toBe("[]"); + expect(parseError.message).toContain("Expected a JSON object."); + }); + }, +);