diff --git a/.agents/skills/agenthub-dev/SKILL.md b/.agents/skills/agenthub-dev/SKILL.md index 0cc6cc84..97b04500 100644 --- a/.agents/skills/agenthub-dev/SKILL.md +++ b/.agents/skills/agenthub-dev/SKILL.md @@ -44,7 +44,7 @@ CHANGELOG.md Brief one-line entries linking into changelog/ - Any difference between generations, even a single key name, means a separate folder per generation (e.g. `claude4_6/` vs `claude5/`). - Only an identical wire protocol may share a folder; name it after the newest generation (rename and reroute if needed). This is how `claude5/` serves Claude 4.7, 4.8, and 5. - `auto_client.py` / `autoClient.ts` route model names by explicit version matching only, never a bare substring like `"claude" in model`. -- Conversion must be bijective: a wire message converted to `UniMessage`/`UniEvent` and back must reproduce the original exactly, including thinking signatures, phase labels, and tool-call IDs. Verify against the captured exchange. +- Conversion must be bijective: a wire message converted to `UniMessage`/`UniEvent` and back must reproduce the original exactly, including `fidelity` payloads (thinking signatures, phase labels, reasoning field names) and tool-call IDs. Verify against the captured exchange. - `UniConfig` keys rarely map one-to-one onto provider config keys. **Stop and ask**: list every non-obvious mapping and confirm it with the user before coding. Never decide silently. - Implement Python and TypeScript together with identical behavior. diff --git a/CHANGELOG.md b/CHANGELOG.md index 73aa1dba..f746fcf1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ Here, we record the addition and removal times of models, major functional updates, bug fixes, and release times of key versions. Each entry keeps one brief line; the full details of a change live in a file under [changelog/](changelog/). +- [2026-07-20] Content items now carry an opaque `fidelity` payload that absorbs the former `signature`/`phase` fields (breaking), and OpenAI-compatible clients use it to replay thinking through exactly the reasoning field the upstream produced instead of always sending both spellings. ([details](changelog/2026-07-20-reasoning-field-fidelity.md)) + - [2026-07-17] Add the `agenthub-dev` skill that fixes the model-support development workflow, and the `changelog/` details directory. ([details](changelog/2026-07-17-agenthub-dev-skill.md)) - [2026-07-14] Raise `EmptyResponseError` when a model completes a response with thinking output only, since sending it back would fail with a 400 error. It and `ToolCallArgumentParseError` now inherit the new `AgentHubError` base class. diff --git a/README.md b/README.md index f2e4fa7a..f0a98103 100644 --- a/README.md +++ b/README.md @@ -433,7 +433,7 @@ Example UniMessage: {"type": "text", "text": "How are you doing?"}, {"type": "image_url", "image_url": "https://example.com/image.jpg"}, {"type": "inline_data", "mime_type": "image/jpeg", "data": "base64-encoded-image"}, - {"type": "thinking", "thinking": "I am thinking.", "signature": "0x123456"}, + {"type": "thinking", "thinking": "I am thinking.", "fidelity": {"signature": "0x123456"}}, {"type": "inline_thinking", "mime_type": "image/jpeg", "data": "base64-encoded-image"}, {"type": "tool_call", "name": "math", "arguments": {"expression": "2 + 3"}, "tool_call_id": "123"}, {"type": "tool_result", "text": "2 + 3 = 5", "images": [], "tool_call_id": "123"} diff --git a/changelog/2026-07-20-reasoning-field-fidelity.md b/changelog/2026-07-20-reasoning-field-fidelity.md new file mode 100644 index 00000000..6c31be14 --- /dev/null +++ b/changelog/2026-07-20-reasoning-field-fidelity.md @@ -0,0 +1,39 @@ +# Add the `fidelity` field and replay the exact reasoning field the upstream produced + +## The bug + +OpenAI Chat Completions-compatible servers spell the streamed thinking field differently — vLLM & SiliconFlow use `reasoning_content` while OpenRouter uses `reasoning` — and when sending assistant history back, the `openai`, `glm5_1`, and `kimi_k2_6` clients always set **both** fields on the message. Strict upstreams reject the spelling they did not emit (e.g. a server that returned `reasoning_content` refuses a request containing `reasoning`), breaking multi-turn conversations. + +## The `fidelity` field + +Fixing this needs a place to record which wire field carried the thinking. Rather than overloading `signature`, content items now carry a single dedicated field: + +- `fidelity` (`dict[str, Any]` / `Record`, optional) — an arbitrary JSON-style object of wire-level data a client records to reproduce the original message on replay. Opaque to consumers: pass it back unchanged. + +It replaces and absorbs the former item-level `signature` and `phase` fields: + +| Client | Old | New | +| --- | --- | --- | +| `claude5` / `claude4_6` | `signature: ` on thinking (also holds redacted-thinking data) | `fidelity: {"signature": }` | +| `gemini3` | `signature: ` on text / thinking / inline / tool_call items (key present even when `None`) | `fidelity: {"signature": }`, omitted entirely when absent | +| `gpt5_5` | `signature: json.dumps({"id": ..., "encrypted_content": ...})` on thinking; `phase:

` on text | `fidelity: {"id": ..., "encrypted_content": ...}` (no more JSON-in-a-string); `fidelity: {"phase":

}` | +| `openai` / `glm5_1` / `kimi_k2_6` / `deepseek_v4` | nothing recorded; both reasoning spellings sent back | `fidelity: {"reasoning_field": "reasoning_content" \| "reasoning"}` per thinking delta | + +## The reasoning-field fix + +On receive, the OpenAI-compatible clients record the wire field name that carried each thinking delta. On send, the message conversion replays the thinking through exactly that field. Fallbacks keep the old maximum-compatibility behavior: thinking without a recorded `reasoning_field` (hand-written histories, foreign-protocol fidelity), mixed fields within one message, and the ambiguous case where one chunk carries both spellings (such deltas record no fidelity) all still send both fields. + +## Concatenation rules + +`concat_uni_events_to_uni_message` / `concatUniEventsToUniMessage` now key on `fidelity`: + +- text: a phase change starts a new item (same-phase and phaseless deltas merge, per the GPT-5.5 `phase` guide); an incoming fidelity payload (e.g. a signature) merges into the open item's fidelity and finishes it. +- thinking: an incoming fidelity payload finishes the open item (Claude signature deltas, GPT-5.5 reasoning markers), and a run of deltas carrying **equal** fidelity concatenates into one item (the OpenAI-compatible per-delta `reasoning_field` tags). + +## Breaking change + +Histories recorded by earlier versions carry `signature` / `phase` at the top level of content items; clients no longer read those fields. To replay an old history, move each item's `signature`/`phase` into `fidelity` (`{"signature": ...}` for Claude/Gemini, `{"id": ..., "encrypted_content": ...}` parsed from the GPT-5.5 JSON string, `{"phase": ...}` for GPT-5.5 text). + +## Tests + +Offline fake-stream suites `src_py/tests/test_reasoning_fidelity.py` and `src_ts/tests/reasoning-fidelity.test.ts` cover both reasoning field spellings, the ambiguous both-fields case, and the no-fidelity fallback across the `openai`, `glm5_1`, and `kimi_k2_6` clients. diff --git a/llmsdk_docs/gpt5_5/README.md b/llmsdk_docs/gpt5_5/README.md index 1b9bba9d..34e01deb 100644 --- a/llmsdk_docs/gpt5_5/README.md +++ b/llmsdk_docs/gpt5_5/README.md @@ -15,6 +15,7 @@ The `docs/` folder contains detailed guides on various GPT-5.5 features: - [images-vision.md](./docs/images-vision.md) - Image and vision capabilities - [latest-model.md](./docs/latest-model.md) - Latest model features and updates - [migrate-to-responses.md](./docs/migrate-to-responses.md) - Migration guide to Responses API +- [reasoning.md](./docs/reasoning.md) - Reasoning effort, summaries, encrypted reasoning, and the `phase` parameter - [text.md](./docs/text.md) - Text generation and completion ## Examples diff --git a/llmsdk_docs/gpt5_5/docs/reasoning.md b/llmsdk_docs/gpt5_5/docs/reasoning.md new file mode 100644 index 00000000..7df1750b --- /dev/null +++ b/llmsdk_docs/gpt5_5/docs/reasoning.md @@ -0,0 +1,58 @@ +# Reasoning models + +Source: https://developers.openai.com/api/docs/guides/reasoning (Responses API; applies to GPT-5.6 variants, GPT-5.5, and GPT-5.4) + +Page outline: Get started with reasoning / Reasoning effort / Reasoning mode / How reasoning works / Managing the context window / Controlling costs / Allocating space for reasoning / Handling incomplete responses / Preserve reasoning across calls / Continue reasoning with stored responses / Preserve reasoning without stored responses / Reasoning summaries / `phase` parameter / Advice on prompting / Use case examples + +## Reasoning effort + +The `reasoning.effort` parameter guides how much the model thinks: `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, or `max`. Lower values prioritize speed and cost; higher values enable more thorough analysis. Defaults vary by model; GPT-5.5 defaults to `medium`. + +## Reasoning summaries + +Models can emit summaries of their internal reasoning when `reasoning.summary` is set (`concise`, `detailed`, or `auto`). Summaries appear in the `summary` array of reasoning output items and require explicit opt-in. + +## Preserve reasoning without stored responses + +When `store` is `false`, reasoning items include an `encrypted_content` property by default. Pass these encrypted tokens back on later calls to preserve reasoning context without storing responses server-side. + +## `phase` parameter + +> For long-running or tool-heavy flows with GPT-5.5 and GPT-5.4 in the Responses API, use the assistant message `phase` field to avoid early stopping. + +- Use `phase: "commentary"` for intermediate assistant updates, such as preambles before tool calls. +- Use `phase: "final_answer"` for the completed answer. +- These are the only two `phase` values. +- Don't add `phase` to user messages. +- `phase` is optional at the API level, but OpenAI recommends using it. When replaying assistant history manually, preserve each original `phase` value. Missing or dropped `phase` can cause preambles to be treated as final answers in those workflows. +- Using `previous_response_id` is usually the simplest path because prior assistant state is preserved. + +Official example (input array with `phase` on assistant messages): + +```python +from openai import OpenAI + +client = OpenAI() + +response = client.responses.create( + model="gpt-5.6", + input=[ + { + "role": "assistant", + "phase": "commentary", + "content": "I'll inspect the logs and then summarize root cause and remediation.", + }, + { + "role": "assistant", + "phase": "final_answer", + "content": "Root cause: cache invalidation race.", + }, + { + "role": "user", + "content": "Great—now give me a rollout-safe fix plan.", + }, + ], +) + +print(response.output_text) +``` diff --git a/skills/agenthub-python/SKILL.md b/skills/agenthub-python/SKILL.md index 72b44fc0..8b6fa6c5 100644 --- a/skills/agenthub-python/SKILL.md +++ b/skills/agenthub-python/SKILL.md @@ -100,7 +100,7 @@ Agent loop rules: - Send every tool result with the exact `tool_call_id` from its originating `tool_call`. Do not invent, normalize, or reuse IDs across unrelated tool calls. - If streamed tool-call arguments cannot be parsed, AgentHub raises `ToolCallArgumentParseError`. Do not execute the tool from partial arguments; let the agent runtime retry or re-prompt the model. -- Preserve `thinking` and `inline_thinking` items. Do not strip `phase` or `signature` fields. +- Preserve `thinking` and `inline_thinking` items. Do not strip or modify `fidelity` fields. - Do not accumulate `usage_metadata` across events. Take the latest `usage_metadata` as the usage of the current request. - For embedding models, each `UniMessage` in the `messages` array produces **one embedding vector**. Within a single message, all items in `content_items` are aggregated into a single embedding. Set `embedding_config.dimensions` in the config to control vector size. diff --git a/skills/agenthub-python/reference/data-models.md b/skills/agenthub-python/reference/data-models.md index 7809462a..cf518185 100644 --- a/skills/agenthub-python/reference/data-models.md +++ b/skills/agenthub-python/reference/data-models.md @@ -54,12 +54,12 @@ Fields: message = { "role": "user", "content_items": [ - {"type": "text", "text": "Hello", "phase": None, "signature": "sig"}, + {"type": "text", "text": "Hello", "fidelity": {"phase": "commentary"}}, {"type": "image_url", "image_url": "https://example.com/image.jpg"}, - {"type": "inline_data", "data": b"...", "mime_type": "image/png", "signature": "sig"}, - {"type": "thinking", "thinking": "Reasoning", "signature": "sig"}, - {"type": "inline_thinking", "data": b"...", "mime_type": "image/png", "signature": "sig"}, - {"type": "tool_call", "name": "get_weather", "arguments": {"location": "Paris"}, "tool_call_id": "call_1", "signature": "sig"}, + {"type": "inline_data", "data": b"...", "mime_type": "image/png", "fidelity": {"signature": "sig"}}, + {"type": "thinking", "thinking": "Reasoning", "fidelity": {"signature": "sig"}}, + {"type": "inline_thinking", "data": b"...", "mime_type": "image/png", "fidelity": {"signature": "sig"}}, + {"type": "tool_call", "name": "get_weather", "arguments": {"location": "Paris"}, "tool_call_id": "call_1", "fidelity": {"signature": "sig"}}, {"type": "tool_result", "text": "22 C", "tool_call_id": "call_1"}, {"type": "embedding", "embedding": [0.1, 0.2]}, ], @@ -76,16 +76,16 @@ Fields: Content items: -- `text`: Text chunk; `phase` marks sub-stage; `signature` verifies signed content. +- `text`: Text chunk; may carry `fidelity`. - `image_url`: Image URL or data URI. -- `inline_data`: Inline media bytes with MIME type; may carry `signature`. -- `thinking`: Text reasoning content; may carry `signature`. -- `inline_thinking`: Binary reasoning artifact; may carry `signature`. -- `tool_call`: Complete model tool request with name, args, ID, and optional `signature`. +- `inline_data`: Inline media bytes with MIME type; may carry `fidelity`. +- `thinking`: Text reasoning content; may carry `fidelity`. +- `inline_thinking`: Binary reasoning artifact; may carry `fidelity`. +- `tool_call`: Complete model tool request with name, args, ID, and optional `fidelity`. - `tool_result`: Tool output text for a `tool_call_id`; may include image URLs. - `embedding`: Numeric embedding vector. -Preserve `phase` and `signature`; never drop either field. +`fidelity` is an arbitrary JSON object of wire-level data the client recorded to reproduce the original message on replay — thinking signatures, phase labels, the upstream reasoning field name, and the like. It is opaque: pass it back unchanged, never modify or drop it. ## UniEvent diff --git a/skills/agenthub-typescript/SKILL.md b/skills/agenthub-typescript/SKILL.md index 697026ee..8fa879c9 100644 --- a/skills/agenthub-typescript/SKILL.md +++ b/skills/agenthub-typescript/SKILL.md @@ -102,7 +102,7 @@ Keep these points in mind for agent loops: - Send every tool result with the exact `tool_call_id` from its originating `tool_call`. Do not invent, normalize, or reuse IDs across unrelated tool calls. - If streamed tool-call arguments cannot be parsed, AgentHub raises `ToolCallArgumentParseError`. Do not execute the tool from partial arguments; let the agent runtime retry or re-prompt the model. -- Preserve `thinking` and `inline_thinking` items. Do not strip `phase` or `signature` fields. +- Preserve `thinking` and `inline_thinking` items. Do not strip or modify `fidelity` fields. - Do not accumulate `usage_metadata` across events. Take the latest `usage_metadata` as the usage of the current request. - For embedding models, each `UniMessage` in the `messages` array produces **one embedding vector**. Within a single message, all items in `content_items` are aggregated into a single embedding. Set `embedding_config.dimensions` in the config to control vector size. diff --git a/skills/agenthub-typescript/reference/data-models.md b/skills/agenthub-typescript/reference/data-models.md index 7e6785e9..9ff5e4b7 100644 --- a/skills/agenthub-typescript/reference/data-models.md +++ b/skills/agenthub-typescript/reference/data-models.md @@ -54,12 +54,12 @@ Fields: const message = { role: "user", content_items: [ - { type: "text", text: "Hello", phase: null, signature: "sig" }, + { type: "text", text: "Hello", fidelity: { phase: "commentary" } }, { type: "image_url", image_url: "https://example.com/image.jpg" }, - { type: "inline_data", data: Buffer.from("..."), mime_type: "image/png", signature: "sig" }, - { type: "thinking", thinking: "Reasoning", signature: "sig" }, - { type: "inline_thinking", data: Buffer.from("..."), mime_type: "image/png", signature: "sig" }, - { type: "tool_call", name: "get_weather", arguments: { location: "Paris" }, tool_call_id: "call_1", signature: "sig" }, + { type: "inline_data", data: Buffer.from("..."), mime_type: "image/png", fidelity: { signature: "sig" } }, + { type: "thinking", thinking: "Reasoning", fidelity: { signature: "sig" } }, + { type: "inline_thinking", data: Buffer.from("..."), mime_type: "image/png", fidelity: { signature: "sig" } }, + { type: "tool_call", name: "get_weather", arguments: { location: "Paris" }, tool_call_id: "call_1", fidelity: { signature: "sig" } }, { type: "tool_result", text: "22 C", tool_call_id: "call_1" }, { type: "embedding", embedding: [0.1, 0.2] }, ], @@ -76,16 +76,16 @@ Fields: Content items: -- `text`: Text chunk; `phase` marks sub-stage; `signature` verifies signed content. +- `text`: Text chunk; may carry `fidelity`. - `image_url`: Image URL or data URI. -- `inline_data`: Inline media bytes with MIME type; may carry `signature`. -- `thinking`: Text reasoning content; may carry `signature`. -- `inline_thinking`: Binary reasoning artifact; may carry `signature`. -- `tool_call`: Complete model tool request with name, args, ID, and optional `signature`. +- `inline_data`: Inline media bytes with MIME type; may carry `fidelity`. +- `thinking`: Text reasoning content; may carry `fidelity`. +- `inline_thinking`: Binary reasoning artifact; may carry `fidelity`. +- `tool_call`: Complete model tool request with name, args, ID, and optional `fidelity`. - `tool_result`: Tool output text for a `tool_call_id`; may include image URLs. - `embedding`: Numeric embedding vector. -Preserve `phase` and `signature`; never drop either field. +`fidelity` is an arbitrary JSON object of wire-level data the client recorded to reproduce the original message on replay — thinking signatures, phase labels, the upstream reasoning field name, and the like. It is opaque: pass it back unchanged, never modify or drop it. ## UniEvent diff --git a/src_py/agenthub/base_client.py b/src_py/agenthub/base_client.py index ad81fac7..36488086 100644 --- a/src_py/agenthub/base_client.py +++ b/src_py/agenthub/base_client.py @@ -101,28 +101,40 @@ def concat_uni_events_to_uni_message(self, events: list[UniEvent]) -> UniMessage for event in events: # Merge content_items from all events for item in event["content_items"]: + last_fidelity = (content_items[-1].get("fidelity") or {}) if content_items else {} + item_fidelity = item.get("fidelity") or {} if item["type"] == "text": + # a delta announcing a different phase starts a new item; same-phase and + # phaseless deltas merge until a signature finishes the item if ( content_items and content_items[-1]["type"] == "text" - and content_items[-1].get("signature") is None # no signature yet - and item.get("phase") is None # no new phase + and last_fidelity.get("signature") is None # not finished by a signature yet + and ( + item_fidelity.get("phase") is None # phaseless deltas continue the item + or item_fidelity.get("phase") == last_fidelity.get("phase") # same phase merges + ) ): content_items[-1]["text"] += item["text"] - if "signature" in item: # finish the current item if signature is not None - content_items[-1]["signature"] = item["signature"] - elif item["text"] or item.get("phase") is not None: # text or new phase starts an item + if item_fidelity: # a signature finishes the current item + content_items[-1]["fidelity"] = {**last_fidelity, **item_fidelity} + elif item["text"] or item_fidelity.get("phase") is not None: # text or new phase starts an item content_items.append(item.copy()) elif item["type"] == "thinking": + # a new item starts only when the open item's fidelity is non-empty and + # differs from the incoming delta's; everything else merges into it if ( content_items and content_items[-1]["type"] == "thinking" - and content_items[-1].get("signature") is None # no signature yet + and ( + not last_fidelity # not finished by fidelity yet + or last_fidelity == item_fidelity # a run of equal fidelity is one item + ) ): content_items[-1]["thinking"] += item["thinking"] - if "signature" in item: # finish the current item if signature is not None - content_items[-1]["signature"] = item["signature"] - elif item["thinking"] or item.get("signature"): # omit empty thinking items + if item_fidelity: # fidelity finishes the current item + content_items[-1]["fidelity"] = item_fidelity + elif item["thinking"] or item_fidelity: # omit empty thinking items content_items.append(item.copy()) elif item["type"] == "partial_tool_call": # Skip partial_tool_call items - they should already be converted to tool_call diff --git a/src_py/agenthub/claude4_6/client.py b/src_py/agenthub/claude4_6/client.py index 5bf62fe1..42469303 100644 --- a/src_py/agenthub/claude4_6/client.py +++ b/src_py/agenthub/claude4_6/client.py @@ -204,10 +204,14 @@ async def transform_uni_message_to_model_input(self, messages: list[UniMessage]) content_blocks.append(await self._convert_image_url_to_source(item["image_url"])) elif item["type"] == "thinking": if item["thinking"] == REDACTED_THINKING: - content_blocks.append({"type": "redacted_thinking", "data": item["signature"]}) + content_blocks.append({"type": "redacted_thinking", "data": item["fidelity"]["signature"]}) else: content_blocks.append( - {"type": "thinking", "thinking": item["thinking"], "signature": item["signature"]} + { + "type": "thinking", + "thinking": item["thinking"], + "signature": item["fidelity"]["signature"], + } ) elif item["type"] == "tool_call": content_blocks.append( @@ -263,7 +267,9 @@ def transform_model_output_to_uni_event(self, model_output: BetaRawMessageStream {"type": "partial_tool_call", "name": block.name, "arguments": "", "tool_call_id": block.id} ) elif block.type == "redacted_thinking": - content_items.append({"type": "thinking", "thinking": REDACTED_THINKING, "signature": block.data}) + content_items.append( + {"type": "thinking", "thinking": REDACTED_THINKING, "fidelity": {"signature": block.data}} + ) elif claude_event_type == "content_block_delta": event_type = "delta" @@ -277,7 +283,7 @@ def transform_model_output_to_uni_event(self, model_output: BetaRawMessageStream {"type": "partial_tool_call", "name": "", "arguments": delta.partial_json, "tool_call_id": ""} ) elif delta.type == "signature_delta": - content_items.append({"type": "thinking", "thinking": "", "signature": delta.signature}) + content_items.append({"type": "thinking", "thinking": "", "fidelity": {"signature": delta.signature}}) elif claude_event_type == "content_block_stop": event_type = "stop" diff --git a/src_py/agenthub/claude5/client.py b/src_py/agenthub/claude5/client.py index a5edb4b9..8ca199e1 100644 --- a/src_py/agenthub/claude5/client.py +++ b/src_py/agenthub/claude5/client.py @@ -204,10 +204,14 @@ async def transform_uni_message_to_model_input(self, messages: list[UniMessage]) content_blocks.append(await self._convert_image_url_to_source(item["image_url"])) elif item["type"] == "thinking": if item["thinking"] == REDACTED_THINKING: - content_blocks.append({"type": "redacted_thinking", "data": item["signature"]}) + content_blocks.append({"type": "redacted_thinking", "data": item["fidelity"]["signature"]}) else: content_blocks.append( - {"type": "thinking", "thinking": item["thinking"], "signature": item["signature"]} + { + "type": "thinking", + "thinking": item["thinking"], + "signature": item["fidelity"]["signature"], + } ) elif item["type"] == "tool_call": content_blocks.append( @@ -263,7 +267,9 @@ def transform_model_output_to_uni_event(self, model_output: BetaRawMessageStream {"type": "partial_tool_call", "name": block.name, "arguments": "", "tool_call_id": block.id} ) elif block.type == "redacted_thinking": - content_items.append({"type": "thinking", "thinking": REDACTED_THINKING, "signature": block.data}) + content_items.append( + {"type": "thinking", "thinking": REDACTED_THINKING, "fidelity": {"signature": block.data}} + ) elif claude_event_type == "content_block_delta": event_type = "delta" @@ -277,7 +283,7 @@ def transform_model_output_to_uni_event(self, model_output: BetaRawMessageStream {"type": "partial_tool_call", "name": "", "arguments": delta.partial_json, "tool_call_id": ""} ) elif delta.type == "signature_delta": - content_items.append({"type": "thinking", "thinking": "", "signature": delta.signature}) + content_items.append({"type": "thinking", "thinking": "", "fidelity": {"signature": delta.signature}}) elif claude_event_type == "content_block_stop": event_type = "stop" diff --git a/src_py/agenthub/deepseek_v4/client.py b/src_py/agenthub/deepseek_v4/client.py index b54d6a17..99e27168 100644 --- a/src_py/agenthub/deepseek_v4/client.py +++ b/src_py/agenthub/deepseek_v4/client.py @@ -201,7 +201,15 @@ def transform_model_output_to_uni_event(self, model_output: ChatCompletionChunk) if getattr(delta, "reasoning_content", None): event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning_content")}) + # record the wire field so a replay through another OpenAI-compatible + # client reproduces the exact field DeepSeek produced + content_items.append( + { + "type": "thinking", + "thinking": getattr(delta, "reasoning_content"), + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ) if delta.content: event_type = "delta" diff --git a/src_py/agenthub/gemini3/client.py b/src_py/agenthub/gemini3/client.py index e6612952..bfaa87d5 100644 --- a/src_py/agenthub/gemini3/client.py +++ b/src_py/agenthub/gemini3/client.py @@ -26,7 +26,9 @@ from ..base_client import LLMClient from ..types import ( + ContentItem, EventType, + Fidelity, FinishReason, PartialContentItem, PromptCaching, @@ -188,6 +190,19 @@ def transform_uni_config_to_model_config(self, config: UniConfig) -> types.Gener return types.GenerateContentConfig(**config_params) if config_params else None + @staticmethod + def _part_fidelity(part: types.Part) -> dict[str, Fidelity]: + """Wrap a part's thought signature as a fidelity payload, or nothing when absent.""" + if part.thought_signature is None: + return {} + + return {"fidelity": {"signature": part.thought_signature}} + + @staticmethod + def _item_thought_signature(item: ContentItem) -> str | bytes | None: + """Read the thought signature recorded in an item's fidelity payload.""" + return (item.get("fidelity") or {}).get("signature") + async def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> list[types.Content]: """ Transform universal message format to Gemini's Content format. @@ -204,26 +219,34 @@ async def transform_uni_message_to_model_input(self, messages: list[UniMessage]) parts = [] for item in msg["content_items"]: if item["type"] == "text": - parts.append(types.Part(text=item["text"], thought_signature=item.get("signature"))) + parts.append(types.Part(text=item["text"], thought_signature=self._item_thought_signature(item))) elif item["type"] == "image_url": image_url = item["image_url"] image_data = await self._get_image_bytes_and_mime_type(image_url) parts.append(types.Part.from_bytes(**image_data)) elif item["type"] == "inline_data": inline_data = types.Blob(data=item["data"], mime_type=item["mime_type"]) - parts.append(types.Part(inline_data=inline_data, thought_signature=item.get("signature"))) + parts.append( + types.Part(inline_data=inline_data, thought_signature=self._item_thought_signature(item)) + ) elif item["type"] == "thinking": parts.append( - types.Part(text=item["thinking"], thought=True, thought_signature=item.get("signature")) + types.Part( + text=item["thinking"], thought=True, thought_signature=self._item_thought_signature(item) + ) ) elif item["type"] == "inline_thinking": inline_data = types.Blob(data=item["data"], mime_type=item["mime_type"]) parts.append( - types.Part(inline_data=inline_data, thought=True, thought_signature=item.get("signature")) + types.Part( + inline_data=inline_data, thought=True, thought_signature=self._item_thought_signature(item) + ) ) elif item["type"] == "tool_call": function_call = types.FunctionCall(name=item["name"], args=item["arguments"]) - parts.append(types.Part(function_call=function_call, thought_signature=item.get("signature"))) + parts.append( + types.Part(function_call=function_call, thought_signature=self._item_thought_signature(item)) + ) elif item["type"] == "tool_result": if "tool_call_id" not in item: raise ValueError("tool_call_id is required for tool result.") @@ -277,21 +300,19 @@ def transform_model_output_to_uni_event(self, model_output: types.GenerateConten "name": part.function_call.name, "arguments": part.function_call.args or {}, "tool_call_id": part.function_call.name, - "signature": part.thought_signature, + **self._part_fidelity(part), } ) elif part.thought: if part.text is not None: - content_items.append( - {"type": "thinking", "thinking": part.text, "signature": part.thought_signature} - ) + content_items.append({"type": "thinking", "thinking": part.text, **self._part_fidelity(part)}) elif part.inline_data is not None: content_items.append( { "type": "inline_thinking", "data": part.inline_data.data, "mime_type": part.inline_data.mime_type, - "signature": part.thought_signature, + **self._part_fidelity(part), } ) elif part.inline_data is not None: @@ -300,11 +321,11 @@ def transform_model_output_to_uni_event(self, model_output: types.GenerateConten "type": "inline_data", "data": part.inline_data.data, "mime_type": part.inline_data.mime_type, - "signature": part.thought_signature, + **self._part_fidelity(part), } ) elif part.text is not None: - content_items.append({"type": "text", "text": part.text, "signature": part.thought_signature}) + content_items.append({"type": "text", "text": part.text, **self._part_fidelity(part)}) else: raise ValueError(f"Unknown output: {part}") @@ -415,7 +436,7 @@ async def _streaming_response_internal( "name": item["name"], "arguments": json.dumps(item["arguments"], ensure_ascii=False), "tool_call_id": item["tool_call_id"], - "signature": item.get("signature"), + "fidelity": item.get("fidelity"), } ], "usage_metadata": None, diff --git a/src_py/agenthub/glm5_1/client.py b/src_py/agenthub/glm5_1/client.py index ce518068..c39e42ab 100644 --- a/src_py/agenthub/glm5_1/client.py +++ b/src_py/agenthub/glm5_1/client.py @@ -116,6 +116,7 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> li content_parts = [] # may be empty for tool results tool_calls = [] # may be empty for no tool calls thinking = "" + thinking_fields: set[str | None] = set() for item in msg["content_items"]: if item["type"] == "text": content_parts.append({"type": "text", "text": item["text"]}) @@ -123,6 +124,7 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> li raise ValueError("GLM-5 does not support image inputs.") elif item["type"] == "thinking": thinking += item["thinking"] + thinking_fields.add((item.get("fidelity") or {}).get("reasoning_field")) elif item["type"] == "tool_call": tool_calls.append( { @@ -160,8 +162,15 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> li message["tool_calls"] = tool_calls if thinking: - message["reasoning_content"] = thinking # vLLM & siliconflow compatibility - message["reasoning"] = thinking # openrouter compatibility + # send thinking back through the exact field the upstream produced (recorded + # in the item fidelity); servers may reject the spelling they did not emit + if thinking_fields == {"reasoning_content"}: + message["reasoning_content"] = thinking + elif thinking_fields == {"reasoning"}: + message["reasoning"] = thinking + else: + message["reasoning_content"] = thinking # vLLM & siliconflow compatibility + message["reasoning"] = thinking # openrouter compatibility # message may be empty for tool results if len(message.keys()) > 1: @@ -192,15 +201,29 @@ def transform_model_output_to_uni_event(self, model_output: ChatCompletionChunk) event_type = "delta" content_items.append({"type": "text", "text": delta.content}) - # vLLM & siliconflow compatibility - if getattr(delta, "reasoning_content", None): + # the thinking field name differs by server: vLLM & siliconflow use reasoning_content + # while openrouter uses reasoning; record the wire field that carried each delta + # so a replay can reproduce exactly the field the upstream produced + reasoning_content = getattr(delta, "reasoning_content", None) + reasoning = getattr(delta, "reasoning", None) + if reasoning_content and reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning_content")}) - - # openrouter compatibility - elif getattr(delta, "reasoning", None): + # ambiguous origin: record no fidelity so a replay sends both fields back + content_items.append({"type": "thinking", "thinking": reasoning_content}) + elif reasoning_content: + event_type = "delta" + content_items.append( + { + "type": "thinking", + "thinking": reasoning_content, + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ) + elif reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning")}) + content_items.append( + {"type": "thinking", "thinking": reasoning, "fidelity": {"reasoning_field": "reasoning"}} + ) if delta.tool_calls: event_type = "delta" diff --git a/src_py/agenthub/gpt5_5/client.py b/src_py/agenthub/gpt5_5/client.py index aaec5f8f..eafb2033 100644 --- a/src_py/agenthub/gpt5_5/client.py +++ b/src_py/agenthub/gpt5_5/client.py @@ -120,12 +120,13 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> Re for item in msg["content_items"]: if item["type"] == "text": - if msg["role"] == "assistant" and item.get("phase"): # split different phases - if last_phase is not None and content_items: + phase = (item.get("fidelity") or {}).get("phase") + if msg["role"] == "assistant" and phase: # split different phases + if last_phase is not None and last_phase != phase and content_items: input_list.append({"role": msg["role"], "content": content_items, "phase": last_phase}) content_items = [] - last_phase = item["phase"] + last_phase = phase if msg["role"] == "user": content_items.append({"type": "input_text", "text": item["text"]}) @@ -134,15 +135,15 @@ def transform_uni_message_to_model_input(self, messages: list[UniMessage]) -> Re elif item["type"] == "image_url": content_items.append({"type": "input_image", "image_url": item["image_url"]}) elif item["type"] == "thinking": - signature = json.loads(item["signature"]) + fidelity = item["fidelity"] input_list.append( { "type": "reasoning", - "id": signature["id"], + "id": fidelity["id"], "summary": [{"type": "summary_text", "text": item["thinking"]}] if item["thinking"] else [], - "encrypted_content": signature["encrypted_content"], + "encrypted_content": fidelity["encrypted_content"], } ) elif item["type"] == "tool_call": @@ -217,16 +218,16 @@ def transform_model_output_to_uni_event(self, model_output: ResponseStreamEvent) # adding the following thinking item leads to 400 invalid request error, why? # elif model_output.item.type == "reasoning": # event_type = "delta" - # signature = { + # fidelity = { # "id": model_output.item.id, # "encrypted_content": model_output.item.encrypted_content, # } - # content_items.append({"type": "thinking", "thinking": "", "signature": json.dumps(signature)}) + # content_items.append({"type": "thinking", "thinking": "", "fidelity": fidelity}) elif model_output.item.type == "message": if hasattr(model_output.item, "phase"): event_type = "delta" content_items.append( - {"type": "text", "text": "", "phase": getattr(model_output.item, "phase", None)} + {"type": "text", "text": "", "fidelity": {"phase": getattr(model_output.item, "phase", None)}} ) else: event_type = "unused" @@ -237,11 +238,11 @@ def transform_model_output_to_uni_event(self, model_output: ResponseStreamEvent) # not sure about the signature of openai, need to check if model_output.item.type == "reasoning": event_type = "delta" - signature = { + fidelity = { "id": model_output.item.id, "encrypted_content": model_output.item.encrypted_content, } - content_items.append({"type": "thinking", "thinking": "", "signature": json.dumps(signature)}) + content_items.append({"type": "thinking", "thinking": "", "fidelity": fidelity}) else: event_type = "unused" diff --git a/src_py/agenthub/kimi_k2_6/client.py b/src_py/agenthub/kimi_k2_6/client.py index 0bca7289..a3578147 100644 --- a/src_py/agenthub/kimi_k2_6/client.py +++ b/src_py/agenthub/kimi_k2_6/client.py @@ -144,6 +144,7 @@ async def transform_uni_message_to_model_input( content_parts = [] # may be empty for tool results tool_calls = [] # may be empty for no tool calls thinking = "" + thinking_fields: set[str | None] = set() for item in msg["content_items"]: if item["type"] == "text": content_parts.append({"type": "text", "text": item["text"]}) @@ -152,6 +153,7 @@ async def transform_uni_message_to_model_input( content_parts.append({"type": "image_url", "image_url": {"url": base64_image}}) elif item["type"] == "thinking": thinking += item["thinking"] + thinking_fields.add((item.get("fidelity") or {}).get("reasoning_field")) elif item["type"] == "tool_call": tool_calls.append( { @@ -197,8 +199,15 @@ async def transform_uni_message_to_model_input( message["tool_calls"] = tool_calls if thinking: - message["reasoning_content"] = thinking # vLLM & siliconflow compatibility - message["reasoning"] = thinking # openrouter compatibility + # send thinking back through the exact field the upstream produced (recorded + # in the item fidelity); servers may reject the spelling they did not emit + if thinking_fields == {"reasoning_content"}: + message["reasoning_content"] = thinking + elif thinking_fields == {"reasoning"}: + message["reasoning"] = thinking + else: + message["reasoning_content"] = thinking # vLLM & siliconflow compatibility + message["reasoning"] = thinking # openrouter compatibility # message may be empty for tool results if len(message.keys()) > 1: @@ -229,15 +238,29 @@ def transform_model_output_to_uni_event(self, model_output: ChatCompletionChunk) event_type = "delta" content_items.append({"type": "text", "text": delta.content}) - # vLLM & siliconflow compatibility - if getattr(delta, "reasoning_content", None): + # the thinking field name differs by server: vLLM & siliconflow use reasoning_content + # while openrouter uses reasoning; record the wire field that carried each delta + # so a replay can reproduce exactly the field the upstream produced + reasoning_content = getattr(delta, "reasoning_content", None) + reasoning = getattr(delta, "reasoning", None) + if reasoning_content and reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning_content")}) - - # openrouter compatibility - elif getattr(delta, "reasoning", None): + # ambiguous origin: record no fidelity so a replay sends both fields back + content_items.append({"type": "thinking", "thinking": reasoning_content}) + elif reasoning_content: + event_type = "delta" + content_items.append( + { + "type": "thinking", + "thinking": reasoning_content, + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ) + elif reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning")}) + content_items.append( + {"type": "thinking", "thinking": reasoning, "fidelity": {"reasoning_field": "reasoning"}} + ) if delta.tool_calls: event_type = "delta" diff --git a/src_py/agenthub/openai/client.py b/src_py/agenthub/openai/client.py index 36a2092c..369bb52f 100644 --- a/src_py/agenthub/openai/client.py +++ b/src_py/agenthub/openai/client.py @@ -129,6 +129,7 @@ async def transform_uni_message_to_model_input( content_parts = [] # may be empty for tool results tool_calls = [] # may be empty for no tool calls thinking = "" + thinking_fields: set[str | None] = set() for item in msg["content_items"]: if item["type"] == "text": content_parts.append({"type": "text", "text": item["text"]}) @@ -137,6 +138,7 @@ async def transform_uni_message_to_model_input( content_parts.append({"type": "image_url", "image_url": {"url": base64_image}}) elif item["type"] == "thinking": thinking += item["thinking"] + thinking_fields.add((item.get("fidelity") or {}).get("reasoning_field")) elif item["type"] == "tool_call": tool_calls.append( { @@ -182,8 +184,15 @@ async def transform_uni_message_to_model_input( message["tool_calls"] = tool_calls if thinking: - message["reasoning_content"] = thinking # vLLM & siliconflow compatibility - message["reasoning"] = thinking # openrouter compatibility + # send thinking back through the exact field the upstream produced (recorded + # in the item fidelity); servers may reject the spelling they did not emit + if thinking_fields == {"reasoning_content"}: + message["reasoning_content"] = thinking + elif thinking_fields == {"reasoning"}: + message["reasoning"] = thinking + else: + message["reasoning_content"] = thinking # vLLM & siliconflow compatibility + message["reasoning"] = thinking # openrouter compatibility # message may be empty for tool results if len(message.keys()) > 1: @@ -214,15 +223,29 @@ def transform_model_output_to_uni_event(self, model_output: ChatCompletionChunk) event_type = "delta" content_items.append({"type": "text", "text": delta.content}) - # vLLM & siliconflow compatibility - if getattr(delta, "reasoning_content", None): + # the thinking field name differs by server: vLLM & siliconflow use reasoning_content + # while openrouter uses reasoning; record the wire field that carried each delta + # so a replay can reproduce exactly the field the upstream produced + reasoning_content = getattr(delta, "reasoning_content", None) + reasoning = getattr(delta, "reasoning", None) + if reasoning_content and reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning_content")}) - - # openrouter compatibility - elif getattr(delta, "reasoning", None): + # ambiguous origin: record no fidelity so a replay sends both fields back + content_items.append({"type": "thinking", "thinking": reasoning_content}) + elif reasoning_content: + event_type = "delta" + content_items.append( + { + "type": "thinking", + "thinking": reasoning_content, + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ) + elif reasoning: event_type = "delta" - content_items.append({"type": "thinking", "thinking": getattr(delta, "reasoning")}) + content_items.append( + {"type": "thinking", "thinking": reasoning, "fidelity": {"reasoning_field": "reasoning"}} + ) if delta.tool_calls: for tool_call in delta.tool_calls: diff --git a/src_py/agenthub/types.py b/src_py/agenthub/types.py index 2339ba91..cc7c36e3 100644 --- a/src_py/agenthub/types.py +++ b/src_py/agenthub/types.py @@ -42,12 +42,16 @@ class PromptCaching(StrEnum): AspectRatio = Literal["1:1", "2:3", "3:2", "3:4", "4:3", "9:16", "16:9", "21:9"] ImageSize = Literal["1K", "2K"] +# Arbitrary JSON-style payload of wire-fidelity data recorded by a client, such as +# thinking signatures, phase labels, or the upstream reasoning field name. Opaque to +# consumers: pass it back unchanged so a replay reproduces the original wire message. +Fidelity = dict[str, Any] + class TextContentItem(TypedDict): type: Literal["text"] text: str - phase: NotRequired[str | None] - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class ImageContentItem(TypedDict): @@ -59,20 +63,20 @@ class InlineDataContentItem(TypedDict): type: Literal["inline_data"] data: bytes mime_type: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class ThinkingContentItem(TypedDict): type: Literal["thinking"] thinking: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class InlineThinkingContentItem(TypedDict): type: Literal["inline_thinking"] data: bytes mime_type: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class ToolCallContentItem(TypedDict): @@ -80,7 +84,7 @@ class ToolCallContentItem(TypedDict): name: str arguments: dict[str, Any] tool_call_id: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class PartialToolCallContentItem(TypedDict): @@ -88,7 +92,7 @@ class PartialToolCallContentItem(TypedDict): name: str arguments: str tool_call_id: str - signature: NotRequired[str | bytes] + fidelity: NotRequired[Fidelity] class ToolResultContentItem(TypedDict): diff --git a/src_py/tests/test_reasoning_fidelity.py b/src_py/tests/test_reasoning_fidelity.py new file mode 100644 index 00000000..bae110ef --- /dev/null +++ b/src_py/tests/test_reasoning_fidelity.py @@ -0,0 +1,244 @@ +# Copyright 2025 Prism Shadow. and/or its affiliates +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import inspect +from collections.abc import AsyncIterator +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any + +import pytest + +from agenthub import AutoLLMClient + + +@dataclass +class ReasoningReplayCase: + model: str + client_type: str + + +REASONING_REPLAY_CASES = [ + ReasoningReplayCase(model="gpt-5.5", client_type="openai"), + ReasoningReplayCase(model="glm-5.1", client_type="glm-5.1"), + ReasoningReplayCase(model="kimi-k2.6", client_type="kimi-k2.6"), +] + + +def _create_auto_client(case: ReasoningReplayCase) -> AutoLLMClient: + return AutoLLMClient(model=case.model, api_key="test-key", client_type=case.client_type) + + +async def _stream_from_chunks(chunks: list[object]) -> AsyncIterator[object]: + for chunk in chunks: + yield chunk + + +class _FakeOpenAICompatibleCompletions: + def __init__(self, chunks: list[object]) -> None: + self._chunks = chunks + + async def create(self, **_kwargs: object) -> AsyncIterator[object]: + return _stream_from_chunks(self._chunks) + + +class _FakeOpenAICompatibleClient: + def __init__(self, chunks: list[object]) -> None: + self.base_url = "https://api.test.invalid/v1" + self.chat = SimpleNamespace(completions=_FakeOpenAICompatibleCompletions(chunks)) + + +def _install_fake_openai_compatible_stream(client: AutoLLMClient, chunks: list[object]) -> None: + client._client._client = _FakeOpenAICompatibleClient(chunks) # noqa: SLF001 + + +def _delta_chunk(text: str | None = None, **reasoning_fields: str) -> object: + return SimpleNamespace( + choices=[ + SimpleNamespace( + delta=SimpleNamespace(content=text, tool_calls=None, **reasoning_fields), + finish_reason=None, + ) + ], + usage=None, + ) + + +def _stop_chunk(finish_reason: str = "stop") -> object: + return SimpleNamespace( + choices=[SimpleNamespace(delta=SimpleNamespace(content=None, tool_calls=None), finish_reason=finish_reason)], + usage=SimpleNamespace( + prompt_tokens=1, + completion_tokens=1, + prompt_tokens_details=None, + completion_tokens_details=SimpleNamespace(reasoning_tokens=1), + prompt_cache_hit_tokens=0, + prompt_cache_miss_tokens=1, + ), + ) + + +def _user_message() -> dict[str, Any]: + return {"role": "user", "content_items": [{"type": "text", "text": "Create a memo."}]} + + +async def _transform_history(client: AutoLLMClient, history: list[dict[str, Any]]) -> list[dict[str, Any]]: + model_input = client.transform_uni_message_to_model_input(history) + if inspect.isawaitable(model_input): + model_input = await model_input + + return model_input + + +async def _run_turn_and_replay(client: AutoLLMClient) -> tuple[dict[str, Any], dict[str, Any]]: + """Run one fake streamed turn, then rebuild the request payload from the stored history.""" + async for _event in client.streaming_response_stateful(_user_message(), {}): + pass + + history = client.get_history() + model_input = await _transform_history(client, history) + return history[-1], model_input[-1] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("case", REASONING_REPLAY_CASES, ids=[case.client_type for case in REASONING_REPLAY_CASES]) +async def test_replay_preserves_reasoning_content_field(case: ReasoningReplayCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _delta_chunk(reasoning_content="Let me think"), + _delta_chunk(reasoning_content=" about the memo."), + _delta_chunk(text="Here is the memo."), + _stop_chunk(), + ], + ) + + history_message, replayed_message = await _run_turn_and_replay(client) + thinking_items = [item for item in history_message["content_items"] if item["type"] == "thinking"] + assert thinking_items == [ + { + "type": "thinking", + "thinking": "Let me think about the memo.", + "fidelity": {"reasoning_field": "reasoning_content"}, + } + ] + assert replayed_message["reasoning_content"] == "Let me think about the memo." + assert "reasoning" not in replayed_message + + +@pytest.mark.asyncio +@pytest.mark.parametrize("case", REASONING_REPLAY_CASES, ids=[case.client_type for case in REASONING_REPLAY_CASES]) +async def test_replay_preserves_reasoning_field(case: ReasoningReplayCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _delta_chunk(reasoning="Let me think"), + _delta_chunk(reasoning=" about the memo."), + _delta_chunk(text="Here is the memo."), + _stop_chunk(), + ], + ) + + history_message, replayed_message = await _run_turn_and_replay(client) + thinking_items = [item for item in history_message["content_items"] if item["type"] == "thinking"] + assert thinking_items == [ + { + "type": "thinking", + "thinking": "Let me think about the memo.", + "fidelity": {"reasoning_field": "reasoning"}, + } + ] + assert replayed_message["reasoning"] == "Let me think about the memo." + assert "reasoning_content" not in replayed_message + + +@pytest.mark.asyncio +@pytest.mark.parametrize("case", REASONING_REPLAY_CASES, ids=[case.client_type for case in REASONING_REPLAY_CASES]) +async def test_replay_keeps_both_fields_when_origin_is_ambiguous(case: ReasoningReplayCase): + client = _create_auto_client(case) + _install_fake_openai_compatible_stream( + client, + [ + _delta_chunk(reasoning_content="Let me think.", reasoning="Let me think."), + _delta_chunk(text="Here is the memo."), + _stop_chunk(), + ], + ) + + history_message, replayed_message = await _run_turn_and_replay(client) + thinking_items = [item for item in history_message["content_items"] if item["type"] == "thinking"] + assert thinking_items == [{"type": "thinking", "thinking": "Let me think."}] + assert replayed_message["reasoning_content"] == "Let me think." + assert replayed_message["reasoning"] == "Let me think." + + +@pytest.mark.asyncio +@pytest.mark.parametrize("case", REASONING_REPLAY_CASES, ids=[case.client_type for case in REASONING_REPLAY_CASES]) +async def test_replay_of_thinking_without_fidelity_sends_both_fields(case: ReasoningReplayCase): + client = _create_auto_client(case) + history = [ + _user_message(), + { + "role": "assistant", + "content_items": [ + {"type": "thinking", "thinking": "Let me think."}, + {"type": "text", "text": "Here is the memo."}, + ], + }, + ] + + model_input = await _transform_history(client, history) + replayed_message = model_input[-1] + assert replayed_message["reasoning_content"] == "Let me think." + assert replayed_message["reasoning"] == "Let me think." + + +def _text_delta_event(text: str, phase: str | None = None) -> dict[str, Any]: + item: dict[str, Any] = {"type": "text", "text": text} + if phase is not None: + item["fidelity"] = {"phase": phase} + + return { + "role": "assistant", + "event_type": "delta", + "content_items": [item], + "usage_metadata": None, + "finish_reason": None, + } + + +def test_concat_splits_text_items_only_on_phase_change(): + client = _create_auto_client(REASONING_REPLAY_CASES[0]) + message = client.concat_uni_events_to_uni_message( + [ + _text_delta_event("", phase="commentary"), + _text_delta_event("I'll inspect the logs."), + _text_delta_event("", phase="final_answer"), + _text_delta_event("Root cause:"), + _text_delta_event(" cache invalidation race."), + _text_delta_event("", phase="final_answer"), + _text_delta_event(" Remediation follows."), + ] + ) + + assert message["content_items"] == [ + {"type": "text", "text": "I'll inspect the logs.", "fidelity": {"phase": "commentary"}}, + { + "type": "text", + "text": "Root cause: cache invalidation race. Remediation follows.", + "fidelity": {"phase": "final_answer"}, + }, + ] diff --git a/src_ts/src/baseClient.ts b/src_ts/src/baseClient.ts index 837c983b..04eb70b4 100644 --- a/src_ts/src/baseClient.ts +++ b/src_ts/src/baseClient.ts @@ -14,6 +14,7 @@ import { EmptyResponseError } from "./errors"; import { + Fidelity, FinishReason, ContentItem, UniConfig, @@ -22,6 +23,22 @@ import { UsageMetadata, } from "./types"; +/** + * Whether a content item carries a non-empty fidelity payload. + */ +function hasFidelity(fidelity?: Fidelity): boolean { + return fidelity != null && Object.keys(fidelity).length > 0; +} + +/** + * Compare two fidelity payloads by value. Fidelity dicts are built with a + * stable key order by each client, so JSON serialization is a faithful + * equality check. + */ +function fidelityEquals(a?: Fidelity, b?: Fidelity): boolean { + return JSON.stringify(a ?? {}) === JSON.stringify(b ?? {}); +} + /** * Abstract base class for LLM clients. * @@ -85,32 +102,42 @@ export abstract class LLMClient { for (const item of event.content_items) { if (item.type === "text") { const lastItem = contentItems[contentItems.length - 1]; + const itemFidelity = item.fidelity ?? {}; + // a delta announcing a different phase starts a new item; same-phase and + // phaseless deltas merge until a signature finishes the item if ( lastItem && lastItem.type === "text" && - lastItem.signature == null && // no signature yet - item.phase == null // no new phase + lastItem.fidelity?.signature == null && // not finished by a signature yet + (itemFidelity.phase == null || // phaseless deltas continue the item + itemFidelity.phase === lastItem.fidelity?.phase) // same phase merges ) { lastItem.text += item.text; - if (item.signature) { - lastItem.signature = item.signature; + if (hasFidelity(item.fidelity)) { + // a signature finishes the current item + lastItem.fidelity = { ...lastItem.fidelity, ...item.fidelity }; } - } else if (item.text || item.phase != null) { + } else if (item.text || itemFidelity.phase != null) { // text or new phase starts an item contentItems.push({ ...item }); } } else if (item.type === "thinking") { const lastItem = contentItems[contentItems.length - 1]; + // a new item starts only when the open item's fidelity is non-empty and + // differs from the incoming delta's; everything else merges into it if ( lastItem && lastItem.type === "thinking" && - lastItem.signature == null + (!hasFidelity(lastItem.fidelity) || // not finished by fidelity yet + // a run of equal fidelity is one item + fidelityEquals(lastItem.fidelity, item.fidelity)) ) { lastItem.thinking += item.thinking; - if (item.signature) { - lastItem.signature = item.signature; + if (hasFidelity(item.fidelity)) { + // fidelity finishes the current item + lastItem.fidelity = item.fidelity; } - } else if (item.thinking || item.signature) { + } else if (item.thinking || hasFidelity(item.fidelity)) { contentItems.push({ ...item }); } } else if (item.type === "partial_tool_call") { diff --git a/src_ts/src/claude4_6/client.ts b/src_ts/src/claude4_6/client.ts index 9f3d0f2c..97ddf7f5 100644 --- a/src_ts/src/claude4_6/client.ts +++ b/src_ts/src/claude4_6/client.ts @@ -269,13 +269,13 @@ export class Claude4_6Client extends LLMClient { if (item.thinking === REDACTED_THINKING) { contentBlocks.push({ type: "redacted_thinking", - data: item.signature, + data: item.fidelity?.signature, }); } else { contentBlocks.push({ type: "thinking", thinking: item.thinking, - signature: item.signature, + signature: item.fidelity?.signature, }); } } else if (item.type === "tool_call") { @@ -346,7 +346,7 @@ export class Claude4_6Client extends LLMClient { contentItems.push({ type: "thinking", thinking: REDACTED_THINKING, - signature: block.data, + fidelity: { signature: block.data }, }); } } else if (claudeEventType === "content_block_delta") { @@ -367,7 +367,7 @@ export class Claude4_6Client extends LLMClient { contentItems.push({ type: "thinking", thinking: "", - signature: delta.signature, + fidelity: { signature: delta.signature }, }); } } else if (claudeEventType === "content_block_stop") { diff --git a/src_ts/src/claude5/client.ts b/src_ts/src/claude5/client.ts index ac13d5d3..af32daa5 100644 --- a/src_ts/src/claude5/client.ts +++ b/src_ts/src/claude5/client.ts @@ -271,13 +271,13 @@ export class Claude5Client extends LLMClient { if (item.thinking === REDACTED_THINKING) { contentBlocks.push({ type: "redacted_thinking", - data: item.signature, + data: item.fidelity?.signature, }); } else { contentBlocks.push({ type: "thinking", thinking: item.thinking, - signature: item.signature, + signature: item.fidelity?.signature, }); } } else if (item.type === "tool_call") { @@ -348,7 +348,7 @@ export class Claude5Client extends LLMClient { contentItems.push({ type: "thinking", thinking: REDACTED_THINKING, - signature: block.data, + fidelity: { signature: block.data }, }); } } else if (claudeEventType === "content_block_delta") { @@ -369,7 +369,7 @@ export class Claude5Client extends LLMClient { contentItems.push({ type: "thinking", thinking: "", - signature: delta.signature, + fidelity: { signature: delta.signature }, }); } } else if (claudeEventType === "content_block_stop") { diff --git a/src_ts/src/deepseek_v4/client.ts b/src_ts/src/deepseek_v4/client.ts index da946d4e..da3fa343 100644 --- a/src_ts/src/deepseek_v4/client.ts +++ b/src_ts/src/deepseek_v4/client.ts @@ -230,10 +230,13 @@ export class DeepSeekV4Client extends LLMClient { // eslint-disable-next-line @typescript-eslint/no-explicit-any if ((delta as any)?.reasoning_content) { eventType = "delta"; + // record the wire field so a replay through another OpenAI-compatible + // client reproduces the exact field DeepSeek produced contentItems.push({ type: "thinking", // eslint-disable-next-line @typescript-eslint/no-explicit-any thinking: (delta as any).reasoning_content, + fidelity: { reasoning_field: "reasoning_content" }, }); } diff --git a/src_ts/src/gemini3/client.ts b/src_ts/src/gemini3/client.ts index 5c2f2111..9be71ef9 100644 --- a/src_ts/src/gemini3/client.ts +++ b/src_ts/src/gemini3/client.ts @@ -40,6 +40,7 @@ import * as path from "path"; import { LLMClient } from "../baseClient"; import { EventType, + Fidelity, FinishReason, PartialContentItem, PromptCaching, @@ -51,6 +52,25 @@ import { UsageMetadata, } from "../types"; +/** + * Wrap a part's thought signature as a fidelity payload, or nothing when absent. + */ +function partFidelity(part: Part): { fidelity?: Fidelity } { + if (part.thoughtSignature == null) { + return {}; + } + return { fidelity: { signature: part.thoughtSignature } }; +} + +/** + * Read the thought signature recorded in an item's fidelity payload. + */ +function itemThoughtSignature(item: { + fidelity?: Fidelity; +}): string | undefined { + return item.fidelity?.signature; +} + /** * Gemini 3-specific LLM client implementation. */ @@ -312,7 +332,7 @@ export class Gemini3Client extends LLMClient { if (item.type === "text") { parts.push({ text: item.text, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "image_url") { const urlValue = item.image_url; @@ -332,13 +352,13 @@ export class Gemini3Client extends LLMClient { mimeType: item.mime_type, data: item.data.toString("base64"), }, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "thinking") { parts.push({ text: item.thinking, thought: true, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "inline_thinking") { parts.push({ @@ -347,7 +367,7 @@ export class Gemini3Client extends LLMClient { data: item.data.toString("base64"), }, thought: true, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "tool_call") { const functionCall: FunctionCall = { @@ -356,7 +376,7 @@ export class Gemini3Client extends LLMClient { }; parts.push({ functionCall: functionCall, - thoughtSignature: item.signature as string | undefined, + thoughtSignature: itemThoughtSignature(item), } as Part); } else if (item.type === "tool_result") { if (!item.tool_call_id) { @@ -426,21 +446,21 @@ export class Gemini3Client extends LLMClient { name: part.functionCall.name || "", arguments: part.functionCall.args || {}, tool_call_id: part.functionCall.name || "", - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } else if (part.thought) { if (part.text !== undefined) { contentItems.push({ type: "thinking", thinking: part.text, - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } else if (part.inlineData) { contentItems.push({ type: "inline_thinking", data: Buffer.from(part.inlineData.data || "", "base64"), mime_type: part.inlineData.mimeType || "application/octet-stream", - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } } else if (part.inlineData) { @@ -448,13 +468,13 @@ export class Gemini3Client extends LLMClient { type: "inline_data", data: Buffer.from(part.inlineData.data || "", "base64"), mime_type: part.inlineData.mimeType || "application/octet-stream", - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } else if (part.text !== undefined) { contentItems.push({ type: "text", text: part.text, - signature: part.thoughtSignature as string | undefined, + ...partFidelity(part), }); } else { throw new Error(`Unknown output: ${JSON.stringify(part)}`); @@ -596,7 +616,7 @@ export class Gemini3Client extends LLMClient { name: item.name, arguments: JSON.stringify(item.arguments), tool_call_id: item.tool_call_id, - signature: item.signature, + fidelity: item.fidelity, }, ], usage_metadata: null, diff --git a/src_ts/src/glm5_1/client.ts b/src_ts/src/glm5_1/client.ts index 5adb7351..a38188bb 100644 --- a/src_ts/src/glm5_1/client.ts +++ b/src_ts/src/glm5_1/client.ts @@ -159,6 +159,7 @@ export class GLM5_1Client extends LLMClient { // eslint-disable-next-line @typescript-eslint/no-explicit-any const toolCalls: any[] = []; let thinking = ""; + const thinkingFields = new Set(); for (const item of msg.content_items) { if (item.type === "text") { @@ -167,6 +168,7 @@ export class GLM5_1Client extends LLMClient { throw new Error("GLM-5 does not support image inputs."); } else if (item.type === "thinking") { thinking += item.thinking; + thinkingFields.add(item.fidelity?.reasoning_field); } else if (item.type === "tool_call") { toolCalls.push({ id: item.tool_call_id, @@ -208,8 +210,22 @@ export class GLM5_1Client extends LLMClient { } if (thinking) { - message.reasoning_content = thinking; - message.reasoning = thinking; + // send thinking back through the exact field the upstream produced (recorded + // in the item fidelity); servers may reject the spelling they did not emit + if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning_content") + ) { + message.reasoning_content = thinking; + } else if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning") + ) { + message.reasoning = thinking; + } else { + message.reasoning_content = thinking; // vLLM & siliconflow compatibility + message.reasoning = thinking; // openrouter compatibility + } } if (Object.keys(message).length > 1) { @@ -238,24 +254,31 @@ export class GLM5_1Client extends LLMClient { contentItems.push({ type: "text", text: delta.content }); } - // vLLM & siliconflow compatibility + // the thinking field name differs by server: vLLM & siliconflow use + // reasoning_content while openrouter uses reasoning; record the wire + // field that carried each delta so a replay can reproduce exactly the + // field the upstream produced + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const reasoningContent = (delta as any)?.reasoning_content; // eslint-disable-next-line @typescript-eslint/no-explicit-any - if ((delta as any)?.reasoning_content) { + const reasoning = (delta as any)?.reasoning; + if (reasoningContent && reasoning) { + eventType = "delta"; + // ambiguous origin: record no fidelity so a replay sends both fields back + contentItems.push({ type: "thinking", thinking: reasoningContent }); + } else if (reasoningContent) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning_content, + thinking: reasoningContent, + fidelity: { reasoning_field: "reasoning_content" }, }); - } - // openrouter compatibility - // eslint-disable-next-line @typescript-eslint/no-explicit-any - else if ((delta as any)?.reasoning) { + } else if (reasoning) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning, + thinking: reasoning, + fidelity: { reasoning_field: "reasoning" }, }); } diff --git a/src_ts/src/gpt5_5/client.ts b/src_ts/src/gpt5_5/client.ts index 63395a0e..64bd9e13 100644 --- a/src_ts/src/gpt5_5/client.ts +++ b/src_ts/src/gpt5_5/client.ts @@ -156,9 +156,14 @@ export class GPT5_5Client extends LLMClient { for (const item of msg.content_items) { if (item.type === "text") { - if (msg.role === "assistant" && item.phase) { + const phase = item.fidelity?.phase; + if (msg.role === "assistant" && phase) { // split different phases - if (lastPhase !== null && contentItems.length > 0) { + if ( + lastPhase !== null && + lastPhase !== phase && + contentItems.length > 0 + ) { inputList.push({ role: msg.role, content: contentItems, @@ -166,7 +171,7 @@ export class GPT5_5Client extends LLMClient { }); contentItems = []; } - lastPhase = item.phase; + lastPhase = phase; } if (msg.role === "user") { contentItems.push({ type: "input_text", text: item.text }); @@ -179,15 +184,14 @@ export class GPT5_5Client extends LLMClient { image_url: item.image_url, }); } else if (item.type === "thinking") { - const signatureStr = item.signature || "{}"; - const signature = JSON.parse(signatureStr); + const fidelity = item.fidelity ?? {}; inputList.push({ type: "reasoning", - id: signature.id, + id: fidelity.id, summary: item.thinking ? [{ type: "summary_text", text: item.thinking }] : [], - encrypted_content: signature.encrypted_content, + encrypted_content: fidelity.encrypted_content, }); } else if (item.type === "tool_call") { inputList.push({ @@ -263,21 +267,21 @@ export class GPT5_5Client extends LLMClient { // adding the following thinking item leads to 400 invalid request error, why? // } else if (item.type === "reasoning") { // eventType = "delta"; - // const signature = { + // const fidelity = { // id: item.id, // encrypted_content: item.encrypted_content, // }; // contentItems.push({ // type: "thinking", // thinking: "", - // signature: JSON.stringify(signature), + // fidelity, // }); } else if (item.type === "message") { // eslint-disable-next-line @typescript-eslint/no-explicit-any const phase = (item as any).phase as string | undefined; if (phase != null) { eventType = "delta"; - contentItems.push({ type: "text", text: "", phase }); + contentItems.push({ type: "text", text: "", fidelity: { phase } }); } else { eventType = "unused"; } @@ -289,14 +293,14 @@ export class GPT5_5Client extends LLMClient { const item = modelOutput.item; if (item.type === "reasoning") { eventType = "delta"; - const signature = { + const fidelity = { id: item.id, encrypted_content: item.encrypted_content, }; contentItems.push({ type: "thinking", thinking: "", - signature: JSON.stringify(signature), + fidelity, }); } else { eventType = "unused"; diff --git a/src_ts/src/kimi_k2_6/client.ts b/src_ts/src/kimi_k2_6/client.ts index 09db7cbe..e565ff3f 100644 --- a/src_ts/src/kimi_k2_6/client.ts +++ b/src_ts/src/kimi_k2_6/client.ts @@ -207,6 +207,7 @@ export class KimiK2_6Client extends LLMClient { // eslint-disable-next-line @typescript-eslint/no-explicit-any const toolCalls: any[] = []; let thinking = ""; + const thinkingFields = new Set(); for (const item of msg.content_items) { if (item.type === "text") { @@ -222,6 +223,7 @@ export class KimiK2_6Client extends LLMClient { }); } else if (item.type === "thinking") { thinking += item.thinking; + thinkingFields.add(item.fidelity?.reasoning_field); } else if (item.type === "tool_call") { toolCalls.push({ id: item.tool_call_id, @@ -283,8 +285,22 @@ export class KimiK2_6Client extends LLMClient { } if (thinking) { - message.reasoning_content = thinking; - message.reasoning = thinking; + // send thinking back through the exact field the upstream produced (recorded + // in the item fidelity); servers may reject the spelling they did not emit + if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning_content") + ) { + message.reasoning_content = thinking; + } else if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning") + ) { + message.reasoning = thinking; + } else { + message.reasoning_content = thinking; // vLLM & siliconflow compatibility + message.reasoning = thinking; // openrouter compatibility + } } if (Object.keys(message).length > 1) { @@ -313,24 +329,31 @@ export class KimiK2_6Client extends LLMClient { contentItems.push({ type: "text", text: delta.content }); } - // vLLM & siliconflow compatibility + // the thinking field name differs by server: vLLM & siliconflow use + // reasoning_content while openrouter uses reasoning; record the wire + // field that carried each delta so a replay can reproduce exactly the + // field the upstream produced // eslint-disable-next-line @typescript-eslint/no-explicit-any - if ((delta as any)?.reasoning_content) { + const reasoningContent = (delta as any)?.reasoning_content; + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const reasoning = (delta as any)?.reasoning; + if (reasoningContent && reasoning) { + eventType = "delta"; + // ambiguous origin: record no fidelity so a replay sends both fields back + contentItems.push({ type: "thinking", thinking: reasoningContent }); + } else if (reasoningContent) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning_content, + thinking: reasoningContent, + fidelity: { reasoning_field: "reasoning_content" }, }); - } - // openrouter compatibility - // eslint-disable-next-line @typescript-eslint/no-explicit-any - else if ((delta as any)?.reasoning) { + } else if (reasoning) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning, + thinking: reasoning, + fidelity: { reasoning_field: "reasoning" }, }); } diff --git a/src_ts/src/openai/client.ts b/src_ts/src/openai/client.ts index 1d0e7fc2..1011f0ec 100644 --- a/src_ts/src/openai/client.ts +++ b/src_ts/src/openai/client.ts @@ -179,6 +179,7 @@ export class OpenaiClient extends LLMClient { // eslint-disable-next-line @typescript-eslint/no-explicit-any const toolCalls: any[] = []; let thinking = ""; + const thinkingFields = new Set(); for (const item of msg.content_items) { if (item.type === "text") { @@ -194,6 +195,7 @@ export class OpenaiClient extends LLMClient { }); } else if (item.type === "thinking") { thinking += item.thinking; + thinkingFields.add(item.fidelity?.reasoning_field); } else if (item.type === "tool_call") { toolCalls.push({ id: item.tool_call_id, @@ -254,8 +256,22 @@ export class OpenaiClient extends LLMClient { } if (thinking) { - message.reasoning_content = thinking; - message.reasoning = thinking; + // send thinking back through the exact field the upstream produced (recorded + // in the item fidelity); servers may reject the spelling they did not emit + if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning_content") + ) { + message.reasoning_content = thinking; + } else if ( + thinkingFields.size === 1 && + thinkingFields.has("reasoning") + ) { + message.reasoning = thinking; + } else { + message.reasoning_content = thinking; // vLLM & siliconflow compatibility + message.reasoning = thinking; // openrouter compatibility + } } if (Object.keys(message).length > 1) { @@ -284,24 +300,31 @@ export class OpenaiClient extends LLMClient { contentItems.push({ type: "text", text: delta.content }); } - // vLLM & siliconflow compatibility + // the thinking field name differs by server: vLLM & siliconflow use + // reasoning_content while openrouter uses reasoning; record the wire + // field that carried each delta so a replay can reproduce exactly the + // field the upstream produced // eslint-disable-next-line @typescript-eslint/no-explicit-any - if ((delta as any)?.reasoning_content) { + const reasoningContent = (delta as any)?.reasoning_content; + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const reasoning = (delta as any)?.reasoning; + if (reasoningContent && reasoning) { + eventType = "delta"; + // ambiguous origin: record no fidelity so a replay sends both fields back + contentItems.push({ type: "thinking", thinking: reasoningContent }); + } else if (reasoningContent) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning_content, + thinking: reasoningContent, + fidelity: { reasoning_field: "reasoning_content" }, }); - } - // openrouter compatibility - // eslint-disable-next-line @typescript-eslint/no-explicit-any - else if ((delta as any)?.reasoning) { + } else if (reasoning) { eventType = "delta"; contentItems.push({ type: "thinking", - // eslint-disable-next-line @typescript-eslint/no-explicit-any - thinking: (delta as any).reasoning, + thinking: reasoning, + fidelity: { reasoning_field: "reasoning" }, }); } diff --git a/src_ts/src/types.ts b/src_ts/src/types.ts index d08a65f1..2bbacac0 100644 --- a/src_ts/src/types.ts +++ b/src_ts/src/types.ts @@ -47,12 +47,19 @@ export type AspectRatio = | "21:9"; export type ImageSize = "1K" | "2K"; +/** + * Arbitrary JSON-style payload of wire-fidelity data recorded by a client, + * such as thinking signatures, phase labels, or the upstream reasoning field + * name. Opaque to consumers: pass it back unchanged so a replay reproduces + * the original wire message. + */ +// eslint-disable-next-line @typescript-eslint/no-explicit-any +export type Fidelity = Record; + export interface TextContentItem { type: "text"; text: string; - phase?: string | null; - // signature is always base64 encode string in typescript - signature?: string; + fidelity?: Fidelity; } export interface ImageContentItem { @@ -64,20 +71,20 @@ export interface InlineDataContentItem { type: "inline_data"; data: Buffer; mime_type: string; - signature?: string; + fidelity?: Fidelity; } export interface ThinkingContentItem { type: "thinking"; thinking: string; - signature?: string; + fidelity?: Fidelity; } export interface InlineThinkingContentItem { type: "inline_thinking"; data: Buffer; mime_type: string; - signature?: string; + fidelity?: Fidelity; } export interface ToolCallContentItem { @@ -86,7 +93,7 @@ export interface ToolCallContentItem { // eslint-disable-next-line @typescript-eslint/no-explicit-any arguments: Record; tool_call_id: string; - signature?: string; + fidelity?: Fidelity; } export interface PartialToolCallContentItem { @@ -94,7 +101,7 @@ export interface PartialToolCallContentItem { name: string; arguments: string; tool_call_id: string; - signature?: string; + fidelity?: Fidelity; } export interface ToolResultContentItem { diff --git a/src_ts/tests/reasoning-fidelity.test.ts b/src_ts/tests/reasoning-fidelity.test.ts new file mode 100644 index 00000000..fb76c6eb --- /dev/null +++ b/src_ts/tests/reasoning-fidelity.test.ts @@ -0,0 +1,275 @@ +// Copyright 2025 Prism Shadow. and/or its affiliates +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { expect, describe, test } from "@jest/globals"; +import { AutoLLMClient, TextContentItem, UniEvent, UniMessage } from "../src"; + +type FakeOpenAICompatibleClient = { + baseURL: string; + chat: { + completions: { + create: () => Promise>; + }; + }; +}; + +interface ReasoningReplayCase { + model: string; + clientType: string; +} + +const REASONING_REPLAY_CASES: ReasoningReplayCase[] = [ + { model: "gpt-5.5", clientType: "openai" }, + { model: "glm-5.1", clientType: "glm-5.1" }, + { model: "kimi-k2.6", clientType: "kimi-k2.6" }, +]; + +function streamFromChunks(chunks: unknown[]): AsyncIterable { + return { + async *[Symbol.asyncIterator]() { + for (const chunk of chunks) { + yield chunk; + } + }, + }; +} + +function installFakeOpenAICompatibleStream( + client: AutoLLMClient, + chunks: unknown[], +): void { + const fakeClient: FakeOpenAICompatibleClient = { + baseURL: "https://api.test.invalid/v1", + chat: { + completions: { + create: async () => streamFromChunks(chunks), + }, + }, + }; + const routedClient = ( + client as unknown as { _client: { _client: FakeOpenAICompatibleClient } } + )._client; + routedClient._client = fakeClient; +} + +function createAutoClient(testCase: ReasoningReplayCase): AutoLLMClient { + return new AutoLLMClient({ + model: testCase.model, + apiKey: "test-key", + clientType: testCase.clientType, + }); +} + +function deltaChunk(delta: { + content?: string; + reasoning_content?: string; + reasoning?: string; +}): unknown { + return { + choices: [{ delta, finish_reason: null }], + usage: null, + }; +} + +function stopChunk(finishReason: string = "stop"): unknown { + return { + choices: [{ delta: {}, finish_reason: finishReason }], + usage: { + prompt_tokens: 1, + completion_tokens: 1, + completion_tokens_details: { reasoning_tokens: 1 }, + prompt_cache_hit_tokens: 0, + prompt_cache_miss_tokens: 1, + }, + }; +} + +function userMessage(): UniMessage { + return { + role: "user", + content_items: [{ type: "text", text: "Create a memo." }], + }; +} + +async function transformHistory( + client: AutoLLMClient, + history: UniMessage[], +): Promise[]> { + return (await client.transformUniMessageToModelInput(history)) as Record< + string, + unknown + >[]; +} + +async function runTurnAndReplay(client: AutoLLMClient): Promise<{ + historyMessage: UniMessage; + replayedMessage: Record; +}> { + const events: UniEvent[] = []; + for await (const event of client.streamingResponseStateful({ + message: userMessage(), + config: {}, + })) { + events.push(event); + } + + const history = client.getHistory(); + const modelInput = await transformHistory(client, history); + const historyMessage = history[history.length - 1]; + const replayedMessage = modelInput[modelInput.length - 1]; + if (!historyMessage || !replayedMessage) { + throw new Error("history or model input is empty"); + } + + return { historyMessage, replayedMessage }; +} + +function thinkingItems(message: UniMessage): unknown[] { + return message.content_items.filter((item) => item.type === "thinking"); +} + +describe.each(REASONING_REPLAY_CASES)( + "Reasoning field fidelity for $clientType", + (testCase) => { + test("replay preserves the reasoning_content field", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + deltaChunk({ reasoning_content: "Let me think" }), + deltaChunk({ reasoning_content: " about the memo." }), + deltaChunk({ content: "Here is the memo." }), + stopChunk(), + ]); + + const { historyMessage, replayedMessage } = + await runTurnAndReplay(client); + expect(thinkingItems(historyMessage)).toEqual([ + { + type: "thinking", + thinking: "Let me think about the memo.", + fidelity: { reasoning_field: "reasoning_content" }, + }, + ]); + expect(replayedMessage.reasoning_content).toBe( + "Let me think about the memo.", + ); + expect(replayedMessage).not.toHaveProperty("reasoning"); + }); + + test("replay preserves the reasoning field", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + deltaChunk({ reasoning: "Let me think" }), + deltaChunk({ reasoning: " about the memo." }), + deltaChunk({ content: "Here is the memo." }), + stopChunk(), + ]); + + const { historyMessage, replayedMessage } = + await runTurnAndReplay(client); + expect(thinkingItems(historyMessage)).toEqual([ + { + type: "thinking", + thinking: "Let me think about the memo.", + fidelity: { reasoning_field: "reasoning" }, + }, + ]); + expect(replayedMessage.reasoning).toBe("Let me think about the memo."); + expect(replayedMessage).not.toHaveProperty("reasoning_content"); + }); + + test("replay keeps both fields when the origin is ambiguous", async () => { + const client = createAutoClient(testCase); + installFakeOpenAICompatibleStream(client, [ + deltaChunk({ + reasoning_content: "Let me think.", + reasoning: "Let me think.", + }), + deltaChunk({ content: "Here is the memo." }), + stopChunk(), + ]); + + const { historyMessage, replayedMessage } = + await runTurnAndReplay(client); + expect(thinkingItems(historyMessage)).toEqual([ + { type: "thinking", thinking: "Let me think." }, + ]); + expect(replayedMessage.reasoning_content).toBe("Let me think."); + expect(replayedMessage.reasoning).toBe("Let me think."); + }); + + test("replay of thinking without fidelity sends both fields", async () => { + const client = createAutoClient(testCase); + const history: UniMessage[] = [ + userMessage(), + { + role: "assistant", + content_items: [ + { type: "thinking", thinking: "Let me think." }, + { type: "text", text: "Here is the memo." }, + ], + }, + ]; + + const modelInput = await transformHistory(client, history); + const replayedMessage = modelInput[modelInput.length - 1]; + if (!replayedMessage) { + throw new Error("model input is empty"); + } + + expect(replayedMessage.reasoning_content).toBe("Let me think."); + expect(replayedMessage.reasoning).toBe("Let me think."); + }); + }, +); + +function textDeltaEvent(text: string, phase?: string): UniEvent { + const item: TextContentItem = { type: "text", text }; + if (phase !== undefined) { + item.fidelity = { phase }; + } + + return { + role: "assistant", + event_type: "delta", + content_items: [item], + usage_metadata: null, + finish_reason: null, + }; +} + +test("concatenation splits text items only on phase change", () => { + const client = createAutoClient(REASONING_REPLAY_CASES[0]); + const message = client.concatUniEventsToUniMessage([ + textDeltaEvent("", "commentary"), + textDeltaEvent("I'll inspect the logs."), + textDeltaEvent("", "final_answer"), + textDeltaEvent("Root cause:"), + textDeltaEvent(" cache invalidation race."), + textDeltaEvent("", "final_answer"), + textDeltaEvent(" Remediation follows."), + ]); + + expect(message.content_items).toEqual([ + { + type: "text", + text: "I'll inspect the logs.", + fidelity: { phase: "commentary" }, + }, + { + type: "text", + text: "Root cause: cache invalidation race. Remediation follows.", + fidelity: { phase: "final_answer" }, + }, + ]); +});