Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions .agents/skills/agenthub-dev/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
name: agenthub-dev
description: Fixed workflow for developing AgentHub itself — adding or updating model support. Use when asked to support a new model or protocol version in this repository, sync llmsdk_docs, or implement a provider client. Covers doc syncing, live API capture, paired Python/TypeScript implementation, and model-scoped e2e testing.
---

# AgentHub Development Workflow

Adding or updating model support follows the stages below, in order. Where a stage says **stop and ask**, pause and ask the user; do not continue until the issue is resolved, and never fill the gap yourself.

## Directory map

```
llmsdk_docs/<model_version>/ Official docs snapshot, one folder per model generation (README.md + docs/)
api_captures/<protocol>/ Git-ignored raw API captures: request payloads + stream events
src_py/agenthub/<protocol>/ Python client, one folder per wire protocol
src_py/agenthub/auto_client.py Routes model names to protocol clients by explicit version
src_ts/src/<protocol>/ TypeScript client, mirrors the Python folder
src_ts/src/autoClient.ts TypeScript routing, mirrors auto_client.py
src_py/tests/test_client.py Parameterized e2e tests (env-gated AVAILABLE_MODELS)
src_ts/tests/client.test.ts Same for TypeScript
changelog/ One detail file per CHANGELOG.md entry
CHANGELOG.md Brief one-line entries linking into changelog/
```

## Stage 1 — Sync official docs into `llmsdk_docs/`

- Sources must be the model vendor's official documentation site (e.g. docs.anthropic.com, platform.openai.com, ai.google.dev). Never use third-party mirrors, blog posts, or model memory.
- Save the snapshot under `llmsdk_docs/<model_version>/` following the existing folder conventions, and list the folder in `llmsdk_docs/README.md`. Running this workflow is the explicit request that the repository rule against editing `llmsdk_docs/` asks for.
- When the fetched docs differ from an existing snapshot, the new official docs win: update the old files in place.
- The snapshot must be complete enough to implement from: request/response schemas, streaming event sequence, thinking output, tool calling, usage fields, and error responses.
- **Stop and ask** if the official URL is uncertain or a page cannot be fetched. The user can paste the content manually.

## Stage 2 — Capture a live API exchange into `api_captures/`

- Gate: the provider's API key environment variable must be set and usable. Use the same environment variables and base URLs as `src_py/tests/test_client.py` (`AVAILABLE_MODELS` gating and `_create_client`). **Stop and ask** the user to supply the key if it is missing; the workflow must not continue without it.
- Using the provider's official SDK, or raw HTTP exactly as documented, run one streaming tool-call request with thinking enabled, then send the tool result back so the capture also shows how assistant turns are re-sent.
- Save the complete exchange unmodified under `api_captures/<protocol>/` (git-ignored), e.g. `round1.request.json` plus `round1.stream.jsonl` with every raw stream event in order. Never save credentials.
- **Stop and ask** on any API error (invalid key, insufficient quota, rate limit). Do not mock the response or continue from docs alone.
- The capture is the primary implementation reference and outranks the docs: where they disagree, implement what the API actually returned.

## Stage 3 — Implement the Python and TypeScript clients

- One folder per wire protocol, named after the newest model generation that uses it. Diff the new protocol (capture + docs) against the closest existing folder:
- Any difference between generations, even a single key name, means a separate folder per generation (e.g. `claude4_6/` vs `claude5/`).
- Only an identical wire protocol may share a folder; name it after the newest generation (rename and reroute if needed). This is how `claude5/` serves Claude 4.7, 4.8, and 5.
- `auto_client.py` / `autoClient.ts` route model names by explicit version matching only, never a bare substring like `"claude" in model`.
- Conversion must be bijective: a wire message converted to `UniMessage`/`UniEvent` and back must reproduce the original exactly, including `fidelity` payloads (thinking signatures, phase labels, reasoning field names) and tool-call IDs. Verify against the captured exchange.
- `UniConfig` keys rarely map one-to-one onto provider config keys. **Stop and ask**: list every non-obvious mapping and confirm it with the user before coding. Never decide silently.
- Implement Python and TypeScript together with identical behavior.

## Stage 4 — Verify

- Register the model in the env-gated `AVAILABLE_MODELS` lists of both test files with correct capability flags. Do not add model-specific test functions or files.
- Static checks: `make lint` in `src_py/`; `npm run lint` and `npm run build` in `src_ts/`.
- Run only the new model's e2e tests; the full suites are slow and spend real API quota:
- `cd src_py && uv run pytest -vvv tests/test_client.py -k "<model-name>"`
- `cd src_ts && npm run test -- -t "<model-name>"`
- Leave unrelated tests to CI.

## Record and ship

- Write `changelog/YYYY-MM-DD-<slug>.md` with the specifics: protocol differences found, config mapping decisions, notable capture findings.
- Add one brief line at the top of `CHANGELOG.md` linking to that file. The root file keeps a single line per change.
- Commit on a feature branch and open a PR with `gh pr create --base dev`; direct pushes to `dev` are rejected.
1 change: 1 addition & 0 deletions .claude
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,9 @@ vite.config.js.timestamp-*
vite.config.ts.timestamp-*
.vite/

# Raw LLM API captures used as implementation reference (see .agents/skills/agenthub-dev)
api_captures/

# Conversation monitor cache
cache/
generated_image*
Expand Down
10 changes: 9 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
# Changelog

Here, we record the addition and removal times of models, major functional updates, bug fixes, and release times of key versions.
Here, we record the addition and removal times of models, major functional updates, bug fixes, and release times of key versions. Each entry keeps one brief line; the full details of a change live in a file under [changelog/](changelog/).

- [2026-07-20] Release version 0.4.0. Content items now carry an opaque `fidelity` payload that absorbs the former `signature`/`phase` fields (breaking), and OpenAI-compatible clients use it to replay thinking through exactly the reasoning field the upstream produced. ([details](changelog/2026-07-20-reasoning-field-fidelity.md))

- [2026-07-17] Add the `agenthub-dev` skill that fixes the model-support development workflow, and the `changelog/` details directory. ([details](changelog/2026-07-17-agenthub-dev-skill.md))

- [2026-07-14] Raise `EmptyResponseError` when a model completes a response with thinking output only, since sending it back would fail with a 400 error. It and `ToolCallArgumentParseError` now inherit the new `AgentHubError` base class.

- [2026-06-10] Support Claude 5 models.

- [2026-06-01] Release version 0.3.3. Support OpenAI-compatible embedding input format.

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ https://github.com/user-attachments/assets/c49a21a1-5bf9-4768-a76d-f73c9a03ca87
| Model Name | Vendor | Example Model ID | Input Modalities | Output Modalities |
| -------------- | ----------------------------------- | ---------------------- | ---------------- | ------------------------------ |
| Gemini 3-3.5 | Official/Google Vertex AI | `gemini-3.5-flash` | Text, Image | Text, Image, Speech, Embedding |
| Claude 4.6-4.8 | Official/Amazon Bedrock/UModelVerse | `claude-opus-4-8` | Text, Image | Text |
| Claude 4.6-5 | Official/Amazon Bedrock/UModelVerse | `claude-opus-4-8` | Text, Image | Text |
| GPT-5.4/5.5 | Official/UModelVerse | `gpt-5.5` | Text, Image | Text, Embedding |
| Kimi-K2.5/K2.6 | Official/OpenRouter/SiliconFlow | `kimi-k2.6` | Text, Image | Text |
| DeepSeek V4 | Official/OpenRouter/SiliconFlow | `deepseek-v4-pro` | Text | Text |
Expand Down Expand Up @@ -433,7 +433,7 @@ Example UniMessage:
{"type": "text", "text": "How are you doing?"},
{"type": "image_url", "image_url": "https://example.com/image.jpg"},
{"type": "inline_data", "mime_type": "image/jpeg", "data": "base64-encoded-image"},
{"type": "thinking", "thinking": "I am thinking.", "signature": "0x123456"},
{"type": "thinking", "thinking": "I am thinking.", "fidelity": {"signature": "0x123456"}},
{"type": "inline_thinking", "mime_type": "image/jpeg", "data": "base64-encoded-image"},
{"type": "tool_call", "name": "math", "arguments": {"expression": "2 + 3"}, "tool_call_id": "123"},
{"type": "tool_result", "text": "2 + 3 = 5", "images": [], "tool_call_id": "123"}
Expand Down
6 changes: 6 additions & 0 deletions changelog/2026-07-17-agenthub-dev-skill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Add the agenthub-dev skill and the changelog details directory

- Added `.agents/skills/agenthub-dev/SKILL.md`, fixing the model-support development workflow: sync official docs into `llmsdk_docs/`, capture a live streaming tool-call exchange with thinking into the git-ignored `api_captures/`, implement paired Python/TypeScript protocol clients with bijective message conversion, and verify with model-scoped e2e tests only.
- The workflow makes four situations hard stops that require asking the user: unclear or unfetchable official docs, a missing provider API key, any live API request error, and non-obvious `UniConfig` key mappings.
- Added `api_captures/` to `.gitignore` as the home for raw API captures; where docs and captures disagree, the capture wins.
- Added the `changelog/` directory: each `CHANGELOG.md` entry keeps one brief line and links to a detail file here (see `changelog/README.md`).
39 changes: 39 additions & 0 deletions changelog/2026-07-20-reasoning-field-fidelity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Add the `fidelity` field and replay the exact reasoning field the upstream produced

## The bug

OpenAI Chat Completions-compatible servers spell the streamed thinking field differently — vLLM & SiliconFlow use `reasoning_content` while OpenRouter uses `reasoning` — and when sending assistant history back, the `openai`, `glm5_1`, and `kimi_k2_6` clients always set **both** fields on the message. Strict upstreams reject the spelling they did not emit (e.g. a server that returned `reasoning_content` refuses a request containing `reasoning`), breaking multi-turn conversations.

## The `fidelity` field

Fixing this needs a place to record which wire field carried the thinking. Rather than overloading `signature`, content items now carry a single dedicated field:

- `fidelity` (`dict[str, Any]` / `Record<string, any>`, optional) — an arbitrary JSON-style object of wire-level data a client records to reproduce the original message on replay. Opaque to consumers: pass it back unchanged.

It replaces and absorbs the former item-level `signature` and `phase` fields:

| Client | Old | New |
| --- | --- | --- |
| `claude5` / `claude4_6` | `signature: <sig>` on thinking (also holds redacted-thinking data) | `fidelity: {"signature": <sig>}` |
| `gemini3` | `signature: <thought_signature>` on text / thinking / inline / tool_call items (key present even when `None`) | `fidelity: {"signature": <thought_signature>}`, omitted entirely when absent |
| `gpt5_5` | `signature: json.dumps({"id": ..., "encrypted_content": ...})` on thinking; `phase: <p>` on text | `fidelity: {"id": ..., "encrypted_content": ...}` (no more JSON-in-a-string); `fidelity: {"phase": <p>}` |
| `openai` / `glm5_1` / `kimi_k2_6` / `deepseek_v4` | nothing recorded; both reasoning spellings sent back | `fidelity: {"reasoning_field": "reasoning_content" \| "reasoning"}` per thinking delta |

## The reasoning-field fix

On receive, the OpenAI-compatible clients record the wire field name that carried each thinking delta. On send, the message conversion replays the thinking through exactly that field. Fallbacks keep the old maximum-compatibility behavior: thinking without a recorded `reasoning_field` (hand-written histories, foreign-protocol fidelity), mixed fields within one message, and the ambiguous case where one chunk carries both spellings (such deltas record no fidelity) all still send both fields.

## Concatenation rules

`concat_uni_events_to_uni_message` / `concatUniEventsToUniMessage` now key on `fidelity`:

- text: a phase change starts a new item (same-phase and phaseless deltas merge, per the GPT-5.5 `phase` guide); an incoming fidelity payload (e.g. a signature) merges into the open item's fidelity and finishes it.
- thinking: an incoming fidelity payload finishes the open item (Claude signature deltas, GPT-5.5 reasoning markers), and a run of deltas carrying **equal** fidelity concatenates into one item (the OpenAI-compatible per-delta `reasoning_field` tags).

## Breaking change

Histories recorded by earlier versions carry `signature` / `phase` at the top level of content items; clients no longer read those fields. To replay an old history, move each item's `signature`/`phase` into `fidelity` (`{"signature": ...}` for Claude/Gemini, `{"id": ..., "encrypted_content": ...}` parsed from the GPT-5.5 JSON string, `{"phase": ...}` for GPT-5.5 text).

## Tests

Offline fake-stream suites `src_py/tests/test_reasoning_fidelity.py` and `src_ts/tests/reasoning-fidelity.test.ts` cover both reasoning field spellings, the ambiguous both-fields case, and the no-fidelity fallback across the `openai`, `glm5_1`, and `kimi_k2_6` clients.
7 changes: 7 additions & 0 deletions changelog/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Changelog Details

One file per entry in [../CHANGELOG.md](../CHANGELOG.md). The root file keeps a single brief line per change; the full story lives here.

- File name: `YYYY-MM-DD-short-slug.md`, matching the entry date.
- Content: what changed and why, affected modules, and decisions worth keeping (protocol differences, config mappings, migration notes).
- Link the root entry to its file: `- [YYYY-MM-DD] Brief description. ([details](changelog/YYYY-MM-DD-short-slug.md))`
10 changes: 10 additions & 0 deletions llmsdk_docs/claude5/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Claude 5 SDK Documentation

This directory contains documentation for using Anthropic's Claude 5 API.

## Documentation

The `docs/` folder contains detailed guides on various Claude 5 features:

- [introducing-claude-fable-5-and-claude-mythos-5.md](./docs/introducing-claude-fable-5-and-claude-mythos-5.md) - What's new in Claude 5
- [migration-guide.md](./docs/migration-guide.md) - Migration guide from Claude 4.8 to Claude 5
Loading
Loading