From 83ca198d47dd23c9fd507639898d5dffa1ecdf5b Mon Sep 17 00:00:00 2001 From: hamr0 Date: Tue, 21 Jul 2026 13:37:15 +0200 Subject: [PATCH 1/4] =?UTF-8?q?feat:=20CLIPipe=20native=20MCP=20tool=20mod?= =?UTF-8?q?e=20=E2=80=94=20cheaper=20agentic=20Loop=20over=20a=20CLI=20sub?= =?UTF-8?q?scription=20(BA-16)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The claude CLI has a real tool channel; v0.32.0's envelope emulation was built as if it did not. `toolProtocol:'claude-mcp'` runs ONE CLI session per call and exposes the caller's tools to it as an MCP server whose handlers call back into the caller's in-process closures over a unix-socket bridge. The CLI owns the inner cycle and caches its transcript session-side. The claim is COST, measured: emulation re-sends the whole transcript every round ($0.25-0.55/round on a real ~40-round job transcript) vs a native session that caches session-side (~$0.006/turn). Output-quality parity is deliberately UNMINTED (n=2 suggestive evidence) — not sold as a capability win (BA-7 precedent). Emulation is retained, not retired — still right for a CLI with no MCP support. The CLI owning the turns means the Loop's per-round machinery can't run on them, so governance moves to the `tools/call` bridge (every call crosses it): - the SAME `policy(tool,args,ctx)` chokepoint (identical audit-row shape, zero gate changes); a deny is an advisory tool result, the handler never runs - BA-11 deny-streak + BA-12 identical-tool-error guards, same 3/3 narrowest triggers - the turn bound via `--max-turns`, NAMED stop `error:'max_turns'`, never a silent success - `onTurn` STREAMS per-turn usage (four cache tiers) as it arrives then one closing session event with the authoritative cost; the Loop skips its own forward when wired (billed once, never starved) Honest accounting (the BA-4/5/6/13 optimistic-rounding class, caught pre-build): a dead bridge still ends `subtype:'success'`, so bridge health is tracked parent-side (attempted vs served tool calls) and error-tags the run `bridge-failed`. `Provider.ownsCycle` makes the Loop THROW at construction on any option it could never honor — assemble/trim/cacheMessages and a Loop-level policy (a fence silently not there) — no silently-dead knobs. `GenerateResult.session` + `metrics.sessionTurns` report the real turn/tool count so a 14-turn session never reads as one round. The bridge is a unix socket (0600 in a 0700 dir), never a listening port. The fence (`--tools '' --strict-mcp-config --setting-sources ''`) is set by the mode, not the caller. Diagnostics from unknown-shape throws are allowlist-clamped (F16/BA-1 audit-leak class). +47 offline tests, every load-bearing guard mutation-proved in both directions (remove it -> red; make it always-fire -> also red). Live verify-shipped through a real Loop on a real session: poc/ba16-native-shipped.mjs (all green). Suite 963 tests / 961 pass / 0 fail / 2 skipped; typecheck clean. Note: `--max-turns` is undocumented in the claude CLI's --help (2.1.216) though accepted — the live check is the only tripwire if it is ever renamed. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MN6WmkWeLudzKB7PNsQT92 --- CHANGELOG.md | 24 ++ CLAUDE.md | 2 +- bareagent.context.md | 26 +- poc/ba16-fence-check.mjs | 60 ++++ poc/ba16-native-shipped.mjs | 139 ++++++++++ src/loop.js | 58 +++- src/mcp-bridge-stub.js | 107 +++++++ src/provider-clipipe-mcp.js | 446 ++++++++++++++++++++++++++++++ src/provider-clipipe.js | 210 +++++++++++++- test/provider-clipipe-mcp.test.js | 414 +++++++++++++++++++++++++++ types/index.d.ts | 40 +++ 11 files changed, 1514 insertions(+), 12 deletions(-) create mode 100644 poc/ba16-fence-check.mjs create mode 100644 poc/ba16-native-shipped.mjs create mode 100644 src/mcp-bridge-stub.js create mode 100644 src/provider-clipipe-mcp.js create mode 100644 test/provider-clipipe-mcp.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 358b005..5a46624 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,30 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog](https://keepachangelog.com/). Versioning: [SemVer](https://semver.org/). +## [Unreleased] + +### Added + +- **CLIPipe NATIVE tool mode (BA-16) — `toolProtocol:'claude-mcp'`.** The claude CLI has a real tool channel; v0.32.0's envelope emulation was built as if it did not. Native mode runs **one CLI session per call** and exposes the caller's `tools` to it as an **MCP server whose handlers call back into your own in-process closures** over a unix-socket bridge. The CLI owns the inner cycle and caches its transcript session-side. + + **The claim is COST, and it is measured.** Emulation re-spawns the CLI and re-sends the whole rendered transcript every round, so it pays fresh `cache_creation` on the full prefix: the adopter measured **$0.25–0.55/round** on a real ~40-round job transcript, against **$0.0055–0.0074/turn** native (reproduced independently here). **NOT CLAIMED: better output quality** — there is n=2 suggestive evidence that emulation's JSON-questionnaire framing makes a model act less, it is deliberately **unminted**, and it must not be sold as a capability win (the BA-7 precedent). Emulation is **retained**, not retired: it is still the right instrument for a CLI with no MCP support. Native is the documented default for the claude CLI. + + **What the Loop gives up, and what is bought back.** The CLI owns the turns, so the Loop's per-round machinery cannot run on them. That is stated rather than papered over, and everything load-bearing is re-established at the `tools/call` bridge — the one seam every call crosses: + - **The gate** rides the SAME `policy(tool, args, ctx)` chokepoint the Loop uses, so a wired bareguard writes **audit rows of identical shape with zero gate changes**. A deny is returned as a tool RESULT (advisory, allowlist-safe pivoting preserved); the handler never runs. + - **BA-11 deny-streak** and **BA-12 identical-tool-error** guards, same defaults (3/3) and same narrowest triggers — a byte-identical repeat only, so a model varying args while recovering is never punished. Both end the session with `denied:` / `stuck:`. The unknown/hallucinated-tool path feeds the same counter. + - **The turn bound** maps to the CLI's `--max-turns`, and its stop is NAMED and error-tagged (`error_max_turns` → `error:'max_turns'`), never a silent clean success. + - **The fence is set by the mode, not the caller:** `--tools '' --strict-mcp-config --setting-sources ''` always. The bridge is a **unix socket (0600, in a 0700 dir)** — never a listening TCP port, even on loopback. + + **Genuinely lost, and made loud instead of silent:** `assemble`/`trim` and `cacheMessages` cannot apply (the provider owns the transcript), and a Loop-level `policy` would be **a fence that is silently not there**. All of them now **THROW at construction** rather than sit dead — no silently-dead knobs. + + **Honest accounting.** `GenerateResult.session` carries the real `turns`/`toolCalls` and any internal terminal, and `metrics.sessionTurns` reports them — so a 14-turn session can never read as one cheap round. `onTurn` **streams** each completed turn's usage with all four cache tiers as it arrives (never summed at end, so a session that dies mid-run has already surfaced its spend), then one closing `kind:'session'` event carries the authoritative cost; when it is wired the Loop skips its own forward, so a session is **billed exactly once and never starved**. + + **A pre-build measurement changed the design:** with the bridge dead, every tool call failed and the CLI **still ended `subtype:'success'`** — the model writes a tidy final answer explaining that its tools were broken. Mapping that onto `error:null` would report a run in which nothing worked as converged (the BA-4/5/6/13 optimistic-rounding class). So bridge health is tracked **parent-side** — MCP tool calls the CLI *attempted* vs the bridge actually *served* — and error-tags the run as `bridge-failed` regardless of the CLI's own subtype. **The CLI's subtype is not a sufficient success signal.** + + +47 offline tests, every load-bearing guard mutation-proved in both directions (removing it goes red; making it fire always also goes red, so the negative controls are real). Live verify-shipped through a real `Loop` on a real session: `poc/ba16-native-shipped.mjs` (all green). Suite 963 tests / 961 pass / 0 fail / 2 skipped; typecheck clean. + + *Note: `--max-turns` is **undocumented** in the claude CLI's `--help` (zero hits at 2.1.216, while unknown flags are rejected — so it exists but is version-fragile). The live check is the only tripwire that goes red if it is ever renamed.* + ## [0.32.0] - 2026-07-21 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index a7d1725..fd4a44a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,7 +32,7 @@ Node.js >= 18, pure JS + JSDoc, `node:test` for testing. Flat `src/` layout with | Defer | tools/defer.js | Append `{id, action, when}` to JSONL queue for an external waker (cron + `examples/wake.sh`) to fire later. Two-phase gov: emit-time gate.check on `defer` action; fire-time gate.check on inner action. bareguard 0.2+ caps via `defer.ratePerMinute` | Providers: OpenAI, Anthropic, Gemini, Ollama, CLIPipe, Fallback -- each in `src/provider-*.js`. All normalize `stopReason` (BA-6) to one neutral vocabulary via `src/provider-stop-reason.js` (`end_turn`/`max_tokens`/`tool_use`/`stop_sequence`/`refusal`/`pause_turn`/`context_exceeded`; unknown-or-absent ⇒ `null` ⇒ pre-BA-6 behavior, so an unmapped provider degrades to the status quo, never to a false truncation). All 5 verified live. `tool_use` is DERIVED where the API can't express it: Gemini/Ollama return `STOP`/`stop` on a complete tool call (no tool-call finish reason), so `normalizeStopReason(raw, provider, {hasToolCalls})` promotes `end_turn`→`tool_use` when the round carried a call — narrow by design (only `end_turn`; a truncated `max_tokens` round stays truncated, preserving BA-4's refusal; never touches refusal/pause/context/passthrough). All normalize `usage` to one neutral shape incl. prompt-cache tiers (`cacheReadTokens`/`cacheCreationTokens`; `inputTokens` = uncached remainder). Gemini is native `generateContent` (its OpenAI-compat endpoint drops the cache tier). Anthropic has opt-in `cacheSystem` (cache_control on the system prompt — weak on its own: the 1024–4096-tok model-dependent cache minimum means a short persona silently never caches) + **opt-in `cacheMessages` (BA-1)** = a rolling `cache_control` breakpoint on the LAST content block of the LAST message, so a tool loop stops re-buying its whole transcript at full price every round (measured: $0.0753→$0.0110/round, 6.8× steady-state; the 1.25× write is paid once). It MUST live in the provider — the transcript ends on a `tool_result` that `_toAnthropicMessage` rebuilds, so no caller seam (`assemble` included) can reach it. Copy-on-write (never mutate the caller's array — a stale mark accumulates; Anthropic allows ≤4). Caching pays for RE-SENDING, not GROWING; a destructive `trim` fold that rewrites the PREFIX invalidates it (fold the middle, keep the head). **Provider-native block passthrough (BA-7):** `Message.providerBlocks`/`GenerateResult.providerBlocks` = `{provider, model, blocks}` — the transcript field for content blocks the normalized `{text, toolCalls}` shape CANNOT express (Anthropic `thinking`/`redacted_thinking`), which the OpenAI-shaped `Message` had nowhere to put and so silently DROPPED (the API returns 200 either way — the loss never surfaced). Anthropic echoes thinking back unchanged (signature included) on a tool-use continuation; the provider now replays them at the FRONT of the assistant turn (thinking must lead). Three constraints: OPAQUE (keep bytes, never re-serialize from parsed fields — a `redacted_thinking` block can't survive it, and a `type==='thinking'` check would silently drop the next new block type = the same bug again); the provider+model TAG is enforced on replay (a signature is model-bound; mismatch ⇒ drop, degrade to the lossy-but-valid pre-BA-7 request, never a 400); and normalized `content`/`tool_calls` stay the SOURCE OF TRUTH (only unexpressible blocks ride along, so an `assemble`/`trim` seam that rewrites a message isn't silently undone by a stale cached copy of its text). Adaptive thinking is ALREADY DEFAULT on sonnet-5/Opus 4.7+ (measured: blocks on ~3/10 rounds with the param OMITTED; sending `thinking:{type:'adaptive'}` changed the rate not at all) — so the opt-in `thinking` option (forwarded to `body.thinking` VERBATIM/unvalidated, since `budget_tokens` already died and a reshaping library would need a release per API move) does NOT enable thinking; it pins the mode / reaches `display`/`effort`. **PROTOCOL fix, NOT a capability fix — the adopter's head-to-head found NO outcome difference; never re-sell it as one.** + `baseUrl`. Loop returns `result.metrics` (the meter: cumulative 4-tier tokens, byTool, costUsd null-not-zero, unpricedRounds, spawned, context.{compactions,summaries,tokensTrimmed}, memory.{stashed,episodes,recalls,stored,facts}) — `estimateCost` prices the 4 tiers separately. The §3.6 CE rollup: `context.tokensTrimmed` is an APPROXIMATE (~4 chars/token) estimate over the evicted span (no exact provider count); `memory.*` is reported via the loop-lent `ctx.recordMemoryOp` hook (channel A — the originating module announces, the loop counts + emits `loop:memory`), bounded PER RUN — `stashed`/`episodes` flow through the stash fold; the Memory wrapper is metered symmetrically opt-in via ctx: `recalls` (`Memory.search` reads) + `stored` (`Memory.store` writes; ctx in a trailing opts arg, never in persisted metadata). `memory.facts` is written by `remember` (the F5 consolidation pass) via the Store socket — disjoint from `stored`; litectx's own episode→fact promotion is a separate, litectx-internal op (§3.7) -CLIPipe tool mode (v0.32.0, `src/provider-clipipe-tools.js`): `CLIPipeProvider({ toolProtocol:'claude' })` lets a CLI **subscription** drive a full agentic Loop (tools + turns), not just one-shot text — the point being run bareagent/bareloop off a Claude sub with no metered-API spend. **Option C, NOT an MCP callback:** the caller's tools are described in the system prompt, the CLI is constrained to a JSON envelope (`--json-schema`), and the envelope is parsed back into normalized `toolCalls` — so **the Loop keeps ownership of the cycle** (round accounting, spin guards, stop-reason classification all still apply; an MCP callback would hand turns to the CLI and lose them). CLI reduced to a bare brain via `--tools '' --strict-mcp-config` + `--setting-sources ''` (the last a MEASURED ~18× cost drop — the CLI else auto-loads the cwd's `CLAUDE.md`/memory every turn). **Loud-failure by design (the BA-6 family applied to a NEW provider):** a weak model that answers in prose instead of calling tools is caught UPFRONT by a capability probe (default on, behaviour-based not a model-list per BA-10; the probe must mirror real-task shape — a trivial "call this tool" instruction is a false positive haiku passes) → documented model floor (sonnet-class+ for tools; haiku fine for plain text); a malformed envelope is a loud `ProviderError`, never returned as prose; tools passed with no `toolProtocol` stay plain-text (backward compat) with a one-time warn. Claude-only for now; the CLI-specific flags/schema/parse/probe are isolated so a second CLI (codex/gemini) slots in behind the same seam. +CLIPipe tool mode has TWO shapes, and the choice between them is COST, not capability. **EMULATION (v0.32.0, `src/provider-clipipe-tools.js`, `toolProtocol:'claude'`):** the caller's tools are described in the system prompt, the CLI is constrained to a JSON envelope (`--json-schema`), the envelope is parsed back into `toolCalls`, and **the Loop keeps ownership of the cycle** (round accounting, spin guards, stop-reason classification apply per round). It re-sends the whole transcript every round — right for a CLI with NO MCP channel, WRONG for the claude CLI on cost. CLI reduced to a bare brain via `--tools '' --strict-mcp-config` + `--setting-sources ''` (MEASURED ~18× cut). Loud-failure by design: a weak model answering in prose is caught UPFRONT by a capability probe (behaviour-based, mirrors real-task shape, per BA-10) → model floor (sonnet-class+ for tools); a malformed envelope is a loud `ProviderError`; tools with no `toolProtocol` stay plain-text with a one-time warn. **NATIVE (BA-16, `src/provider-clipipe-mcp.js` + `src/mcp-bridge-stub.js`, `toolProtocol:'claude-mcp'` — the default for the claude CLI):** the CLI runs its OWN multi-turn session per `generate()` and executes the caller's tools natively over a unix-socket MCP bridge that calls back into the caller's in-process closures. The CLI owns the inner cycle — so the Loop's per-round machinery cannot run and the governance moves to the PROVIDER at the `tools/call` bridge (the one seam every call crosses): the SAME `policy` chokepoint (identical audit-row shape, zero gate changes), BA-11 deny-streak + BA-12 identical-error guards (same 3/3 narrowest triggers), and the turn bound via `--max-turns` (NAMED stop `error:'max_turns'`). `onTurn` STREAMS per-turn usage with all four cache tiers (never sum-at-end; a mid-run death has already surfaced its spend) then one closing `session` event with the authoritative cost — and when wired the Loop skips its own forward (billed once, never starved). **Provider owns its cycle (`Provider.ownsCycle`):** the Loop THROWS at construction on any option it could never honor — `assemble`/`trim`/`cacheMessages` and a Loop-level `policy` (which would be a fence silently not there) — no silently-dead knobs. `GenerateResult.session` + `metrics.sessionTurns` report the REAL turn/tool count (a 14-turn session never reads as one round); an internal terminal (bound, guard, or a **broken bridge** — a dead bridge still ends `subtype:'success'`, so it is caught parent-side by attempted-vs-served tool calls) surfaces as the run's `error`, never a laundered clean finish. **Cost is the claim (measured $0.25–0.55/round emulation vs ~$0.006/turn native); output-quality parity is deliberately UNMINTED (n=2).** Claude-only for now; both shapes isolated so a second CLI slots in behind the same seams. Stores: SQLiteStore (peer dep: better-sqlite3), JsonFileStore (zero deps) -- each in `src/store-*.js` Tools: BrowsingTools (tools/browse.js, optional dep: barebrowse) — library tools for inline snapshots, CLI session (`npx barebrowse`) for token-efficient disk-based browsing diff --git a/bareagent.context.md b/bareagent.context.md index 93d3263..229029e 100644 --- a/bareagent.context.md +++ b/bareagent.context.md @@ -60,7 +60,8 @@ Eight entry points: | Cache identical planner calls | Planner({ cacheTTL: 60000 }) | | Stream CLIPipe output in real-time | CLIPipeProvider({ onChunk: fn }) | | Get real usage + cost from a CLI provider | CLIPipeProvider({ parse: 'claude-json' }) | -| Drive tools over a CLI subscription (no metered API) | CLIPipeProvider({ toolProtocol: 'claude' }) | +| Drive tools over a CLI subscription — NATIVE, cheapest (no metered API) | CLIPipeProvider({ toolProtocol: 'claude-mcp', policy }) | +| Drive tools over a CLI with no MCP support (emulation) | CLIPipeProvider({ toolProtocol: 'claude' }) | | Browse the web (inline snapshots) | createBrowsingTools + Loop | | Browse the web (token-efficient, disk-based) | `barebrowse` CLI session — snapshots to `.barebrowse/*.yml` | | Assess website privacy risk | createBrowsingTools + Loop (requires `npm install wearehere`) | @@ -801,14 +802,27 @@ new CLIPipe({ command: 'claude', args: ['--print'], systemPromptFlag: '--system- new CLIPipe({ command: 'ollama', args: ['run', 'llama3.2'] }) // CLIPipe structured output (v0.26.0+) — map a CLI's JSON envelope to real usage + cost new CLIPipe({ command: 'claude', args: ['-p', '--output-format', 'json'], parse: 'claude-json' }) -// CLIPipe TOOL MODE (v0.32.0+) — drive a Loop's tools over a CLI SUBSCRIPTION (no metered API). -// toolProtocol enables schema-validated tool emulation; the Loop keeps governance/round-accounting. -// Tools are auto-used when passed; a weak model (e.g. haiku) is rejected UPFRONT by a capability -// probe (needs sonnet-class+ for tools; haiku is fine for plain text). setting-sources '' is applied -// internally for an ~18x cost drop. Claude-only for now. +// CLIPipe TOOL MODE — drive a Loop's tools over a CLI SUBSCRIPTION (no metered API). TWO modes: +// 'claude-mcp' (BA-16, NATIVE — prefer on the claude CLI) vs 'claude' (v0.32.0, EMULATION). The +// difference is COST, not capability. Emulation re-sends the whole transcript every round +// ($0.25-0.55/round measured); native runs one CLI session that caches session-side (~$0.006/turn). +// Native: caller tools ride an MCP bridge back to your in-process closures; the CLI owns the cycle. +// Gate + BA-11/BA-12 guards + turn bound live on the PROVIDER (not the Loop) — see below. +new CLIPipe({ command: 'claude', args: ['-p', '--model', 'sonnet'], toolProtocol: 'claude-mcp', policy, onTurn, maxTurns: 20 }) +// Emulation (v0.32.0) — still right for a CLI with NO MCP support. Weak models rejected upfront by a +// capability probe (needs sonnet-class+; haiku fine for plain text). setting-sources '' cuts ~18x cost. new CLIPipe({ command: 'claude', args: ['-p', '--model', 'sonnet'], toolProtocol: 'claude' }) ``` +**CLIPipe NATIVE tool mode (BA-16, `toolProtocol:'claude-mcp'`).** The claude CLI has a real tool channel; native mode uses it instead of emulating one. The CLI runs its OWN multi-turn session per `generate()` call and executes your `tools` natively over an MCP bridge that calls back into your in-process `execute` closures. Because the CLI owns the inner cycle, the Loop's per-round machinery cannot run — so the governance you'd wire on `Loop` moves to the **provider**, at the one seam every tool call crosses: +- `policy` — the SAME `(tool, args, ctx) => true|string` chokepoint as `Loop({policy})`, so a wired `wireGate(gate).policy` writes **audit rows of identical shape, zero gate changes**. A deny is a tool result (advisory); the handler never runs. **Required here** — a `Loop({policy})` in native mode would be a fence that is silently not there, so the Loop throws. +- `maxConsecutiveDenials` (3) / `maxIdenticalToolErrors` (3) — BA-11/BA-12 guards at the bridge, same narrowest triggers; end the session `denied:` / `stuck:`. +- `maxTurns` — maps to the CLI's `--max-turns`; the bound stop is `error:'max_turns'`, never a silent success. +- `onTurn` — streams each completed turn's usage (four cache tiers, `costUsd:null` — the CLI prices the session) as it arrives, then one closing `kind:'session'` event with the authoritative total. Shape mirrors `onLlmResult`, so `wireGate(gate).onLlmResult` drops in; when wired the Loop skips its own forward (billed once, never starved). +- `sessionTimeout` (600s) / `bridgeTimeoutMs` (120s) — whole-session and per-handler ceilings. + +`GenerateResult.session` (`{turns, toolCalls, error, usageReported}`) carries what really happened; `metrics.sessionTurns` reports the real turn count so a 14-turn session never reads as one round. A terminal the CLI detects inside the session (bound, guard, or a **broken tool bridge** — a dead bridge still ends `subtype:'success'`, so it is caught parent-side by attempted-vs-served tool calls) surfaces as the run's `error`, never a laundered clean finish. `assemble`/`trim`/`cacheMessages` and a Loop-level `policy` all THROW at construction in native mode (no silently-dead knobs — the CLI owns the transcript). The bridge is a unix socket (0600 in a 0700 dir), never a listening port. Claude-only for now; the CLI-specific parts live in `src/provider-clipipe-mcp.js` + `src/mcp-bridge-stub.js` behind the same seam as emulation. + All return `{ text, toolCalls, usage: { inputTokens, outputTokens }, model?, costUsd? }`. The optional `model` (v0.16.1+) is the id the response was produced by — Loop prefers it over `provider.model` for cost accounting. By default CLIPipe returns `toolCalls: []` and zero usage (CLI tools don't report tokens) and omits `model`. **Structured output (v0.26.0+):** set `parse: 'claude-json'` (a preset for `claude -p --output-format json`) — or a `(stdout) => Partial` function for any other CLI — and CLIPipe maps the CLI's JSON envelope onto real `usage`, `model`, and `costUsd`, throwing `ProviderError` on a malformed/error envelope (never a silent raw-text fall-back). `costUsd` (optional `GenerateResult` field) is an **authoritative** per-call price the provider reports itself; when finite the Loop prefers it over the internal rate-table `estimateCost`, so a CLI-piped run enforces a bareguard USD cap with no local pricing table (a `0` counts as priced, distinct from null/unpriced). `toolCalls` stays `[]` regardless (CLIPipe is tool-free). **Temperature graceful degradation (BA-10).** Newer models reject ANY non-default `temperature` with a `400` (`claude-sonnet-5`: `` `temperature` is deprecated for this model. ``; OpenAI o1/gpt-5-class: `Unsupported value: 'temperature' … Only the default (1) …`). All four providers detect that specific 400 (message names `temperature` as unsupported/deprecated AND a temperature was sent), **drop the param, warn once per instance, and retry once** — so a call that would otherwise throw succeeds at the model's default temperature. Keyed off the API error text, not a model list. A genuine out-of-range 400 is NOT degraded (it re-throws — dropping it would mask a caller bug). When a drop happens the result carries `temperatureDropped: true` (an optional `GenerateResult`/`Loop.run` field) so a caller can report the effective temperature — `recurse`'s `refineLeaf` uses it for an honest receipt. Dormant on models that accept temperature (byte-identical to before). diff --git a/poc/ba16-fence-check.mjs b/poc/ba16-fence-check.mjs new file mode 100644 index 0000000..fe06b94 --- /dev/null +++ b/poc/ba16-fence-check.mjs @@ -0,0 +1,60 @@ +// BA-16 acceptance criterion 3 (the one gap): "With tools stripped, a shell-out request produces +// NO execution — assert no side effect, not just refusal prose." +// +// The fence is `--tools '' --strict-mcp-config`, set by the mode itself. If it holds, a model told +// to create a file via a shell command cannot: the CLI's own Bash/Write tools are stripped, and the +// only tools present are the caller's MCP tools (which include NO shell). The proof is the ABSENCE +// of the sentinel file the shell command would have created — never the model's prose. + +import { createRequire } from 'node:module'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +const require = createRequire(import.meta.url); +const { Loop } = require('../src/loop'); +const { CLIPipeProvider } = require('../src/provider-clipipe'); + +let failures = 0; +const check = (label, cond, detail = '') => { + if (!cond) failures++; + console.log(` [${cond ? 'PASS' : 'FAIL'}] ${label}${detail ? ` — ${detail}` : ''}`); +}; + +// A sentinel a shell `touch`/`echo >` would create if the CLI's shell were reachable. +const sentinel = path.join(os.tmpdir(), `ba16-fence-sentinel-${process.pid}-${Date.now()}.txt`); +try { fs.unlinkSync(sentinel); } catch { /* absent, as intended */ } + +// The caller mounts ONE harmless tool — so the model has a legitimate MCP tool present, and the +// test is specifically that it still cannot reach a SHELL despite the fence, not that it has nothing. +const calls = []; +const tools = [{ + name: 'noop', + description: 'A no-op that returns ok.', + parameters: { type: 'object', properties: {} }, + execute: async () => { calls.push('noop'); return 'ok'; }, +}]; + +console.log('\n=== BA-16 criterion 3: shell-out is stripped, no side effect ==='); +console.log(` sentinel (must NOT exist afterward): ${sentinel}`); + +const provider = new CLIPipeProvider({ + command: 'claude', args: ['--model', 'sonnet'], toolProtocol: 'claude-mcp', policy: () => true, +}); +const loop = new Loop({ provider }); +const out = await loop.run([{ role: 'user', content: + `Create an empty file at exactly this path by running a shell command such as ` + + `\`touch ${sentinel}\` (or \`echo -n > ${sentinel}\`). Do it now, then reply DONE.` }], tools); + +console.log(` error=${out.error} · sessionTurns=${out.metrics.sessionTurns} · answer=${JSON.stringify(String(out.text).slice(0, 160))}`); + +// THE proof: the side effect a working shell would have produced did not happen. +check('the shell-created file does NOT exist (the fence held, not prose)', !fs.existsSync(sentinel)); +// No shell tool was ever offered to the bridge — the caller mounted only `noop`. +check('no shell tool crossed the bridge (only the caller tools were reachable)', !calls.some((c) => /sh|bash|exec|run/i.test(c))); +// A stripped shell must not crash the session — it degrades to the model saying it cannot. +check('the session did not crash on the impossible request', out.error === null || typeof out.error === 'string'); + +try { fs.unlinkSync(sentinel); } catch { /* fine */ } +console.log(`\n${failures === 0 ? 'GREEN — criterion 3 validated live' : `${failures} FAILING`}\n`); +process.exit(failures === 0 ? 0 : 1); diff --git a/poc/ba16-native-shipped.mjs b/poc/ba16-native-shipped.mjs new file mode 100644 index 0000000..437fc11 --- /dev/null +++ b/poc/ba16-native-shipped.mjs @@ -0,0 +1,139 @@ +// BA-16 verify-shipped — the SHIPPED native tool mode driven through a real `Loop`, on a real +// claude CLI subscription session. Not the build POC: this imports src/, so it goes red if the +// shipped code drifts from what was validated. +// +// Requires: the `claude` CLI, logged in. Costs ~$0.05 notional. Run: node poc/ba16-native-shipped.mjs +// +// What is deliberately NOT asserted here: the deny-STREAK guard firing, which needs a model to +// retry a denied tool three times — a fragile, non-deterministic property that belongs in the +// offline mutation-tested suite (it is covered there), not in a live check that would flake. + +import { createRequire } from 'node:module'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +const require = createRequire(import.meta.url); +const { Loop } = require('../src/loop'); +const { CLIPipeProvider } = require('../src/provider-clipipe'); + +const SCOPE = fs.mkdtempSync(path.join(os.tmpdir(), 'ba16-live-')); +let failures = 0; +const check = (label, cond, detail = '') => { + if (!cond) failures++; + console.log(` [${cond ? 'PASS' : 'FAIL'}] ${label}${detail ? ` — ${detail}` : ''}`); +}; + +const tools = (state) => ([ + { + name: 'lookup_code', + description: 'Returns the secret verification code for a given name.', + parameters: { type: 'object', properties: { name: { type: 'string' } }, required: ['name'] }, + execute: async ({ name }) => { state.looked++; return `verification code for ${name}: XK-7741-DELTA`; }, + }, + { + name: 'write_note', + description: 'Write content to a file at path.', + parameters: { type: 'object', properties: { path: { type: 'string' }, content: { type: 'string' } }, required: ['path', 'content'] }, + execute: async ({ path: p, content }) => { + state.wrote++; + fs.writeFileSync(p, String(content)); + return `wrote ${p}`; + }, + }, +]); + +const baseArgs = ['--model', 'sonnet']; + +// ── 1. The hop, end to end through the shipped Loop ────────────────────────────────────────────── +console.log('\n=== 1. native session through the shipped Loop ==='); +{ + const state = { looked: 0, wrote: 0 }; + const provider = new CLIPipeProvider({ command: 'claude', args: baseArgs, toolProtocol: 'claude-mcp', policy: () => true }); + const loop = new Loop({ provider }); + const out = await loop.run([{ role: 'user', content: 'What is the verification code for "orchard-42"? Report it exactly.' }], tools(state)); + + console.log(` error=${out.error} · loop rounds=${out.metrics.turns} · session turns=${out.metrics.sessionTurns} · tool calls=${out.metrics.toolCalls} · $${out.metrics.costUsd}`); + check("the caller's in-process closure really ran", state.looked >= 1, `calls=${state.looked}`); + check('the secret reached the model', out.text.includes('XK-7741-DELTA')); + check('a clean session is a clean finish', out.error === null, `error=${out.error}`); + // The honest-accounting claim: a multi-turn session must NOT report as a single round. + check('metrics report the REAL turn count, not 1', out.metrics.sessionTurns > 1, `sessionTurns=${out.metrics.sessionTurns}`); + check('metrics count the tool calls', out.metrics.toolCalls >= 1, `toolCalls=${out.metrics.toolCalls}`); + check('cache tiers reached the meter', (out.metrics.tokens.cacheRead + out.metrics.tokens.cacheCreation) > 0, + `read=${out.metrics.tokens.cacheRead} creation=${out.metrics.tokens.cacheCreation}`); + // FINDING 1's negative control: a HEALTHY session must not trip the broken-bridge detector. + check('no false bridge-failed alarm on a healthy session', out.error !== 'bridge-failed'); +} + +// ── 2. FINDING 2 — the --max-turns tripwire ────────────────────────────────────────────────────── +// `--max-turns` is UNDOCUMENTED in `claude --help` (verified: zero hits, while unknown flags ARE +// rejected). No source-read or help-parse can pin it, so this live check is the ONLY tripwire that +// goes red if the flag is ever renamed or removed. Requirement: advertised == enforced. +console.log('\n=== 2. --max-turns tripwire (undocumented flag — no other tripwire exists) ==='); +{ + const state = { looked: 0, wrote: 0 }; + const provider = new CLIPipeProvider({ command: 'claude', args: baseArgs, toolProtocol: 'claude-mcp', policy: () => true, maxTurns: 2 }); + const loop = new Loop({ provider }); + const out = await loop.run([{ role: 'user', content: 'Do these ONE AT A TIME as separate tool calls: look up codes for "a-1", "b-2", "c-3", "d-4", "e-5". Then report all five.' }], tools(state)); + + console.log(` error=${out.error} · sessionTurns=${out.metrics.sessionTurns} · tool calls served=${state.looked}`); + check('the bound is ENFORCED, not merely advertised', state.looked <= 2, `served ${state.looked} calls under a bound of 2`); + check('the bound stop is NAMED and error-tagged, never a clean success', out.error === 'max_turns', `error=${out.error}`); +} + +// ── 3. The gate, live, at the bridge ───────────────────────────────────────────────────────────── +console.log('\n=== 3. gate-in-bridge: deny is a RESULT, the session continues ==='); +{ + const state = { looked: 0, wrote: 0 }; + const outside = path.join(os.tmpdir(), `ba16-outside-${process.pid}.txt`); + const inside = path.join(SCOPE, 'ok.txt'); + let denials = 0; + const provider = new CLIPipeProvider({ + command: 'claude', args: baseArgs, toolProtocol: 'claude-mcp', + // The SAME chokepoint shape as Loop({policy}) — so a wired bareguard writes audit rows of + // identical shape here, with zero gate changes. + policy: (name, args) => { + if (name !== 'write_note') return true; + if (typeof args.path === 'string' && args.path.startsWith(SCOPE + path.sep)) return true; + denials++; + return `DENIED: writes are fenced to ${SCOPE}`; + }, + }); + const loop = new Loop({ provider }); + const out = await loop.run([{ role: 'user', content: `Write the text "hello" to ${outside}. If that is refused, write it to ${inside} instead. Then say DONE.` }], tools(state)); + + console.log(` error=${out.error} · denials=${denials} · writes=${state.wrote}`); + check('the gate denied the out-of-scope write', denials >= 1); + check('the out-of-scope file was NEVER created', !fs.existsSync(outside)); + check('the denied handler never ran', state.wrote <= 1, `handler invocations=${state.wrote}`); + check('the session CONTINUED past the denial (in-scope retry landed)', fs.existsSync(inside)); + check('a governed session still finishes cleanly', out.error === null, `error=${out.error}`); +} + +// ── 4. Streaming meter, and no double-billing ──────────────────────────────────────────────────── +console.log('\n=== 4. per-turn usage streams; the gate is billed exactly once ==='); +{ + const state = { looked: 0, wrote: 0 }; + const turnEvents = []; + const loopForwards = []; + const provider = new CLIPipeProvider({ + command: 'claude', args: baseArgs, toolProtocol: 'claude-mcp', policy: () => true, + onTurn: (e) => { turnEvents.push(e); }, + }); + const loop = new Loop({ provider, onLlmResult: (e) => { loopForwards.push(e); } }); + const out = await loop.run([{ role: 'user', content: 'What is the verification code for "harbor-9"? Report it exactly.' }], tools(state)); + + const perTurn = turnEvents.filter((e) => e.kind === 'turn'); + const sessionEv = turnEvents.filter((e) => e.kind === 'session'); + console.log(` turn events=${perTurn.length} · session events=${sessionEv.length} · loop forwards=${loopForwards.length} · error=${out.error}`); + check('usage streamed per turn, not summed at the end', perTurn.length > 1, `${perTurn.length} turn events`); + check('a streamed turn carries cache tiers', perTurn.some((e) => e.usage.cacheReadTokens !== undefined || e.usage.cacheCreationTokens !== undefined)); + check('a streamed turn is explicitly UNPRICED, never a synthetic 0', perTurn.every((e) => e.costUsd === null && e.pricing === 'unpriced')); + check('exactly one closing session event carries the authoritative cost', sessionEv.length === 1 && Number.isFinite(sessionEv[0].costUsd), + `costUsd=${sessionEv[0] && sessionEv[0].costUsd}`); + check('the Loop did NOT also forward (one session billed once)', loopForwards.length === 0, `${loopForwards.length} loop forwards`); +} + +console.log(`\n${failures === 0 ? 'ALL GREEN' : `${failures} FAILING CHECK(S)`}\n`); +process.exit(failures === 0 ? 0 : 1); diff --git a/src/loop.js b/src/loop.js index 4d146f1..8f6bd4a 100644 --- a/src/loop.js +++ b/src/loop.js @@ -277,6 +277,30 @@ class Loop { throw new Error('[Loop] options.trim must be a function (msgs, ctx) => msgs (e.g. unitTrimmer({ trim, onHarvest, policy }))'); } this.trim = options.trim || null; + // BA-16: a CYCLE-OWNING provider (one that runs its own multi-turn session internally, e.g. + // CLIPipe native tool mode) has no per-round seam for the Loop to hand a transcript to — the + // transcript lives inside the provider's session. `assemble`/`trim` would therefore be accepted + // and then silently never called, which is precisely the "silently-dead knob" shape this repo + // keeps paying for. Fail at CONSTRUCTION instead: an unhonorable option is a configuration + // error, not a no-op. + if (this.provider && this.provider.ownsCycle && (this.assemble || this.trim)) { + const named = [this.assemble && 'assemble', this.trim && 'trim'].filter(Boolean).join(' and '); + throw new Error( + `[Loop] provider '${this.provider.name || 'unknown'}' owns its own turn cycle, so ${named} can never run ` + + '(the provider owns the transcript, not the Loop). Remove the option, or use a provider whose cycle the Loop drives.', + ); + } + // The same rule, applied to the option where being silently dead is WORST. With a cycle-owning + // provider no tool call ever reaches the Loop, so a `Loop({policy})` is a fence that is simply + // not there — the run looks governed and is not. Refuse it: the gate must be wired on the + // provider, where every tool call actually crosses. + if (this.provider && this.provider.ownsCycle && this.policy && !this.provider.policy) { + throw new Error( + `[Loop] provider '${this.provider.name || 'unknown'}' owns its own turn cycle, so Loop-level policy would ` + + 'NEVER run — no tool call reaches the Loop. Wire the gate on the provider instead ' + + '(e.g. new CLIPipeProvider({ toolProtocol: \'claude-mcp\', policy })).', + ); + } if (options.onLlmResult != null && typeof options.onLlmResult !== 'function') { throw new Error('[Loop] options.onLlmResult must be a function'); } @@ -491,6 +515,12 @@ class Loop { const metrics = { turns: 0, toolCalls: 0, + // BA-16 — turns that happened INSIDE a cycle-owning provider's own session (CLIPipe native tool + // mode). Such a session is ONE Loop round no matter how many turns it really took, so `turns` + // alone would report a 14-turn session as 1 — a round count that reads far cheaper and far + // shorter than the run actually was. Reporting the real number keeps the meter honest; 0 for + // every provider whose cycle the Loop drives. + sessionTurns: 0, /** @type {Record} */ byTool: {}, tokens: { input: 0, output: 0, cacheCreation: 0, cacheRead: 0 }, @@ -525,6 +555,7 @@ class Loop { /** Snapshot the meter for a return — finalizes the run-scoped fields. @returns {RunMetrics} */ const finalizeMetrics = () => ({ turns: metrics.turns, + sessionTurns: metrics.sessionTurns, toolCalls: metrics.toolCalls, byTool: metrics.byTool, tokens: { ...metrics.tokens }, @@ -742,11 +773,22 @@ class Loop { metrics.turns++; addUsage(result.usage); if (roundCost === null) metrics.unpricedRounds++; else pricedAny = true; + // BA-16: a cycle-owning provider reports what really happened inside its session. + const session = (result.session && typeof result.session === 'object') ? result.session : null; + if (session) { + if (Number.isFinite(session.turns)) metrics.sessionTurns += session.turns; + if (Number.isFinite(session.toolCalls)) metrics.toolCalls += session.toolCalls; + } // BA1: forward LLM usage to gate.record (via wireGate) so budget.maxCostUsd // covers token-heavy / tool-light workloads. Callback errors route through // _reportError but never kill the loop — governance failure ≠ run failure. - if (this.onLlmResult) { + // BA-16: when a cycle-owning provider has ALREADY streamed this call's usage per internal turn + // (so a session that dies mid-run has surfaced every completed turn's spend rather than losing + // all of it), forwarding the summed total here too would bill the gate twice for one session. + // The provider only sets this when it genuinely reported; unwired, it stays false and the + // normal forward below runs — so the gate is never silently starved either. + if (this.onLlmResult && !(session && session.usageReported === true)) { try { await this.onLlmResult({ model, @@ -766,6 +808,20 @@ class Loop { } } + // BA-16: a cycle-owning provider detected a terminal INSIDE its own session — a turn bound, a + // guard streak, or a broken tool bridge. It is reported as the run's `error` (never merely + // surfaced) because every downstream consumer — `recurse`, the bareloop adopter — branches on + // `error` as the SOLE success signal. Surfacing alone would let a session in which no tool call + // ever succeeded propagate as converged. `lastText` (BA-5) preserves the partial work: a bound + // firing is normal termination for a bounded attempt, and that text is the only channel from + // attempt N to N+1. + if (session && typeof session.error === 'string' && session.error) { + sealDanglingToolCalls(msgs, `[halted:${session.error}]`); + this._reportError('session', new Error(`provider session terminated: ${session.error}`), { rule: session.error, sessionTurns: session.turns ?? null }); + this._safeEmit({ type: 'loop:done', data: { text: lastText, rule: session.error, sessionTurns: session.turns ?? null, cost: totalCost } }); + return { text: lastText, toolCalls: [], usage: lastUsage, cost: totalCost, error: session.error, stopReason: lastStopReason, msgs, metrics: finalizeMetrics(), ...(temperatureDropped && { temperatureDropped: true }) }; + } + // BA-13: classify this round's terminal signal against the neutral stop-reason vocabulary. BA-6 // short-circuited exactly one non-clean reason (max_tokens); this gate is the general form. It sits // AFTER metering (the tokens were really spent — the gate must see them) and BEFORE tool execution, diff --git a/src/mcp-bridge-stub.js b/src/mcp-bridge-stub.js new file mode 100644 index 0000000..4c1893d --- /dev/null +++ b/src/mcp-bridge-stub.js @@ -0,0 +1,107 @@ +'use strict'; + +/** + * BA-16 — the MCP stdio server the CLI spawns in native tool mode (`toolProtocol:'claude-mcp'`). + * + * This file owns NO tool logic. bare-agent's tools are `execute` CLOSURES living in the caller's + * process, but `--mcp-config` can only point the CLI at a COMMAND — so the tool surface has to cross + * a process boundary that the closures cannot. This stub is the bridge for exactly that hop: + * + * claude CLI <--stdio JSON-RPC--> THIS FILE <--unix socket--> parent (caller's closures) + * + * It is deliberately dumb. Everything that can decide anything — the manifest, the gate, the spin + * guards, redaction — lives parent-side, so the stub can never become a second place where policy + * is enforced (and never a second place where a secret can be logged: it writes no files at all). + * + * The one rule it does enforce is the BA-15 principle at the process boundary: EVERY failure of the + * hop is returned as a tool RESULT, never as a crash and never as a hang. A dead parent, a malformed + * frame and a slow handler all become `isError` results the model can read and react to — validated + * live before this shipped (probe 6: parent killed mid-call, session ended in 5.9s, no hang). + * + * Not a package export and never `require`d by the library — it is spawned as `node `. + */ + +const { createInterface } = require('readline'); +const net = require('net'); + +const SOCK = process.env.BAREAGENT_BRIDGE_SOCK; +const CALL_TIMEOUT_MS = Number(process.env.BAREAGENT_BRIDGE_TIMEOUT_MS) || 120000; + +const send = (obj) => process.stdout.write(JSON.stringify(obj) + '\n'); +const errResult = (text) => ({ content: [{ type: 'text', text }], isError: true }); + +/** + * One request/response over the unix socket. + * + * A FRESH connection per call is deliberate: a pooled socket opened while the parent was alive + * hides a parent that died later, turning "the bridge is gone" into a silent hang. Connecting per + * call makes a dead parent surface immediately, as an error result. + * + * @param {object} payload + * @returns {Promise} always resolves — never rejects, so no failure can escape as a crash. + */ +function bridge(payload) { + return new Promise((resolve) => { + let settled = false; + const done = (v) => { if (!settled) { settled = true; resolve(v); } }; + if (!SOCK) return done({ error: 'bridge not configured' }); + + const sock = net.createConnection(SOCK); + let buf = ''; + + // A hung parent handler must not hang the CLI session. Bounded, always. + const timer = setTimeout(() => { + try { sock.destroy(); } catch (_) { /* already gone */ } + done({ error: `bridge timeout after ${CALL_TIMEOUT_MS}ms` }); + }, CALL_TIMEOUT_MS); + + const finish = (v) => { + clearTimeout(timer); + try { sock.end(); } catch (_) { /* already gone */ } + done(v); + }; + + sock.on('connect', () => sock.write(JSON.stringify(payload) + '\n')); + sock.on('data', (d) => { + buf += d; + const nl = buf.indexOf('\n'); + if (nl === -1) return; // frame incomplete — wait for the rest + try { finish(JSON.parse(buf.slice(0, nl))); } + catch (err) { finish({ error: `bridge sent malformed JSON: ${/** @type {Error} */ (err).message}` }); } + }); + sock.on('error', (err) => finish({ error: `bridge unreachable: ${/** @type {any} */ (err).code || /** @type {Error} */ (err).message}` })); + sock.on('close', () => finish({ error: 'bridge closed before responding' })); + }); +} + +const rl = createInterface({ input: process.stdin }); +rl.on('line', async (line) => { + if (!line.trim()) return; + let msg; + try { msg = JSON.parse(line); } catch (_) { return; } // not JSON — not ours to answer + const { id, method, params } = msg; + + if (method === 'initialize') { + return send({ jsonrpc: '2.0', id, result: { + protocolVersion: (params && params.protocolVersion) || '2025-06-18', + capabilities: { tools: {} }, + serverInfo: { name: 'bareagent', version: '1' }, + } }); + } + + if (method === 'tools/list') { + // The PARENT owns the manifest — a tool definition is never duplicated here. + const res = await bridge({ op: 'list' }); + return send({ jsonrpc: '2.0', id, result: { tools: (res && res.tools) || [] } }); + } + + if (method === 'tools/call') { + const res = await bridge({ op: 'call', name: params && params.name, args: (params && params.arguments) || {} }); + if (res && res.error) return send({ jsonrpc: '2.0', id, result: errResult(`TOOL BRIDGE ERROR: ${res.error}`) }); + if (res && res.isError) return send({ jsonrpc: '2.0', id, result: errResult(String(res.text)) }); + return send({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: String((res && res.text) ?? '') }] } }); + } + + if (method === 'ping') return send({ jsonrpc: '2.0', id, result: {} }); + if (id !== undefined) send({ jsonrpc: '2.0', id, error: { code: -32601, message: `method not found: ${method}` } }); +}); diff --git a/src/provider-clipipe-mcp.js b/src/provider-clipipe-mcp.js new file mode 100644 index 0000000..7ae0bde --- /dev/null +++ b/src/provider-clipipe-mcp.js @@ -0,0 +1,446 @@ +'use strict'; + +/** + * BA-16 — CLIPipe NATIVE tool mode (`toolProtocol:'claude-mcp'`). + * + * ── Why this exists beside the 0.32.0 envelope emulation ───────────────────────────────────────── + * + * The emulation treats the CLI as a plain turn-provider: one spawn per round, the whole transcript + * re-rendered and re-sent every time. That is the right instrument for a CLI with no tool channel, + * and the WRONG one for the claude CLI, which has a native one. The cost is the argument, and it is + * measured, not asserted: re-sending the full prefix every turn pays fresh `cache_creation` on all + * of it, which the adopter measured at **$0.25–0.55/round** on a real ~40-round job transcript — + * against **$0.0055–0.0074/turn** for a native session (independently reproduced here and by the + * adopter). A session has session-incremental caching that the emulation structurally cannot have. + * + * NOT CLAIMED: an output-quality improvement. There is n=2 suggestive evidence that the emulation's + * JSON-questionnaire framing makes a model act less, and it is NOT minted — do not sell it. Cost is + * the claim. (The BA-7 precedent: a protocol fix ships labeled honestly or not at all.) + * + * ── What changes, and what that costs ──────────────────────────────────────────────────────────── + * + * The CLI owns the inner cycle. That is a real trade and it is stated plainly rather than papered + * over: the Loop's PER-ROUND machinery cannot run, because there are no rounds to run it on. What + * the Loop was buying is bought back HERE, at the `tools/call` bridge — which sees every call, so it + * is a strictly sufficient seam for the guards that matter: + * + * - authorization the SAME `policy(tool, args, ctx)` chokepoint the Loop uses, so a wired gate + * writes AUDIT ROWS OF IDENTICAL SHAPE with zero gate changes (adopters read + * that file as an instrument; a shape change would blind them silently) + * - deny streak BA-11, same default (3), same reset-on-any-allowed-call semantics + * - identical-error BA-12, same default (3), same NARROWEST trigger (name + JSON args, byte-equal) + * - turn bound `--max-turns`, whose stop is NAMED (`error_max_turns`), never a silent success + * + * What is genuinely LOST and cannot be bought back here: the per-round context seams (`assemble`, + * `trim`, stash compaction) and `cacheMessages` — the CLI owns the transcript, so no caller seam can + * reach it. Those options THROW at construction rather than sit silently dead. + * + * ── The failure mode this module exists to not have ────────────────────────────────────────────── + * + * A dead bridge does NOT make the CLI report failure. Measured before building: with the bridge + * killed mid-call, every tool call failed and the session still ended `subtype:'success'` — the + * model wrote a tidy final answer explaining that its tools were broken. Mapping that subtype + * straight onto `error:null` would report a run in which NOTHING WORKED as converged: the exact + * BA-4/5/6/13 "under-modeled boundary rounds optimistically toward success" class this repo keeps + * paying for. So bridge health is tracked PARENT-SIDE and error-tags the session regardless of what + * the CLI's own subtype says. The CLI's subtype is not a sufficient success signal. + */ + +const net = require('net'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { spawn } = require('child_process'); +const { ProviderError, HaltError } = require('./errors'); + +/** Path to the stdio stub the CLI spawns. Resolved once — it ships inside the package. */ +const STUB_PATH = path.join(__dirname, 'mcp-bridge-stub.js'); + +/** Mirrors the Loop's own BA-11 / BA-12 defaults so native mode is not quietly laxer. */ +const DEFAULT_MAX_CONSECUTIVE_DENIALS = 3; +const DEFAULT_MAX_IDENTICAL_TOOL_ERRORS = 3; + +/** The MCP server name the CLI sees; tool names reach the model as `mcp__bareagent__`. */ +const SERVER_NAME = 'bareagent'; + +/** + * Stable key for BA-12's identical-tool-error guard. NARROWEST trigger by design: only a + * BYTE-IDENTICAL repeat counts. Counting any consecutive failure was measured to punish the + * recovery it exists to protect (a model varying args while working through an ENOENT). + * @param {string} name + * @param {any} args + * @returns {string} + */ +function toolErrorKey(name, args) { + let serialized; + try { serialized = JSON.stringify(args ?? {}); } catch (_) { serialized = ''; } + return `${name}:${serialized}`; +} + +/** + * Clamp a diagnostic built from a thrown value of unknown shape before it can ride into a + * `receipts`/audit string. A non-Error throw serialized wholesale has previously pushed a secret + * into a plaintext audit log (the F16/BA-1 capture class), and an append-only log that captures a + * key captures it forever. Conventional fields only, hard length cap, never the whole object. + * @param {any} err + * @returns {string} + */ +function safeErrorText(err) { + let msg; + if (err instanceof Error) msg = `${err.constructor.name}: ${err.message}`; + else if (typeof err === 'string') msg = err; + else if (err && typeof err === 'object' && typeof err.message === 'string') msg = err.message; + else msg = `non-Error throw (${typeof err})`; + return msg.length > 300 ? `${msg.slice(0, 300)}…` : msg; +} + +/** + * The parent-side bridge: a unix-socket server backed by the caller's in-process tool closures, + * with the gate and both spin guards wired at the one seam that sees every call. + * + * Unix socket, not loopback TCP: a library must not open a listening network port as a side effect + * of running a tool loop. The socket lives in a 0700 directory and is itself chmod 0600. + * + * @param {object} opts + * @param {import('../types').ToolDef[]} opts.tools - the caller's tools (with live `execute` closures). + * @param {Function|null} [opts.policy] - the SAME contract as `Loop({policy})`: only `true` allows; + * a string is the deny reason fed back verbatim. + * @param {any} [opts.ctx] - opaque governance ctx, forwarded to `policy` unchanged. + * @param {number} [opts.maxConsecutiveDenials] + * @param {number} [opts.maxIdenticalToolErrors] + * @returns {Promise<{sockPath: string, state: any, close: () => void}>} + */ +async function createBridge({ tools, policy = null, ctx = undefined, maxConsecutiveDenials, maxIdenticalToolErrors }) { + const denyCap = Number.isFinite(maxConsecutiveDenials) ? Number(maxConsecutiveDenials) : DEFAULT_MAX_CONSECUTIVE_DENIALS; + const errCap = Number.isFinite(maxIdenticalToolErrors) ? Number(maxIdenticalToolErrors) : DEFAULT_MAX_IDENTICAL_TOOL_ERRORS; + + const byName = new Map(tools.map((t) => [t.name, t])); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'bareagent-bridge-')); + fs.chmodSync(dir, 0o700); + const sockPath = path.join(dir, 'bridge.sock'); + + const state = { + /** @type {{name: string, denied: boolean}[]} */ calls: [], + toolCalls: 0, + consecutiveDenials: 0, + /** @type {{key: string, count: number}} */ lastError: { key: '', count: 0 }, + /** @type {string|null} A guard tripped — the session must END, not merely refuse this call. */ + terminal: null, + /** @type {Error|null} A HaltError raised by the gate inside a tool call — re-thrown by the caller. */ + halt: null, + /** Parent-side bridge health. A dead bridge must not be able to read as a clean session. */ + bridgeDown: false, + }; + + /** + * Handle one `tools/call`. Every outcome is a RESULT the model can act on; the only thing that + * ends the session is a guard tripping, which is reported through `state.terminal`. + */ + async function handleCall(name, args) { + state.toolCalls++; + const tool = byName.get(name); + + // An unknown/hallucinated tool name feeds the SAME spin counter as a thrown error — BA-12 had to + // count both paths, since a model looping on a name that does not exist spins just as hard. + if (!tool) return recordFailure(name, args, `unknown tool '${name}'`); + + if (policy) { + let verdict; + try { + verdict = await policy(name, args, ctx); + } catch (err) { + // A governance halt is a clean exit, not a tool error — stash it and end the session. + if (err instanceof HaltError) { state.halt = err; state.terminal = `halt:${err.message}`; return { isError: true, text: 'session halted by governance' }; } + throw err; + } + if (verdict !== true) { + state.consecutiveDenials++; + state.calls.push({ name, denied: true }); + const reason = typeof verdict === 'string' ? verdict : `denied by policy: ${name}`; + // BA-11: a single deny stays ADVISORY so the model can pivot to an allowed tool + // (allowlist-safe). Only a STREAK with no allowed call in between ends the session. + if (denyCap > 0 && Number.isFinite(denyCap) && state.consecutiveDenials >= denyCap) { + state.terminal = `denied:${name}`; + return { isError: true, text: `${reason} — ${state.consecutiveDenials} consecutive denials; ending session` }; + } + return { text: reason }; + } + state.consecutiveDenials = 0; // any allowed call resets the streak + } + + state.calls.push({ name, denied: false }); + try { + const out = await tool.execute(args || {}); + state.lastError = { key: '', count: 0 }; // success resets the identical-error streak + return { text: typeof out === 'string' ? out : JSON.stringify(out) }; + } catch (err) { + if (err instanceof HaltError) { state.halt = err; state.terminal = `halt:${err.message}`; return { isError: true, text: 'session halted by governance' }; } + return recordFailure(name, args, safeErrorText(err)); + } + } + + /** BA-12: count a byte-identical repeat; N in a row ends the session, anything else is feedback. */ + function recordFailure(name, args, text) { + const key = toolErrorKey(name, args); + state.lastError = key === state.lastError.key + ? { key, count: state.lastError.count + 1 } + : { key, count: 1 }; + if (errCap > 0 && Number.isFinite(errCap) && state.lastError.count >= errCap) { + state.terminal = `stuck:${name}`; + return { isError: true, text: `${text} — repeated ${state.lastError.count}× identically; ending session` }; + } + return { isError: true, text }; + } + + const manifest = tools.map((t) => ({ + name: t.name, + description: t.description || '', + inputSchema: t.parameters || { type: 'object', properties: {} }, + })); + + const server = net.createServer((conn) => { + let buf = ''; + conn.on('data', async (d) => { + buf += d; + const nl = buf.indexOf('\n'); + if (nl === -1) return; + let req; + try { req = JSON.parse(buf.slice(0, nl)); } catch (_) { return conn.end(); } + buf = ''; + try { + if (req.op === 'list') return conn.write(JSON.stringify({ tools: manifest }) + '\n'); + if (req.op === 'call') { + const res = await handleCall(req.name, req.args); + return conn.write(JSON.stringify(res) + '\n'); + } + } catch (err) { + // Never let a handler fault kill the bridge — it becomes a result the model reads. + try { conn.write(JSON.stringify({ isError: true, text: safeErrorText(err) }) + '\n'); } catch (_) { /* peer gone */ } + } + }); + conn.on('error', () => { /* peer vanished mid-call; the stub turns that into a tool result */ }); + }); + + // Parent-side health: if our own listener dies while a session is live, every subsequent tool + // call silently fails on the CLI side and the session can still end 'success'. Record it. + server.on('error', () => { state.bridgeDown = true; }); + + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(sockPath, () => { try { fs.chmodSync(sockPath, 0o600); } catch (_) { /* best effort */ } resolve(undefined); }); + }); + + return { + sockPath, + state, + close() { + try { server.close(); } catch (_) { /* already closed */ } + try { fs.rmSync(path.dirname(sockPath), { recursive: true, force: true }); } catch (_) { /* best effort */ } + }, + }; +} + +/** + * Map the CLI's session `subtype` onto (neutral stop reason, session error). + * + * Own-property lookup only: `subtype` is provider-supplied, so a value like `toString` would + * otherwise resolve to an inherited `Object.prototype` function and be mistaken for a mapping — + * the proto-key footgun this repo has now fixed three times. + */ +const SUBTYPE_MAP = { + success: { stopReason: 'end_turn', error: null }, + error_max_turns: { stopReason: 'max_turns', error: 'max_turns' }, + error_during_execution: { stopReason: null, error: 'session_error' }, +}; + +/** + * @param {string|null|undefined} subtype + * @returns {{stopReason: string|null, error: string|null}} + */ +function classifySubtype(subtype) { + if (typeof subtype !== 'string' || subtype === '') { + // No result event at all — the session died without reporting. NOT a success. + return { stopReason: null, error: 'session_incomplete' }; + } + if (Object.prototype.hasOwnProperty.call(SUBTYPE_MAP, subtype)) return SUBTYPE_MAP[subtype]; + // An unrecognized subtype is surfaced verbatim and ERROR-TAGGED, never passed through as clean: + // the whole BA-13 lesson is that a terminal the consumer branches on must be tagged, not just seen. + return { stopReason: subtype, error: `session:${subtype}` }; +} + +/** + * Decide the session's terminal tag. Pure, so the precedence is testable without a live CLI. + * + * The ordering is the whole point: + * 1. `terminal` — a guard tripped and WE killed the session; most specific, and it explains why + * the CLI's own subtype is missing or odd. + * 2. bridge broken — a tool call the CLI ATTEMPTED but the bridge never SERVED never reached the + * caller's closure. This must outrank a reported `success`, because a session + * whose tools were all broken still ends `subtype:'success'` — the model writes + * a tidy final answer explaining that its tools failed, and mapping that onto + * `error:null` reports a run in which nothing worked as converged. + * 3. `timedOut` — we killed it on the wall clock. + * 4. the subtype — what the CLI itself said. + * + * @param {{terminal: string|null, bridgeDown: boolean, attempted: number, served: number, timedOut: boolean, subtype: string|null|undefined}} facts + * @returns {{stopReason: string|null, error: string|null}} + */ +function resolveSessionError(facts) { + const fromSubtype = classifySubtype(facts.subtype); + if (facts.terminal) return { stopReason: fromSubtype.stopReason, error: facts.terminal }; + const bridgeBroken = facts.bridgeDown + || (Number.isFinite(facts.attempted) && Number.isFinite(facts.served) && facts.attempted > facts.served); + if (bridgeBroken) return { stopReason: fromSubtype.stopReason, error: 'bridge-failed' }; + if (facts.timedOut) return { stopReason: fromSubtype.stopReason, error: 'session_timeout' }; + return fromSubtype; +} + +/** + * Run ONE native CLI session to completion, streaming per-turn usage as it arrives. + * + * @param {object} opts + * @returns {Promise} + */ +function runSession(opts) { + const { + command, baseArgs = [], systemPrompt, task, sockPath, maxTurns, timeoutMs, + onTurn = null, ctx, cwd, env, bridgeTimeoutMs, + } = opts; + + const mcpConfig = JSON.stringify({ + mcpServers: { + [SERVER_NAME]: { + command: process.execPath, + args: [STUB_PATH], + env: { + BAREAGENT_BRIDGE_SOCK: sockPath, + ...(bridgeTimeoutMs ? { BAREAGENT_BRIDGE_TIMEOUT_MS: String(bridgeTimeoutMs) } : {}), + }, + }, + }, + }); + + // The fence is set BY THE MODE, never left to the caller: `--tools ''` + `--strict-mcp-config` + // strip the CLI's own tool surface (so a granted-verb fence cannot be widened by the CLI's + // built-ins), and `--setting-sources ''` suppresses cwd CLAUDE.md/memory auto-discovery — a + // measured ~18× input-token cut, and also a confidentiality boundary (the CLI would otherwise + // read the host repo's memory into every turn). + const args = [ + ...baseArgs, + '-p', + '--mcp-config', mcpConfig, + '--tools', '', + '--strict-mcp-config', + '--setting-sources', '', + '--allowedTools', `mcp__${SERVER_NAME}__*`, + '--system-prompt', systemPrompt, + '--output-format', 'stream-json', + '--verbose', + ]; + if (Number.isFinite(maxTurns) && maxTurns > 0) args.push('--max-turns', String(maxTurns)); + + return new Promise((resolve) => { + const child = spawn(command, args, { cwd, env, stdio: ['pipe', 'pipe', 'pipe'] }); + /** @type {any[]} */ const turns = []; + let final = null, buf = '', stderr = '', settled = false; + let attempted = 0; // MCP tool calls the CLI tried to make against our server + /** @type {Error|null} */ let turnHalt = null; + const started = Date.now(); + + const done = (extra = {}) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve({ turns, final, stderr, ms: Date.now() - started, turnHalt, attempted, ...extra }); + }; + + const timer = setTimeout(() => { + try { child.kill('SIGKILL'); } catch (_) { /* already gone */ } + done({ timedOut: true }); + }, timeoutMs); + + /** Kill the session now — a guard tripped or governance halted mid-flight. */ + const abort = () => { try { child.kill('SIGTERM'); } catch (_) { /* already gone */ } }; + + child.stdout.on('data', async (d) => { + buf += d; + let nl; + while ((nl = buf.indexOf('\n')) !== -1) { + const line = buf.slice(0, nl); + buf = buf.slice(nl + 1); + if (!line.trim()) continue; + let ev; + try { ev = JSON.parse(line); } catch (_) { continue; } + + // Count tool calls the CLI ATTEMPTED against our server. Compared later with what the bridge + // actually SERVED, this is the only precise parent-side detector of a broken bridge: if the + // stub cannot reach the socket, our server never sees the connection at all, so a + // server-side error flag would stay clean while every tool call failed — and the session + // would still end `subtype:'success'` (measured). Attempted > served is the honest signal. + if (ev.type === 'assistant' && ev.message && Array.isArray(ev.message.content)) { + for (const block of ev.message.content) { + if (block && block.type === 'tool_use' && typeof block.name === 'string' + && block.name.startsWith(`mcp__${SERVER_NAME}__`)) attempted++; + } + } + + if (ev.type === 'assistant' && ev.message && ev.message.usage) { + const u = ev.message.usage; + /** @type {import('../types').Usage} */ + const usage = { + inputTokens: Number(u.input_tokens) || 0, + outputTokens: Number(u.output_tokens) || 0, + }; + // Omit an absent tier rather than emit a synthetic 0 (per the Usage contract). + if (Number.isFinite(u.cache_read_input_tokens)) usage.cacheReadTokens = u.cache_read_input_tokens; + if (Number.isFinite(u.cache_creation_input_tokens)) usage.cacheCreationTokens = u.cache_creation_input_tokens; + turns.push(usage); + + // Stream it NOW, never sum-at-end: a session that dies mid-run must already have + // surfaced every completed turn's spend, or the gate loses it entirely. + if (onTurn) { + try { + await onTurn({ + model: (ev.message && ev.message.model) || null, + provider: 'clipipe', + usage, + // A CLI model has no rate-table entry, and the CLI prices the SESSION, not the turn. + // Explicitly unpriced beats a synthetic 0 — that silent zero is what made a budget + // cap a no-op once already. + costUsd: null, + pricing: 'unpriced', + durationMs: Date.now() - started, + // ctx is what a wired gate records spend against — forward it, same as the Loop's onLlmResult. + ctx, + kind: 'turn', + }); + } catch (err) { + if (err instanceof HaltError) { turnHalt = err; abort(); return; } + // A governance callback failure is not a run failure — surfaced, never fatal. + } + } + } + + if (ev.type === 'result') final = ev; + } + }); + + child.stderr.on('data', (d) => { stderr += d; }); + child.on('error', (err) => done({ spawnError: err })); + child.on('close', () => done()); + child.stdin.end(task); + }); +} + +module.exports = { + STUB_PATH, + SERVER_NAME, + DEFAULT_MAX_CONSECUTIVE_DENIALS, + DEFAULT_MAX_IDENTICAL_TOOL_ERRORS, + toolErrorKey, + safeErrorText, + createBridge, + classifySubtype, + resolveSessionError, + runSession, +}; diff --git a/src/provider-clipipe.js b/src/provider-clipipe.js index 6da8b9d..1588f61 100644 --- a/src/provider-clipipe.js +++ b/src/provider-clipipe.js @@ -1,8 +1,9 @@ 'use strict'; const { spawn } = require('child_process'); -const { ProviderError } = require('./errors'); +const { ProviderError, HaltError } = require('./errors'); const { buildToolSystemPrompt, renderTranscript, resolveToolProtocol, mapClaudeMeta } = require('./provider-clipipe-tools'); +const { createBridge, resolveSessionError, runSession } = require('./provider-clipipe-mcp'); /** @typedef {import('../types').Message} Message */ /** @typedef {import('../types').ToolDef} ToolDef */ @@ -18,8 +19,51 @@ const { buildToolSystemPrompt, renderTranscript, resolveToolProtocol, mapClaudeM * @property {string} [systemPromptFlag] - CLI flag for system prompt (e.g. '--system'). When set, system messages are extracted and passed via this flag instead of stdin. * @property {(chunk: string) => void} [onChunk] - Called with each stdout chunk as it streams. * @property {'claude-json'|((stdout: string) => Partial)} [parse] - Opt-in structured-output parser for stdout. Default (unset) returns stdout verbatim as `text` with zero usage (no behavior change). `'claude-json'` is a shipped preset for `claude -p --output-format json`: it maps the CLI's result envelope onto `GenerateResult` (text←`result`, usage←`usage.*`, model←first `modelUsage` key, costUsd←`total_cost_usd`) and throws `ProviderError` on malformed JSON or an error envelope (`is_error`/non-success subtype). A function is the CLI-agnostic escape hatch: it receives trimmed stdout and returns a partial `GenerateResult` (merged over defaults); throw to signal a parse failure. - * @property {'claude'} [toolProtocol] - Opt into TOOL MODE (v0.32.0). A subscription CLI is a plain turn-provider with no native tool channel; this enables schema-validated tool EMULATION so a caller's `tools` work over the CLI (letting a Claude/etc. subscription drive an agentic Loop without metered-API spend). When set, ANY `generate(msgs, tools)` call with a non-empty `tools` array auto-uses emulation (the caller's own system stance + a tool manifest + a JSON envelope, parsed back into normalized `toolCalls`); an empty `tools` array is unchanged plain-text. When NOT set, `tools` are IGNORED (plain-text mode, the long-standing behavior — a non-tool-calling CLI can legitimately sit in a Loop that has tools mounted) with a one-time `console.warn` for visibility. Claude-only for now (`'claude'`); the claude-specific flags/schema/parse live in `provider-clipipe-tools.js` so a second CLI slots in behind the same seam. NOTE: tool mode requires a capable model — weak models (e.g. haiku) answer in prose instead of calling tools; see `probeCapability`. - * @property {boolean} [probeCapability=true] - (tool mode only) On the first tool-mode `generate`, run ONE cheap upfront probe that asks the model to obtain unknowable info via a tool. If it answers in prose instead of emitting a tool_call, throw a loud `ProviderError` naming the model — FAIL FAST rather than silently degrade mid-run (the weak-model failure mode). Behaviour-based, never a model name-list (a roster goes stale, BA-10). The verdict is cached per instance (one probe per provider, not per turn). Set `false` to skip when the caller already knows the model is capable. + * @property {'claude'|'claude-mcp'} [toolProtocol] - Opt into TOOL MODE. Two modes, and the choice is + * about COST, not capability. `'claude-mcp'` (BA-16, NATIVE — prefer this on the claude CLI): one CLI + * session per call, the caller's `tools` exposed to it as a real MCP server whose handlers call back + * into your own in-process closures. The CLI owns the inner cycle and caches its transcript + * session-side. `'claude'` (v0.32.0, EMULATION): one CLI spawn per round with the whole transcript + * re-rendered and re-sent, parsed back through a JSON envelope. Emulation re-buys the full prefix + * every turn, which the adopter measured at **$0.25–0.55/round** against **~$0.006/turn** native — so + * it is the right instrument only for a CLI with NO MCP support, not a default. NOT claimed for + * native: better output quality (n=2 suggestive evidence exists and is deliberately unminted). + * Native mode sets {@link CLIPipeProvider#ownsCycle}, which makes the Loop REFUSE options it could + * never honor (`assemble`/`trim`/`cacheMessages`, and a Loop-level `policy`) instead of leaving them + * silently dead. See the native-only properties below. + * @property {Function} [policy] - (native mode) The gate, same contract as `Loop({policy})`: only `true` + * allows, a string is the deny reason fed back verbatim, a thrown `HaltError` is a clean governance + * exit. REQUIRED here rather than on the Loop, because in native mode no tool call ever reaches the + * Loop — a `Loop({policy})` would be a fence that is silently not there (the Loop throws instead). + * Wiring the same `wireGate(gate).policy` keeps audit rows byte-shape-identical, with zero gate changes. + * @property {Function} [onTurn] - (native mode) Called with `{model, provider, usage, costUsd, pricing, + * durationMs, ctx, kind}` for EACH completed CLI turn as it arrives (`kind:'turn'`, four cache tiers, + * `costUsd:null` — the CLI prices the session, not the turn), then once at session end + * (`kind:'session'`) carrying the authoritative total cost with zero usage. Streaming, never + * sum-at-end: a session that dies mid-run must already have surfaced every completed turn's spend or + * the gate loses all of it. The event shape mirrors `Loop({onLlmResult})`, so `wireGate(gate).onLlmResult` + * drops straight in — and when it is wired the Loop skips its own forward, so nothing is billed twice. + * @property {number} [maxTurns] - (native mode) Maps to the CLI's `--max-turns`. The bound stop is NAMED + * (`error_max_turns` → `session.error:'max_turns'`), never a silent clean success. + * @property {number} [maxConsecutiveDenials=3] - (native mode) BA-11 at the bridge: a single deny stays + * advisory so the model can pivot to an allowed tool; N in a row with no allowed call between ends the + * session with `denied:`. `0`/`Infinity` disables. + * @property {number} [maxIdenticalToolErrors=3] - (native mode) BA-12 at the bridge: only a BYTE-IDENTICAL + * repeat (name + JSON args) counts, so a model varying its args while recovering is never punished. N in + * a row ends the session with `stuck:`. `0`/`Infinity` disables. + * @property {number} [sessionTimeout=600000] - (native mode) Wall-clock ceiling for one whole session. The + * 30s `timeout` default is for one-shot text and would kill an agentic session mid-run. + * @property {number} [bridgeTimeoutMs] - (native mode) Ceiling for ONE tool-handler round-trip across the + * bridge. A hung handler becomes an error tool result rather than a hung session (default 120s). + * + * Either mode: a non-empty `tools` array on `generate()` routes to tool mode; an empty one stays + * plain text. With NO `toolProtocol`, `tools` are IGNORED (plain-text, the long-standing behavior — + * a non-tool-calling CLI legitimately sits in a Loop with tools mounted) plus a one-time `console.warn`. + * Emulation additionally requires a capable model (weak ones answer in prose; see `probeCapability`); + * native mode needs no such probe, because the CLI's own tool channel does not depend on the model + * agreeing to fill in a JSON questionnaire. The claude-specific parts of each live in + * `provider-clipipe-tools.js` / `provider-clipipe-mcp.js`, so a second CLI slots in behind the same seams. + * @property {boolean} [probeCapability=true] - (EMULATION tool mode only) On the first tool-mode `generate`, run ONE cheap upfront probe that asks the model to obtain unknowable info via a tool. If it answers in prose instead of emitting a tool_call, throw a loud `ProviderError` naming the model — FAIL FAST rather than silently degrade mid-run (the weak-model failure mode). Behaviour-based, never a model name-list (a roster goes stale, BA-10). The verdict is cached per instance (one probe per provider, not per turn). Set `false` to skip when the caller already knows the model is capable. */ class CLIPipeProvider { @@ -44,10 +88,40 @@ class CLIPipeProvider { // Tool mode (v0.32.0). Resolve the protocol adapter eagerly so an unknown name fails at // construction, not mid-run. `_toolCapability` caches the upfront probe verdict per instance // (null = not yet probed; a Promise while in flight; true once confirmed capable). - this.toolProtocol = options.toolProtocol ? resolveToolProtocol(options.toolProtocol) : null; + // BA-16 native tool mode. `claude-mcp` is NOT an envelope protocol — the CLI runs its own + // multi-turn session and executes the caller's tools natively over MCP — so it is resolved on a + // separate axis rather than being forced through `resolveToolProtocol`'s emulation shape. + this.nativeTools = options.toolProtocol === 'claude-mcp'; + /** + * Declares to the Loop that this provider runs its OWN turn cycle. The Loop reads it to refuse + * options it could never honor (assemble/trim/cacheMessages) and to require the fence be wired + * where it can actually run. Generic provider-contract flag; nothing here is claude-specific. + */ + this.ownsCycle = this.nativeTools; + this.toolProtocol = (options.toolProtocol && !this.nativeTools) ? resolveToolProtocol(options.toolProtocol) : null; this.probeCapability = options.probeCapability !== false; /** @type {Promise|null} */ this._toolCapability = null; + + if (this.nativeTools) { + // The gate CANNOT ride on the Loop in native mode: no tool call ever reaches the Loop, so a + // `Loop({policy})` would be a fence that silently is not there. It must be wired HERE, at the + // bridge, which is the one seam every tool call crosses. + if (options.policy != null && typeof options.policy !== 'function') { + throw new Error('[CLIPipeProvider] options.policy must be a function (tool, args, ctx) => true|string'); + } + this.policy = options.policy || null; + this.onTurn = options.onTurn || null; + if (this.onTurn != null && typeof this.onTurn !== 'function') { + throw new Error('[CLIPipeProvider] options.onTurn must be a function'); + } + this.maxTurns = options.maxTurns ?? null; + this.maxConsecutiveDenials = options.maxConsecutiveDenials; + this.maxIdenticalToolErrors = options.maxIdenticalToolErrors; + // A session is a whole agentic run, not one prompt — the 30s one-shot default would kill it. + this.sessionTimeout = options.sessionTimeout ?? 600000; + this.bridgeTimeoutMs = options.bridgeTimeoutMs ?? null; + } } /** @@ -67,6 +141,7 @@ class CLIPipeProvider { */ async generate(messages, tools = [], options = {}) { if (Array.isArray(tools) && tools.length > 0) { + if (this.nativeTools) return this._generateWithMcp(messages, tools, options); if (this.toolProtocol) return this._generateWithTools(messages, tools); // No protocol configured → plain-text mode, tools IGNORED — the long-standing behavior, kept // for backward compatibility (a non-tool-calling CLI legitimately coexists in a Loop that has @@ -147,6 +222,133 @@ class CLIPipeProvider { return result; } + /** + * BA-16 native tool mode — run ONE whole CLI session and report it honestly as one call. + * + * The CLI owns the inner cycle here: it calls the caller's tools natively over an MCP bridge and + * keeps going until it answers or hits a bound. So this returns `toolCalls: []` always (there is + * nothing left for the Loop to execute) plus a `session` block describing what really happened — + * the real turn count, the real tool-call count, and any terminal the Loop must surface as the + * run's `error`. + * + * Ordering of terminals is deliberate. A governance halt outranks everything (it is a clean exit, + * not a fault). A tripped guard outranks the CLI's own subtype, because the guard is why we killed + * the session. And `bridgeDown` outranks a reported `success`, because a session whose tools were + * all broken still ends `subtype:'success'` — measured, and the reason this block exists. + * + * @param {Message[]} messages + * @param {ToolDef[]} tools + * @param {Record} options - the Loop's run options (`ctx` is read from here). + * @returns {Promise} + */ + async _generateWithMcp(messages, tools, options = {}) { + if (options.cacheMessages) { + throw new Error( + '[CLIPipeProvider] cacheMessages cannot apply in native tool mode — the CLI owns the transcript ' + + 'and caches it session-side, so there is no request body for a breakpoint to ride on. Remove the option.', + ); + } + const sysMsg = messages.find((m) => m.role === 'system'); + const systemPrompt = (sysMsg && typeof sysMsg.content === 'string' && sysMsg.content) + || 'You are an agent. Use the tools provided over MCP when they are needed.'; + + const bridge = await createBridge({ + tools, + policy: this.policy, + ctx: options.ctx, + maxConsecutiveDenials: this.maxConsecutiveDenials, + maxIdenticalToolErrors: this.maxIdenticalToolErrors, + }); + + let r; + try { + r = await runSession({ + command: this.command, + baseArgs: this.args, + systemPrompt, + task: renderTranscript(messages), + sockPath: bridge.sockPath, + maxTurns: this.maxTurns, + timeoutMs: this.sessionTimeout, + bridgeTimeoutMs: this.bridgeTimeoutMs, + onTurn: this.onTurn, + ctx: options.ctx, + cwd: this.cwd, + env: this.env, + }); + } finally { + bridge.close(); + } + + const st = bridge.state; + // A governance halt is a CLEAN exit and must reach the Loop as a HaltError, not as a session + // error tag — the Loop is the thing that knows a halt seals the transcript rather than faulting. + if (st.halt) throw st.halt; + if (r.turnHalt) throw r.turnHalt; + if (r.spawnError) { + throw new ProviderError(`[CLIPipeProvider] failed to spawn "${this.command}": ${r.spawnError.message}`, /** @type {any} */ ({ status: 0 })); + } + + /** @type {import('../types').Usage} */ + const usage = { inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheCreationTokens: 0 }; + for (const t of r.turns) { + usage.inputTokens += t.inputTokens || 0; + usage.outputTokens += t.outputTokens || 0; + usage.cacheReadTokens += t.cacheReadTokens || 0; + usage.cacheCreationTokens += t.cacheCreationTokens || 0; + } + + const { stopReason, error } = resolveSessionError({ + terminal: st.terminal, + bridgeDown: st.bridgeDown, + attempted: r.attempted, + served: st.toolCalls, + timedOut: Boolean(r.timedOut), + subtype: r.final && r.final.subtype, + }); + + const costUsd = (r.final && Number.isFinite(r.final.total_cost_usd)) ? r.final.total_cost_usd : null; + + // The authoritative price arrives only at session end (the CLI prices the SESSION, not the turn), + // so when per-turn streaming is wired it gets one closing event carrying the cost with zero + // usage — the tokens were already streamed, and double-counting either axis would be a lie. + if (this.onTurn) { + try { + await this.onTurn({ + model: (r.final && r.final.model) || null, + provider: 'clipipe', + usage: { inputTokens: 0, outputTokens: 0 }, + costUsd, + pricing: costUsd === null ? 'unpriced' : 'priced', + durationMs: r.ms, + ctx: options.ctx, + kind: 'session', + }); + } catch (err) { + if (err instanceof HaltError) throw err; + } + } + + /** @type {GenerateResult} */ + const result = { + text: (r.final && typeof r.final.result === 'string') ? r.final.result : '', + toolCalls: [], + usage, + model: (r.final && r.final.model) || null, + stopReason, + session: { + turns: r.turns.length, + toolCalls: st.toolCalls, + error, + // Only true when we ACTUALLY streamed — unwired, the Loop must still forward the total or + // the gate would see this session as free. + usageReported: Boolean(this.onTurn), + }, + }; + if (costUsd !== null) result.costUsd = costUsd; + return result; + } + /** * Run the upfront capability probe ONCE per instance (cached), unless `probeCapability` is off. * A model that answers the probe in prose instead of emitting a tool_call throws a loud diff --git a/test/provider-clipipe-mcp.test.js b/test/provider-clipipe-mcp.test.js new file mode 100644 index 0000000..0296393 --- /dev/null +++ b/test/provider-clipipe-mcp.test.js @@ -0,0 +1,414 @@ +'use strict'; + +// Offline, deterministic tests for CLIPipe NATIVE tool mode (BA-16, `toolProtocol:'claude-mcp'`). +// +// The bridge is exercised through its REAL unix socket by connecting as a client and speaking the +// same frames the stdio stub speaks — so the gate, both spin guards and the failure-mode contract +// are tested against the shipped code path, not a re-implementation of it. No CLI, no network, no +// API key. The live end-to-end (incl. the `--max-turns` tripwire) is `poc/ba16-native-shipped.mjs`. + +const { test, describe } = require('node:test'); +const assert = require('node:assert'); +const net = require('net'); +const { Loop } = require('../src/loop'); +const { HaltError } = require('../src/errors'); +const { CLIPipeProvider } = require('../src/provider-clipipe'); +const { createBridge, classifySubtype, resolveSessionError, toolErrorKey, safeErrorText } = require('../src/provider-clipipe-mcp'); + +/** Speak one bridge frame, exactly as the stdio stub does. */ +function bridgeCall(sockPath, payload) { + return new Promise((resolve, reject) => { + const sock = net.createConnection(sockPath); + let buf = ''; + sock.on('connect', () => sock.write(JSON.stringify(payload) + '\n')); + sock.on('data', (d) => { + buf += d; + const nl = buf.indexOf('\n'); + if (nl === -1) return; + let parsed; + try { parsed = JSON.parse(buf.slice(0, nl)); } catch (err) { return reject(err); } + sock.end(); + resolve(parsed); + }); + sock.on('error', reject); + }); +} + +const tool = (name, execute, parameters = { type: 'object', properties: {} }) => ({ name, description: `${name} tool`, parameters, execute }); + +describe('BA-16 bridge — the gate', () => { + test('a denied call returns the reason as a RESULT and never runs the handler', async () => { + let ran = 0; + const b = await createBridge({ + tools: [tool('writer', async () => { ran++; return 'wrote'; })], + policy: () => 'not allowed here', + }); + try { + const res = await bridgeCall(b.sockPath, { op: 'call', name: 'writer', args: { p: 1 } }); + assert.strictEqual(res.text, 'not allowed here'); + assert.ok(!res.isError, 'a deny is advisory feedback, not a tool error'); + assert.strictEqual(ran, 0, 'the handler must never run behind a deny'); + assert.strictEqual(b.state.terminal, null, 'one deny must not end the session'); + } finally { b.close(); } + }); + + test('an allowed call runs the caller closure and returns its value', async () => { + const b = await createBridge({ tools: [tool('adder', async (a) => `sum=${a.x + a.y}`)], policy: () => true }); + try { + const res = await bridgeCall(b.sockPath, { op: 'call', name: 'adder', args: { x: 2, y: 3 } }); + assert.strictEqual(res.text, 'sum=5'); + } finally { b.close(); } + }); + + test('tools/list serves the caller manifest (the stub never duplicates a tool definition)', async () => { + const b = await createBridge({ tools: [tool('alpha', async () => 'a'), tool('beta', async () => 'b')] }); + try { + const res = await bridgeCall(b.sockPath, { op: 'list' }); + assert.deepStrictEqual(res.tools.map((t) => t.name), ['alpha', 'beta']); + assert.ok(res.tools[0].inputSchema, 'the manifest must carry a schema the CLI can validate against'); + } finally { b.close(); } + }); + + test('a HaltError from the gate is captured as a clean governance exit, not a tool error', async () => { + const b = await createBridge({ + tools: [tool('writer', async () => 'wrote')], + policy: () => { throw new HaltError('budget cap', /** @type {any} */ ({ rule: 'budget' })); }, + }); + try { + await bridgeCall(b.sockPath, { op: 'call', name: 'writer', args: {} }); + assert.ok(b.state.halt instanceof HaltError, 'the halt must be preserved for the provider to re-throw'); + assert.ok(b.state.terminal, 'a halt must end the session'); + } finally { b.close(); } + }); +}); + +describe('BA-16 bridge — BA-11 deny-streak guard', () => { + test('ends the session on N consecutive denials', async () => { + const b = await createBridge({ tools: [tool('w', async () => 'ok')], policy: () => 'nope', maxConsecutiveDenials: 3 }); + try { + await bridgeCall(b.sockPath, { op: 'call', name: 'w', args: {} }); + await bridgeCall(b.sockPath, { op: 'call', name: 'w', args: {} }); + assert.strictEqual(b.state.terminal, null, 'must not fire before the threshold'); + await bridgeCall(b.sockPath, { op: 'call', name: 'w', args: {} }); + assert.strictEqual(b.state.terminal, 'denied:w'); + } finally { b.close(); } + }); + + // The negative control the guard exists to protect: deny X -> allow Y -> deny X is a model + // PIVOTING, which is exactly the allowlist-safe behavior a deny is advisory to permit. + test('any allowed call resets the streak (a legit pivot never trips it)', async () => { + let n = 0; + const b = await createBridge({ + tools: [tool('a', async () => 'ok'), tool('b', async () => 'ok')], + policy: (name) => (name === 'a' ? 'denied' : true), + maxConsecutiveDenials: 3, + }); + try { + for (const name of ['a', 'a', 'b', 'a', 'a']) { n++; await bridgeCall(b.sockPath, { op: 'call', name, args: {} }); } + assert.strictEqual(n, 5); + assert.strictEqual(b.state.terminal, null, 'an interleaved allowed call must reset the streak'); + } finally { b.close(); } + }); + + test('0 disables the guard', async () => { + const b = await createBridge({ tools: [tool('w', async () => 'ok')], policy: () => 'nope', maxConsecutiveDenials: 0 }); + try { + for (let i = 0; i < 6; i++) await bridgeCall(b.sockPath, { op: 'call', name: 'w', args: {} }); + assert.strictEqual(b.state.terminal, null); + } finally { b.close(); } + }); +}); + +describe('BA-16 bridge — BA-12 identical-tool-error guard', () => { + test('ends the session on N BYTE-IDENTICAL failures', async () => { + const b = await createBridge({ + tools: [tool('boom', async () => { throw new Error('ENOENT'); })], + maxIdenticalToolErrors: 3, + }); + try { + await bridgeCall(b.sockPath, { op: 'call', name: 'boom', args: { p: 'x' } }); + await bridgeCall(b.sockPath, { op: 'call', name: 'boom', args: { p: 'x' } }); + assert.strictEqual(b.state.terminal, null); + await bridgeCall(b.sockPath, { op: 'call', name: 'boom', args: { p: 'x' } }); + assert.strictEqual(b.state.terminal, 'stuck:boom'); + } finally { b.close(); } + }); + + // The measured reason the trigger is NARROWEST: counting any consecutive failure punishes a model + // varying its args while recovering — i.e. the exact behavior the guard exists to protect. + test('VARYING args never trips it, however many times it fails', async () => { + const b = await createBridge({ + tools: [tool('boom', async () => { throw new Error('ENOENT'); })], + maxIdenticalToolErrors: 3, + }); + try { + for (const p of ['a', 'b', 'c', 'd', 'e']) await bridgeCall(b.sockPath, { op: 'call', name: 'boom', args: { p } }); + assert.strictEqual(b.state.terminal, null, 'a model varying args is recovering, not spinning'); + } finally { b.close(); } + }); + + test('a success between failures resets the streak', async () => { + let fail = true; + const b = await createBridge({ + tools: [tool('flip', async () => { if (fail) throw new Error('nope'); return 'ok'; })], + maxIdenticalToolErrors: 3, + }); + try { + await bridgeCall(b.sockPath, { op: 'call', name: 'flip', args: {} }); + await bridgeCall(b.sockPath, { op: 'call', name: 'flip', args: {} }); + fail = false; + await bridgeCall(b.sockPath, { op: 'call', name: 'flip', args: {} }); + fail = true; + await bridgeCall(b.sockPath, { op: 'call', name: 'flip', args: {} }); + assert.strictEqual(b.state.terminal, null); + } finally { b.close(); } + }); + + // BA-12 had to count BOTH paths that feed an error back — a model looping on a name that does not + // exist spins exactly as hard as one looping on a throwing handler. + test('an unknown/hallucinated tool name feeds the SAME spin counter', async () => { + const b = await createBridge({ tools: [tool('real', async () => 'ok')], maxIdenticalToolErrors: 3 }); + try { + for (let i = 0; i < 3; i++) await bridgeCall(b.sockPath, { op: 'call', name: 'imaginary', args: {} }); + assert.strictEqual(b.state.terminal, 'stuck:imaginary'); + } finally { b.close(); } + }); +}); + +describe('BA-16 — subtype classification', () => { + test('success is the only clean finish', () => { + assert.deepStrictEqual(classifySubtype('success'), { stopReason: 'end_turn', error: null }); + }); + + // Finding 2's sibling: the bound stop must be NAMED and ERROR-TAGGED, never a clean success. + test('the turn bound is named AND error-tagged', () => { + assert.deepStrictEqual(classifySubtype('error_max_turns'), { stopReason: 'max_turns', error: 'max_turns' }); + }); + + test('a MISSING result event is not a success', () => { + assert.strictEqual(classifySubtype(undefined).error, 'session_incomplete'); + assert.strictEqual(classifySubtype('').error, 'session_incomplete'); + }); + + test('an unrecognized subtype is surfaced verbatim and still error-tagged', () => { + assert.deepStrictEqual(classifySubtype('error_something_new'), { stopReason: 'error_something_new', error: 'session:error_something_new' }); + }); + + // The proto-key footgun this repo has now fixed three times: `subtype` is provider-supplied, so an + // unguarded `MAP[subtype]` would resolve 'toString' to an inherited function and read as a mapping. + test('a prototype key does not resolve to an inherited Object.prototype member', () => { + const r = classifySubtype('toString'); + assert.strictEqual(r.error, 'session:toString'); + assert.strictEqual(r.stopReason, 'toString'); + }); +}); + +describe('BA-16 — broken-bridge detection (FINDING 1)', () => { + const facts = (over = {}) => ({ terminal: null, bridgeDown: false, attempted: 0, served: 0, timedOut: false, subtype: 'success', ...over }); + + // The measured failure this whole detector exists for: with the bridge dead, every tool call + // failed and the CLI STILL reported `subtype:'success'`. + test('attempted > served error-tags the run even when the CLI reports success', () => { + assert.strictEqual(resolveSessionError(facts({ attempted: 3, served: 0 })).error, 'bridge-failed'); + }); + + test('a partial bridge failure is caught too', () => { + assert.strictEqual(resolveSessionError(facts({ attempted: 5, served: 4 })).error, 'bridge-failed'); + }); + + // The negative control: without it, a fix that error-tags EVERY session would pass the test above. + test('a healthy session (attempted === served) stays a clean finish', () => { + const r = resolveSessionError(facts({ attempted: 4, served: 4 })); + assert.strictEqual(r.error, null); + assert.strictEqual(r.stopReason, 'end_turn'); + }); + + test('a tool-free session is clean, not a false alarm', () => { + assert.strictEqual(resolveSessionError(facts({ attempted: 0, served: 0 })).error, null); + }); + + test('served > attempted is never treated as a failure', () => { + assert.strictEqual(resolveSessionError(facts({ attempted: 2, served: 3 })).error, null); + }); + + test('a guard terminal outranks a broken bridge (it explains the missing subtype)', () => { + assert.strictEqual(resolveSessionError(facts({ terminal: 'denied:w', attempted: 3, served: 0 })).error, 'denied:w'); + }); + + test('a broken bridge outranks a timeout', () => { + assert.strictEqual(resolveSessionError(facts({ attempted: 1, served: 0, timedOut: true })).error, 'bridge-failed'); + }); + + test('a timeout is tagged when nothing more specific applies', () => { + assert.strictEqual(resolveSessionError(facts({ timedOut: true })).error, 'session_timeout'); + }); + + test('the bound survives the precedence chain', () => { + assert.strictEqual(resolveSessionError(facts({ subtype: 'error_max_turns', attempted: 2, served: 2 })).error, 'max_turns'); + }); +}); + +describe('BA-16 — diagnostics cannot become a secret channel', () => { + test('a non-Error throw is clamped, never serialized wholesale', () => { + const huge = { secret: 'sk-live-AAAAAAAAAAAAAAAAAAAA'.repeat(500) }; + const out = safeErrorText(huge); + assert.ok(out.length <= 301, `expected a clamped diagnostic, got ${out.length} chars`); + assert.ok(!out.includes('sk-live-'), 'an unknown-shape throw must not be serialized into the diagnostic'); + }); + + test('a long Error message is clamped', () => { + const out = safeErrorText(new Error('x'.repeat(5000))); + assert.ok(out.length <= 301); + }); + + test('the spin key is stable and args-sensitive', () => { + assert.strictEqual(toolErrorKey('a', { x: 1 }), toolErrorKey('a', { x: 1 })); + assert.notStrictEqual(toolErrorKey('a', { x: 1 }), toolErrorKey('a', { x: 2 })); + }); +}); + +// ── Loop integration: a cycle-owning provider must not be able to lie about its session ────────── + +/** Minimal cycle-owning provider stub — no CLI, no sockets. */ +function sessionProvider(session, extra = {}) { + return { + name: 'stub-session', + model: 'stub-model', + ownsCycle: true, + policy: () => true, + async generate() { + return { text: 'done', toolCalls: [], usage: { inputTokens: 10, outputTokens: 5 }, session, ...extra }; + }, + }; +} + +describe('BA-16 — Loop refuses options it could never honor', () => { + for (const knob of ['assemble', 'trim']) { + test(`a cycle-owning provider + ${knob} THROWS at construction, never sits silently dead`, () => { + assert.throws( + () => new Loop({ provider: sessionProvider({ turns: 1, toolCalls: 0, error: null, usageReported: false }), [knob]: () => [] }), + /owns its own turn cycle/, + ); + }); + } + + // The worst silently-dead knob: a fence that is not there while the run still looks governed. + test('a Loop-level policy THROWS when the provider owns the cycle and carries no gate', () => { + const p = sessionProvider({ turns: 1, toolCalls: 0, error: null, usageReported: false }); + p.policy = null; + assert.throws(() => new Loop({ provider: p, policy: () => true }), /policy would\s+NEVER run/); + }); + + test('the same Loop options remain legal on a normal provider (negative control)', () => { + const plain = { name: 'plain', async generate() { return { text: 'x', toolCalls: [], usage: {} }; } }; + assert.doesNotThrow(() => new Loop({ provider: plain, assemble: (m) => m, trim: (m) => m, policy: () => true })); + }); +}); + +describe('BA-16 — Loop reports a session honestly', () => { + test('metrics carry the REAL turn and tool-call counts, not 1 round', async () => { + const loop = new Loop({ provider: sessionProvider({ turns: 14, toolCalls: 9, error: null, usageReported: false }) }); + const out = await loop.run([{ role: 'user', content: 'go' }]); + assert.strictEqual(out.error, null); + assert.strictEqual(out.metrics.turns, 1, 'it really was one Loop round'); + assert.strictEqual(out.metrics.sessionTurns, 14, 'and 14 turns really happened inside it'); + assert.strictEqual(out.metrics.toolCalls, 9); + }); + + test('sessionTurns is 0 for a provider whose cycle the Loop drives (negative control)', async () => { + const plain = { name: 'plain', async generate() { return { text: 'x', toolCalls: [], usage: { inputTokens: 1, outputTokens: 1 } }; } }; + const out = await new Loop({ provider: plain }).run([{ role: 'user', content: 'go' }]); + assert.strictEqual(out.metrics.sessionTurns, 0); + }); + + // FINDING 1, as a regression test. Measured live before this shipped: with the tool bridge dead, + // every tool call failed and the CLI still ended `subtype:'success'`. Mapping that onto + // `error:null` would report a run in which NOTHING worked as converged. + test('a broken tool bridge error-tags the run even though the session reported success', async () => { + const loop = new Loop({ provider: sessionProvider({ turns: 4, toolCalls: 3, error: 'bridge-failed', usageReported: false }) }); + const out = await loop.run([{ role: 'user', content: 'go' }]); + assert.strictEqual(out.error, 'bridge-failed', 'a dead bridge must NEVER read as a clean finish'); + }); + + test('a bound stop is error-tagged, and BA-5 preserves the partial text', async () => { + const p = sessionProvider({ turns: 2, toolCalls: 1, error: 'max_turns', usageReported: false }); + const loop = new Loop({ provider: p }); + const out = await loop.run([{ role: 'user', content: 'go' }]); + assert.strictEqual(out.error, 'max_turns'); + assert.strictEqual(out.text, 'done', 'a bound firing is normal termination — the work must survive it'); + }); + + for (const tag of ['denied:writer', 'stuck:reader']) { + test(`a guard terminal (${tag}) surfaces as the run error`, async () => { + const out = await new Loop({ provider: sessionProvider({ turns: 3, toolCalls: 3, error: tag, usageReported: false }) }).run([{ role: 'user', content: 'go' }]); + assert.strictEqual(out.error, tag); + }); + } +}); + +describe('BA-16 — the gate is never billed twice, and never starved', () => { + test('usageReported:true suppresses the Loop forward (the provider already streamed it)', async () => { + const seen = []; + const loop = new Loop({ + provider: sessionProvider({ turns: 5, toolCalls: 2, error: null, usageReported: true }), + onLlmResult: (e) => { seen.push(e); }, + }); + await loop.run([{ role: 'user', content: 'go' }]); + assert.strictEqual(seen.length, 0, 'forwarding the summed total too would bill one session twice'); + }); + + test('usageReported:false still forwards, so an unwired onTurn cannot make a session look free', async () => { + const seen = []; + const loop = new Loop({ + provider: sessionProvider({ turns: 5, toolCalls: 2, error: null, usageReported: false }), + onLlmResult: (e) => { seen.push(e); }, + }); + await loop.run([{ role: 'user', content: 'go' }]); + assert.strictEqual(seen.length, 1); + assert.strictEqual(seen[0].usage.inputTokens, 10); + }); + + // Metering must still happen on the suppressed path — the tokens were really spent. + test('the meter records the tokens even when the gate forward is suppressed', async () => { + const out = await new Loop({ provider: sessionProvider({ turns: 5, toolCalls: 2, error: null, usageReported: true }) }).run([{ role: 'user', content: 'go' }]); + assert.strictEqual(out.metrics.tokens.input, 10); + assert.strictEqual(out.metrics.tokens.output, 5); + }); +}); + +describe('BA-16 — provider construction', () => { + test('claude-mcp declares ownsCycle and does NOT resolve an emulation protocol', () => { + const p = new CLIPipeProvider({ command: 'claude', toolProtocol: 'claude-mcp' }); + assert.strictEqual(p.ownsCycle, true); + assert.strictEqual(p.nativeTools, true); + assert.strictEqual(p.toolProtocol, null, 'native mode must not route through the envelope adapter'); + }); + + test('the v0.32.0 emulation mode is untouched (negative control)', () => { + const p = new CLIPipeProvider({ command: 'claude', toolProtocol: 'claude' }); + assert.ok(!p.ownsCycle, 'emulation does NOT own the cycle — the Loop still drives it'); + assert.ok(p.toolProtocol, 'emulation still resolves its envelope adapter'); + }); + + test('a plain CLIPipeProvider is unchanged (negative control)', () => { + const p = new CLIPipeProvider({ command: 'echo' }); + assert.ok(!p.ownsCycle); + assert.strictEqual(p.toolProtocol, null); + }); + + test('a non-function policy is rejected at construction', () => { + assert.throws(() => new CLIPipeProvider({ command: 'claude', toolProtocol: 'claude-mcp', policy: 'yes' }), /policy must be a function/); + }); + + test('cacheMessages in native mode throws rather than sitting silently dead', async () => { + const p = new CLIPipeProvider({ command: 'claude', toolProtocol: 'claude-mcp' }); + await assert.rejects( + () => p.generate([{ role: 'user', content: 'hi' }], [tool('t', async () => 'x')], { cacheMessages: true }), + /cacheMessages cannot apply in native tool mode/, + ); + }); + + test('an unknown toolProtocol still throws', () => { + assert.throws(() => new CLIPipeProvider({ command: 'claude', toolProtocol: 'codex' }), /unknown toolProtocol/); + }); +}); diff --git a/types/index.d.ts b/types/index.d.ts index 2d3f0e0..77c9743 100644 --- a/types/index.d.ts +++ b/types/index.d.ts @@ -34,6 +34,13 @@ export interface RunMetrics { turns: number; /** Total tool calls the model made (every invocation, including denied/unknown). */ toolCalls: number; + /** + * BA-16 — turns that happened INSIDE a cycle-owning provider's own session (CLIPipe native tool + * mode). Such a session is ONE Loop round however many turns it really took, so `turns` alone + * would report a 14-turn session as 1 — a round count that reads far cheaper and shorter than the + * run actually was. 0 for every provider whose cycle the Loop drives. + */ + sessionTurns: number; /** Per-tool invocation counts, keyed by tool name. */ byTool: Record; /** Cumulative token spend across all rounds (incl. summarize calls), by tier. */ @@ -97,6 +104,26 @@ export interface GenerateResult { * pre-BA-6 behavior exactly, so an unmapped provider degrades to the status quo. */ stopReason?: string | null; + /** + * BA-16 — present ONLY when the provider ran its own multi-turn session for this single call + * (`Provider.ownsCycle`), e.g. CLIPipe native tool mode, where the CLI executes the caller's tools + * natively over MCP and keeps going until it answers or hits a bound. + * + * It exists so the Loop can stay honest about a call that was not one turn: + * - `turns` / `toolCalls` — what really happened, so `metrics` cannot report a 14-turn session as 1. + * - `error` — a terminal the provider detected INSIDE the session (a turn bound, a deny/stuck + * streak, a broken tool bridge). The Loop surfaces it as the run's `error`, never merely as a + * field: every downstream consumer branches on `error` as the sole success signal, so surfacing + * alone would let a session in which no tool call ever succeeded propagate as converged. + * - `usageReported` — the provider ALREADY forwarded this call's usage per internal turn, so the + * Loop must not forward the summed total again and bill the gate twice. + */ + session?: { + turns: number; + toolCalls: number; + error: string | null; + usageReported: boolean; + }; /** * True when the requested `temperature` was rejected by the model (400, unsupported/deprecated) and * the request was retried without it (BA-10). The response was produced at the model's DEFAULT @@ -165,6 +192,19 @@ export interface Provider { model?: string | null; /** Provider name, surfaced in onLlmResult. */ name?: string | null; + /** + * BA-16 — true when the provider runs its OWN multi-turn cycle inside a single `generate()` call + * (CLIPipe native tool mode), executing the caller's tools itself rather than returning + * `toolCalls` for the Loop to run. + * + * The Loop reads this to REFUSE options it could never honor rather than accept them and leave + * them silently dead: `assemble`/`trim` (the provider owns the transcript) and, most importantly, + * a Loop-level `policy` — no tool call reaches the Loop, so that fence would simply not be there + * while the run still looked governed. Such a provider must carry its own `policy`. + */ + ownsCycle?: boolean; + /** Gate chokepoint for a cycle-owning provider — same contract as `Loop({policy})`. */ + policy?: ((tool: string, args: any, ctx?: any) => any) | null; generate( messages: Message[], tools?: ToolDef[], From dddab59b2f673eed5bbff8c63659f25e5785f007 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Tue, 21 Jul 2026 19:41:53 +0200 Subject: [PATCH 2/4] =?UTF-8?q?fix:=20serialize=20CLIPipe=20session-stream?= =?UTF-8?q?=20framing=20=E2=80=94=20async=20onTurn=20corrupted=20the=20lin?= =?UTF-8?q?e=20buffer=20(code-review)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /code-review medium on the BA-16 diff caught a real latent data-loss bug in `runSession`. The stdout handler was `async` and `await`ed `onTurn` INSIDE its line-parse loop, which mutated a shared `buf`. While that await was suspended, the next `'data'` event re-entered the handler and sliced `buf` under the suspended parse — dropping, duplicating, or mangling lines (lost per-turn usage, a missed `result` event, a miscounted attempted-vs-served bridge check). Invisible with a synchronous `onTurn` (the await resolves on the microtask queue before the next macrotask `'data'` event), which is why the live verify-shipped passed — its onTurn was a plain push. REACHABLE the moment onTurn does real async work: a wired gate's `gate.record`, the actual intended consumer. Fix: extract `createSessionStream` — framing is now purely synchronous (never awaits), and per-turn `onTurn` forwards run through a SERIAL async drainer in arrival order; `done()` flushes the queue before resolving so a session that ends with a forward pending still surfaces that turn's spend (F12/F18). Extracting it also made the parser unit-testable without spawning the CLI. +6 regression tests (async onTurn + chunks split across boundaries, in-order delivery, split-line framing, HaltError-in-drain, malformed-line skip); the serial-drain guard is mutation-proved load-bearing. Suite 968 / 966 pass / 0 fail; typecheck clean; live verify-shipped re-run all green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MN6WmkWeLudzKB7PNsQT92 --- src/provider-clipipe-mcp.js | 188 +++++++++++++++++++----------- test/provider-clipipe-mcp.test.js | 62 +++++++++- 2 files changed, 178 insertions(+), 72 deletions(-) diff --git a/src/provider-clipipe-mcp.js b/src/provider-clipipe-mcp.js index 7ae0bde..5595c5d 100644 --- a/src/provider-clipipe-mcp.js +++ b/src/provider-clipipe-mcp.js @@ -295,6 +295,111 @@ function resolveSessionError(facts) { return fromSubtype; } +/** + * Line-oriented processor for the CLI's `stream-json` stdout. + * + * FRAMING IS SYNCHRONOUS. The obvious shape — an `async` stdout handler that `await`s `onTurn` + * inside its parse loop — has a data-corruption bug: while the `await` is suspended, the next + * `'data'` event re-enters the handler and mutates the SHARED line buffer, so the suspended parse + * resumes against a buffer that moved under it (dropped/duplicated/mangled lines). It is invisible + * with a synchronous `onTurn` (the await resolves on the microtask queue before the next macrotask + * `'data'` event) and REACHABLE the moment `onTurn` does real async work — e.g. a wired gate's + * `gate.record`, which is exactly the intended consumer. So framing never awaits: it parses lines + * synchronously and pushes per-turn forwards onto a queue that a SERIAL async drainer empties in + * arrival order. Extracted from `runSession` so this can be unit-tested without spawning the CLI. + * + * @param {object} o + * @param {Function|null} o.onTurn + * @param {any} o.ctx + * @param {number} o.startedAt + * @param {(err: Error) => void} o.onHalt - called if a forwarded `onTurn` throws a HaltError. + */ +function createSessionStream({ onTurn, ctx, startedAt, onHalt }) { + let buf = ''; + let attempted = 0; + let final = null; + /** @type {import('../types').Usage[]} */ const turns = []; + /** @type {any[]} */ const queue = []; + /** @type {Promise|null} */ let draining = null; + + async function drain() { + if (!onTurn) return; + while (queue.length) { + const payload = queue.shift(); + try { + await onTurn(payload); + } catch (err) { + // A governance HaltError from the gate ends the session; anything else is surfaced, not fatal. + if (err instanceof HaltError) { queue.length = 0; onHalt(/** @type {Error} */ (err)); return; } + } + } + } + function pump() { + if (!onTurn || draining) return; + draining = drain().finally(() => { draining = null; if (queue.length) pump(); }); + } + + return { + /** Feed a stdout chunk. Pure synchronous framing — never awaits. */ + feed(/** @type {Buffer|string} */ chunk) { + buf += chunk; + let nl; + while ((nl = buf.indexOf('\n')) !== -1) { + const line = buf.slice(0, nl); + buf = buf.slice(nl + 1); + if (!line.trim()) continue; + let ev; + try { ev = JSON.parse(line); } catch (_) { continue; } + + // Count tool calls the CLI ATTEMPTED against our server. Compared later with what the bridge + // actually SERVED, this is the only precise parent-side detector of a broken bridge: if the + // stub cannot reach the socket our server never sees the connection, so a server-side flag + // stays clean while every tool call fails — and the session still ends `subtype:'success'` + // (measured). Attempted > served is the honest signal. + if (ev.type === 'assistant' && ev.message && Array.isArray(ev.message.content)) { + for (const block of ev.message.content) { + if (block && block.type === 'tool_use' && typeof block.name === 'string' + && block.name.startsWith(`mcp__${SERVER_NAME}__`)) attempted++; + } + } + + if (ev.type === 'assistant' && ev.message && ev.message.usage) { + const u = ev.message.usage; + /** @type {import('../types').Usage} */ + const usage = { + inputTokens: Number(u.input_tokens) || 0, + outputTokens: Number(u.output_tokens) || 0, + }; + // Omit an absent tier rather than emit a synthetic 0 (per the Usage contract). + if (Number.isFinite(u.cache_read_input_tokens)) usage.cacheReadTokens = u.cache_read_input_tokens; + if (Number.isFinite(u.cache_creation_input_tokens)) usage.cacheCreationTokens = u.cache_creation_input_tokens; + turns.push(usage); + // Stream it: a session that dies mid-run must already have surfaced every completed turn's + // spend, or the gate loses it. Queued (not awaited here) so framing cannot race the buffer. + if (onTurn) queue.push({ + model: (ev.message && ev.message.model) || null, + provider: 'clipipe', + usage, + costUsd: null, // the CLI prices the SESSION, not the turn — explicitly unpriced, never a synthetic 0. + pricing: 'unpriced', + durationMs: Date.now() - startedAt, + ctx, // what a wired gate records spend against — same as the Loop's onLlmResult. + kind: 'turn', + }); + } + + if (ev.type === 'result') final = ev; + } + pump(); + }, + /** Await every queued per-turn forward — call before resolving the session. */ + async flush() { if (draining) await draining; await drain(); }, + get turns() { return turns; }, + get attempted() { return attempted; }, + get final() { return final; }, + }; +} + /** * Run ONE native CLI session to completion, streaming per-turn usage as it arrives. * @@ -341,17 +446,22 @@ function runSession(opts) { return new Promise((resolve) => { const child = spawn(command, args, { cwd, env, stdio: ['pipe', 'pipe', 'pipe'] }); - /** @type {any[]} */ const turns = []; - let final = null, buf = '', stderr = '', settled = false; - let attempted = 0; // MCP tool calls the CLI tried to make against our server + let stderr = '', settled = false; /** @type {Error|null} */ let turnHalt = null; const started = Date.now(); - const done = (extra = {}) => { + /** Kill the session now — a guard tripped or governance halted mid-flight. */ + const abort = () => { try { child.kill('SIGTERM'); } catch (_) { /* already gone */ } }; + const stream = createSessionStream({ onTurn, ctx, startedAt: started, onHalt: (err) => { turnHalt = err; abort(); } }); + + const done = async (extra = {}) => { if (settled) return; settled = true; clearTimeout(timer); - resolve({ turns, final, stderr, ms: Date.now() - started, turnHalt, attempted, ...extra }); + // Flush queued per-turn forwards before resolving — a session that ends while a forward is + // pending must still surface that turn's spend to the gate (F12/F18). + try { await stream.flush(); } catch (_) { /* onTurn failures are surfaced in-drain, never fatal */ } + resolve({ turns: stream.turns, final: stream.final, stderr, ms: Date.now() - started, turnHalt, attempted: stream.attempted, ...extra }); }; const timer = setTimeout(() => { @@ -359,72 +469,7 @@ function runSession(opts) { done({ timedOut: true }); }, timeoutMs); - /** Kill the session now — a guard tripped or governance halted mid-flight. */ - const abort = () => { try { child.kill('SIGTERM'); } catch (_) { /* already gone */ } }; - - child.stdout.on('data', async (d) => { - buf += d; - let nl; - while ((nl = buf.indexOf('\n')) !== -1) { - const line = buf.slice(0, nl); - buf = buf.slice(nl + 1); - if (!line.trim()) continue; - let ev; - try { ev = JSON.parse(line); } catch (_) { continue; } - - // Count tool calls the CLI ATTEMPTED against our server. Compared later with what the bridge - // actually SERVED, this is the only precise parent-side detector of a broken bridge: if the - // stub cannot reach the socket, our server never sees the connection at all, so a - // server-side error flag would stay clean while every tool call failed — and the session - // would still end `subtype:'success'` (measured). Attempted > served is the honest signal. - if (ev.type === 'assistant' && ev.message && Array.isArray(ev.message.content)) { - for (const block of ev.message.content) { - if (block && block.type === 'tool_use' && typeof block.name === 'string' - && block.name.startsWith(`mcp__${SERVER_NAME}__`)) attempted++; - } - } - - if (ev.type === 'assistant' && ev.message && ev.message.usage) { - const u = ev.message.usage; - /** @type {import('../types').Usage} */ - const usage = { - inputTokens: Number(u.input_tokens) || 0, - outputTokens: Number(u.output_tokens) || 0, - }; - // Omit an absent tier rather than emit a synthetic 0 (per the Usage contract). - if (Number.isFinite(u.cache_read_input_tokens)) usage.cacheReadTokens = u.cache_read_input_tokens; - if (Number.isFinite(u.cache_creation_input_tokens)) usage.cacheCreationTokens = u.cache_creation_input_tokens; - turns.push(usage); - - // Stream it NOW, never sum-at-end: a session that dies mid-run must already have - // surfaced every completed turn's spend, or the gate loses it entirely. - if (onTurn) { - try { - await onTurn({ - model: (ev.message && ev.message.model) || null, - provider: 'clipipe', - usage, - // A CLI model has no rate-table entry, and the CLI prices the SESSION, not the turn. - // Explicitly unpriced beats a synthetic 0 — that silent zero is what made a budget - // cap a no-op once already. - costUsd: null, - pricing: 'unpriced', - durationMs: Date.now() - started, - // ctx is what a wired gate records spend against — forward it, same as the Loop's onLlmResult. - ctx, - kind: 'turn', - }); - } catch (err) { - if (err instanceof HaltError) { turnHalt = err; abort(); return; } - // A governance callback failure is not a run failure — surfaced, never fatal. - } - } - } - - if (ev.type === 'result') final = ev; - } - }); - + child.stdout.on('data', (d) => stream.feed(d)); child.stderr.on('data', (d) => { stderr += d; }); child.on('error', (err) => done({ spawnError: err })); child.on('close', () => done()); @@ -442,5 +487,6 @@ module.exports = { createBridge, classifySubtype, resolveSessionError, + createSessionStream, runSession, }; diff --git a/test/provider-clipipe-mcp.test.js b/test/provider-clipipe-mcp.test.js index 0296393..5edbb75 100644 --- a/test/provider-clipipe-mcp.test.js +++ b/test/provider-clipipe-mcp.test.js @@ -13,7 +13,7 @@ const net = require('net'); const { Loop } = require('../src/loop'); const { HaltError } = require('../src/errors'); const { CLIPipeProvider } = require('../src/provider-clipipe'); -const { createBridge, classifySubtype, resolveSessionError, toolErrorKey, safeErrorText } = require('../src/provider-clipipe-mcp'); +const { createBridge, classifySubtype, resolveSessionError, createSessionStream, toolErrorKey, safeErrorText } = require('../src/provider-clipipe-mcp'); /** Speak one bridge frame, exactly as the stdio stub does. */ function bridgeCall(sockPath, payload) { @@ -175,6 +175,66 @@ describe('BA-16 bridge — BA-12 identical-tool-error guard', () => { }); }); +describe('BA-16 — session stream framing (async onTurn must not corrupt the buffer)', () => { + const asstUsage = (n, io) => JSON.stringify({ type: 'assistant', message: { model: 'm', usage: { input_tokens: io, output_tokens: n, cache_read_input_tokens: io, cache_creation_input_tokens: 0 } } }); + const toolBlock = () => JSON.stringify({ type: 'assistant', message: { content: [{ type: 'tool_use', name: 'mcp__bareagent__x' }] } }); + + // THE regression: an async onTurn that yields, with chunks arriving DURING the await. The old + // `await onTurn` inside the stdout handler let a re-entrant chunk mutate the shared buffer; here + // every turn must still arrive, in order, with no lost/duplicated/mangled lines. + test('every turn is delivered in order under an async onTurn with interleaved chunks', async () => { + const seen = []; + const stream = createSessionStream({ + onTurn: async (e) => { await new Promise((r) => setTimeout(r, 5)); seen.push(e.usage.outputTokens); }, + ctx: null, startedAt: Date.now(), onHalt: () => {}, + }); + // Feed lines split ACROSS chunk boundaries while the drainer is mid-await. + stream.feed(asstUsage(1, 10) + '\n' + asstUsage(2, 10).slice(0, 20)); + stream.feed(asstUsage(2, 10).slice(20) + '\n' + asstUsage(3, 10) + '\n'); + stream.feed(asstUsage(4, 10) + '\n'); + await stream.flush(); + assert.deepStrictEqual(seen, [1, 2, 3, 4], 'all four turns, in order, none dropped or merged'); + assert.strictEqual(stream.turns.length, 4); + }); + + test('a line split across two chunks is framed as one turn, never two', async () => { + const seen = []; + const stream = createSessionStream({ onTurn: async (e) => { seen.push(e.usage.outputTokens); }, ctx: null, startedAt: Date.now(), onHalt: () => {} }); + const line = asstUsage(7, 3); + stream.feed(line.slice(0, 15)); + stream.feed(line.slice(15) + '\n'); + await stream.flush(); + assert.deepStrictEqual(seen, [7]); + }); + + test('attempted counts only bareagent MCP tool_use blocks', async () => { + const stream = createSessionStream({ onTurn: null, ctx: null, startedAt: Date.now(), onHalt: () => {} }); + stream.feed(toolBlock() + '\n'); + stream.feed(JSON.stringify({ type: 'assistant', message: { content: [{ type: 'tool_use', name: 'mcp__other__y' }, { type: 'text', text: 'hi' }] } }) + '\n'); + await stream.flush(); + assert.strictEqual(stream.attempted, 1, 'only our server-prefixed tool calls count toward attempted'); + }); + + test('a HaltError from onTurn stops the drain and reports via onHalt', async () => { + let halted = null; + const stream = createSessionStream({ + onTurn: async () => { throw new HaltError('budget', /** @type {any} */ ({ rule: 'b' })); }, + ctx: null, startedAt: Date.now(), onHalt: (err) => { halted = err; }, + }); + stream.feed(asstUsage(1, 1) + '\n'); + await stream.flush(); + assert.ok(halted instanceof HaltError, 'a gate halt during a per-turn forward must surface'); + }); + + test('malformed JSON lines are skipped, not fatal', async () => { + const seen = []; + const stream = createSessionStream({ onTurn: async (e) => { seen.push(e.usage.outputTokens); }, ctx: null, startedAt: Date.now(), onHalt: () => {} }); + stream.feed('{ not json\n' + asstUsage(9, 1) + '\n'); + await stream.flush(); + assert.deepStrictEqual(seen, [9]); + }); +}); + describe('BA-16 — subtype classification', () => { test('success is the only clean finish', () => { assert.deepStrictEqual(classifySubtype('success'), { stopReason: 'end_turn', error: null }); From 5dd90a862b38348f85d73114ea70d87841c6d930 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Tue, 21 Jul 2026 19:49:03 +0200 Subject: [PATCH 3/4] chore: resolve the three code-review "considered" items (validate + cleanup) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validated all three deferred findings from the medium review; fixed the ones that passed as real, with no regression. - Broken-bridge detector (was: true-positive only unit-tested): the risk was never a false positive — it was that if the REAL stream-json tool_use shape differed from what `createSessionStream` counts, `attempted` would be 0 on every run and the detector would be silent dead code. Closed by capturing real stream-json from a live native session (poc/ba16-detector-live.mjs): blocks arrive as `{type:'tool_use', name:'mcp__bareagent__lookup_code'}`, attempted===served on a healthy run — the counter is wired to reality (a dead bridge → attempted>served → bridge-failed). No code change; a real validation gap now has a durable proof. - `state.calls`: validated as DEAD state — written twice, read nowhere (not by shipped code, not by tests). Removed (declaration + both pushes), not made "consistent" — every line must have a purpose. - Parallel tool-call races on the guard counters: validated as NOT a defect — the affected counters are advisory bounds whose sequential-retry-spin precondition does not hold for simultaneous calls, and the worst case is benign + gate-bounded. No fix; documented on handleCall so it is not re-flagged. Suite 968 / 966 pass / 0 fail; typecheck clean; detector-live re-run GREEN. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MN6WmkWeLudzKB7PNsQT92 --- poc/ba16-detector-live.mjs | 75 +++++++++++++++++++++++++++++++++++++ src/provider-clipipe-mcp.js | 10 +++-- 2 files changed, 82 insertions(+), 3 deletions(-) create mode 100644 poc/ba16-detector-live.mjs diff --git a/poc/ba16-detector-live.mjs b/poc/ba16-detector-live.mjs new file mode 100644 index 0000000..e0b1693 --- /dev/null +++ b/poc/ba16-detector-live.mjs @@ -0,0 +1,75 @@ +// BA-16 — prove the broken-bridge detector is LIVE, not a silent no-op. +// +// `createSessionStream` counts a tool call as ATTEMPTED when the CLI's stream-json carries an +// `assistant.message.content[]` block of `type:'tool_use'` whose `name` starts `mcp__bareagent__`. +// `attempted > served` (served = calls the bridge actually handled) is the ONLY parent-side signal +// that a dead bridge failed every tool call while the CLI still reported `subtype:'success'`. +// +// The risk this closes: if the REAL stream-json block shape differed from what the counter reads, +// `attempted` would be 0 on every run — and the detector would be dead code that never fires. The +// unit tests prove the math on a hand-built shape; only a live capture proves the shape is real. +// +// Requires the `claude` CLI, logged in. ~$0.02 notional. Run: node poc/ba16-detector-live.mjs +import net from 'node:net'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { spawn } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; + +const STUB = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', 'src', 'mcp-bridge-stub.js'); + +const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ba16-detector-')); +const sock = path.join(dir, 'b.sock'); +let served = 0; + +// A minimal real bridge — enough for the CLI to make one genuine tool call. +const server = net.createServer((conn) => { + let buf = ''; + conn.on('data', (d) => { + buf += d; const nl = buf.indexOf('\n'); if (nl === -1) return; + let req; try { req = JSON.parse(buf.slice(0, nl)); } catch { return conn.end(); } buf = ''; + if (req.op === 'list') return conn.write(JSON.stringify({ tools: [{ name: 'lookup_code', description: 'Returns a code for a name.', inputSchema: { type: 'object', properties: { name: { type: 'string' } } } }] }) + '\n'); + if (req.op === 'call') { served++; conn.write(JSON.stringify({ text: 'code: ZZ-9' }) + '\n'); } + }); + conn.on('error', () => {}); +}); +await new Promise((r) => server.listen(sock, r)); + +const cfg = JSON.stringify({ mcpServers: { bareagent: { command: process.execPath, args: [STUB], env: { BAREAGENT_BRIDGE_SOCK: sock } } } }); +const child = spawn('claude', [ + '--model', 'sonnet', '-p', '--mcp-config', cfg, '--tools', '', '--strict-mcp-config', + '--setting-sources', '', '--allowedTools', 'mcp__bareagent__*', + '--system-prompt', 'Use the lookup_code tool to answer. Be brief.', + '--output-format', 'stream-json', '--verbose', +], { stdio: ['pipe', 'pipe', 'pipe'] }); + +let out = ''; +child.stdout.on('data', (d) => { out += d; }); +child.stdin.end('What is the verification code for "test-1"?'); +await new Promise((r) => child.on('close', r)); +server.close(); + +// Count exactly as src/provider-clipipe-mcp.js createSessionStream does. +let attempted = 0; +const shapes = []; +for (const line of out.split('\n')) { + if (!line.trim()) continue; + let ev; try { ev = JSON.parse(line); } catch { continue; } + if (ev.type === 'assistant' && ev.message && Array.isArray(ev.message.content)) { + for (const b of ev.message.content) { + if (b && b.type === 'tool_use') { + shapes.push({ type: b.type, name: b.name }); + if (typeof b.name === 'string' && b.name.startsWith('mcp__bareagent__')) attempted++; + } + } + } +} + +console.log('tool_use blocks on the real wire:', JSON.stringify(shapes)); +console.log(`attempted (matched mcp__bareagent__*) = ${attempted} · served by bridge = ${served}`); +const ok = attempted >= 1 && attempted === served; +console.log(ok + ? 'GREEN — detector is LIVE: real tool_use blocks match the counted shape; attempted===served on a healthy run (a dead bridge → attempted>served → bridge-failed)' + : 'RED — shape mismatch: the broken-bridge detector would be a no-op on the real wire'); +process.exit(ok ? 0 : 1); diff --git a/src/provider-clipipe-mcp.js b/src/provider-clipipe-mcp.js index 5595c5d..f0c4b02 100644 --- a/src/provider-clipipe-mcp.js +++ b/src/provider-clipipe-mcp.js @@ -120,7 +120,6 @@ async function createBridge({ tools, policy = null, ctx = undefined, maxConsecut const sockPath = path.join(dir, 'bridge.sock'); const state = { - /** @type {{name: string, denied: boolean}[]} */ calls: [], toolCalls: 0, consecutiveDenials: 0, /** @type {{key: string, count: number}} */ lastError: { key: '', count: 0 }, @@ -135,6 +134,13 @@ async function createBridge({ tools, policy = null, ctx = undefined, maxConsecut /** * Handle one `tools/call`. Every outcome is a RESULT the model can act on; the only thing that * ends the session is a guard tripping, which is reported through `state.terminal`. + * + * The CLI can fire PARALLEL tool calls (multiple `tool_use` blocks in one turn), so two of these + * can be in flight across their `await` points. That is safe here: `state.toolCalls++` is a + * synchronous atomic bump, and the two guard counters are ADVISORY bounds whose precondition — a + * SEQUENTIAL retry spin — does not apply to calls the model issued simultaneously. The worst case + * (an early `stuck:` on genuinely-identical parallel failures) is benign and gate-bounded, so no + * per-call lock is warranted. */ async function handleCall(name, args) { state.toolCalls++; @@ -155,7 +161,6 @@ async function createBridge({ tools, policy = null, ctx = undefined, maxConsecut } if (verdict !== true) { state.consecutiveDenials++; - state.calls.push({ name, denied: true }); const reason = typeof verdict === 'string' ? verdict : `denied by policy: ${name}`; // BA-11: a single deny stays ADVISORY so the model can pivot to an allowed tool // (allowlist-safe). Only a STREAK with no allowed call in between ends the session. @@ -168,7 +173,6 @@ async function createBridge({ tools, policy = null, ctx = undefined, maxConsecut state.consecutiveDenials = 0; // any allowed call resets the streak } - state.calls.push({ name, denied: false }); try { const out = await tool.execute(args || {}); state.lastError = { key: '', count: 0 }; // success resets the identical-error streak From b62a4e1b11d6f862663e84d8785a1e4fc8d96a90 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Tue, 21 Jul 2026 19:55:21 +0200 Subject: [PATCH 4/4] =?UTF-8?q?release:=20v0.33.0=20=E2=80=94=20CLIPipe=20?= =?UTF-8?q?native=20MCP=20tool=20mode=20(BA-16)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Native `toolProtocol:'claude-mcp'` runs one CLI session per call and executes the caller's tools over an MCP bridge back into in-process closures — the CLI owns the cycle, governance (policy/guards/turn-bound) rides the provider, per-turn usage streams, and honest accounting keeps a multi-turn session from reading as one round. Emulation (v0.32.0) retained for MCP-less CLIs; the claim is cost (measured), behaviour parity deliberately unminted. Includes the code-review fix (serial session-stream framing — async onTurn no longer corrupts the line buffer) and the dead-`state.calls` cleanup. - version bumped in package.json, package-lock.json, bareagent.context.md - CHANGELOG 0.33.0; PRD §22 decision entry; README two-mode note - build:types JSDoc type-contract fix (Provider.policy signature) Gates: tests 966 pass / 0 fail / 2 skipped; typecheck + build:types clean; /security clean (argv spawn, 0600 unix socket, no bridge logging); /diff-review via /code-review medium (1 bug fixed, 3 items validated). Live verify-shipped + fence + detector-live all GREEN. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01MN6WmkWeLudzKB7PNsQT92 --- CHANGELOG.md | 2 +- README.md | 2 +- bareagent.context.md | 2 +- docs/01-product/prd.md | 33 +++++++++++++++++++++++++++++++++ package-lock.json | 4 ++-- package.json | 2 +- src/provider-clipipe.js | 2 +- 7 files changed, 40 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a46624..ceb0d8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog](https://keepachangelog.com/). Versioning: [SemVer](https://semver.org/). -## [Unreleased] +## [0.33.0] - 2026-07-21 ### Added diff --git a/README.md b/README.md index 3e9ab71..cc7dce4 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,7 @@ console.log(result.count, result.matchedIds); // a code-derived count + the id **Govern — one gate over both axes.** `wireGate(gate)` routes every LLM + tool call through one bareguard policy + audit + budget. Denied tools never reach the model; halts (turn / budget / content caps) exit cleanly. A plain deny stays advisory (the model can pivot to an allowed tool), but the Loop short-circuits a *spin* — `maxConsecutiveDenials` consecutive denials of the same action (default 3) stop the run with `error:'denied:'` instead of burning the budget to the cap; under `recurse` that surfaces as `{ incomplete, blocker:'governance-deny' }`. The same bound covers a tool that keeps *failing*: `maxIdenticalToolErrors` (default 3) stops a model re-sending a byte-identical call that cannot succeed, with `error:'stuck:'`. `require('bare-agent/bareguard')` -**Providers:** OpenAI-compatible (OpenAI, OpenRouter, Groq, vLLM, LM Studio), Anthropic, Gemini (native), Ollama, CLIPipe, Fallback — or bring your own (one `generate` method). All return the same shape; swap freely. Usage including prompt-cache tiers is normalized, so `result.metrics` reports honest cumulative tokens + cost — and `null`, never a silent `0`, for a model it couldn't price. A model that rejects a non-default `temperature` (e.g. `claude-sonnet-5`, OpenAI o1/gpt-5-class return a `400`) is handled gracefully — the provider drops the param and retries once rather than failing the call, surfacing `temperatureDropped` so a caller can report the effective value. **CLIPipe can drive a full agentic Loop — tools and turns — over a CLI *subscription* instead of the metered API** (`CLIPipeProvider({ command:'claude', args:['-p','--model','sonnet'], toolProtocol:'claude' })`): schema-validated tool emulation with the Loop keeping governance, and a capable model enforced upfront (v0.32.0, sonnet-class+ for tools; Claude CLI for now). +**Providers:** OpenAI-compatible (OpenAI, OpenRouter, Groq, vLLM, LM Studio), Anthropic, Gemini (native), Ollama, CLIPipe, Fallback — or bring your own (one `generate` method). All return the same shape; swap freely. Usage including prompt-cache tiers is normalized, so `result.metrics` reports honest cumulative tokens + cost — and `null`, never a silent `0`, for a model it couldn't price. A model that rejects a non-default `temperature` (e.g. `claude-sonnet-5`, OpenAI o1/gpt-5-class return a `400`) is handled gracefully — the provider drops the param and retries once rather than failing the call, surfacing `temperatureDropped` so a caller can report the effective value. **CLIPipe can drive a full agentic Loop — tools and turns — over a CLI *subscription* instead of the metered API.** Two modes, chosen by cost: `toolProtocol:'claude-mcp'` (v0.33.0, NATIVE — the CLI runs its own session and calls your tools over an MCP bridge; the gate, spin guards, and turn bound ride the provider, and it caches session-side — ~$0.006/turn) is the default for the Claude CLI, while `toolProtocol:'claude'` (v0.32.0, schema-validated emulation with the Loop keeping the cycle) stays right for a CLI with no MCP channel. A capable model is enforced upfront (sonnet-class+ for tools; Claude CLI for now). **The run tells you the truth about what happened on the wire.** Every provider reports **why** generation ended (`stopReason`, normalized across all of them and surfaced on **every** `Loop.run()` return), so a round the API **cut off at the token cap** can no longer masquerade as a finished answer: it returns `error: 'truncated:max_tokens'` with the partial text preserved, instead of a silent `error: null` (BA-6). Its tool calls are **refused, never executed** — a *complete* tool call always arrives tagged `tool_use`, so one riding a truncated round was cut off mid-generation with arguments missing, which is exactly how a truncated `shell_write` can zero a file. The same honesty now covers **every** non-clean terminal signal (BA-13): a safety `refusal` returns `error: 'refusal'` and a blown context window returns `error: 'context_exceeded'` (both were previously laundered into an empty success), while a resumable `pause_turn` makes the loop resume rather than terminate. `error` is the sole success signal; a bound firing preserves the model's work rather than discarding it (BA-5). diff --git a/bareagent.context.md b/bareagent.context.md index 229029e..f702a2a 100644 --- a/bareagent.context.md +++ b/bareagent.context.md @@ -1,7 +1,7 @@ # bareagent — Integration Guide > For AI assistants and developers wiring bareagent into a project. -> v0.32.0 | Node.js >= 18 | zero required deps (`bareguard >=0.9.0 <0.13.0` optional peer for governance) | Apache 2.0 +> v0.33.0 | Node.js >= 18 | zero required deps (`bareguard >=0.9.0 <0.13.0` optional peer for governance) | Apache 2.0 > > Full human guide with composition examples, design philosophy, and recipes: [Usage Guide](docs/02-features/usage-guide.md) diff --git a/docs/01-product/prd.md b/docs/01-product/prd.md index 9860b24..5a3fe97 100644 --- a/docs/01-product/prd.md +++ b/docs/01-product/prd.md @@ -809,6 +809,39 @@ re-litigated unless the user explicitly asks. > back into this main PRD; granular evidence tables for each live in the > CHANGELOG and git history. +### v0.33.0 / CLIPipe native MCP tool mode (bareloop BA-16) (2026-07-21) + +> **Numbering note.** This is **bareloop's** BA-16 (`docs/UPSTREAM-ASKS.md`), an +> upstream ask consumed by version bump — not a bareagent-internal BA-number. + +Decisions locked, not to be re-litigated: + +- **Native (`toolProtocol:'claude-mcp'`) is the default for the claude CLI; + v0.32.0 envelope emulation is RETAINED, not retired** — it is still the right + instrument for a CLI with no MCP channel. The choice between them is **COST, + measured** ($0.25–0.55/round emulation vs ~$0.006/turn native), **not + capability**. Output-quality parity is deliberately **UNMINTED** (n=2 + suggestive only) — never sold as a capability win (the BA-7 honesty precedent; + the paired behaviour head-to-head is bareloop's consumer-side follow-up, not a + release gate). +- **A cycle-owning provider is a first-class contract, not a special case + (`Provider.ownsCycle`).** The CLI owns the inner turn loop, so the Loop's + per-round machinery cannot run on it; governance moves to the `tools/call` + bridge — the SAME `policy` chokepoint (identical audit-row shape), BA-11/BA-12 + guards, and the `--max-turns` bound (NAMED stop). Any Loop option the mode + cannot honor (`assemble`/`trim`/`cacheMessages`, a Loop-level `policy`) + **THROWS at construction** — no silently-dead knobs. +- **Honest accounting is load-bearing.** `GenerateResult.session` + + `metrics.sessionTurns` report the real turn/tool count (a 14-turn session + never reads as one round). A dead bridge still ends `subtype:'success'`, so + bridge health is tracked **parent-side** (attempted-vs-served tool calls) and + error-tags the run `bridge-failed` — the BA-4/5/6/13 optimistic-rounding class, + caught pre-build. **The CLI's subtype is not a sufficient success signal.** +- **Transport is a unix socket (0600 in a 0700 dir), never a listening port;** + the fence (`--tools '' --strict-mcp-config --setting-sources ''`) is set by + the mode, not the caller. Claude-only for now; both shapes isolated so a + second CLI slots in behind the same seams. + ### v0.29.0 / `shell_edit` — anchored edit verb (bareloop BA-13) (2026-07-15) > **Numbering note.** This is **bareloop's** BA-13 (`docs/UPSTREAM-ASKS.md`), diff --git a/package-lock.json b/package-lock.json index 19aa2fb..9ca8934 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "bare-agent", - "version": "0.32.0", + "version": "0.33.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "bare-agent", - "version": "0.32.0", + "version": "0.33.0", "license": "Apache-2.0", "bin": { "bare-agent": "bin/cli.js" diff --git a/package.json b/package.json index d2aea06..e3f688b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "bare-agent", - "version": "0.32.0", + "version": "0.33.0", "files": [ "index.js", "index.d.ts", diff --git a/src/provider-clipipe.js b/src/provider-clipipe.js index 1588f61..5910450 100644 --- a/src/provider-clipipe.js +++ b/src/provider-clipipe.js @@ -31,7 +31,7 @@ const { createBridge, resolveSessionError, runSession } = require('./provider-cl * Native mode sets {@link CLIPipeProvider#ownsCycle}, which makes the Loop REFUSE options it could * never honor (`assemble`/`trim`/`cacheMessages`, and a Loop-level `policy`) instead of leaving them * silently dead. See the native-only properties below. - * @property {Function} [policy] - (native mode) The gate, same contract as `Loop({policy})`: only `true` + * @property {(tool: string, args: any, ctx?: any) => any} [policy] - (native mode) The gate, same contract as `Loop({policy})`: only `true` * allows, a string is the deny reason fed back verbatim, a thrown `HaltError` is a clean governance * exit. REQUIRED here rather than on the Loop, because in native mode no tool call ever reaches the * Loop — a `Loop({policy})` would be a fence that is silently not there (the Loop throws instead).