diff --git a/docs/reference/presets.md b/docs/reference/presets.md index 28378b78..cc4145e4 100644 --- a/docs/reference/presets.md +++ b/docs/reference/presets.md @@ -179,7 +179,7 @@ Pipelines below are taken exactly from the `presets` map in `config/config.go`. | `medium` | `cache_aware_summarizer` → `format` → `dedup` → `toon` → `cmdfilter` → `searchfold` → `textclean` → `extract_llm` → `extract` → `cachesplit` → `toolfilter` | The picker's tier 2: `conservative` plus `extract_llm` (`housellm` minus its cold-cache sweep), plus [`cache_aware_summarizer`](../components/advanced-offload.md#two-ways-to-use-cache_state) with `cache_state: pre_expiry`. The summary is prepared in the background, in place of one keep-alive ping, and kept in reserve. It is used only when the user comes back after the cache expired, so the history never changes while the user works. Spends on its own, gated by `extract_llm`'s economic gate and 3,000-token trigger, and by the summarizer's 90%-of-window trigger. Needs keep-alive to prepare a summary; with cache strategy `none` the summarizer almost never acts. | | `high` | `cache_aware_summarizer` → `format` → `dedup` → `toon` → `cmdfilter` → `searchfold` → `textclean` → `extract_llm` → `extract` → `cachesplit` → `toolfilter` | The picker's tier 3: `medium` with `cache_aware_summarizer` at its default `cache_state: any`. The summary is made at 90% of the window and used on the next request, warm or cold. `summarize` is no longer in any picker tier; it stays available by name (see [Two ways to use `cache_state`](../components/advanced-offload.md#two-ways-to-use-cache_state) for when it is the better choice). | | `xhigh` | `cache_aware_summarizer` → `format` → `dedup` → `toon` → `cmdfilter` → `searchfold` → `textclean` → `extract_llm` → `extract_llm_sweep` → `extract` → `cachesplit` → `toolfilter` | The picker's tier 4, and the deepest cut it offers: `high` plus `extract_llm_sweep` (it asks at the keep-alive ping and removes content only on a cold request), i.e. `housellm` plus [`cache_aware_summarizer`](../components/advanced-offload.md#two-ways-to-use-cache_state) at its default trigger (`min_request_frac: 0.9`, `cache_state: any`). The summarizer sends the conversation itself plus one appended instruction, so its call reads the provider's cached prefix instead of paying full price. On Anthropic traffic it depends on #402, #406, #413 and #431. | -| `ultrahigh` | `cache_aware_summarizer` → `format` → `dedup` → `toon` → `cmdfilter` → `searchfold` → `textclean` → `extract_llm` → `extract_llm_sweep` → `extract` → `cachesplit` → `toolfilter` | **Experimental** (see the README's "Experimental: ultrahigh"). The picker's tier 5, **opt-in only**: `xhigh` plus **tool deferral** (`toolfilter` with `defer: auto`, #485). A tool that the user's last 3 sessions declared and never used, that this project did not use in its last 10 sessions, and that weighs at least 200 tokens (counted with the proxy's own o200k tokenizer, about 340 Claude tokens) is deferred. An MCP server is one unit: all its tools are deferred together when no tool of it was used, and the 200-token minimum applies to the server's total. The set is decided once per conversation thread and never changes within it. Without 3 sessions of history it defers nothing. **On Claude models** the tool is sent with `defer_loading: true` and is never removed; the model sees its name in a short list and loads it through the API's search tool (`tool_search_tool_bm25`). **When the request has Claude Code's own `ToolSearch`** (#898), the proxy adds no `tool_search_tool_bm25`: the model loads through `ToolSearch`, and the proxy adds `tool_reference` blocks for its own deferred tools to a `select:Name1,Name2` result, on the next request. Any other query passes through. An unknown `ToolSearch` description, a non-`select:` query that names a deferred tool, or a call to a tool no search loaded disables the mode for that description (24 h): nothing is deferred. See [the search matrix](../tool-search-matrix.md). A thread that started with the server-side search keeps it. This needs a provider check: picking the preset in the preset picker starts it, and the first session after that is not deferred. **On GPT, GLM and Kimi** (#825), which ignore `defer_loading`, the tool is removed from `tools` and listed by name; the model loads it through Context Guru's own search tool, `context_guru_tools`, which adds it back for the rest of the thread. No provider check runs there, so the first request of a thread can already defer. Off with `toolfilter: {defer: off}`, or in Claude Code with `ENABLE_TOOL_SEARCH=false`. Two account lists change the rule: `keep` (never defer these) and `always_defer` (#878: defer these in every new thread, without the history rule and the project veto; empty by default). The keep list wins over `always_defer`, and a tool on `always_defer` still stays loaded when the thread uses it, when `tool_choice` forces it, when it is a core tool, or when it weighs under 200 tokens. Each list takes a tool name, or `mcp__` for a whole server. The per-thread record is kept on disk in every setup (the session-state table with `--dashboard`, else `context-guru-tooldefer.db` next to the dashboard database path), so a restart never moves a tool; with no durable place, it defers nothing. A Responses request that declares its own `tool_search` (Codex) keeps it, and nothing is deferred on it (#876). | +| `ultrahigh` | `cache_aware_summarizer` → `format` → `dedup` → `toon` → `cmdfilter` → `searchfold` → `textclean` → `extract_llm` → `extract_llm_sweep` → `extract` → `cachesplit` → `toolfilter` | **Experimental** (see the README's "Experimental: ultrahigh"). The picker's tier 5, **opt-in only**: `xhigh` plus **tool deferral** (`toolfilter` with `defer: auto`, #485). A tool that the user's last 3 sessions declared and never used, that this project did not use in its last 10 sessions, and that weighs at least 200 tokens (counted with the proxy's own o200k tokenizer, about 340 Claude tokens) is deferred. An MCP server is one unit: all its tools are deferred together when no tool of it was used, and the 200-token minimum applies to the server's total. The set is decided once per conversation thread and never changes within it. Without 3 sessions of history it defers nothing. **On Claude models** the tool is sent with `defer_loading: true` and is never removed; the model sees its name in a short list and loads it through the API's search tool (`tool_search_tool_bm25`). **When the request has Claude Code's own `ToolSearch`** (#898), the proxy adds no `tool_search_tool_bm25`: the model loads through `ToolSearch`, and the proxy adds `tool_reference` blocks for its own deferred tools to a `select:Name1,Name2` result, on the next request. Any other query passes through. A non-`select:` query passes through untouched and disables nothing. An unknown `ToolSearch` description, or a call to a deferred tool whose own result is a "not loaded", "no such tool" or "does not exist" error, disables the mode for that description (24 h): nothing is deferred. See [the search matrix](../tool-search-matrix.md). A thread that started with the server-side search keeps it. This needs a provider check: picking the preset in the preset picker starts it, and the first session after that is not deferred. **On GPT, GLM and Kimi** (#825), which ignore `defer_loading`, the tool is removed from `tools` and listed by name; the model loads it through Context Guru's own search tool, `context_guru_tools`, which adds it back for the rest of the thread. No provider check runs there, so the first request of a thread can already defer. Off with `toolfilter: {defer: off}`, or in Claude Code with `ENABLE_TOOL_SEARCH=false`. Two account lists change the rule: `keep` (never defer these) and `always_defer` (#878: defer these in every new thread, without the history rule and the project veto; empty by default). The keep list wins over `always_defer`, and a tool on `always_defer` still stays loaded when the thread uses it, when `tool_choice` forces it, when it is a core tool, or when it weighs under 200 tokens. Each list takes a tool name, or `mcp__` for a whole server. The per-thread record is kept on disk in every setup (the session-state table with `--dashboard`, else `context-guru-tooldefer.db` next to the dashboard database path), so a restart never moves a tool; with no durable place, it defers nothing. A Responses request that declares its own `tool_search` (Codex) keeps it, and nothing is deferred on it (#876). | | `codesmart` | `format` → `textclean` → `searchfold` → `dedup` → `failed_run` → `cmdfilter` → `extract_llm` → `extract` → `linecap` → `cachesplit` | The SWE-bench-winning cache-aware config: structural offloaders + a cheap-model relevance-trimmer (`extract_llm`, routed to `CHEAP_MODEL`, gated so most turns make no model call) + deterministic `extract`. `extract_llm` no-ops (→ deterministic) when no cheap model is configured. **Changed 2026-08:** the lossless trio replaced `toon`, which acted 0 times on 5,752 production requests, and `linecap` was added. Re-measure before quoting the published SWE-bench numbers against it. | | `codesafe` | `format` → `textclean` → `searchfold` → `dedup` → `failed_run` → `cmdfilter` → `extract` → `collapse` → `linecap` → `cachesplit` | `codesmart` minus the LLM pass — **deterministic-only, zero model calls by policy**. The safe control / the choice when you don't want an LLM on the hot path. | | `off` | *(empty)* | Passthrough — no components. The baseline / A-B control, and what the [Claude Code plugin](../how-to/install-plugin.md) installs by default; the plugin's actual default saving mechanism is the `auto` cache strategy (keep-alive), not a pipeline component. | diff --git a/docs/tool-search-matrix.md b/docs/tool-search-matrix.md index ba3f9563..a907ca17 100644 --- a/docs/tool-search-matrix.md +++ b/docs/tool-search-matrix.md @@ -7,7 +7,7 @@ This page lists which search tool each client and model uses when `ultrahigh` de | Client and model | Search tool | Who answers | What the proxy defers | On a break or disable | Issue | |---|---|---|---|---|---| -| Claude Code, `ToolSearch` on, Claude model | `ToolSearch` (the client's) | Claude Code. The proxy adds `tool_reference` blocks for its own tools to the result | Tools the history rule picks | The proxy disables the `ToolSearch` description hash. It defers nothing on new threads with that hash. The live thread stops deferring (one cache rewrite). It never switches to another search tool | #898, this page | +| Claude Code, `ToolSearch` on, Claude model | `ToolSearch` (the client's) | Claude Code. Its search already finds the proxy's deferred tools (it indexes every tool it has). The proxy's added `tool_reference` is a safety net with nothing to add today: `client_loads` stays 0 (#871, comment 6642059429) | Tools the history rule picks | The proxy disables the `ToolSearch` description hash. It defers nothing on new threads with that hash. The live thread stops deferring (one cache rewrite). It never switches to another search tool | #898, this page | | Claude Code, search off on purpose (`ENABLE_TOOL_SEARCH=false`) | None | - | Nothing | - | #485 | | Claude Code, search off, not on purpose, Claude model | Anthropic's server-side search (`tool_search_tool_bm25`) | The provider | Tools the history rule picks | A 400 turns deferral off for the tenant and model | #485 | | Codex, gpt-5.5 (declares its own `tool_search`, `execution: client`) | Codex's own `tool_search` | Codex | Nothing. The proxy adds no `context_guru_tools`. Codex's loads count as uses in the history | - | #876, #897 | @@ -70,7 +70,10 @@ The code is in `tooldefer/tooldefer.go`: the rule in its package comment, the th documents the query syntax, so the hash is the format version. An unknown hash, or a disabled one, defers nothing on the thread. A second probe checks that the API accepts the proxy's `tool_reference` blocks for the model. The mode is fixed per thread. -- **Search calls.** Every `ToolSearch` call goes to Claude Code. On the next request, the proxy adds +- **Search calls.** Every `ToolSearch` call goes to Claude Code. Its own search indexes every tool + Claude Code has, also the tools the proxy deferred, so it already finds them and the model loads + them. The proxy's addition is a safety net that has nothing to add today (`client_loads` stays 0; + #871, comment 6642059429). On the next request, the proxy adds a `tool_reference` for each of its own deferred tools that the query names. Claude Code's result stays. The cap in `max_results` covers the combined result. - **Accepted query.** Only `select:Name1,Name2`. The prefix is read in any case, names are split on @@ -83,10 +86,11 @@ The code is in `tooldefer/tooldefer.go`: the rule in its package comment, the th | Reason | Trigger | |---|---| | `unknown_description` | The `ToolSearch` description hash is not on the verified list | -| `non_select_query_for_our_tool` | A query names a deferred tool of the proxy in a form other than `select:` | | `break` | A call to a deferred tool got an error that says the tool is not loaded or does not exist. A direct call that worked is a use (`direct_calls`), not a break | - **Who is affected:** the tenant, for that description hash, for 24 hours. +- **A keyword query passes through.** A query that names a deferred tool in a form other than + `select:` disables nothing. Claude Code's own search finds it (#871). - **What one break costs:** one cache rewrite on the live thread, because `tools` changes once. - **How to clear the mark:** restart the proxy, or wait 24 hours. - **How to verify a new Claude Code version:** run the live probe, then add the new hash to @@ -94,7 +98,7 @@ The code is in `tooldefer/tooldefer.go`: the rule in its package comment, the th ## What production monitoring watches -- Metric `cg_tooldefer_client_disabled_total{reason="unknown_description|non_select_query_for_our_tool|break"}` -- `/stats` fields `tool_defer.client_search_disabled` (the same three reasons), `client_loads`, `direct_calls` +- Metric `cg_tooldefer_client_disabled_total{reason="unknown_description|break"}` +- `/stats` fields `tool_defer.client_search_disabled` (the same two reasons), `client_loads`, `direct_calls` - Log line `cg.tooldefer_client_search_disabled` with fields `reason`, `desc_hash` (8 characters), `session`, `thread`. It holds no query text. diff --git a/proxy/promexport.go b/proxy/promexport.go index 9e157565..f41aa3cc 100644 --- a/proxy/promexport.go +++ b/proxy/promexport.go @@ -586,9 +586,8 @@ func (h *Handler) renderMetrics() string { // deferring for that client version. if _, g := h.deferPeek(); g != nil { promHeaderProc(&b, "cg_tooldefer_client_disabled_total", - "Times tool deferral through Claude Code's ToolSearch was disabled, by reason. unknown_description: the ToolSearch description hash was not verified. non_select_query_for_our_tool: a query named a deferred tool in a form other than select:. break: a deferred tool's own result said it is not loaded. unknown_description and non_select_query_for_our_tool count once per tenant and hash per 24 h; break counts every time.", "counter") + "Times tool deferral through Claude Code's ToolSearch was disabled, by reason. unknown_description: the ToolSearch description hash was not verified. break: a deferred tool's own result said it is not loaded. unknown_description counts once per tenant and hash per 24 h; break counts every time.", "counter") promLine(&b, "cg_tooldefer_client_disabled_total", `reason="unknown_description"`, float64(g.DisabledUnknown.Load())) - promLine(&b, "cg_tooldefer_client_disabled_total", `reason="non_select_query_for_our_tool"`, float64(g.DisabledNonSel.Load())) promLine(&b, "cg_tooldefer_client_disabled_total", `reason="break"`, float64(g.DisabledBreak.Load())) } diff --git a/proxy/tooldefer.go b/proxy/tooldefer.go index acc6c5c8..affe46e4 100644 --- a/proxy/tooldefer.go +++ b/proxy/tooldefer.go @@ -778,12 +778,10 @@ func (h *Handler) deferStats() any { "client_loads": g.ClientLoads.Load(), "direct_calls": g.DirectCalls.Load(), "mixed_searches": g.MixedSearches.Load(), - // Client-search mode (#898) disabled, by reason. unknown_description and - // non_select_query_for_our_tool count once per tenant and hash per 24 h; break counts every - // time. The same numbers are cg_tooldefer_client_disabled_total. + // Client-search mode (#898) disabled, by reason. unknown_description counts + // once per tenant and hash per 24 h; break counts every time. The same numbers are cg_tooldefer_client_disabled_total. "client_search_disabled": map[string]int64{ tooldefer.ReasonUnknownDescription: g.DisabledUnknown.Load(), - tooldefer.ReasonNonSelectQuery: g.DisabledNonSel.Load(), tooldefer.ReasonBreak: g.DisabledBreak.Load(), }, "probes_pass": pass, diff --git a/tooldefer/apply.go b/tooldefer/apply.go index f45cee09..92dfae2c 100644 --- a/tooldefer/apply.go +++ b/tooldefer/apply.go @@ -234,20 +234,14 @@ func Apply(env Env, r Req) (o Outcome) { o.Client = true a := AugmentSearches(rs, r.Session, r.Thread, out, set) reason := "" - switch { - case a.Bad: - reason = ReasonNonSelectQuery - } - if reason == "" { - // A break is a call to a deferred tool whose own result says it is not loaded. A call no - // search loaded that worked is a use (the API accepts it), counted as a direct call. - // A tool tool_choice kept loaded on this request is declared in full, so it never breaks. - broke, direct := ReadCalls(a.Body, set) - if broke && !forcedAny(r.Orig, set) { - reason = ReasonBreak - } - o.Direct = direct + // A break is a call to a deferred tool whose own result says it is not loaded. A call no + // search loaded that worked is a use (the API accepts it), counted as a direct call. + // A tool tool_choice kept loaded on this request is declared in full, so it never breaks. + broke, direct := ReadCalls(a.Body, set) + if broke && !forcedAny(r.Orig, set) { + reason = ReasonBreak } + o.Direct = direct if reason != "" { // A format change. Disable the description hash and stop deferring on this thread. This // changes `tools` once, so the live thread pays one cache rewrite, on purpose: a broken diff --git a/tooldefer/gate.go b/tooldefer/gate.go index 542e8e7e..b86da863 100644 --- a/tooldefer/gate.go +++ b/tooldefer/gate.go @@ -42,14 +42,12 @@ type Gate struct { MixedSearches atomic.Int64 // searches in the same response as a client tool call // Client-search mode (#898): disables by reason. The production monitor reads these. DisabledUnknown atomic.Int64 // reason unknown_description - DisabledNonSel atomic.Int64 // reason non_select_query_for_our_tool DisabledBreak atomic.Int64 // reason break } // The reasons ClientDisabled is set, as logged and exported. const ( ReasonUnknownDescription = "unknown_description" - ReasonNonSelectQuery = "non_select_query_for_our_tool" ReasonBreak = "break" ) @@ -232,8 +230,6 @@ func (g *Gate) DisableClient(tenant, hash, reason string) bool { switch reason { case ReasonUnknownDescription: g.DisabledUnknown.Add(1) - case ReasonNonSelectQuery: - g.DisabledNonSel.Add(1) default: g.DisabledBreak.Add(1) } diff --git a/tooldefer/toolsearch.go b/tooldefer/toolsearch.go index 1daac105..68edf488 100644 --- a/tooldefer/toolsearch.go +++ b/tooldefer/toolsearch.go @@ -19,8 +19,10 @@ import ( // tools to it. Claude Code's own result stays as it is. The API expands a tool_reference for a tool // with defer_loading, with no server-side search in `tools` (measured live, #898). // -// The accepted query is only the exact select form (selectNames). Every other query passes through. -// The proxy does not read a query in any other way. The measured behavior of Claude Code's search: +// The accepted query is only the exact select form (selectNames). Every other query passes through, +// also one that names a deferred tool: Claude Code's own search indexes every tool it has, so it +// answers such a query itself and the model loads the tool (#871). The proxy does not read a query +// in any other way. The measured behavior of Claude Code's search: // - `select:` is matched in any case ("Select:"), names are split on "," and trimmed, so a space // after the comma is fine; // - a name it does not know is dropped; when no name is known, the result is the text @@ -28,10 +30,9 @@ import ( // - the result is capped at max_results (default 5). // // Fail closed on a format change, never switch to another search tool: a hash of ToolSearch's -// description (DescHash) is the version key. An unknown hash defers nothing on the thread. A query that -// names one of our tools in any other form, or a call to a deferred tool that no result loaded, marks -// the hash disabled (Gate.DisableClient) and the thread stops deferring: `tools` changes once, so that -// thread pays one cache rewrite. +// description (DescHash) is the version key. An unknown hash defers nothing on the thread. A call to a +// deferred tool whose own result says it is not loaded marks the hash disabled (Gate.DisableClient) +// and the thread stops deferring: `tools` changes once, so that thread pays one cache rewrite. // // Every added list is written to the thread's record by tool_use id the first time, and replayed from // there on every later request. The bytes are then the same on every request, also after a restart. @@ -117,9 +118,6 @@ type Augment struct { // Loads is the deferred tools that results written now carry; Answered how many calls were new. Loads []string Answered int - // Bad is true when a new query names one of our tools in a form other than the exact select form. - // Body is then the input. - Bad bool } // AugmentSearches adds, to the result of every ToolSearch call in body, a tool_reference for each @@ -171,10 +169,6 @@ func AugmentSearches(rs store.Store, session, thread string, body []byte, deferr if b.Get("type").String() != "tool_result" || !isCall { continue } - if _, isSel := selectNames(c.query); !isSel && namesOurs(c.query, deferred) { - a.Bad = true - return a // the format is not what the proxy knows: it adds nothing - } res := b.Get("content") // The result shapes the proxy extends: the no-match text, or references only. An error or // any other shape stands as the client wrote it, and books nothing. @@ -275,16 +269,6 @@ func ourNames(sel []string, deferred map[string]bool, res gjson.Result, max int) return out } -// namesOurs reports whether a query holds the exact name of a deferred tool. -func namesOurs(q string, deferred map[string]bool) bool { - for n := range deferred { - if strings.Contains(q, n) { - return true - } - } - return false -} - // ReadCalls reads the proxy's deferred tools that the model called in the transcript. broke: a call's // own tool_result is an error that says the tool is not loaded or does not exist. direct: calls that // no search loaded and that did not fail that way. The API accepts a direct call, so it is a use, not a diff --git a/tooldefer/toolsearch_test.go b/tooldefer/toolsearch_test.go index 4a99443f..479d00e6 100644 --- a/tooldefer/toolsearch_test.go +++ b/tooldefer/toolsearch_test.go @@ -114,9 +114,9 @@ func TestClientModeAddsOnlyOurToolsToSelectQueries(t *testing.T) { } } -// A keyword query that does not name our tools passes through. One that names them is a format -// change: the hash is disabled and the thread stops deferring. -func TestClientModeNonSelectQueryDisables(t *testing.T) { +// A keyword query passes through untouched, also one that names a deferred tool: Claude Code's own +// search finds it (#871). Nothing is disabled and the thread keeps deferring. +func TestClientModeKeywordQueryPassesThrough(t *testing.T) { e, _ := clientEnv(t, filepath.Join(t.TempDir(), FileName)) Apply(e, creq(body(catalogue(), ""))) o := Apply(e, creq(body(catalogue(), "", searchRound("t1", "cron jobs", noMatch)...))) @@ -124,20 +124,20 @@ func TestClientModeNonSelectQueryDisables(t *testing.T) { t.Fatalf("keyword query: %s", show(o)) } o = Apply(e, creq(body(catalogue(), "", searchRound("t2", "+Workflow script", noMatch)...))) - if o.Changed || o.ClientDisabled != ReasonNonSelectQuery || e.Gate.DisabledNonSel.Load() != 1 { - t.Fatalf("a non-select query for our tool: %s (%q)", show(o), o.ClientDisabled) + if !o.Changed || result(o.Body, 3) != noMatch || o.ClientDisabled != "" { + t.Fatalf("a keyword query for our tool: %s (%q)", show(o), o.ClientDisabled) } h := DescHash(body(catalogue(), "")) - if !e.Gate.ClientDisabled("", h) { - t.Fatal("hash not marked disabled") + if e.Gate.ClientDisabled("", h) { + t.Fatal("hash marked disabled") } - // The live thread stays undeferred, and a new thread defers nothing. - if o = Apply(e, creq(body(catalogue(), "", searchRound("t3", "x", noMatch)...))); o.Changed { - t.Fatalf("live thread still deferred: %s", show(o)) + // The live thread keeps deferring, and a new thread defers too. + if o = Apply(e, creq(body(catalogue(), "", searchRound("t3", "x", noMatch)...))); !o.Changed { + t.Fatalf("live thread stopped deferring: %s", show(o)) } r := creq(body(catalogue(), "")) r.Thread = "new" - if o = Apply(e, r); o.Changed || o.Why != "client_search_disabled" { + if o = Apply(e, r); !o.Changed || o.Why == "client_search_disabled" { t.Fatalf("new thread: %s", show(o)) } }