diff --git a/README.md b/README.md index f46bb0b..4bfb86b 100644 --- a/README.md +++ b/README.md @@ -339,9 +339,15 @@ which is why the medium pins survived the generation jump unchanged; the fable-class tier is built for long-horizon frontier work - a session-model choice, not a dispatch target. -The effort ladder - the second knob. Unset effort means `high` on -current models (not medium), and `xhigh` is absent on some models that -support `max` (e.g. the 4.6 generation). The vendor recommendation is +The effort ladder - the second knob. Unset effort means `high` on every +model that supports effort, the one exception being Opus 4.7, which +defaults to `xhigh`. Support itself is an explicit list rather than a +version cutoff: Fable 5, Opus 5, Sonnet 5, Opus 4.8 and Opus 4.7 take the +whole ladder, Opus 4.6 and Sonnet 4.6 take everything but `xhigh`, and a +model absent from that list - Haiku 4.5 among them - has no effort knob at +all. Setting a level a model does not support runs the highest supported +level at or below it, so `xhigh` becomes `high` on Opus 4.6. The vendor +recommendation is per-generation ([effort reference](https://platform.claude.com/docs/en/build-with-claude/effort)): Opus 4.7 and 4.8 start coding and agentic work at `xhigh`, while Opus 5 @@ -445,10 +451,11 @@ chapter for the full set of rules. ## Dispatch counter -Every Agent dispatch is logged by a PostToolUse hook (agent name + model, -nothing else) to `/model-routing/dispatches.jsonl`, self-pruned to -30 days. Stats show how much work routing actually kept off your session -model - real counts, not invented dollar savings: +Every Agent dispatch is logged by a PostToolUse hook (agent name, model, the +session model, and the session effort level - nothing else) to +`/model-routing/dispatches.jsonl`, self-pruned to 30 days. Stats show +how much work routing actually kept off your session model - real counts, not +invented dollar savings: ```text /model-routing:stats @@ -486,6 +493,14 @@ tool, so they are invisible to the dispatch report - but `tokens` reads their transcripts (nested under `subagents/workflows/`) and counts their volume against the parent session like any other subagent. +### The effort line + +`report` also prints how many dispatches ran on an agent type carrying no effort pin, and therefore inherited whatever the session was set to. That is the failure a tier-only report scores as a win: a mechanical errand sent to `general-purpose` with `model=sonnet` still thinks as hard as your session does, while `scout` would have run the same errand at `low`. + +Effort is reconstructed rather than observed, because transcripts do not record it. The sources are read in the order Claude Code applies them: `CLAUDE_CODE_EFFORT_LEVEL` first (the only place `max` is accepted), then `effortLevel` from the settings cascade (local, then project, then user - these accept `low`/`medium`/`high`/`xhigh` only), then the documented model default described in [the effort ladder](#model-tiers-and-effort-ladder). Levels that came from that last rung are counted separately in the report, so an inferred default never reads as a setting somebody chose, and a level the session model does not support is recorded as the one Claude Code falls back to instead. Two cases contribute nothing to the line rather than a guess: a session on a model the docs give no effort support, and a session whose model could not be read at all, since the same configured level means different things on different models. + +The line states its limits rather than hiding them, and there are more than the sources suggest. Four documented states override the reconstruction and none of them are visible to a hook: a `/effort` or `--effort` choice made inside a running session, ultracode (which sends `xhigh`), an organization effort cap, and the model-default hold that Fable 5, Opus 4.8 and Opus 4.7 apply on first run over a level you previously set. Only the pins of the bundled agents are known here too - an agent from another plugin may pin its own effort and will still be counted as inheriting. Read the figure as what the visible sources resolve to, not as a measurement. + Embed the one-liner in your status line by appending the command's output to whatever your `statusLine.command` already prints. Delete the `.jsonl` any time to reset; a missing file just means zero. @@ -514,9 +529,10 @@ There is deliberately no config subsystem - four override paths cover it: frontmatter pin when present, else from the session level. - **Permanent**: edit the `model:` / `effort:` frontmatter in `agents/*.md`. A directory-source install picks the change up next - session. Keep `PINNED_MODELS` in `hooks/dispatch-counter.mjs` in step - - the CI sync test fails the build when the two drift, because that drift - silently corrupts the stats (it happened once; see 0.7.1). + session. Keep `AGENT_PINS` in `hooks/dispatch-counter.mjs` in step - it + carries both the model and the effort column, and a CI sync test fails the + build when either drifts from the frontmatter, because that drift silently + corrupts the stats (it happened once; see 0.7.1). - **Reset**: `git checkout -- agents` in the plugin checkout, or reinstall from the marketplace. diff --git a/hooks/dispatch-counter.mjs b/hooks/dispatch-counter.mjs index bd044b3..7d3a0f6 100644 --- a/hooks/dispatch-counter.mjs +++ b/hooks/dispatch-counter.mjs @@ -18,6 +18,10 @@ // the agent's frontmatter pin) ranks below the recorded session model. Entries // missing either side fall back to a cheap-agent/cheap-tier heuristic. Counts // dispatches, not tokens - honest bookkeeping, no dollar fiction. +// Each entry also records the session's effort level and which source it came +// from - CLAUDE_CODE_EFFORT_LEVEL, else the settings cascade, else the model's +// documented default - so the report can show the second knob: dispatches on +// agent types with no known pin inherit it, pinned agents do not. // Log lives in /model-routing/dispatches.jsonl and self-prunes to 30d. import { appendFileSync, closeSync, existsSync, fstatSync, mkdirSync, openSync, readFileSync, readSync, writeFileSync } from "node:fs"; @@ -50,30 +54,146 @@ function sessionFilterFromArgs(argv) { const i = argv.indexOf("--session"); return i >= 0 && argv[i + 1] ? String(argv[i + 1]).toLowerCase() : null; } -// Frontmatter pins of the bundled agents. A bare dispatch (no model param) -// still runs on the pinned model, so classification must resolve through -// this table or bare implementer dispatches (pin=sonnet since 0.6.0) get -// miscounted as session-tier work. Keep in sync with agents/*.md. -const PINNED_MODELS = { - "model-routing:scout": "sonnet", - "model-routing:test-runner": "haiku", - "model-routing:e2e-runner": "sonnet", - "model-routing:verifier": "haiku", - "model-routing:implementer": "sonnet", - "model-routing:reviewer": "opus", +// Frontmatter pins of the bundled agents, model and effort together. A bare +// dispatch (no model param) still runs on the pinned model, so classification +// must resolve through this table or bare implementer dispatches (pin=sonnet +// since 0.6.0) get miscounted as session-tier work; the effort column is the +// second cost knob, which moves cost as hard as tier does. One table rather +// than two because they are two columns of one fact - the agent's frontmatter - +// and asking "is this agent pinned" through separate tables let them disagree. +// Keep in sync with agents/*.md; one sync test guards both columns. +const AGENT_PINS = { + "model-routing:scout": { model: "sonnet", effort: "low" }, + "model-routing:test-runner": { model: "haiku", effort: "low" }, + "model-routing:e2e-runner": { model: "sonnet", effort: "medium" }, + "model-routing:verifier": { model: "haiku", effort: "low" }, + "model-routing:implementer": { model: "sonnet", effort: "medium" }, + "model-routing:reviewer": { model: "opus", effort: "high" }, }; +const pinnedModel = (agent) => AGENT_PINS[agent]?.model ?? null; +const pinnedEffort = (agent) => AGENT_PINS[agent]?.effort ?? null; // Unpinned agent types that are inherently cheap dispatch targets. const CHEAP_AGENTS = new Set(["Explore"]); -function dataFile() { - const cfg = process.env.CLAUDE_CONFIG_DIR?.trim() +function configDir() { + return process.env.CLAUDE_CONFIG_DIR?.trim() ? resolve(process.env.CLAUDE_CONFIG_DIR) : join(homedir(), ".claude"); - const dir = join(cfg, "model-routing"); +} + +function dataFile() { + const dir = join(configDir(), "model-routing"); if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); return join(dir, "dispatches.jsonl"); } +// The session's reasoning effort, and where it came from. Transcripts record no +// effort at all, so it is reconstructed from the sources Claude Code documents, +// in the precedence Claude Code itself applies: +// 1. CLAUDE_CODE_EFFORT_LEVEL - overrides settings for the session, and is the +// only place `max` is accepted. `auto` means "use the model default". +// 2. the settings cascade, local > project > user. Settings accept only +// low/medium/high/xhigh; `max` and `ultracode` are session-only there and +// are rejected in a settings file. +// 3. the model default, which is what an unset key actually means - `high` +// wherever effort is supported, `xhigh` on Opus 4.7. Recording it beats +// omitting the most common configuration of all. +// A `/effort` change or a `--effort` flag inside a running session is invisible +// to all three, and an agent may carry its own effort pin this script cannot +// see; the report states both limits rather than implying precision it lacks. +// Reads one key and never emits anything else from these files, which routinely +// hold secrets. +const SETTINGS_EFFORTS = new Set(["low", "medium", "high", "xhigh"]); +const ENV_EFFORTS = new Set([...SETTINGS_EFFORTS, "max"]); + +// Which levels each model offers, transcribed from the Claude Code docs (Model +// configuration, "Adjust effort level") rather than inferred from version +// numbers: that table is an explicit enumeration and it states "Models not +// listed here do not support effort". Being rankable by TIER_PATTERNS is +// therefore NOT the test - claude-3-5-sonnet ranks fine and has no effort knob +// at all, and Haiku 4.5 is absent from the table entirely. Add a row when the +// docs add a model; assert nothing for one that is not listed. +// Deliberately a SECOND table rather than columns added to TIER_PATTERNS: tier +// is a property of the family (every opus ranks 3) while effort support is a +// property of the version, so one ordered list would have to repeat the tier +// across every version row and add a family fallback carrying no efforts. A +// consistency test pins the two together instead - every model named here must +// be rankable there. +const EFFORT_SUPPORT = [ + [/fable-5|opus-5|sonnet-5|opus-4-8|opus-4-7/, ["low", "medium", "high", "xhigh", "max"]], + [/opus-4-6|sonnet-4-6/, ["low", "medium", "high", "max"]], +]; +const EFFORT_LADDER = ["low", "medium", "high", "xhigh", "max"]; +const effortLevelsFor = (m) => (m ? EFFORT_SUPPORT.find(([re]) => re.test(m))?.[1] ?? null : null); + +// "The default effort is high on every model that supports effort, except Opus +// 4.7, which defaults to xhigh." No support, no default - an unlisted or +// unrecognized session model never receives a fabricated level. +function defaultEffortFor(sessionModel) { + if (!effortLevelsFor(sessionModel)) return null; + return /opus-4-7/.test(sessionModel) ? "xhigh" : "high"; +} + +// "If you set a level the active model does not support, Claude Code falls back +// to the highest supported level at or below the one you set. For example, +// xhigh runs as high on Opus 4.6." So a configured level is not necessarily the +// level that applies, and the clamp closes the gap this hook CAN see. Others it +// cannot: an organization effort cap, and the model-default hold that Fable 5, +// Opus 4.8 and Opus 4.7 apply on first run "even if you previously set a +// different level", overriding a persisted setting until an explicit choice is +// made. The report names both rather than claiming more than it knows. +// No session model means the +// clamp cannot be computed, so nothing is recorded: a configured `high` on a +// session whose transcript could not be read might have been a Haiku 4.5 +// session, where the level does not exist at all. That matches how the rest of +// this report treats an unknown session - excluded, never guessed. +function clampEffort(level, sessionModel) { + const levels = effortLevelsFor(sessionModel); + if (!levels) return null; + if (levels.includes(level)) return level; + for (let i = EFFORT_LADDER.indexOf(level) - 1; i >= 0; i--) { + if (levels.includes(EFFORT_LADDER[i])) return EFFORT_LADDER[i]; + } + return null; +} + +function sessionEffort(cwd, sessionModel) { + const fromDefault = () => { + const def = defaultEffortFor(sessionModel); + return def ? { effort: def, effortFrom: "default" } : null; + }; + const asRan = (level, from) => { + const ran = clampEffort(level, sessionModel); + return ran ? { effort: ran, effortFrom: from } : null; + }; + const env = process.env.CLAUDE_CODE_EFFORT_LEVEL?.trim(); + if (env) { + if (env === "auto") return fromDefault(); + // An unrecognized override is still an override: settings were bypassed, + // and by what is unknown, so assert nothing rather than reporting the + // level the session did NOT run on. + return ENV_EFFORTS.has(env) ? asRan(env, "env") : null; + } + for (const f of [ + join(cwd, ".claude", "settings.local.json"), + join(cwd, ".claude", "settings.json"), + join(configDir(), "settings.json"), + ]) { + let parsed; + // An unreadable or malformed file is ignored wholesale by the harness too, + // so deferring to the next rung matches what actually happens. + try { parsed = JSON.parse(readFileSync(f, "utf-8")); } catch { continue; } + if (!parsed || typeof parsed !== "object" || !Object.hasOwn(parsed, "effortLevel")) continue; + // A file that DEFINES the key ends the walk whether or not the value is + // usable. Falling through on a bad value would log the next rung's level, + // which is not the level this session ran on - a wrong data point is worse + // than a missing one. + const v = parsed.effortLevel; + return typeof v === "string" && SETTINGS_EFFORTS.has(v) ? asRan(v, "settings") : null; + } + return fromDefault(); +} + function readEntries(file) { if (!existsSync(file)) return []; return readFileSync(file, "utf-8") @@ -122,6 +242,15 @@ function firstModelIn(file, bytes) { return m?.[1] ?? null; } +// Session model for the tokens report. The head names the model the session +// STARTED on, but a long session can push the first assistant message past the +// head window (measured here: 6 of 60 transcripts, all multi-MB), and +// returning null there dropped those sessions out of the routed-down math as +// "session unknown" - and they are the biggest ones, so the loss was +// concentrated exactly where it mattered. Fall back to the tail, accepting the +// lower precision: a late-window model beats no model. +const sessionModelOf = (file) => firstModelIn(file, 262144) ?? lastModelIn(file, 262144); + function lastModelIn(file, bytes) { // Model in effect NOW: the last model named in the transcript tail. The // dispatch hook uses this so /model switches, opusplan's plan->execute @@ -136,7 +265,7 @@ function lastModelIn(file, bytes) { // CLAUDE_CODE_SUBAGENT_MODEL env override (recorded by the hook as e.env), // else the explicit model param, else the agent's frontmatter pin, else // unknown (session-model inheritance). -const effectiveModel = (e) => e.env ?? e.model ?? PINNED_MODELS[e.agent] ?? null; +const effectiveModel = (e) => e.env ?? e.model ?? pinnedModel(e.agent) ?? null; function isRoutedDown(e) { // With the session model recorded (0.5.3+ entries), judge by tier: an @@ -209,7 +338,7 @@ if (process.argv[2] === "stats" || process.argv[2] === "report") { for (const e of entries) { const key = e.env ? `${e.agent} (env=${e.env})` : e.model ? `${e.agent} (model=${e.model})` - : PINNED_MODELS[e.agent] ? `${e.agent} (pin=${PINNED_MODELS[e.agent]})` + : pinnedModel(e.agent) ? `${e.agent} (pin=${pinnedModel(e.agent)})` : e.agent; const s = byAgent.get(key) ?? { n: 0, down: 0, up: 0, unknown: 0 }; s.n++; @@ -244,7 +373,7 @@ if (process.argv[2] === "stats" || process.argv[2] === "report") { // routed-down tier would need rework >~20% of the time the price edge // is gone - here inverted, >20% of cheap-capable dispatches leaking UP // is the same signal that the tier assignment is not holding. - const BUNDLED = new Set([...Object.keys(PINNED_MODELS), ...CHEAP_AGENTS]); + const BUNDLED = new Set([...Object.keys(AGENT_PINS), ...CHEAP_AGENTS]); const capable = entries.filter((e) => !BUNDLED.has(e.agent)); // An env override means the dispatch did NOT inherit the session model, // no matter that the call itself was bare - not a leak. @@ -256,6 +385,32 @@ if (process.argv[2] === "stats" || process.argv[2] === "report") { leakLines.push("", `Tier leaks: ${leaks.length} of ${capable.length} unpinned dispatches inherited a strong session model bare (${Math.round(rate * 100)}%).`); if (rate > LEAK_WARN) leakLines.push(` ! above the 20% rework threshold - pass an explicit model= on general-purpose/custom dispatches (sonnet default).`); } + // Effort, the knob the tier columns cannot show. A dispatch on an agent type + // with no known pin inherits the session level, so cheap-tier work can still + // think at the session's expense - the failure mode a tier-only report calls a + // success. Counted only over entries that recorded an effort: pre-0.15 logs + // have none, and neither does a session on a model the docs list as having no + // effort support at all. + const effortLines = []; + const withEffort = entries.filter((e) => e.effort); + if (withEffort.length) { + const inherited = withEffort.filter((e) => !pinnedEffort(e.agent)); + const byLevel = [...inherited.reduce((m, e) => m.set(e.effort, (m.get(e.effort) ?? 0) + 1), new Map())] + .sort((a, b) => b[1] - a[1]) + .map(([lvl, n]) => `${n} at ${lvl}`) + .join(", "); + // Scoped to `inherited`, not to `withEffort`: the sentence below reads as a + // subset of the count in the line above it, and a pinned agent on the model + // default would otherwise print "8 of these" under a total of 2. + const inferred = inherited.filter((e) => e.effortFrom === "default").length; + effortLines.push( + "", + `Effort: ${inherited.length} of ${withEffort.length} dispatches ran on an agent type carrying no pin this plugin knows about, and so inherited the session level${byLevel ? ` (${byLevel})` : ""}.`, + ` The bundled agents pin theirs in frontmatter, so routing a mechanical errand through a role agent buys a cheaper effort as well as a cheaper tier. An agent from anywhere else may pin its own effort, which is invisible here and counted as inherited.`, + ...(inferred ? [` ${inferred} of these levels are the documented model default rather than an observed setting.`] : []), + ` Source order is CLAUDE_CODE_EFFORT_LEVEL, then settings effortLevel, then the model default. Four states can override that and none are visible here: a /effort or --effort choice inside a running session, ultracode, an organization effort cap, and the model-default hold Fable 5 / Opus 4.8 / Opus 4.7 apply on first run over a previously set level.`, + ); + } // Grouped sections instead of per-row v/- markers: the reader should not // need a legend to see what ran cheap and what ran at the session tier. const groups = { down: [], top: [], up: [], unknown: [] }; @@ -290,6 +445,7 @@ if (process.argv[2] === "stats" || process.argv[2] === "report") { "By session model:", ...sessionRows.map(([m, s]) => ` ${m}: ${s.down} of ${s.cmp} routed down (${s.cmp ? Math.round((s.down / s.cmp) * 100) : 0}%)${s.n > s.cmp ? ` - ${s.n - s.cmp} not comparable` : ""}`), ...leakLines, + ...effortLines, "", `Log: ${dataFile()} - history kept 30 days.`, ]; @@ -308,12 +464,7 @@ if (process.argv[2] === "tokens") { const sf = sessionFilterFromArgs(process.argv); const winLabel = (win.ago ? `${win.days}d ending ${win.ago}d ago` : `${win.days}d`) + (sf ? `, ${sf} sessions` : ""); - const projRoot = (() => { - const cfg = process.env.CLAUDE_CONFIG_DIR?.trim() - ? resolve(process.env.CLAUDE_CONFIG_DIR) - : join(homedir(), ".claude"); - return join(cfg, "projects"); - })(); + const projRoot = join(configDir(), "projects"); const sessionModelCache = new Map(); const perModel = new Map(); // model -> {agents, in, out, cr, cw, down} const perSession = new Map(); // session model -> {agents, vol, downVol} @@ -399,7 +550,7 @@ if (process.argv[2] === "tokens") { if (isMainSession) { // Scoped by the same --session filter as the agents, so the denominator // always describes the same population as the headline above it. - if (!sessionModelCache.has(p)) sessionModelCache.set(p, firstModelIn(p, 262144)); + if (!sessionModelCache.has(p)) sessionModelCache.set(p, sessionModelOf(p)); const sessionModel = sessionModelCache.get(p); if (sf && !(sessionModel && shortModel(sessionModel).toLowerCase().includes(sf))) continue; mainSessions++; @@ -414,7 +565,7 @@ if (process.argv[2] === "tokens") { const anchored = p.match(/^(.*?)[\\/]subagents[\\/]/); const sessionJsonl = anchored ? anchored[1] + ".jsonl" : null; if (sessionJsonl && !sessionModelCache.has(sessionJsonl)) { - sessionModelCache.set(sessionJsonl, firstModelIn(sessionJsonl, 262144)); + sessionModelCache.set(sessionJsonl, sessionModelOf(sessionJsonl)); } const sessionModel = sessionJsonl ? sessionModelCache.get(sessionJsonl) : null; const tsess = tierOf(sessionModel); @@ -491,7 +642,7 @@ if (process.argv[2] === "tokens") { "", "By session model:", ...sessionRows.map(([m, s]) => ` ${m}: ${fmtN(s.vol)} across ${s.agents} agents - ${s.cmpVol ? `${Math.round((s.downVol / s.cmpVol) * 100)}% below session tier` : "not tier-comparable"}${s.cmpVol && s.vol > s.cmpVol ? ` (${fmtN(s.vol - s.cmpVol)} not comparable)` : ""}`), - ...(unknownAgents ? ["", `${unknownAgents} agents not tier-comparable (${fmtN(unknownVol)}) - unrecognized agent model or unknown session tier, excluded from routed-down math; extend TIER_PATTERNS in dispatch-counter.mjs.`] : []), + ...(unknownAgents ? ["", `${unknownAgents} agents not tier-comparable (${fmtN(unknownVol)}), excluded from routed-down math - either the agent ran an unrecognized model family (extend TIER_PATTERNS in dispatch-counter.mjs) or no model could be read from the parent session transcript, which happens when the transcript is gone or names no model anywhere.`] : []), ...(unreadable ? ["", `${unreadable} transcript(s) could not be read (too large to load as one string, or unreadable) - the totals below understate by whatever they held.`] : []), ...(mainVolTotal ? [ "", @@ -501,7 +652,7 @@ if (process.argv[2] === "tokens") { ] : []), "", "Volume = tokens the subagent processed; cache reads are billed at the subagent's model rate, which is where routing saves.", - "Session model is sampled at session START - a mid-session /model switch or fallback attributes later subagents to the start model (the dispatch report does not have this limit).", + "Session model is read from the head of each session transcript - the model it started on - so a mid-session /model switch or fallback attributes later subagents to the start model (the dispatch report does not have this limit). In a session long enough that its head names no model at all, the tail is read instead and that session is attributed to its LAST model, which is the opposite bias for those sessions.", ]; process.stdout.write(out.join("\n")); process.exit(0); @@ -518,6 +669,12 @@ try { const event = JSON.parse(raw); if (event.tool_name !== "Agent" && event.tool_name !== "Task") process.exit(0); const input = event.tool_input ?? {}; + // Which main model this dispatch was routed FROM - the last model named in + // the session transcript, i.e. the one in effect at dispatch time (survives + // /model switches, opusplan handoffs, and quota fallbacks). Resolved before + // the entry because the effort default depends on it. + const session = event.transcript_path ? lastModelIn(event.transcript_path, 262144) : null; + const effort = sessionEffort(event.cwd ?? process.cwd(), session); const entry = { ts: Date.now(), agent: input.subagent_type ?? "general-purpose", @@ -526,10 +683,11 @@ try { // frontmatter pin - when set, it is the model every subagent actually // ran on, so record it rather than guessing from pins. ...(process.env.CLAUDE_CODE_SUBAGENT_MODEL ? { env: process.env.CLAUDE_CODE_SUBAGENT_MODEL } : {}), - // Which main model this dispatch was routed FROM - the last model named - // in the session transcript, i.e. the one in effect at dispatch time - // (survives /model switches, opusplan handoffs, and quota fallbacks). - session: event.transcript_path ? lastModelIn(event.transcript_path, 262144) : null, + session, + // The session's effort level plus its source, so the report can separate an + // observed setting from the documented default it fell back to. Omitted + // entirely when neither could be established. + ...(effort ?? {}), }; const file = dataFile(); appendFileSync(file, JSON.stringify(entry) + "\n"); diff --git a/hooks/dispatch-counter.test.mjs b/hooks/dispatch-counter.test.mjs index 7a6cc52..78629f2 100644 --- a/hooks/dispatch-counter.test.mjs +++ b/hooks/dispatch-counter.test.mjs @@ -12,11 +12,18 @@ import { fileURLToPath } from "node:url"; const SCRIPT = join(dirname(fileURLToPath(import.meta.url)), "dispatch-counter.mjs"); -function run(args, configDir, stdin, extraEnv) { +function run(args, configDir, stdin, extraEnv, cwd) { return execFileSync(process.execPath, [SCRIPT, ...args].filter(Boolean), { - // CLAUDE_CODE_SUBAGENT_MODEL is blanked by default so a developer's own - // override cannot leak into the hermetic tests; set via extraEnv to test. - env: { ...process.env, CLAUDE_CONFIG_DIR: configDir, CLAUDE_CODE_SUBAGENT_MODEL: "", ...(extraEnv ?? {}) }, + // CLAUDE_CODE_SUBAGENT_MODEL and CLAUDE_CODE_EFFORT_LEVEL are blanked by + // default so a developer's own overrides cannot leak into the hermetic + // tests; set either via extraEnv to test it. + env: { ...process.env, CLAUDE_CONFIG_DIR: configDir, CLAUDE_CODE_SUBAGENT_MODEL: "", CLAUDE_CODE_EFFORT_LEVEL: "", ...(extraEnv ?? {}) }, + // Default the working directory to the temp config dir, never the directory + // the suite happens to run from: the hook reads /.claude/settings*.json + // for the session effort, so an ambient cwd would let a real settings file + // two levels up decide a test outcome. Tests that need a specific project + // cascade pass their own cwd. + cwd: cwd ?? configDir, input: stdin ?? "", encoding: "utf-8", }); @@ -133,18 +140,24 @@ test("tokens with no transcripts explains where it looked", () => { } finally { rmSync(cfg, { recursive: true, force: true }); } }); -test("PINNED_MODELS mirrors agents/*.md frontmatter", () => { +test("AGENT_PINS mirrors agents/*.md frontmatter, both columns", () => { // The exact drift class 0.7.1 fixes: a pin changed in frontmatter but not // in the stats table. Bidirectional deepEqual catches a stale pin, a new - // agent missing from the table, and an orphaned table entry alike. + // agent missing from the table, and an orphaned table entry alike - and it + // now covers effort as well as model, because both live in one table. const agentsDir = join(dirname(SCRIPT), "..", "agents"); const fromFrontmatter = Object.fromEntries( readdirSync(agentsDir).filter((f) => f.endsWith(".md")).map((f) => { const fm = readFileSync(join(agentsDir, f), "utf-8").match(/^---\r?\n([\s\S]*?)\r?\n---/)[1]; - return [`model-routing:${f.replace(/\.md$/, "")}`, fm.match(/^model:\s*(\S+)/m)?.[1] ?? null]; + return [`model-routing:${f.replace(/\.md$/, "")}`, { + model: fm.match(/^model:\s*(\S+)/m)?.[1] ?? null, + effort: fm.match(/^effort:\s*(\S+)/m)?.[1] ?? null, + }]; })); - const table = readFileSync(SCRIPT, "utf-8").match(/const PINNED_MODELS = \{([\s\S]*?)\};/)[1]; - const pinned = Object.fromEntries([...table.matchAll(/"([^"]+)":\s*"([^"]+)"/g)].map((m) => [m[1], m[2]])); + const table = readFileSync(SCRIPT, "utf-8").match(/const AGENT_PINS = \{([\s\S]*?)^\};/m)[1]; + const pinned = Object.fromEntries( + [...table.matchAll(/"([^"]+)":\s*\{\s*model:\s*"([^"]+)",\s*effort:\s*"([^"]+)"\s*\}/g)] + .map((m) => [m[1], { model: m[2], effort: m[3] }])); assert.deepEqual(pinned, fromFrontmatter); }); @@ -569,3 +582,270 @@ test("tokens reaches Workflow-spawned agents nested under subagents/workflows/", assert.match(run(["tokens", "--session", "fable"], cfg), /No subagent transcripts found[\s\S]*fable sessions/); } finally { rmSync(cfg, { recursive: true, force: true }); } }); + +// Effort has three sources with a precedence between them, so these tests pin +// every rung: a temp CLAUDE_CONFIG_DIR for the user file, a temp cwd for the two +// project files, and an explicit env var for the override. Nothing here may read +// the directory the suite happens to run from - run() defaults cwd to the temp +// config dir for exactly that reason. +function freshCwd() { + return mkdtempSync(join(tmpdir(), "mr-cwd-")); +} + +function logEntries(cfg) { + return readFileSync(join(cfg, "model-routing", "dispatches.jsonl"), "utf-8") + .split("\n").filter(Boolean).map((l) => JSON.parse(l)); +} + +// One dispatch through the hook, returning the logged entry. `settings` maps a +// rung to its JSON: "user" -> /settings.json, "project" -> +// /.claude/settings.json, "local" -> /.claude/settings.local.json; a +// string value is written verbatim so a rung can hold unparseable content. +// sessionModel defaults to a supported model so the precedence cases exercise +// the real clamp path - with no model the clamp cannot run and every one of them +// would assert against an early return instead. Pass null to test that. +function dispatchWithSettings({ settings = {}, env, sessionModel = "claude-opus-5" }) { + const cfg = freshConfigDir(); + const cwd = freshCwd(); + const write = (p, v) => writeFileSync(p, typeof v === "string" ? v : JSON.stringify(v)); + try { + if (settings.user !== undefined) write(join(cfg, "settings.json"), settings.user); + if (settings.project !== undefined || settings.local !== undefined) mkdirSync(join(cwd, ".claude"), { recursive: true }); + if (settings.project !== undefined) write(join(cwd, ".claude", "settings.json"), settings.project); + if (settings.local !== undefined) write(join(cwd, ".claude", "settings.local.json"), settings.local); + const event = { tool_name: "Agent", tool_input: { subagent_type: "x" }, cwd }; + if (sessionModel) { + const sess = join(cfg, "sess.jsonl"); + writeFileSync(sess, JSON.stringify({ message: { model: sessionModel } }) + "\n"); + event.transcript_path = sess; + } + run([], cfg, JSON.stringify(event), env); + return logEntries(cfg)[0]; + } finally { + rmSync(cfg, { recursive: true, force: true }); + rmSync(cwd, { recursive: true, force: true }); + } +} + +test("hook records the session effort from the user settings file", () => { + const e = dispatchWithSettings({ settings: { user: { effortLevel: "high" } } }); + assert.equal(e.effort, "high"); + assert.equal(e.effortFrom, "settings"); +}); + +test("the project settings file outranks the user one for effort", () => { + // The middle rung: without a test here, dropping or misordering it passes. + const e = dispatchWithSettings({ settings: { user: { effortLevel: "high" }, project: { effortLevel: "medium" } } }); + assert.equal(e.effort, "medium"); +}); + +test("the local settings file outranks the project one for effort", () => { + const e = dispatchWithSettings({ settings: { project: { effortLevel: "medium" }, local: { effortLevel: "low" } } }); + assert.equal(e.effort, "low"); +}); + +test("CLAUDE_CODE_EFFORT_LEVEL overrides every settings rung", () => { + // The env var wins in Claude Code, so reporting the settings value would name + // an effort the session did not run on. + const e = dispatchWithSettings({ + settings: { user: { effortLevel: "high" }, local: { effortLevel: "high" } }, + env: { CLAUDE_CODE_EFFORT_LEVEL: "low" }, + }); + assert.equal(e.effort, "low"); + assert.equal(e.effortFrom, "env"); +}); + +test("max is accepted from the env var but rejected from a settings file", () => { + // Settings reject max as session-only; the env var accepts it. + const fromEnv = dispatchWithSettings({ env: { CLAUDE_CODE_EFFORT_LEVEL: "max" } }); + assert.equal(fromEnv.effort, "max"); + const fromSettings = dispatchWithSettings({ settings: { user: { effortLevel: "max" } } }); + assert.equal("effort" in fromSettings, false); +}); + +test("ultracode is accepted from no source", () => { + // It is a Claude Code setting, not a model effort level: the persisted key + // and the env var both reject it, so it must never appear as a level. + assert.equal("effort" in dispatchWithSettings({ env: { CLAUDE_CODE_EFFORT_LEVEL: "ultracode" } }), false); + assert.equal("effort" in dispatchWithSettings({ settings: { user: { effortLevel: "ultracode" } } }), false); +}); + +test("a model the docs list as having no effort support gets no level", () => { + // The support table is an enumeration, not a version threshold: Haiku 4.5 is + // absent from it, and claude-3-5-sonnet ranks fine under TIER_PATTERNS while + // predating adaptive reasoning entirely. Neither may be handed a default, and + // neither may be handed a configured level as though it had run. + const haiku = dispatchWithSettings({ sessionModel: "claude-haiku-4-5" }); + assert.equal("effort" in haiku, false); + const old = dispatchWithSettings({ sessionModel: "claude-3-5-sonnet-20241022", settings: { user: { effortLevel: "high" } } }); + assert.equal("effort" in old, false); +}); + +test("a level the model does not support is recorded as the level that ran", () => { + // "xhigh runs as high on Opus 4.6" - so logging xhigh there would name an + // effort the session never used. + const clamped = dispatchWithSettings({ settings: { user: { effortLevel: "xhigh" } }, sessionModel: "claude-opus-4-6" }); + assert.equal(clamped.effort, "high"); + assert.equal(clamped.effortFrom, "settings"); + // Control: the same level on a model that does support it is untouched. + const kept = dispatchWithSettings({ settings: { user: { effortLevel: "xhigh" } }, sessionModel: "claude-opus-5" }); + assert.equal(kept.effort, "xhigh"); +}); + +test("CLAUDE_CODE_EFFORT_LEVEL=auto records the model default, not the settings value", () => { + const e = dispatchWithSettings({ + settings: { user: { effortLevel: "low" } }, + env: { CLAUDE_CODE_EFFORT_LEVEL: "auto" }, + sessionModel: "claude-opus-5", + }); + assert.equal(e.effort, "high"); + assert.equal(e.effortFrom, "default"); +}); + +test("an unset effortLevel records the documented model default", () => { + // The most common configuration of all: nothing set anywhere. Omitting it + // left the report blank for most users. + const opus5 = dispatchWithSettings({ sessionModel: "claude-opus-5" }); + assert.equal(opus5.effort, "high"); + assert.equal(opus5.effortFrom, "default"); + // Opus 4.7 is the documented exception. + const opus47 = dispatchWithSettings({ sessionModel: "claude-opus-4-7" }); + assert.equal(opus47.effort, "xhigh"); +}); + +test("no default is invented for a model absent from the support table", () => { + // Named for the gate that actually fires: effort SUPPORT, not rankability. + // An unknown family fails both tests, which is why the claude-3-5-sonnet case + // above is the one that discriminates between them. + const e = dispatchWithSettings({ sessionModel: "claude-zephyr-1" }); + assert.equal("effort" in e, false); +}); + +test("every model in the effort table is rankable by the tier table", () => { + // The two model tables answer different questions at different granularity - + // tier by family, effort support by version - so they stay separate, but they + // must not disagree about which models exist. A model with effort support and + // no tier would be excluded from the routed-down math while still reporting an + // effort, which is the drift this pins. + const src = readFileSync(SCRIPT, "utf-8"); + const rows = src.match(/const EFFORT_SUPPORT = \[([\s\S]*?)^\];/m)[1]; + const families = [...rows.matchAll(/\/([^/]+)\//g)].flatMap((m) => m[1].split("|")); + assert.ok(families.length >= 6, `expected the documented model list, parsed ${families.length}`); + // Every level named in a support row must exist on the ladder clampEffort + // walks: a stray one would pass `levels.includes` and then fall out of the + // downward walk silently, since indexOf returns -1 for it. + const ladder = new Set([...src.match(/const EFFORT_LADDER = \[([^\]]*)\]/)[1].matchAll(/"([^"]+)"/g)].map((m) => m[1])); + for (const level of [...rows.matchAll(/"([^"]+)"/g)].map((m) => m[1])) { + assert.ok(ladder.has(level), `${level} is in EFFORT_SUPPORT but not on EFFORT_LADDER`); + } + // [\s\S] rather than . so reformatting either table across lines fails this + // test on its subject, not on its own parsing. + const tiers = src.match(/const TIER_PATTERNS = \[([\s\S]*?)\];/)[1]; + for (const family of families) { + const ranked = [...tiers.matchAll(/\/([^/]+)\//g)] + .some((m) => m[1].split("|").some((p) => new RegExp(p).test(`claude-${family}`))); + assert.ok(ranked, `${family} has effort support but no tier in TIER_PATTERNS`); + } +}); + +test("an effort value outside the ladder is ignored rather than logged", () => { + // A typo or a future level must not become a fabricated data point. + const e = dispatchWithSettings({ settings: { user: { effortLevel: "turbo" } } }); + assert.equal("effort" in e, false); +}); + +test("a bad value stops the walk instead of falling through to the next rung", () => { + // Falling through would log the user-level "high" for a session whose local + // file tried to set something else: a wrong data point replacing a missing + // one. The harness would not have read past that rung either. + const e = dispatchWithSettings({ + settings: { user: { effortLevel: "high" }, local: { effortLevel: "turbo" } }, + sessionModel: "claude-opus-5", + }); + assert.equal("effort" in e, false); +}); + +test("a malformed settings file defers to the next rung", () => { + // A file the harness cannot parse is ignored wholesale, so the next rung + // decides - unlike a parseable file carrying a bad value. + const e = dispatchWithSettings({ settings: { user: { effortLevel: "medium" }, local: "{ not json" } }); + assert.equal(e.effort, "medium"); +}); + +test("nothing is recorded when the session model could not be read", () => { + // The clamp needs the model: the same configured level means different things + // on different models, and on Haiku 4.5 it means nothing at all. So an + // unreadable session model records no effort rather than the configured value, + // matching how the rest of the report excludes an unknown session. + const fromSettings = dispatchWithSettings({ settings: { user: { effortLevel: "high" } }, sessionModel: null }); + assert.equal("effort" in fromSettings, false); + const fromEnv = dispatchWithSettings({ env: { CLAUDE_CODE_EFFORT_LEVEL: "max" }, sessionModel: null }); + assert.equal("effort" in fromEnv, false); +}); + +test("report separates inherited effort from pinned, and flags inferred levels", () => { + const cfg = freshConfigDir(); + const now = Date.now(); + writeLog(cfg, [ + { ts: now, agent: "general-purpose", model: "sonnet", session: "claude-opus-5", effort: "high", effortFrom: "settings" }, + { ts: now, agent: "general-purpose", model: "haiku", session: "claude-opus-5", effort: "high", effortFrom: "default" }, + // Frontmatter-pinned: carries its own effort, so it never inherits. Two of + // them, both on the model default, so a count scoped to the whole + // population instead of the inherited subset would print 3 of these under a + // total of 2 - the fixture has to be able to tell those apart. + { ts: now, agent: "model-routing:scout", session: "claude-opus-5", effort: "high", effortFrom: "default" }, + { ts: now, agent: "model-routing:reviewer", session: "claude-opus-5", effort: "high", effortFrom: "default" }, + ]); + try { + const out = run(["report"], cfg); + assert.match(out, /Effort: 2 of 4 dispatches ran on an agent type carrying no pin this plugin knows about/); + assert.match(out, /\(2 at high\)/); + // An inferred default must never read as an observed setting, and the count + // must be the inherited subset - 1 here, not the 3 in the whole population. + assert.match(out, /1 of these levels are the documented model default rather than an observed setting/); + // The source order and the invisible-change limit are stated, not implied. + assert.match(out, /CLAUDE_CODE_EFFORT_LEVEL, then settings effortLevel, then the model default/); + assert.match(out, /an agent from anywhere else may pin its own effort/i); + } finally { rmSync(cfg, { recursive: true, force: true }); } +}); + +test("report omits the effort section when no entry recorded one", () => { + const cfg = freshConfigDir(); + writeLog(cfg, [{ ts: Date.now(), agent: "general-purpose", model: "sonnet", session: "claude-opus-5" }]); + try { + assert.doesNotMatch(run(["report"], cfg), /^Effort:/m); + } finally { rmSync(cfg, { recursive: true, force: true }); } +}); + +test("tokens falls back to the transcript tail when the head names no model", () => { + const cfg = freshConfigDir(); + const dir = join(cfg, "projects", "proj", "sess-1", "subagents"); + mkdirSync(dir, { recursive: true }); + // A head larger than the 256KB read window that names no model: the first + // assistant message lands past it, exactly as in real multi-MB sessions, + // which used to drop the whole session out of the routed-down math. + const pad = JSON.stringify({ type: "user", filler: "x".repeat(300000) }); + writeFileSync(join(cfg, "projects", "proj", "sess-1.jsonl"), pad + "\n" + usageLine("claude-opus-5", 5000) + "\n"); + writeFileSync(join(dir, "agent-a.jsonl"), usageLine("claude-sonnet-5", 1000) + "\n"); + try { + const out = run(["tokens"], cfg); + assert.match(out, /opus-5: 1k across 1 agents - 100% below session tier/); + assert.doesNotMatch(out, /session unknown/); + // The footer must admit the tail case: for these sessions the attribution + // is the LAST model, the opposite bias from the start-model promise. + assert.match(out, /the tail is read instead and that session is attributed to its LAST model/); + } finally { rmSync(cfg, { recursive: true, force: true }); } +}); + +test("the not-comparable note names the unreadable-session cause, not just the tier table", () => { + const cfg = freshConfigDir(); + const dir = join(cfg, "projects", "proj", "sess-1", "subagents"); + mkdirSync(dir, { recursive: true }); + // No parent sess-1.jsonl at all: the session model cannot be read, which + // extending TIER_PATTERNS would never have fixed. + writeFileSync(join(dir, "agent-a.jsonl"), usageLine("claude-sonnet-5", 1000) + "\n"); + try { + const out = run(["tokens"], cfg); + assert.match(out, /no model could be read from the parent session transcript/); + } finally { rmSync(cfg, { recursive: true, force: true }); } +}); diff --git a/skills/model-routing/SKILL.md b/skills/model-routing/SKILL.md index b95a494..43366a2 100644 --- a/skills/model-routing/SKILL.md +++ b/skills/model-routing/SKILL.md @@ -29,8 +29,11 @@ thinking lightly often beats a weaker model thinking hard. Pick both: which model, and how hard it thinks. The full ladder is `low / medium / high / xhigh / max`. On current -models the API default is `high` - an unset effort IS high effort, not -medium. The per-model recommendation moves with the generation: Opus +models the default is `high` - an unset effort IS high effort, not +medium - with one documented exception: Opus 4.7 defaults to `xhigh`. +Which levels exist at all is a per-model list rather than a version +cutoff, and setting a level the model does not support runs the highest +supported level at or below it. The per-model recommendation moves with the generation: Opus 4.7 and 4.8 are told to start coding and agentic work at `xhigh`, while Opus 5 is told to start at `high`, step up to `xhigh` for demanding coding and agentic work, and use `low` and `medium` liberally as the