diff --git a/.changes/harness-ops/docs-harness-ops-held-rows-pilot-7b29b2d0.md b/.changes/harness-ops/docs-harness-ops-held-rows-pilot-7b29b2d0.md new file mode 100644 index 00000000000..1a7f5ad6d7a --- /dev/null +++ b/.changes/harness-ops/docs-harness-ops-held-rows-pilot-7b29b2d0.md @@ -0,0 +1,16 @@ +--- +bump: patch +--- + +### Fixed + +- `observability` no longer describes `cli-flag-verify` as a PostToolUse hook: guardrails runs it at push time in the `reference-verify` git hook, so the report marks hallucination-guard catches as not collected rather than reading an empty count as clean. +- `plugins` no longer says `claude plugin list` shows the highest version across scopes: on 2.1.322 it prints one row per install record with its own scope and version, and still never says which row loads here. +- `plugins` no longer tells a user to run `/reload-plugins` after every `Run /reload-plugins to apply.` summary: since 2.1.268 the `/plugin` panel can apply changes itself, so the two-step checks what the panel did (at the install pointer) and asks for a bare reload only when the plugin is still not active. + +### Changed + +- `plugins`: the shallow-clone reason for comparing only installs at the clone's HEAD points at the host-marketplace section that describes the re-clone, instead of restating it beside an unanchored link. +- `lanes`: the cloud-session note points at agent view's "List sessions as JSON" section, and the schtasks payload limit points at the `/tr` row the config already cites. +- `audit-skill-visibility`: the usage-counters reference names the binary reading the plugin-use definition rests on, and drops an unsourced polling rate and public-report remark. +- Nine eval cases cover held re-audit rows across `changelog`, `plugins`, `audit-skill-visibility`, `lanes`, `audit-performance`, `observability` and `audit-native-overlap`. diff --git a/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/case.yaml b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/case.yaml new file mode 100644 index 00000000000..f950cf91aeb --- /dev/null +++ b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/case.yaml @@ -0,0 +1,18 @@ +schema_version: "1.1" +name: ano-summarized-fetch-not-evidence +graders: + - name: fetch-answer-is-summary + type: regex + pattern: '\b(?:summary|summaries|summari[sz]\w*)|\bsmall(?:er)? model\b|\bparaphras\w*|\bmodel[’'']?s? (?:answer|reading|summary)\b|\bnot the (?:raw )?page\b' + flags: 'i' + target: last_message + - name: absence-not-evidence + type: regex + pattern: '\babsence\b[^.\n]{0,80}\bnot\b[^.\n]{0,40}\bevidence|\bnot\b[^.\n]{0,30}\bevidence of absence\b|\b(?:don[’'']?t|do not|cannot|can[’'']?t|shouldn[’'']?t|should not)\b[^.\n]{0,40}\brecord' + flags: 'i' + target: last_message + - name: uses-raw-page + type: regex + pattern: '\braw\b[^.\n]{0,60}\b(?:markdown|\.md|page|bytes|endpoint)|fetch-docs\.sh|\bskills\.md\b' + flags: 'i' + target: last_message diff --git a/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/prompt.md b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/prompt.md new file mode 100644 index 00000000000..7129f22a542 --- /dev/null +++ b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/prompt.md @@ -0,0 +1,10 @@ +--- +description: "One question audit-native-overlap answers: whether a WebFetch answer that omits a bundled skill proves Claude Code ships none (no: a summarizing fetch returns a model's answer about the page, so absence is not evidence; the claim needs the raw markdown). Reaches SKILL.md 'Any claim about what Claude Code itself ships'. Source: the consumer's own writing. routine. Read-only tools. Held-rows rows: harness-ops 15." +tags: [harness-ops, audit-native-overlap, reference-answer, routine] +max_turns: 15 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Glob, Grep] +expected_outcome: "The answer says no: a summarizing fetch returns a model's answer about the page, so absence from it is not evidence, and the claim must rest on the raw markdown page (fetch-docs.sh)." +--- + +/harness-ops:audit-native-overlap before a report: I asked WebFetch about https://code.claude.com/docs/en/skills and its answer didn't mention any bundled skill called `explain-usage`. Can I record that Claude Code ships no such skill? diff --git a/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/absence-not-evidence.json b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/absence-not-evidence.json new file mode 100644 index 00000000000..5d22af6df37 --- /dev/null +++ b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/absence-not-evidence.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Absence from that answer is not evidence of absence.", + "why": "direct" + }, + { + "answer": "You can't record that from a fetch answer.", + "why": "can't record" + } + ], + "fail": [ + { + "answer": "Yes, record it.", + "why": "wrong" + } + ] +} diff --git a/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/fetch-answer-is-summary.json b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/fetch-answer-is-summary.json new file mode 100644 index 00000000000..98382992291 --- /dev/null +++ b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/fetch-answer-is-summary.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "No. A summarizing fetch returns a small model's answer about the page.", + "why": "summary" + }, + { + "answer": "WebFetch hands back a paraphrase, not the page.", + "why": "paraphrase" + } + ], + "fail": [ + { + "answer": "No, read the raw markdown first.", + "why": "no reason" + } + ] +} diff --git a/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/uses-raw-page.json b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/uses-raw-page.json new file mode 100644 index 00000000000..0ad78e8073b --- /dev/null +++ b/plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/uses-raw-page.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "Fetch the raw markdown with fetch-docs.sh and search skills.md.", + "why": "raw" + } + ], + "fail": [ + { + "answer": "Ask WebFetch again with a sharper question.", + "why": "refetch summary" + } + ] +} diff --git a/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/case.yaml b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/case.yaml new file mode 100644 index 00000000000..0084f77acaf --- /dev/null +++ b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/case.yaml @@ -0,0 +1,23 @@ +schema_version: "1.1" +name: ap-fanout-version-and-kthreads +graders: + - name: subagents-carry-fanout + type: regex + pattern: '(?:hook|statusline|status line)\w*[^\n]{0,200}\b(?:each|every|per)\b[^.\n]{0,40}(?:sub-?agent|child)|\b(?:each|every|per)\b[^.\n]{0,40}sub-?agent[^\n]{0,200}(?:hook|statusline|status line)' + flags: 'i' + target: last_message + - name: version-fixes-slowdowns + type: regex + pattern: '\breleases?\b[^\n]{0,160}\bfix\w*[^\n]{0,120}\b(?:slow\w*|lag|cpu|regression\w*|quadratic|keystroke)|\bfix\w*[^\n]{0,120}\b(?:slowdown|lag|cpu|regression|quadratic|keystroke)\w*[^\n]{0,160}\breleases?\b' + flags: 'i' + target: last_message + - name: names-purge-lever + type: regex + pattern: 'claude purge\b' + flags: 'i' + target: last_message + - name: names-kthread-flag + type: regex + pattern: 'PF_KTHREAD|/proc/[^\s]*stat\b[^\n]{0,120}\bflags?\b|\bflags? field\b' + flags: 'i' + target: last_message diff --git a/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/prompt.md b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/prompt.md new file mode 100644 index 00000000000..8e7611ce75f --- /dev/null +++ b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/prompt.md @@ -0,0 +1,15 @@ +--- +description: "Four questions audit-performance answers from its text before running the engine: why modest concurrency and depth settings overload a host (every subagent carries the parent's statusline and hook fan-out), whether the Claude Code version can be the cause (releases fixed slowdowns), how to shrink ~/.claude.json (`claude purge `), and how the leak count tells kernel threads apart (the PF_KTHREAD flag in /proc//stat). Reaches SKILL.md suspects 2 and 5 and reference/known-performance-issues.md. Source: the consumer's own writing. routine. Read-only tools. Held-rows rows: harness-ops 17, 20, 25, 29." +tags: [harness-ops, audit-performance, reference-answer, routine] +max_turns: 20 +timeout_seconds: 400 +allowed_tools: [Skill, Read, Glob, Grep] +expected_outcome: "The answer says each subagent carries the parent's hook and statusline fan-out so depth times concurrency multiplies it, that releases fixed slowdowns so the version can be the cause, names `claude purge ` for ~/.claude.json, and names the PF_KTHREAD flag in /proc//stat." +--- + +/harness-ops:audit-performance don't run anything yet, I want to understand a few things first: + +1. I set the concurrent-subagent limit to 10 and allowed nesting three deep. Each setting looks modest, so why does my machine fall over when Claude fans out? +2. I'm on Claude Code 2.1.205. Could the version itself be part of the slowness? +3. Someone online said trimming `~/.claude.json` cured their input lag. Is that worth trying, and how? +4. On Linux, how does the leak count tell kernel threads apart from user processes? diff --git a/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/names-kthread-flag.json b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/names-kthread-flag.json new file mode 100644 index 00000000000..4a1d299d998 --- /dev/null +++ b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/names-kthread-flag.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "It reads the PF_KTHREAD bit in the flags field of /proc//stat.", + "why": "flag" + }, + { + "answer": "The flags field of `/proc//stat` marks kernel threads.", + "why": "flags field" + } + ], + "fail": [ + { + "answer": "It checks whether the parent pid is 2.", + "why": "ppid test" + } + ] +} diff --git a/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/names-purge-lever.json b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/names-purge-lever.json new file mode 100644 index 00000000000..74d77d492fc --- /dev/null +++ b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/names-purge-lever.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "Use `claude purge ` to remove one project's entry.", + "why": "lever" + } + ], + "fail": [ + { + "answer": "Edit ~/.claude.json by hand and delete old projects.", + "why": "hand edit" + } + ] +} diff --git a/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/subagents-carry-fanout.json b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/subagents-carry-fanout.json new file mode 100644 index 00000000000..0e7fb4390ba --- /dev/null +++ b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/subagents-carry-fanout.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Every subagent carries the same statusline and hook fan-out as its parent.", + "why": "direct" + }, + { + "answer": "Hooks and the statusline run again for each subagent.", + "why": "reverse" + } + ], + "fail": [ + { + "answer": "Ten subagents three deep is a lot of processes.", + "why": "no fan-out" + } + ] +} diff --git a/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/version-fixes-slowdowns.json b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/version-fixes-slowdowns.json new file mode 100644 index 00000000000..ccf5d35e067 --- /dev/null +++ b/plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/version-fixes-slowdowns.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Yes. Several 2.1.2xx releases fixed long-session slowdowns and keystroke lag.", + "why": "releases fixed" + }, + { + "answer": "Fixes for CPU regressions landed in later releases.", + "why": "reverse" + } + ], + "fail": [ + { + "answer": "The version is unlikely to matter.", + "why": "wrong" + } + ] +} diff --git a/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/case.yaml b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/case.yaml new file mode 100644 index 00000000000..5956aba4e45 --- /dev/null +++ b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/case.yaml @@ -0,0 +1,18 @@ +schema_version: "1.1" +name: asv-absent-key-subagent-use-counts +graders: + - name: absent-key-not-loaded + type: regex + pattern: 'defaultEnabled[^\n]{0,200}\b(?:not|never|ignored|no effect|doesn[’'']?t|does not)\b|\b(?:not|never|doesn[’'']?t|does not)\b[^\n]{0,80}defaultEnabled|\b(?:will be|is|stays|shows as|reads|reported as|counts as|treated as) (?:disabled|hidden|off)\b' + flags: 'i' + target: last_message + - name: subagent-own-window + type: regex + pattern: 'sub-?agent[^\n]{0,250}\b(?:own (?:context )?window|smaller (?:context )?window|shed\w*|(?:truncated?|truncates|truncation|truncating)|cut short|drop\w*|not (?:covered|guaranteed)|doesn[’'']?t (?:cover|carry|apply)|does not (?:cover|carry|apply))|\bmain session\b[^\n]{0,120}\bonly\b|\bonly\b[^\n]{0,60}\bmain session\b' + flags: 'i' + target: last_message + - name: use-count-not-comparable + type: regex + pattern: '\bhooks?\b[^\n]{0,250}\b(?:count\w*|dispatch\w*|inflat\w*|dominat\w*|fires?|fired|every)\b|\bnot (?:directly )?comparable\b|\bincomparable\b' + flags: 'i' + target: last_message diff --git a/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/prompt.md b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/prompt.md new file mode 100644 index 00000000000..3598d4f45f5 --- /dev/null +++ b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/prompt.md @@ -0,0 +1,14 @@ +--- +description: "Three questions audit-skill-visibility answers from its text: an installed plugin with `defaultEnabled: true` and no `enabledPlugins` key (it does not load; the runtime reading beats the docs), whether a fitting listing holds in subagents (a subagent's listing is sized to its own window and can shed), and whether a raw plugin use count ranks usage (hooks and other dispatches count, so hook-heavy plugins dominate). Reaches SKILL.md (verification records, subagent sentence) and reference/usage-counters.md. Source: the consumer's own writing. hard: the docs say defaultEnabled decides an absent key. Read-only tools. Held-rows rows: harness-ops 34, 38, 45." +tags: [harness-ops, audit-skill-visibility, reference-answer, hard] +max_turns: 20 +timeout_seconds: 400 +allowed_tools: [Skill, Read, Glob, Grep] +expected_outcome: "The answer says notes-kit's skills do not load (defaultEnabled does not decide an absent key at runtime), that the fit verdict covers the main session and a subagent's listing is sized to its own window and can shed, and that plugin use counts include hook and other dispatches so they are not comparable across plugin shapes." +--- + +/harness-ops:audit-skill-visibility no need to run the full audit, I just have three questions: + +1. Our `notes-kit` plugin is installed, its marketplace entry sets `defaultEnabled: true`, and no settings file has an `enabledPlugins` entry for it. Will its skills show up? +2. Your last run said my skill listing fits. Does that hold for subagents too? +3. `~/.claude.json` shows `guardrails` with about 200,000 uses and `notes-kit` with 150. Is guardrails really the plugin I use most? diff --git a/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/absent-key-not-loaded.json b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/absent-key-not-loaded.json new file mode 100644 index 00000000000..fe642e60ff7 --- /dev/null +++ b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/absent-key-not-loaded.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "No. `defaultEnabled` does not decide an absent key at runtime, so notes-kit is disabled.", + "why": "direct" + }, + { + "answer": "Its skills won't load: the plugin is treated as disabled with no enabledPlugins entry.", + "why": "treated as disabled" + } + ], + "fail": [ + { + "answer": "Yes, defaultEnabled: true turns it on.", + "why": "wrong" + } + ] +} diff --git a/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/subagent-own-window.json b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/subagent-own-window.json new file mode 100644 index 00000000000..85c3976e1b0 --- /dev/null +++ b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/subagent-own-window.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "Not necessarily. A subagent gets a listing sized to its own window and can shed skills.", + "why": "own window" + }, + { + "answer": "The verdict covers only the main session.", + "why": "only main" + } + ], + "fail": [ + { + "answer": "Yes, subagents see the same listing.", + "why": "wrong" + } + ] +} diff --git a/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/use-count-not-comparable.json b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/use-count-not-comparable.json new file mode 100644 index 00000000000..a6ee64218d0 --- /dev/null +++ b/plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/use-count-not-comparable.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "No. Every hook dispatch counts as a use, so hook-only plugins dominate.", + "why": "hooks count" + }, + { + "answer": "Those numbers are not comparable across plugin shapes.", + "why": "not comparable" + } + ], + "fail": [ + { + "answer": "Yes, guardrails is your most used plugin.", + "why": "wrong" + } + ] +} diff --git a/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/case.yaml b/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/case.yaml new file mode 100644 index 00000000000..8498cc6db92 --- /dev/null +++ b/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/case.yaml @@ -0,0 +1,13 @@ +schema_version: "1.1" +name: cl-drift-dedupe-sees-closed +graders: + - name: dedupe-includes-closed + type: regex + pattern: '--state all\b|\ball states\b|(?:^|[.!?;](?=\s|$))(?!(?:(?![.!?;](?:\s|$))[^\n])*?(?:\b(?:not|no|never|cannot|misses|missed)\b|n[’'']t\b|\bonly (?:open|lists? open|sees? open)))(?:(?![.!?;](?:\s|$))[^\n])*?(?:\b(?:includes?|including|included|covers?|sees?|finds?|searches|lists?)\b(?:(?![.!?;](?:\s|$))[^\n]){0,40}?\bclosed\b|\bclosed\b(?:(?![.!?;](?:\s|$))[^\n]){0,40}?\b(?:too|as well|included)\b)' + flags: 'im' + target: last_message + - name: raises-limit + type: regex + pattern: '--limit 1000\b|\b1,?000\b|\blimit\b[^.\n]{0,40}\braised\b|\b30\b[^.\n]{0,60}\bdefault|\bdefault\b[^.\n]{0,60}\b30\b' + flags: 'i' + target: last_message diff --git a/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/prompt.md b/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/prompt.md new file mode 100644 index 00000000000..479f9e7b5c5 --- /dev/null +++ b/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/prompt.md @@ -0,0 +1,10 @@ +--- +description: "One question about the changelog skill's native-drift step: its duplicate check lists issues in every state with a raised limit (`gh issue list --state all --limit 1000`), so a closed match is found even with 200 open issues. Reaches context/native-drift.md (dedupe step and its recorded fact). Source: the consumer's own writing. routine. Needs --allow-tools Bash for the pre-computed line. Held-rows rows: harness-ops 50." +tags: [harness-ops, changelog, reference-answer, routine] +max_turns: 15 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Glob, Grep, Bash] +expected_outcome: "No apply runs. The answer says the check lists issues in all states (closed included) with a 1000 limit, above the 30 default, so the closed issue is found." +--- + +/harness-ops:changelog quick question about the native-drift step in apply: before it files an item it checks for an existing one. We closed a matching issue last month and have about 200 open issues. Will that check find the closed one? Answer only, don't run apply. diff --git a/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/samples/dedupe-includes-closed.json b/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/samples/dedupe-includes-closed.json new file mode 100644 index 00000000000..cf724235f94 --- /dev/null +++ b/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/samples/dedupe-includes-closed.json @@ -0,0 +1,34 @@ +{ + "pass": [ + { + "answer": "Yes, it runs `gh issue list --state all --limit 1000`, so the closed issue is found and no duplicate is filed.", + "why": "flag with later no" + }, + { + "answer": "Yes: per `context/native-drift.md`, the check runs `gh issue list --state all --limit 1000`.", + "why": "filename dot" + }, + { + "answer": "Yes. It runs `gh issue list --state all`, so closed issues are included.", + "why": "flag" + }, + { + "answer": "Yes, the search covers closed issues as well.", + "why": "prose" + } + ], + "fail": [ + { + "answer": "No, it only sees open issues.", + "why": "wrong" + }, + { + "answer": "No, it won't find the closed one.", + "why": "negated find" + }, + { + "answer": "It lists only open issues, so the closed one is missed.", + "why": "only open" + } + ] +} diff --git a/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/samples/raises-limit.json b/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/samples/raises-limit.json new file mode 100644 index 00000000000..800c6744515 --- /dev/null +++ b/plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/samples/raises-limit.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "It passes `--limit 1000`, so 200 open issues do not crowd it out.", + "why": "limit" + }, + { + "answer": "The default of 30 would miss it, which is why the limit is raised.", + "why": "default 30" + } + ], + "fail": [ + { + "answer": "Yes, it searches closed issues.", + "why": "no limit" + } + ] +} diff --git a/plugins/harness-ops/evals/cl-one-cluster-stays-inline/case.yaml b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/case.yaml new file mode 100644 index 00000000000..b394d2b3f9e --- /dev/null +++ b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/case.yaml @@ -0,0 +1,21 @@ +schema_version: "1.1" +name: cl-one-cluster-stays-inline +context: + scaffold_script: scaffold.sh +graders: + - name: ran-status-on-range + type: tool_used + tool: 'Bash' + input_match: '"command":"(?:[^"\\]|\\.)*?changelog-status\.sh(?:[^"\\]|\\.)*?--range' + min: 1 + - name: stays-inline + type: tool_used + tool: 'Agent' + input_match: '"description":"(?![^"]*[Vv]erif)' + min: 0 + max: 0 + - name: row-for-notes-skill + type: regex + pattern: 'skills/notes/SKILL\.md' + flags: 'i' + target: last_message diff --git a/plugins/harness-ops/evals/cl-one-cluster-stays-inline/prompt.md b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/prompt.md new file mode 100644 index 00000000000..89cfc9aed67 --- /dev/null +++ b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/prompt.md @@ -0,0 +1,12 @@ +--- +description: "A two-release diff with three items, one a behavioral change the scaffolded skill states, so Phase 1 explores and Phase 2 research is reached. One release cluster and one feature cluster: explore and research stay in this session; only the local verifier may run as a subagent. Reaches SKILL.md 'Phase 1. Explore', 'Phase 2. Research' step 1 and context/decisions.md 'Fan-out shape'. Source: the consumer's own writing (a small release). routine: on Opus 5.5 the current text already kept explore and research in-session. Needs --scaffold --allow-tools Bash (Agent granted for the no-spawn grader). Held-rows rows: harness-ops 48, 49." +tags: [harness-ops, changelog, delegation, no-network, routine] +max_turns: 40 +timeout_seconds: 900 +allowed_tools: [Skill, Read, Glob, Grep, Bash, Agent] +expected_outcome: "The skill runs changelog-status.sh on v2.1.260..v2.1.261 with the local changelog, explores and researches the three items in this session without spawning an explorer or researcher (the local verifier may run), and emits a correct row for skills/notes/SKILL.md (its 80% compaction claim)." +--- + +/harness-ops:changelog diff v2.1.260..v2.1.261 + +This machine has no network. A complete copy of the changelog is at ./changelog.md; use it instead of fetching. diff --git a/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/ran-status-on-range.json b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/ran-status-on-range.json new file mode 100644 index 00000000000..b361998b1d8 --- /dev/null +++ b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/ran-status-on-range.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "bash /p/skills/changelog/scripts/changelog-status.sh --changelog ./changelog.md --range v2.1.260..v2.1.261", + "description": "status" + } + } + ], + "why": "range" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "bash /p/skills/changelog/scripts/changelog-status.sh --no-fetch", + "description": "status" + } + } + ], + "why": "no range" + } + ] +} diff --git a/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/row-for-notes-skill.json b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/row-for-notes-skill.json new file mode 100644 index 00000000000..19c4f88343a --- /dev/null +++ b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/row-for-notes-skill.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "| skills/notes/SKILL.md | correct | The compaction default is now 90%, not 80%. |", + "why": "row" + }, + { + "answer": "Owner surface `skills/notes/SKILL.md`: correct the 80% claim.", + "why": "prose" + } + ], + "fail": [ + { + "answer": "No item in range touches this repository.", + "why": "no row" + }, + { + "answer": "The notes skill states 80%; correct it.", + "why": "no path" + } + ] +} diff --git a/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/stays-inline.json b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/stays-inline.json new file mode 100644 index 00000000000..dfa4bf4e6be --- /dev/null +++ b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/samples/stays-inline.json @@ -0,0 +1,49 @@ +{ + "pass": [ + { + "answer": [], + "why": "no call" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "description": "Verify repo-side row facts", + "prompt": "check paths", + "subagent_type": "Explore" + } + } + ], + "why": "local verifier only" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "description": "Explore 2.1.260-261", + "prompt": "grep surfaces", + "subagent_type": "Explore" + } + } + ], + "why": "explorer" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "description": "Research compaction", + "prompt": "docs", + "subagent_type": "claude-code-guide" + } + } + ], + "why": "researcher" + } + ] +} diff --git a/plugins/harness-ops/evals/cl-one-cluster-stays-inline/scaffold.sh b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/scaffold.sh new file mode 100755 index 00000000000..33dacae8459 --- /dev/null +++ b/plugins/harness-ops/evals/cl-one-cluster-stays-inline/scaffold.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# A small plugin whose skill states a default, and a two-release changelog that changes it. +set -euo pipefail +mkdir -p .claude-plugin skills/notes +printf '{"name":"notes-kit","version":"1.0.0","description":"Session notes"}\n' >.claude-plugin/plugin.json +cat >skills/notes/SKILL.md <<'EOF' +--- +description: Save session notes before the conversation is compacted. +--- + +# Notes + +Compaction starts when the conversation reaches 80% of the context window (the +`autoCompactWindow` default), so write the notes file before then. +EOF +cat >changelog.md <<'EOF' +# Claude Code changelog + + +* Changed the `autoCompactWindow` default from 80% to 90% of the context window +* Fixed a crash when a hook printed invalid UTF-8 + + + +* Added a `--tail` flag to `claude agents` + +EOF diff --git a/plugins/harness-ops/evals/la-compaction-resume-and-cloud/case.yaml b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/case.yaml new file mode 100644 index 00000000000..df148767a93 --- /dev/null +++ b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/case.yaml @@ -0,0 +1,24 @@ +schema_version: "1.1" +name: la-compaction-resume-and-cloud +graders: + - name: compaction-not-reset + type: regex + pattern: 'compact\w*[^\n]{0,250}\b(?:not a (?:fresh |true |real |clean )?(?:reset|context|start)|isn[’'']?t a (?:fresh |true |real |clean )?(?:reset|context|start)|lossy|(?:summary|summaries|summari[sz]\w*))|\b(?:lossy|(?:summary|summaries|summari[sz]\w*))[^\n]{0,200}compact' + flags: 'i' + target: last_message + - name: declines-resume-bg + type: regex + pattern: '(?:\bno\b|don[’'']?t|do not|never|avoid|shouldn[’'']?t|should not|\bnot\b)[^.\n]{0,100}--resume|--resume[^\n]{0,200}\b(?:instead|rather|fresh|declin\w*|not a restart|isn[’'']?t a restart)' + flags: 'i' + target: last_message + - name: cloud-not-listed + type: regex + pattern: '(?:does not|doesn[’'']?t|won[’'']?t|will not|\bnot\b|never)[^.\n]{0,40}\b(?:list|show|include|see|report)\w*\b[^.\n]{0,60}\bcloud|\bcloud\b[^.\n]{0,100}\b(?:not (?:listed|shown|included|visible)|aren[’'']?t (?:listed|shown|included)|invisible|missing)|\bnot (?:the )?cloud sessions?\b|\b(?:only (?:lists?|shows?)|lists? only|shows? only)\b[^.\n]{0,60}\b(?:this|your|the local|local)\b[^.\n]{0,20}\b(?:machine|computer|sessions)\b' + flags: 'i' + target: last_message + - name: git-gather-separate + type: tool_used + tool: 'Bash' + input_match: '"command":"(?=(?:[^"\\]|\\.)*\bgit\b)(?:[^"\\]|\\.)*?(?:&&|;|\|\|)' + min: 0 + max: 0 diff --git a/plugins/harness-ops/evals/la-compaction-resume-and-cloud/prompt.md b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/prompt.md new file mode 100644 index 00000000000..ba2f7f12040 --- /dev/null +++ b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/prompt.md @@ -0,0 +1,16 @@ +--- +description: "Three questions the lanes skill answers before any lane runs: auto-compaction is a lossy summary, not a reset; a lane is never restarted with `claude --resume --bg`; `claude agents --json` does not list cloud sessions. Any git gather call stays a separate Bash call. Reaches SKILL.md (repository context, 'A relaunch is the only context reset', Gotchas, verified CLI surface) and context/config.md. Source: the consumer's own writing. routine. Needs --allow-tools Bash for the pre-computed lines. Held-rows rows: harness-ops 64, 66, 71, 74." +tags: [harness-ops, lanes, reference-answer, routine] +max_turns: 20 +timeout_seconds: 400 +allowed_tools: [Skill, Read, Glob, Grep, Bash] +expected_outcome: "Nothing starts or stops. The answer says auto-compaction summarizes older history and is not a fresh context, declines `--resume ... --bg` as a restart (restart launches a fresh session from the prompt file), and says cloud sessions do not show in `claude agents --json`." +--- + +/harness-ops:lanes three questions before I set up a lane on this box: + +1. If a lane runs for a week, does auto-compaction give it a fresh context, so I can skip restarts? +2. Can I restart a lane by resuming its session with `claude --resume --bg`? +3. If I run a lane in the cloud, will `claude agents --json` show it? + +Answer only, don't start or stop anything. diff --git a/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/cloud-not-listed.json b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/cloud-not-listed.json new file mode 100644 index 00000000000..3c0aec04148 --- /dev/null +++ b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/cloud-not-listed.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "No. `claude agents --json` only lists sessions running on this machine, not cloud sessions.", + "why": "only this machine" + }, + { + "answer": "No. That command lists only sessions on this machine.", + "why": "lists only" + }, + { + "answer": "No. `claude agents --json` does not list cloud sessions.", + "why": "direct" + }, + { + "answer": "Cloud lanes are not listed there; the launcher keeps a marker.", + "why": "cloud first" + } + ], + "fail": [ + { + "answer": "Yes, it shows cloud lanes too.", + "why": "wrong" + } + ] +} diff --git a/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/compaction-not-reset.json b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/compaction-not-reset.json new file mode 100644 index 00000000000..58487fa509e --- /dev/null +++ b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/compaction-not-reset.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "No. Auto-compaction summarizes older history in place; it is not a reset.", + "why": "summary" + }, + { + "answer": "Compaction isn't a fresh context, so keep restarts.", + "why": "isn't fresh" + } + ], + "fail": [ + { + "answer": "Yes, compaction gives it a fresh context.", + "why": "wrong" + } + ] +} diff --git a/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/declines-resume-bg.json b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/declines-resume-bg.json new file mode 100644 index 00000000000..dc23c276c86 --- /dev/null +++ b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/declines-resume-bg.json @@ -0,0 +1,18 @@ +{ + "pass": [ + { + "answer": "No, do not use `claude --resume --bg`; restart launches a fresh session.", + "why": "decline" + }, + { + "answer": "`--resume` continues the old transcript; use restart instead.", + "why": "instead" + } + ], + "fail": [ + { + "answer": "Yes, that resumes the lane.", + "why": "wrong" + } + ] +} diff --git a/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/git-gather-separate.json b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/git-gather-separate.json new file mode 100644 index 00000000000..64414fa6b0b --- /dev/null +++ b/plugins/harness-ops/evals/la-compaction-resume-and-cloud/samples/git-gather-separate.json @@ -0,0 +1,34 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "git rev-parse --show-toplevel", + "description": "root" + } + } + ], + "why": "single" + }, + { + "answer": [], + "why": "no call" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Bash", + "input": { + "command": "git remote get-url origin && git rev-parse --show-toplevel", + "description": "gather" + } + } + ], + "why": "chained" + } + ] +} diff --git a/plugins/harness-ops/evals/ob-cost-and-flag-verify/case.yaml b/plugins/harness-ops/evals/ob-cost-and-flag-verify/case.yaml new file mode 100644 index 00000000000..367b7183acc --- /dev/null +++ b/plugins/harness-ops/evals/ob-cost-and-flag-verify/case.yaml @@ -0,0 +1,19 @@ +schema_version: "1.1" +name: ob-cost-and-flag-verify +graders: + - name: cost-is-list-estimate + type: regex + pattern: '\blist[- ]?(?:rate|price|pric\w*)\b|\bestimat\w*\b[^.\n]{0,80}\bnot\b[^.\n]{0,30}\bbill|\bnot (?:a|your|the) bill\b|\bnot what you(?:[’'']ll| will| are)? (?:be )?(?:billed|charged|pay)' + flags: 'i' + target: last_message + - name: no-posttooluse-claim + type: regex + pattern: '\bcli-flag-verify\b(?:(?![.!?;](?:\s|$))[^\n]){0,200}?(?`, which removes one project's entry. Every running -session polls the file at 1 Hz, and the strongest public report of curing input -lag pruned this file rather than the install tree. +`claude purge `, which removes one project's entry. - **Pointer:** [claude-directory: cleaned up automatically](https://code.claude.com/docs/en/claude-directory#cleaned-up-automatically) lists the paths the retention sweep deletes, all under `~/.claude/`, and does not diff --git a/plugins/harness-ops/skills/lanes/context/config.md b/plugins/harness-ops/skills/lanes/context/config.md index 494796272e6..64065440b0c 100644 --- a/plugins/harness-ops/skills/lanes/context/config.md +++ b/plugins/harness-ops/skills/lanes/context/config.md @@ -123,8 +123,9 @@ marketplace repository; this section covers only what the launcher does. has changes, the lane's telemetry cannot be read (no numeric `telemetry.issue`, or a `telemetry.repo` other than origin's), or the lane requests the lane-stop gate. A not-yet-honored `execution_target_fallback` from an authorized author in the lane's telemetry moves one launch to - today's launch. `claude agents --json` does not list cloud sessions, so the - launcher writes `/lanes//-cloud-launch`: `start` skips a lane while it + today's launch. `claude agents --json` lists the sessions on this machine + ([List sessions as JSON](https://code.claude.com/docs/en/agent-view#list-sessions-as-json)), not + cloud sessions, so the launcher writes `/lanes//-cloud-launch`: `start` skips a lane while it stands, and `restart` sends a new session (archive the earlier one at claude.ai). - **`cloud-routine`, `cloud-project`.** Setup steps are printed and nothing launches; `restart` leaves a running local session of the lane up. A lane whose stage may read untrusted input gets no @@ -165,7 +166,8 @@ prompt file is not snapshotted: each run still reads it from the checkout's prom lane can rewrite its next scheduled prompt, but that prompt runs under the same pinned permission mode the lane already has, so the rewrite gives it no privilege it lacks. Calling a short script keeps the Windows `/TR` payload to `"" "