docs(harness-ops): ship eval-gated held rows and three stale-fact fixes

.changes/harness-ops/docs-harness-ops-held-rows-pilot-7b29b2d0.md

@@ -0,0 +1,16 @@
+---
+bump: patch
+---
+
+### Fixed
+
+- `observability` no longer describes `cli-flag-verify` as a PostToolUse hook: guardrails runs it at push time in the `reference-verify` git hook, so the report marks hallucination-guard catches as not collected rather than reading an empty count as clean.
+- `plugins` no longer says `claude plugin list` shows the highest version across scopes: on 2.1.322 it prints one row per install record with its own scope and version, and still never says which row loads here.
+- `plugins` no longer tells a user to run `/reload-plugins` after every `Run /reload-plugins to apply.` summary: since 2.1.268 the `/plugin` panel can apply changes itself, so the two-step checks what the panel did (at the install pointer) and asks for a bare reload only when the plugin is still not active.
+
+### Changed
+
+- `plugins`: the shallow-clone reason for comparing only installs at the clone's HEAD points at the host-marketplace section that describes the re-clone, instead of restating it beside an unanchored link.
+- `lanes`: the cloud-session note points at agent view's "List sessions as JSON" section, and the schtasks payload limit points at the `/tr` row the config already cites.
+- `audit-skill-visibility`: the usage-counters reference names the binary reading the plugin-use definition rests on, and drops an unsourced polling rate and public-report remark.
+- Nine eval cases cover held re-audit rows across `changelog`, `plugins`, `audit-skill-visibility`, `lanes`, `audit-performance`, `observability` and `audit-native-overlap`.

plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/case.yaml

@@ -0,0 +1,18 @@
+schema_version: "1.1"
+name: ano-summarized-fetch-not-evidence
+graders:
+  - name: fetch-answer-is-summary
+    type: regex
+    pattern: '\b(?:summary|summaries|summari[sz]\w*)|\bsmall(?:er)? model\b|\bparaphras\w*|\bmodel[’'']?s? (?:answer|reading|summary)\b|\bnot the (?:raw )?page\b'
+    flags: 'i'
+    target: last_message
+  - name: absence-not-evidence
+    type: regex
+    pattern: '\babsence\b[^.\n]{0,80}\bnot\b[^.\n]{0,40}\bevidence|\bnot\b[^.\n]{0,30}\bevidence of absence\b|\b(?:don[’'']?t|do not|cannot|can[’'']?t|shouldn[’'']?t|should not)\b[^.\n]{0,40}\brecord'
+    flags: 'i'
+    target: last_message
+  - name: uses-raw-page
+    type: regex
+    pattern: '\braw\b[^.\n]{0,60}\b(?:markdown|\.md|page|bytes|endpoint)|fetch-docs\.sh|\bskills\.md\b'
+    flags: 'i'
+    target: last_message

plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "One question audit-native-overlap answers: whether a WebFetch answer that omits a bundled skill proves Claude Code ships none (no: a summarizing fetch returns a model's answer about the page, so absence is not evidence; the claim needs the raw markdown). Reaches SKILL.md 'Any claim about what Claude Code itself ships'. Source: the consumer's own writing. routine. Read-only tools. Held-rows rows: harness-ops 15."
+tags: [harness-ops, audit-native-overlap, reference-answer, routine]
+max_turns: 15
+timeout_seconds: 300
+allowed_tools: [Skill, Read, Glob, Grep]
+expected_outcome: "The answer says no: a summarizing fetch returns a model's answer about the page, so absence from it is not evidence, and the claim must rest on the raw markdown page (fetch-docs.sh)."
+---
+
+/harness-ops:audit-native-overlap before a report: I asked WebFetch about https://code.claude.com/docs/en/skills and its answer didn't mention any bundled skill called `explain-usage`. Can I record that Claude Code ships no such skill?

plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/absence-not-evidence.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Absence from that answer is not evidence of absence.",
+      "why": "direct"
+    },
+    {
+      "answer": "You can't record that from a fetch answer.",
+      "why": "can't record"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Yes, record it.",
+      "why": "wrong"
+    }
+  ]
+}

plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/fetch-answer-is-summary.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "No. A summarizing fetch returns a small model's answer about the page.",
+      "why": "summary"
+    },
+    {
+      "answer": "WebFetch hands back a paraphrase, not the page.",
+      "why": "paraphrase"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "No, read the raw markdown first.",
+      "why": "no reason"
+    }
+  ]
+}

plugins/harness-ops/evals/ano-summarized-fetch-not-evidence/samples/uses-raw-page.json

@@ -0,0 +1,14 @@
+{
+  "pass": [
+    {
+      "answer": "Fetch the raw markdown with fetch-docs.sh and search skills.md.",
+      "why": "raw"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Ask WebFetch again with a sharper question.",
+      "why": "refetch summary"
+    }
+  ]
+}

plugins/harness-ops/evals/ap-fanout-version-and-kthreads/case.yaml

@@ -0,0 +1,23 @@
+schema_version: "1.1"
+name: ap-fanout-version-and-kthreads
+graders:
+  - name: subagents-carry-fanout
+    type: regex
+    pattern: '(?:hook|statusline|status line)\w*[^\n]{0,200}\b(?:each|every|per)\b[^.\n]{0,40}(?:sub-?agent|child)|\b(?:each|every|per)\b[^.\n]{0,40}sub-?agent[^\n]{0,200}(?:hook|statusline|status line)'
+    flags: 'i'
+    target: last_message
+  - name: version-fixes-slowdowns
+    type: regex
+    pattern: '\breleases?\b[^\n]{0,160}\bfix\w*[^\n]{0,120}\b(?:slow\w*|lag|cpu|regression\w*|quadratic|keystroke)|\bfix\w*[^\n]{0,120}\b(?:slowdown|lag|cpu|regression|quadratic|keystroke)\w*[^\n]{0,160}\breleases?\b'
+    flags: 'i'
+    target: last_message
+  - name: names-purge-lever
+    type: regex
+    pattern: 'claude purge\b'
+    flags: 'i'
+    target: last_message
+  - name: names-kthread-flag
+    type: regex
+    pattern: 'PF_KTHREAD|/proc/[^\s]*stat\b[^\n]{0,120}\bflags?\b|\bflags? field\b'
+    flags: 'i'
+    target: last_message

plugins/harness-ops/evals/ap-fanout-version-and-kthreads/prompt.md

@@ -0,0 +1,15 @@
+---
+description: "Four questions audit-performance answers from its text before running the engine: why modest concurrency and depth settings overload a host (every subagent carries the parent's statusline and hook fan-out), whether the Claude Code version can be the cause (releases fixed slowdowns), how to shrink ~/.claude.json (`claude purge <path>`), and how the leak count tells kernel threads apart (the PF_KTHREAD flag in /proc/<pid>/stat). Reaches SKILL.md suspects 2 and 5 and reference/known-performance-issues.md. Source: the consumer's own writing. routine. Read-only tools. Held-rows rows: harness-ops 17, 20, 25, 29."
+tags: [harness-ops, audit-performance, reference-answer, routine]
+max_turns: 20
+timeout_seconds: 400
+allowed_tools: [Skill, Read, Glob, Grep]
+expected_outcome: "The answer says each subagent carries the parent's hook and statusline fan-out so depth times concurrency multiplies it, that releases fixed slowdowns so the version can be the cause, names `claude purge <path>` for ~/.claude.json, and names the PF_KTHREAD flag in /proc/<pid>/stat."
+---
+
+/harness-ops:audit-performance don't run anything yet, I want to understand a few things first:
+
+1. I set the concurrent-subagent limit to 10 and allowed nesting three deep. Each setting looks modest, so why does my machine fall over when Claude fans out?
+2. I'm on Claude Code 2.1.205. Could the version itself be part of the slowness?
+3. Someone online said trimming `~/.claude.json` cured their input lag. Is that worth trying, and how?
+4. On Linux, how does the leak count tell kernel threads apart from user processes?

plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/names-kthread-flag.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "It reads the PF_KTHREAD bit in the flags field of /proc/<pid>/stat.",
+      "why": "flag"
+    },
+    {
+      "answer": "The flags field of `/proc/<pid>/stat` marks kernel threads.",
+      "why": "flags field"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "It checks whether the parent pid is 2.",
+      "why": "ppid test"
+    }
+  ]
+}

plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/names-purge-lever.json

@@ -0,0 +1,14 @@
+{
+  "pass": [
+    {
+      "answer": "Use `claude purge <path>` to remove one project's entry.",
+      "why": "lever"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Edit ~/.claude.json by hand and delete old projects.",
+      "why": "hand edit"
+    }
+  ]
+}

plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/subagents-carry-fanout.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Every subagent carries the same statusline and hook fan-out as its parent.",
+      "why": "direct"
+    },
+    {
+      "answer": "Hooks and the statusline run again for each subagent.",
+      "why": "reverse"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Ten subagents three deep is a lot of processes.",
+      "why": "no fan-out"
+    }
+  ]
+}

plugins/harness-ops/evals/ap-fanout-version-and-kthreads/samples/version-fixes-slowdowns.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Yes. Several 2.1.2xx releases fixed long-session slowdowns and keystroke lag.",
+      "why": "releases fixed"
+    },
+    {
+      "answer": "Fixes for CPU regressions landed in later releases.",
+      "why": "reverse"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "The version is unlikely to matter.",
+      "why": "wrong"
+    }
+  ]
+}

plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/case.yaml

@@ -0,0 +1,18 @@
+schema_version: "1.1"
+name: asv-absent-key-subagent-use-counts
+graders:
+  - name: absent-key-not-loaded
+    type: regex
+    pattern: 'defaultEnabled[^\n]{0,200}\b(?:not|never|ignored|no effect|doesn[’'']?t|does not)\b|\b(?:not|never|doesn[’'']?t|does not)\b[^\n]{0,80}defaultEnabled|\b(?:will be|is|stays|shows as|reads|reported as|counts as|treated as) (?:disabled|hidden|off)\b'
+    flags: 'i'
+    target: last_message
+  - name: subagent-own-window
+    type: regex
+    pattern: 'sub-?agent[^\n]{0,250}\b(?:own (?:context )?window|smaller (?:context )?window|shed\w*|(?:truncated?|truncates|truncation|truncating)|cut short|drop\w*|not (?:covered|guaranteed)|doesn[’'']?t (?:cover|carry|apply)|does not (?:cover|carry|apply))|\bmain session\b[^\n]{0,120}\bonly\b|\bonly\b[^\n]{0,60}\bmain session\b'
+    flags: 'i'
+    target: last_message
+  - name: use-count-not-comparable
+    type: regex
+    pattern: '\bhooks?\b[^\n]{0,250}\b(?:count\w*|dispatch\w*|inflat\w*|dominat\w*|fires?|fired|every)\b|\bnot (?:directly )?comparable\b|\bincomparable\b'
+    flags: 'i'
+    target: last_message

plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/prompt.md

@@ -0,0 +1,14 @@
+---
+description: "Three questions audit-skill-visibility answers from its text: an installed plugin with `defaultEnabled: true` and no `enabledPlugins` key (it does not load; the runtime reading beats the docs), whether a fitting listing holds in subagents (a subagent's listing is sized to its own window and can shed), and whether a raw plugin use count ranks usage (hooks and other dispatches count, so hook-heavy plugins dominate). Reaches SKILL.md (verification records, subagent sentence) and reference/usage-counters.md. Source: the consumer's own writing. hard: the docs say defaultEnabled decides an absent key. Read-only tools. Held-rows rows: harness-ops 34, 38, 45."
+tags: [harness-ops, audit-skill-visibility, reference-answer, hard]
+max_turns: 20
+timeout_seconds: 400
+allowed_tools: [Skill, Read, Glob, Grep]
+expected_outcome: "The answer says notes-kit's skills do not load (defaultEnabled does not decide an absent key at runtime), that the fit verdict covers the main session and a subagent's listing is sized to its own window and can shed, and that plugin use counts include hook and other dispatches so they are not comparable across plugin shapes."
+---
+
+/harness-ops:audit-skill-visibility no need to run the full audit, I just have three questions:
+
+1. Our `notes-kit` plugin is installed, its marketplace entry sets `defaultEnabled: true`, and no settings file has an `enabledPlugins` entry for it. Will its skills show up?
+2. Your last run said my skill listing fits. Does that hold for subagents too?
+3. `~/.claude.json` shows `guardrails` with about 200,000 uses and `notes-kit` with 150. Is guardrails really the plugin I use most?

plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/absent-key-not-loaded.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "No. `defaultEnabled` does not decide an absent key at runtime, so notes-kit is disabled.",
+      "why": "direct"
+    },
+    {
+      "answer": "Its skills won't load: the plugin is treated as disabled with no enabledPlugins entry.",
+      "why": "treated as disabled"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Yes, defaultEnabled: true turns it on.",
+      "why": "wrong"
+    }
+  ]
+}

plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/subagent-own-window.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Not necessarily. A subagent gets a listing sized to its own window and can shed skills.",
+      "why": "own window"
+    },
+    {
+      "answer": "The verdict covers only the main session.",
+      "why": "only main"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Yes, subagents see the same listing.",
+      "why": "wrong"
+    }
+  ]
+}

plugins/harness-ops/evals/asv-absent-key-subagent-use-counts/samples/use-count-not-comparable.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "No. Every hook dispatch counts as a use, so hook-only plugins dominate.",
+      "why": "hooks count"
+    },
+    {
+      "answer": "Those numbers are not comparable across plugin shapes.",
+      "why": "not comparable"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Yes, guardrails is your most used plugin.",
+      "why": "wrong"
+    }
+  ]
+}

plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/case.yaml

@@ -0,0 +1,13 @@
+schema_version: "1.1"
+name: cl-drift-dedupe-sees-closed
+graders:
+  - name: dedupe-includes-closed
+    type: regex
+    pattern: '--state all\b|\ball states\b|(?:^|[.!?;](?=\s|$))(?!(?:(?![.!?;](?:\s|$))[^\n])*?(?:\b(?:not|no|never|cannot|misses|missed)\b|n[’'']t\b|\bonly (?:open|lists? open|sees? open)))(?:(?![.!?;](?:\s|$))[^\n])*?(?:\b(?:includes?|including|included|covers?|sees?|finds?|searches|lists?)\b(?:(?![.!?;](?:\s|$))[^\n]){0,40}?\bclosed\b|\bclosed\b(?:(?![.!?;](?:\s|$))[^\n]){0,40}?\b(?:too|as well|included)\b)'
+    flags: 'im'
+    target: last_message
+  - name: raises-limit
+    type: regex
+    pattern: '--limit 1000\b|\b1,?000\b|\blimit\b[^.\n]{0,40}\braised\b|\b30\b[^.\n]{0,60}\bdefault|\bdefault\b[^.\n]{0,60}\b30\b'
+    flags: 'i'
+    target: last_message

plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "One question about the changelog skill's native-drift step: its duplicate check lists issues in every state with a raised limit (`gh issue list --state all --limit 1000`), so a closed match is found even with 200 open issues. Reaches context/native-drift.md (dedupe step and its recorded fact). Source: the consumer's own writing. routine. Needs --allow-tools Bash for the pre-computed line. Held-rows rows: harness-ops 50."
+tags: [harness-ops, changelog, reference-answer, routine]
+max_turns: 15
+timeout_seconds: 300
+allowed_tools: [Skill, Read, Glob, Grep, Bash]
+expected_outcome: "No apply runs. The answer says the check lists issues in all states (closed included) with a 1000 limit, above the 30 default, so the closed issue is found."
+---
+
+/harness-ops:changelog quick question about the native-drift step in apply: before it files an item it checks for an existing one. We closed a matching issue last month and have about 200 open issues. Will that check find the closed one? Answer only, don't run apply.

plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/samples/dedupe-includes-closed.json

@@ -0,0 +1,34 @@
+{
+  "pass": [
+    {
+      "answer": "Yes, it runs `gh issue list --state all --limit 1000`, so the closed issue is found and no duplicate is filed.",
+      "why": "flag with later no"
+    },
+    {
+      "answer": "Yes: per `context/native-drift.md`, the check runs `gh issue list --state all --limit 1000`.",
+      "why": "filename dot"
+    },
+    {
+      "answer": "Yes. It runs `gh issue list --state all`, so closed issues are included.",
+      "why": "flag"
+    },
+    {
+      "answer": "Yes, the search covers closed issues as well.",
+      "why": "prose"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "No, it only sees open issues.",
+      "why": "wrong"
+    },
+    {
+      "answer": "No, it won't find the closed one.",
+      "why": "negated find"
+    },
+    {
+      "answer": "It lists only open issues, so the closed one is missed.",
+      "why": "only open"
+    }
+  ]
+}

plugins/harness-ops/evals/cl-drift-dedupe-sees-closed/samples/raises-limit.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "It passes `--limit 1000`, so 200 open issues do not crowd it out.",
+      "why": "limit"
+    },
+    {
+      "answer": "The default of 30 would miss it, which is why the limit is raised.",
+      "why": "default 30"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "Yes, it searches closed issues.",
+      "why": "no limit"
+    }
+  ]
+}