Repository navigation
docs: point setup skills' shared reconfiguration and placeholder text at their sources - #15381
Conversation
… at their sources Thirty setup skills restated the plugin-reconfiguration routes (the `claude plugin configure ... --values-stdin` command, its 2.1.285 floor, the install --config fallback, scope details and each route's failure signal) and gave "fixed at session start" as the reason to recheck in a fresh session. The live plugin CLI reference does not state that timing. Each site now keeps the decisions (native configure flow, never uninstall, never write pluginConfigs, fresh-session recheck reporting only an observed value) and cites the plugin-reconfiguration contract's "The routes" and "Caveats" sections by heading, with a pointer record to the CLI reference's plugin configure section. The contract's "The short form setups print" section now asks for that shape, and § The routes gains the PowerShell pipe form two skills used to carry. Seventeen setup skills that fall back to a body Bash call when a pre-computed row reads `[shell command execution disabled by policy]` keep the fallback and gain a pointer record to the skills docs' "Inject dynamic context" section, which documents that placeholder; typos-format's older Claim/Basis record is replaced by it. Eval cases that graded the removed wording now grade the decisions: the native configure route, the option key and value, a pointer at the convention, no uninstall, and the fresh-session recheck. Graders for the fallback order, "already installed", scope and failure signals are removed with their samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
The setup SKILL.md changed in the previous commit, and the changed-skills gate requires a suite case that tags a changed skill. ai-briefing's setup skill had none. The case runs check in a repository with no profile and grades that it reports the missing sources.md as a FAIL routed to apply and writes or edits no file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…setup check step 5 Both setup checks gave "injected when this skill loads" as the reason a changed option shows only in a fresh session. Each keeps the fresh-session and observed-value instruction and now carries a pointer record to the plugins manifest reference's "Reference a saved value" section, read live 2026-11-11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
claude-review reviewed Chunk 1 of 4 (37 files)claude-review, chunk 1 of 4 (blocking only). This lane already reviewed earlier heads (
I also checked the copies in
Each contract copy sits under its own plugin's BlockingNone in this chunk. Earlier threads, re-checked on this head
Checked, no blocking issue
Not verifiedI ran nothing. The validator results and real-model pass tables in the PR description come from the author and are unverified here. Spend (not a hold): none in this chunk. The PR description reports earlier prepaid eval runs. Chunk 2 of 4 (38 files)claude-review, chunk 2 of 4: all 38 listed files read, blocking findings only. The prior-findings file says
The two whole-file deletions in this chunk ( BlockingNone in this chunk. Earlier findings, re-checked on this head
Checked, no blocking issue
Not verifiedI ran nothing. The validator results and real-model pass tables in the PR description come from the author and are unverified here. Spend (not a hold): nothing in this chunk spends credits. The PR description reports prepaid eval runs already spent (about 12 USD for the first two passes, plus passes 3 and 4). Chunk 3 of 4 (35 files)claude-review, chunk 3 of 4 (all 35 files read). Blocking only: this lane already reviewed earlier heads, so this pass reports only blocking findings. I read every hunk of the 35 files in this chunk:
The 5 sample files this chunk deletes whole are in the diff but on no file list, so they are not counted. In BlockingNone in this chunk. Earlier findings, re-checked on this headNone of the earlier threads anchor on a file in this chunk. The headless-only
Checked, no blocking issue
Not verifiedI ran nothing. The validator results and real-model pass tables in the PR description are author-claimed and unverified here. Closing issue: none (the PR states it has no tracking issue). Spend (not a hold): none in this chunk. The PR description reports prepaid eval-run spend already incurred. Chunk 4 of 4 (34 files)claude-review, chunk 4 of 4 (34 files, all read). Blocking only: this lane has already reviewed 3 earlier heads, so this pass reports only blocking findings. I read every hunk of the 34 listed files in this chunk:
BlockingNone in this chunk. Earlier findings, re-checked on this head
Checked, no blocking issue
Not verifiedI had no shell, so I ran nothing. The PR description's Spend (not a hold): none in this chunk. The PR description reports prepaid eval runs already spent. |
|
claude-security-review reviewed Chunk 1 of 4 (37 files)Security review (chunk 1 of 4): no security issues foundcritical-findings: 0 This lane already reviewed an earlier head, so this pass was blocking only: it reports CRITICAL and IMPORTANT findings only. I read every hunk of all 37 files in What this chunk changes
Scope checks
Settled threads and prior findings. I raised no settled thread again. The only prior entry is this lane's earlier summary comment, which reported 0 findings, so nothing carries over. Spend. This chunk spends no API credit and starts nothing that bills. Chunk 2 of 4 (38 files)Security review (chunk 2 of 4): no security issues foundcritical-findings: 0 This lane already reviewed an earlier head, so this review is blocking only: it reports only CRITICAL and IMPORTANT findings. I read every hunk of all 38 files in this chunk's patch. The two deleted What this chunk changes.
Instruction-surface lens. I checked the rules this chunk takes out of the skill bodies against the register (
Other checks.
Settled threads and prior findings. I raised no settled thread again. The only prior finding is this lane's earlier summary, which reported 0 findings, so nothing carries over. Spend. This change spends no API credit and starts nothing that bills. The PR description reports about 12 USD of eval runs, all done before this review. Chunk 3 of 4 (35 files)Security review (chunk 3 of 4): no security issues foundcritical-findings: 0 This review was blocking only: this lane already reviewed an earlier head, so it reports only CRITICAL and IMPORTANT findings. I read every hunk of all 35 listed files in What this chunk changes
Instruction-surface lens
I also checked miro's handling of the sensitive token. The skill still tells the agent to:
The security note below it, which warns that a CLI argument lands in shell history and the process table, is unchanged. The plugin-local contract copy gives Scope checks
Settled threads and prior findings
Spend. This chunk starts nothing that bills. It only changes how eval runs are graded, and those runs spend prepaid API credit. Chunk 4 of 4 (34 files)Security review (chunk 4 of 4): no security issues foundcritical-findings: 0 This lane already reviewed an earlier head ( What this chunk changes
Scope checks
Settled threads and prior findings. No settled thread is raised again. The rate-limit-guard and repo-hygiene threads marked Embedded text. The PR context contains no directive aimed at this lane. Spend. This chunk spends no API credit in CI and starts nothing that bills. The eval runs the description reports, about 12 USD list-price estimate, were prepaid model usage outside this change. |
The first commit removed graders for the install --config fallback, the already-installed rerun, the scope rule, the failure signals, per-key --config, the 2.1.285 floor and skill-quality's fallback and list probes. Each behavior still lives in docs/contracts/plugin-reconfiguration/README.md, which every changed setup skill points the agent at, so each grader comes back re-pointed at that behavior rather than the old skill wording: the fallback regex no longer requires the --values-stdin flag text, and the scope and rerun criteria follow the contract's caveats. mutation-testing's names-values-stdin returns as names-headless-configure, since names-plugin-configure matches only the interactive route. The uninstall reason stays ungraded: neither the skill nor the contract states it. Real-model runs showed three wrong graders, fixed here: no-plugin-cli-write in actionlint, eol-normalizer and ruff-format lacked the --help exclusion the other copies carry; cites-convention missed "reconfiguration convention" and "reconfiguration-convention"; context-budget's fresh-session criterion asked the reply to report a value only the user can observe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
…' into docs/setup-skills-shared-records
…es it Real-model eval runs showed the setup skills' pointer to the published contract URL fails offline: the sandbox blocked raw.githubusercontent.com and code.claude.com in every run, so the agent could not give the routes. The contract is now a registered shared copy at plugins/<p>/reference/plugin-reconfiguration.md for the 31 plugins whose skills cite it, and those skills link the copy by relative path. The contract's one relative link (to the retired-conventions convention) is now the published URL, since a copy cannot resolve it, and its citation section describes the local copy. ai-briefing's setup check now says it writes nothing even when the user asks it to create a missing file; it reports the FAIL and routes to apply. The eval run caught it trying to Write sources.md during check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
…alog Review on #15381: in the 13 cases whose prompt rules out the interactive configure dialog (a CI box, or "I cannot use the interactive configure prompt"), configure-route accepted /plugin configure, so a reply offering only the route the user cannot use passed. It now accepts only `claude plugin configure <plugin>@...`, with the interactive form as a fail sample. The cites-convention grader this PR added is dropped: the skills never tell the reply to name the convention, and with the contract now shipped in each plugin the agent answers from it directly, so the restored route graders grade the substance. The repo-hygiene fresh-session criterion and the skill-quality routes criterion no longer ask for more than the skill instructs. rate-limit-guard's setup check drops its unpointed claims about when the module and the StopFailure hook read a changed value, keeping the fresh-session recheck and the observed-value rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
… cluster The pre-push cross-plugin-source-drift check failed on the 31 new reference/plugin-reconfiguration.md copies: it requires every byte-identical cluster to be registered. They are registered with sync-shared-copies.sh as their dedicated check, as the standards and pr-scope-budget contract copies are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
Review on #15381. already-installed-still-writes (context-budget, repo-hygiene) failed a reply that copies the contract's sentence, "it saves the value on an already-installed plugin", and lost the base version's case -insensitive flag; it now accepts that order and is case-insensitive again, with the contract sentence as a pass sample and a reply that gives the fallback without the behavior as a fail sample. no-plugin-command-run in the same two cases counted a read-only `claude plugin configure --help` as a write; it gets the --help guard the other no-plugin-* copies carry, with a --help probe as a pass sample. No other no-plugin-* grader in the touched cases lacked it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
… names
A base-vs-head comparison (3 runs per version per case, same graders on
both) showed head worse than origin/main on graders for content this PR moved
into the contract: context-budget and repo-hygiene stopped saying the
fallback rerun saves the value on an already-installed plugin, and
skill-quality stopped giving the install fallback and the scope to pass. The
skills only named the contract's sections. Each setup skill now tells the
agent to read the routes and caveats sections and print what they give (the
fallback, its behavior on an installed plugin, the scope), leaving out the
verified-version record, which one repo-hygiene run had quoted. The
contract's short-form section says the same.
already-installed-still-writes also accepts a correct reply worded without
"still" ("prints already installed and only saves the value"), with
negated replies kept as fail samples.
After the fix head is at or above base on every grader in all five cases;
noise-report reads non-inferior at margin 0.05 (+0.04, interval -0.02 to
+0.09).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
kyle-sexton
removed
the
do-not-merge
github-actions
Bot
added
the
merge-path: owner-agent
No related issue: a cross-plugin item the stage-3 skill re-audit withheld for one fix; it has no tracking issue.
Summary
This covers stage-3 re-audit cross-plugin items 3 and 4, both about text repeated across the plugins'
setupskills.claude plugin configure ... --values-stdin, the 2.1.285 floor, theinstall --configfallback, the scope details and each route's failure signal. Each also gave "fixed at session start" as the reason to recheck. The plugin CLI reference § plugin configure and the manifest reference § Reference a saved value, both read live on 2026-11-11, do not state that timing, so the claim is dropped rather than pointed at.[shell command execution disabled by policy]and give it no source. No shared reference in the repository documents the string, so each site gets its own pointer record.Fix
Reconfiguration paragraph (30 setup skills). Each site keeps the decisions:
/plugin configure <plugin>@<marketplace>or a headless route);pluginConfigs;Each site now cites § The routes and § Caveats every setup skill's short form carries by heading, plus a pointer record (as of 2026-11-11) to plugin CLI reference § plugin configure. Plugin-specific rules stay: which key to set, miro's token handling, source-control's
multiplekeys, machine-health's Windows path form.Contract (
docs/contracts/plugin-reconfiguration/README.md).Placeholder (17 setup skills). Each site keeps the body-Bash fallback and adds a pointer record to skills § Inject dynamic context, which documents the setting and the placeholder, with as-of 2026-11-11 and recheck trigger "the placeholder text changes". The older Claim/Basis record in typos-format is replaced.
Evals. Cases that graded the removed wording now grade the decisions:
cites-convention;no-uninstall, without the restated reason;The graders for the fallback order, "already installed", scope, failure signals, per-key
--configand the 2.1.285 floor were first removed. They are restored, re-pointed at the contract that still holds those behaviors (table under Verification). Only the uninstall reason stays ungraded. Cases changed in actionlint, bash-format, biome-format, context-budget, desktop-notification, education (samples only), eol-normalizer, go-format, guardrails, knowledge (legacyevals.json), machine-health, markdown-format, mutation-testing, powershell-format, repo-hygiene, ruff-format, skill-quality and typos-format.Check step 5 in repo-hygiene and rate-limit-guard. Both gave "injected when this skill loads" as the reason a changed option shows only in a fresh session. They keep the fresh-session and observed-value instruction, and each now carries a pointer record (as of 2026-11-11) to plugins manifest reference § Reference a saved value, whose anchor I verified live. No grader cited that wording. rate-limit-guard also drops its unpointed claims about when the module and the StopFailure hook read a changed value (review thread).
Contract shipped inside each plugin. The first real-model runs showed that the published URL is unreachable offline.
docs/contracts/plugin-reconfiguration/README.mdis now a registered shared copy (scripts/shared-copies.txt, plus a cluster entry inscripts/cross-plugin-source-registry.txt) atplugins/<p>/reference/plugin-reconfiguration.mdfor the 31 plugins whose skills cite it. Those skills link the copy by relative path, for example../../reference/plugin-reconfiguration.md#the-routes. The contract's one relative link, to the retired-conventions convention, now uses the published URL, since a copy cannot resolve it. Its citation section describes the local copy. The CLI-reference pointer records are unchanged.Print the routes, not only the section names. The base-vs-head comparison below showed head regressing on content this PR moved into the contract. Each setup skill now tells the agent to read the routes and caveats sections and print what they give (the fallback route, its behavior on an already-installed plugin, the scope to pass), leaving out the verified-version record. The contract's short-form section says the same.
Review-thread grader fixes.
already-installed-still-writes(context-budget, repo-hygiene) is case-insensitive again, and it accepts the contract's own order and a correct reply worded without "still". Replies that negate the behavior are kept as fail samples.no-plugin-command-runin the same two cases has the--helpguard.ai-briefing
check. It now states that it writes nothing even when the user asks it to create a missing file: it reports the FAIL and routes toapply. The eval run had caught it attempting to Writesources.md.Headless-only route grading (review thread). In the 13 cases whose prompt rules out the interactive dialog,
configure-routeaccepts onlyclaude plugin configure <plugin>@…, and the interactive form is a fail sample. Thecites-conventiongrader this PR added is dropped: the skills never tell the reply to name the convention, and the restored route graders now grade the substance.New ai-briefing eval case.
setup-check-missing-sources-read-onlyis new, because the changed-skills gate requires a suite case tagging a changed skill and ai-briefing's setup skill had none. It grades thatcheckreports the missingsources.mdas a FAIL routed toapplyand writes nothing.Changelog. One fragment for each of the 35 plugins touched.
Verification
check-affectedhook passes, including changed-skills on all 35 setup skills.validate-cases.pyon all 19 touched eval dirs: exit 0, 0 FAIL lines. The WARN lines are the usualallowed_toolsand judge-calibration ones.check-matchers.pyon the same dirs: ok.markdownlint-cli2on the 89 changed.mdfiles: 0 issues..jsonparses.scripts/check-changelog-fragments.sh --check: valid.--check-required origin/main: all 35 plugins have a fragment.git grepforvalues-stdin,fixed at session startandalready installedacrossplugins/*/skills/setup/SKILL.md: no hits.Removed graders: where each behavior lives now
Rule applied: a grader may stay removed only if neither the skill body nor the contract it points at holds the behavior any longer.
fallback-after-configureclaude plugin install <p>@<marketplace> -s <scope> --config KEY=VALUE", after the preferredconfigure --values-stdinclaude plugin configure <p>@route, then an install--config <key>=<value>rerun; the--values-stdinflag text is no longer requiredrerun-still-writes(llm) /already-installed-still-writes(regex)already installedand still writes the value"scope-rule/scope-default-userfallback-after-configurecases except eol-normalizer; scope-default-user: eol-normalizerconfigurehas no scope flag; pass the scopeclaude plugin listreports;-sdefaults touser-sdefaults to user" is dropped because the contract does not state itfailure-signals--values-stdinexits 1 on a bad value; the--configrerun exits 0 with a warningrepeatable-per-key--configrepeats per key--configrepeatable per key")configure-minimum-releaseinstall-config-fallback,scope-from-plugin-list--config skills_root=fallback;claude plugin listno-uninstall-with-reason,uninstall-reasonno-uninstall-with-reason); skill-quality (asuninstall-reason)pluginConfigsentry and resets every option to its defaultno-uninstallnames-values-stdin--values-stdinnames-headless-configure(claude plugin configure mutation-testing@). It is not a duplicate: the remainingnames-plugin-configurematches only the interactive/plugin configureNot removed but loosened by the same rule, and left as is, open below: the
values-*graders now accept key and value rather than the JSON string form, andconfigure-routeaccepts the interactive route. The contract still gives both the string form and the headless route.Real-model runs
Each run used the with-plugin arm only (
--ablation none),--runs 1, the default model (Opus 5.5),--judge-model opusand--no-publish. Each went throughdocker-links.py run, whose preconditions held:DOCKER_CONFIGunset, both~/.dockerlinks under/mnt/, and notls/directory. The run set was 19 cases: every case with a changed, added or restored llm grader, plus the new ai-briefing case. The llm graders were not calibrated first, so treat them as uncalibrated.First pass: a defect this PR introduced, fixed since by the plugin-local contract copy (see pass 3). The setup skills now send the agent to the contract and the CLI reference for the headless routes. In every run that asked about them, the sandbox blocked
raw.githubusercontent.comandcode.claude.com. The agent then:curl -o, which tripsno-install;claude plugin configure --help, which tripsno-plugin-command-run.As a result the restored graders fail, along with most
configure-routeandvalues-*checks. The validity gate reports INVALID for 7 first-pass runs because of with-arm network or permission denials, which means the fact the agent needs is not reachable from the plugin.Footnotes:
no-plugin-cli-writematched a read-onlyclaude plugin configure --helpin the three cases whose copy lacked the--helpexclusion the other copies carry. It passed on re-run in all three.cites-conventionmissed "reconfiguration convention" and "reconfiguration-convention". The regex is deterministic, so I re-graded the recorded replies offline: they now match.check, the agent tried to Writesources.mdbecause the user asked; the permission mode refused it. The skill sayscheckcreates nothing.Every other failure in the table comes from the unreachable-contract defect above. A one-run single-arm pass has too few runs (n) to show a gain, so this section reports no scores, only per-grader outcomes. Cost: 8.61 USD (first pass) + 3.36 USD (re-run) = 11.98 USD list-price estimate. The estimate was about 3.8 + 3.1 USD at the 0.2 USD-per-run anchor; the opus judge pushed the average to about 0.45 USD per run. All 26 kept sandboxes were removed by literal path.
Pass 3: with the plugin-local contract copy
Pass 3 re-ran every case that failed or was INVALID in the first pass (16 cases), the same way. Pass 4 then re-ran the 2 cases whose llm criteria were fixed after pass 3. The table shows failing graders as recorded. A
configure-routere-grade of every pass-3 reply against the headless-only regex passes all 14 cases that have one (deterministic, offline).checkfix holdscurl -oon the CLI reference that the kept "fetch live" pointer names. That is the pointer's instruction meeting no network; the grader is rightValidity gate: pass 3 has 15 of 16 cases VALID, with context-budget INVALID as above; pass 4 has 2 of 2 VALID. Real misses from the first pass:
Cost: pass 3 7.03 USD (estimate about 7.0 USD from these cases' first-pass costs) and pass 4 0.81 USD. The total across all four passes is 19.82 USD, list-price estimate. All 18 pass-3 and pass-4 sandboxes and the throwaway plugin copies were removed.
Base vs head, non-inferiority (owner rule D10)
The comparison covers 5 cases: every case where a restored, route or variance-called grader failed in passes 3 and 4. Each version ran 3 times per case, with-arm only, on Opus 5.5 with
--judge-model opus, throughdocker-links.py. Base is origin/main's plugin tree (git archive), and both trees carry head's case files, so both versions are scored by the same graders.The first comparison found head worse beyond noise on three graders:
already-installed-still-writesalready-installed-still-writesinstall-config-fallbackscope-from-plugin-listThe skills named the contract's sections but did not ask for their content, so I fixed the skill text (see Fix) and re-ran those cases on head.
curl -oon the CLI reference the kept "fetch live" pointer namesThe noise report (
--baseline) reads: with-arm change +0.04, 95% interval -0.02 to +0.09. Compare verdict: non-inferior at margin 0.05.skills/audit/SKILL.mdwithsed(blocked by a deny rule) and call Glob, which the case does not allow. Head also attempted fetches of the CLI reference. Read context-budget's numbers as indicative.already-installed-still-writeswidening (a correct reply worded without "still") was re-graded offline on the recorded replies, since it is deterministic. Base stays 3/3 and head is 3/3.Related
Follow-ups:
🤖 Generated with Claude Code
https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm