Repository navigation
docs(harness-ops): ship eval-gated held rows and three stale-fact fixes - #15469
kyle-sexton wants to merge 8 commits into
Conversation
Pilot of the held stage-3 rows on harness-ops (owner decision D24). Nine eval cases cover 23 of the 24 held rows; a detection control (each row's text removed) showed which ones a case can see, and base-vs-head runs on Opus 5.5 and Fable 5.2 gated each detectable row. Shipped (non-inferior on both models): - 113: the shallow-clone reason points at the host-marketplace section that describes the re-clone instead of restating it beside a bare link. - 74: the cloud-session note points at agent view's "List sessions as JSON"; the schtasks limit points at the /tr row the config cites. - 45: usage-counters names the binary reading the plugin-use definition rests on and drops an unsourced polling rate and public-report remark. The new cases stay as coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e66e9efd7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- observability: cli-flag-verify runs at push time in the guardrails reference-verify git hook (entries.json:43-52), not as a PostToolUse hook. - plugins: claude plugin list prints one row per install record on 2.1.322, recorded with version and date; the never-trust-list rule stays. - plugins: since 2.1.268 the /plugin panel runs the reload itself on close; the two-step keeps bare-first, --force only on the cache warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
kyle-sexton
changed the title
|
claude-security-review reviewed Security review (incremental, 944fead → e3ee1b4)I found no security issues in this push. I read every hunk of all 8 files on the incremental list. What changed
Checks
Mode: this review was blocking only ( Spend (not a hold): the eval cases use prepaid API credit or subscription usage when they run. Nothing here can start a real charge. critical-findings: 0 |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
claude-review reviewed Code review: incremental, 944fead → e3ee1b4 (blocking only)This lane already reviewed an earlier head, so this pass reports blocking findings only and drops advisory ones. I read every hunk of the 8 files in the incremental diff. I also read the head What the push fixed. All five earlier findings are addressed, and their threads are settled by the owner's replies at this head:
The changelog fragment now matches the reworded Findings (2, both inline). Both come from the new sentence-anchored form of the graders, and both are new problems on the rewritten lines. Neither repeats a settled point.
Unverified. I ran nothing; I traced both regexes by hand. The Spend (not a finding). The regraded eval cases spend prepaid API credit or subscription usage whenever they run. No plugin manifest changed in this push, so there are no manifest lines. |
- ob-cost-and-flag-verify: the not_contains grader now judges a sentence and skips one that negates the claim, so the case's own expected outcome passes; that wording is a pass sample. - cl-drift-dedupe-sees-closed: dedupe-includes-closed fails a negated or "only open" reply; both added as fail samples. - Case descriptions no longer cite pre-fix text (data-sources 4.5, "highest across scopes"). - scope-semantics: the /plugin summary paragraph returns to the links-only form; the two-step keeps checking what the panel did before a bare reload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
- ob-cost-and-flag-verify: no-posttooluse-claim checks negation only in the words just before PostToolUse, and ends a clause at ; or a terminal . so a filename dot or a later "not" no longer hides a stale claim. Three such answers are fail samples. - cl-drift-dedupe-sees-closed: --state all and "all states" match anywhere again; the negation check applies only to the verb-plus-closed form. Two correct answers that tripped the sentence check are pass samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm
kyle-sexton
removed
the
do-not-merge
|
Hold released: #15468 merged, so the merge-order hold (pr-hold reason=merge-order) ends. Updating from main next. |
No related issue: the held-rows eval pilot for harness-ops (owner decision D24); it has no tracking issue.
Summary
The stage-3 skill re-audit held 24 harness-ops rows (1 cut, 23 rewrites) because no eval suite could see them. This pilot built nine cases for them, ran a detection control (each row's text removed) to find which rows a case can see, and gated each detectable row on base vs head, three runs per arm, with-plugin arm only, on Opus 5.5 and Fable 5.2 (judge opus, deterministic graders only). Three rows ship. The other 21 stay held, with their numbers below.
Fix
plugins): the shallow-clone reason for comparing only installs at the clone's HEAD now points at What background auto-update does with credentials. Before, it restated the re-clone beside an unanchored link.lanes): the cloud-session note points at List sessions as JSON, and the schtasks limit points at the/trrow that the config already cites.audit-skill-visibility):usage-counters.mdnames the binary reading behind the plugin-use definition, and drops an unsourced 1 Hz polling rate and an unlinked "public report".observability):data-sources.md4.5 andoutput-format.mdno longer describecli-flag-verifyas a PostToolUse hook. It runs at push time in guardrails'reference-verifygit hook:plugins/guardrails/git-hooks/entries.json:43-52lists it as apre-pushentry carryinghooks/cli-flag-verify.sh, andhooks.jsonregisters nocli-flag-verifyhook. The section now reports "not collected" instead of reading an empty count as clean.plugins): on Claude Code 2.1.322,claude plugin listprints one row per install record, each with its own scope and version (fresh run plus--help). It no longer prints the highest version across scopes. The record keeps its version, date and trigger, and the rule (never readlistordetailsas what loads here) stays.plugins): changelog 2.1.268, fetched live, says closing/pluginapplies changes with no/reload-pluginsafterwards. Install a plugin says the panel runs the reload itself afterRun /reload-plugins to apply.. The two-step keeps bare-first and--forceonly on the cache warning. That pointer block drops its as-of/trigger under the pointer rule (contract 6.0.0, docs: a stored value is a copied or measured number or string #15468).plugins/harness-ops/evals/stay as coverage.Verification
Detection uses the base and control trees on Opus 5.5. A row counts as detectable when its grader dropped by at least 2 of 3 runs. Ship needs the documented compare verdict, non-inferior at margin 0.05, on both models. All runs used
docker-links.py run --(the link swap: DOCKER_CONFIG was unset, both links point under /mnt/, and there is notls/),--ablation none, andTMPDIR=$HOME/.ev.The compare runs on the three shipped cases were VALID on both models: Opus non-inferior (lower bound -0.04), Fable non-inferior (+0.00). Spot-reads found three graders that failed correct answers (the cloud wording, the HEAD-compare wording, and "not what loads"). I fixed those graders and reran the compares. An OAuth token revocation during the rerun errored the two
pl-cases on three arms, so those were rerun and merged per case.Other checks, rerun after the fix commit:
validate-cases.pyexit 0; markdownlint 0 issues;check-skill.shon harness-ops 18 passed, 0 failed;check-changelog-fragments.sh --checkvalid; typos clean. Row 110 was held as a rewrite; it is now fixed through the fix route. Eval spend is a list-price estimate: $32.73 on Opus, $30.40 on Fable.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_01VcupojQwC9uJAfmSt4EHZm