Skip to content

fix(review): make the code-review second pass unconditional - #937

Open
tend-agent wants to merge 2 commits into
mainfrom
hourly/review-31365030481
Open

fix(review): make the code-review second pass unconditional#937
tend-agent wants to merge 2 commits into
mainfrom
hourly/review-31365030481

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

Since #819 made /tend-ci-runner:code-review reachable again, every tend-review session on numbagg/numbagg has still skipped the mandated second pass — 4 of 4 reviews that reached step 4, zero counter-examples. The reviews still get posted and still read as complete, so the missing pass is invisible from the outside; only the session logs show it.

The reachability bug (#817) is fixed — the skill is in the available-skills listing in every one of these sessions, and no session got a tool-use error. The agent simply never reaches for it, and in all four sessions never even mentions it in its reasoning.

Why the text permits the skip

Step 4 opens with Scale depth to the change… Don't over-analyze trivial changes. Three lines later the second-pass mandate says Scale its depth to how core the change is, **the same way you scale the manual depth above**. That back-reference ties the pass to a scale whose bottom rung is explicitly "skim, not the full checklist" — so for a lockfile-only dependency bump, "scale it to nothing" is a defensible reading of the instruction as written. The pass is never stated to be unconditional.

This change says it is: the depth-scaling sets how deep the pass goes, never whether it happens. The peripheral bullet already names dependency bumps and gives them the short angle set in one pass, so the always-run reading is what the rest of the paragraph already assumes.

Evidence

Run 31365030481. Full log: https://gist.github.com/19b5ab297bb7ac7e1e9a44d595ccde0f

tend-ci-runner:code-review first appears in a numbagg session's available-skills listing between 2026-08-06 and 2026-08-08 (absent in 31113573735, present in 31246101561). Sessions before that are excluded — the skill was genuinely unreachable, and one of them (31088428562) says so in as many words: "I skipped the /code-review pass the workflow calls for. It isn't in this environment's available-skills list." That agent checked, found it missing, and named the deviation — which is the behavior we want, and the reason the four sessions since are worth acting on.

Review session PR Date code-review reachable Second pass run
31246101561 #727 2026-08-08 yes no
31258607744 #726 2026-08-08 yes no
31360815248 #729 2026-08-10 yes no
31361090036 #728 2026-08-10 yes no

Each row is a session that read the diff (gh pr diff) and so reached step 4; none contains a Skill call for tend-ci-runner:code-review, and none mentions it in assistant reasoning.

Gate assessment

  • Evidence level: High — consistent across 4 sessions spanning 3 days and 4 distinct PRs, with zero counter-examples since the skill became reachable. Bar for High is 2–3.
  • Structural vs stochastic: structural contributor in the skill text. The omission is a model behavior, but the text itself licenses it: the mandate delegates its own applicability to a scale that bottoms out at "skim". The same trivial-diff conditions read against the same paragraph produce the same skip, which is what 4/4 shows.
  • Change type: targeted fix — one sentence, in place, no new section. Normal Gate 1 thresholds apply.
  • Passes both gates: yes.

Honest limit: the evidence is from one adopter repo, and I can't prove wording alone fixes what is partly a salience problem. If a later cycle finds sessions still skipping the pass with this text in place, the next step is a harder mechanism (a distinct numbered step, or a step-5 pre-post check that the pass ran) rather than more wording.

@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 10, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The premise holds up: the removed back-reference did tie the pass's applicability to a scale whose bottom rung is "skim, not the full checklist", and the two bullets below already carry the depth-scaling on their own, so the deleted clause was load-bearing only for the reading you're trying to close. The trivial-skip path in step 1 (Do NOT proceed to steps 2–6) never reaches step 4, so "every review that reaches this step" is the right scope rather than an over-claim, and draft mode still lands here.

One suggestion on placement. The mandate is now sentence three, behind a sentence describing what the skill is — and CLAUDE.md's skill-authoring rule is Lead with the rule or recipe; cut motivation. Since the failure you're fixing is partly one of salience (four sessions never mentioned the skill at all), a rule sitting mid-paragraph after two sentences of description is the shape most likely to get skimmed past. Moving it to the front costs no words and puts the imperative where a scanning reader hits it first.

Comment thread plugins/tend-ci-runner/skills/review/SKILL.md Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant