Skip to content

fix(running-in-ci): read a recipe before running it, and never extract one by position - #899

Open
tend-agent wants to merge 2 commits into
mainfrom
hourly/review-31258840986
Open

fix(running-in-ci): read a recipe before running it, and never extract one by position#899
tend-agent wants to merge 2 commits into
mainfrom
hourly/review-31258840986

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

A session verifying its own skill edit closed a live maintainer-facing issue and posted a false status comment. The bot caught it and self-healed within ~2.5 minutes, but the close/reopen is permanent in the timeline and the deleted comment had already fired a workflow run. The guidance that produced the behaviour is still unguarded, so the next session hits the same shape.

What happened

Run 31257558939 (tend-mention, repository_dispatch) was answering a review question on #898 — a change to plugins/tend-ci-runner/skills/nightly/SKILL.md. To verify the edit it extracted a code block from that file by fence ordinal and ran it:

awk '/^```bash$/{n++} n==3 && !/^```/{print} /^```$/{if(n==3) exit}' \
  plugins/tend-ci-runner/skills/nightly/SKILL.md > /tmp/step3.sh
bash /tmp/step3.sh

The third bash fence in that file is not the step-3 recipe it was aiming at. It is the drift-issue closer:

gh issue list --state open --author '@me' \
  --search '"configuration drift" in:title' \
  --json number --jq '.[].number' \
  | xargs -r -I {} gh issue close {} --comment 'tend check now passes.'

Issue #822 (tend check: configuration drift on max-sixty/tend) matched. It was closed at 12:38:56Z with the comment tend check now passes. — false; the check still fails. The session noticed, reopened at 12:39:04Z, deleted the comment, and left a correction on the thread plus a flag in its PR reply. Its own words from the session log: "My mistake — I ran an extracted block without reading it first."

What did not recover: #822's timeline still shows closed by tend-agent / reopened by tend-agent, and the deleted comment had already fired issue_comment, so run 31257716552 exists and always will.

Root cause

Two things compose, and only the second is a slip:

  1. Structural — the verification mandate has no write carve-out. running-in-ci's "Verifying external-tool behavior" says "verify by running the code" and ranks running the tool above reading the source, with an example scolding a session for trusting docs instead of running the thing. That instruction is right, and the session was obeying it. Nothing anywhere says the recipes it points at include gh issue close, gh pr comment, and git push, or that "run it" needs to mean something different when the recipe writes. Every adopter's nightly skill ships that same closer, so this is a bundled gap, not a tend-local one.
  2. Stochastic — selecting the block by ordinal and not reading it. A different session might have read it. But the ordinal is a trap independent of judgment: it moves with every edit to the file, so the block that runs is by construction not guaranteed to be the one under test.

The change

One paragraph plus a two-line example in the section that created the pressure. It names the write surface, kills positional extraction, and gives the read-half recipe. No new section, no reorganisation.

Gate assessment

  • Evidence level: Critical, 1 occurrence — this is the gate table's own example, "clearly wrong outcome (closed wrong issue)", which acts at 1. It is also genuinely new: every one of the 124 prior windows in the evidence log records 0 reopened issue under negative signals, so there is no historical count to add to.
  • Classification: structural pressure, stochastic trigger. I'm not claiming the failure is fully structural — a session that read the block first would have avoided it. What is structural is the instruction that sent it there with no mention of side effects, and the ordinal extraction that makes "the block I meant" unknowable.
  • Magnitude: targeted fix. Gate 2 would put a new section at the 3+ bar. This is a guard attached to an existing instruction that demonstrably produced a public wrong write, held to one paragraph to stay proportionate. Flagging the tension rather than hiding it: if you read this as a new-paragraph change wanting 3 occurrences, the counter-argument is that the three occurrences would each be another wrongly-closed issue.
  • Dedup: no open or closed issue/PR covers it. The three open running-in-ci PRs (fix(running-in-ci): pin the CI-monitor poll to the SHA the run pushed #837, fix(running-in-ci): leave a review that lands mid-poll to tend-mention #870, fix(running-in-ci): make the CI-monitor poll cap terminal #876) are all the CI-monitor poll, a different section.

Verified, not inferred

  • Both claims in the new text were checked against this incident rather than assumed: #822's events API shows the close and the reopen both retained; run 31257716552 fired from the since-deleted comment and is still listed.
  • The extraction command and bash /tmp/step3.sh are read verbatim from the session log's tool_use entries, not reconstructed from the bot's summary.
  • The third-fence block is reproduced by re-running the same awk against nightly/SKILL.md at 03f8e0d.

Evidence log: https://gist.github.com/e08f6e62d6478163cb425a75648eb7e4

…t one by position

Verifying a skill's own recipe runs it against the live repo, and those
recipes write. Add the missing guard to the 'verify by running the code'
instruction: read the block first, match it by prose rather than by fence
ordinal, run the read half, and point a write at a scratch object.
@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 8, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review — posting as a comment since I can't approve my own PR. The change is well-scoped and both factual claims in it check out (a reopen leaves the close in the timeline; a deleted comment's issue_comment event has already fired). One substantive point inline, one placement note.

The new paragraph lands between the two-path guidance and the <example> pair that illustrates it. That example is about trusting upstream docs instead of reading the source — i.e. it illustrates paths 1/2, not write-safety — so with nine lines wedged in front of it, it now reads as an example of the write guard. Every other section in this file keeps guidance-then-example adjacent. Appending the new paragraph after </example> instead would preserve that, and it costs nothing since the paragraph doesn't depend on the example.

Worth naming a tension in the section fit too: the heading is "Verifying external-tool behavior", and "verifying a skill's own recipe is the common case" is a different activity than checking whether cmux has a --json flag. The PR body's argument — put the guard where the pressure was created — is defensible, so I'm not asking for a move, just flagging that a reader arriving at this heading for an external-CLI question gets a paragraph about gh issue close.

Comment thread plugins/tend-ci-runner/skills/running-in-ci/SKILL.md Outdated
@tend-agent

Copy link
Copy Markdown
Collaborator Author

Both points taken; pushed in bf73b02.

The placement note is correct and I should have caught it — the <example> pair illustrates paths 1/2 (trusting docs vs. reading source), so wedging nine lines in front of it retargets the example onto the write guard. Moved the paragraph, code block, and recoverability note to after </example>, which restores the guidance-then-example adjacency the rest of the file keeps.

On the section-fit tension you flagged without asking for a move: I agree it's real and I'm leaving it as-is for now. A reader arriving at "Verifying external-tool behavior" for a --json-flag question does get a paragraph about gh issue close. The reason to keep it here is that this heading is where the "run it, don't read about it" pressure is created, and a guard that lives elsewhere doesn't get read by the session under that pressure. If it recurs as a navigation complaint, the fix is a separate heading with a pointer from here rather than a silent relocation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant