fix(running-in-ci): read a recipe before running it, and never extract one by position - #899
fix(running-in-ci): read a recipe before running it, and never extract one by position#899tend-agent wants to merge 2 commits into
Conversation
…t one by position Verifying a skill's own recipe runs it against the live repo, and those recipes write. Add the missing guard to the 'verify by running the code' instruction: read the block first, match it by prose rather than by fence ordinal, run the read half, and point a write at a scratch object.
tend-agent
left a comment
There was a problem hiding this comment.
Self-review — posting as a comment since I can't approve my own PR. The change is well-scoped and both factual claims in it check out (a reopen leaves the close in the timeline; a deleted comment's issue_comment event has already fired). One substantive point inline, one placement note.
The new paragraph lands between the two-path guidance and the <example> pair that illustrates it. That example is about trusting upstream docs instead of reading the source — i.e. it illustrates paths 1/2, not write-safety — so with nine lines wedged in front of it, it now reads as an example of the write guard. Every other section in this file keeps guidance-then-example adjacent. Appending the new paragraph after </example> instead would preserve that, and it costs nothing since the paragraph doesn't depend on the example.
Worth naming a tension in the section fit too: the heading is "Verifying external-tool behavior", and "verifying a skill's own recipe is the common case" is a different activity than checking whether cmux has a --json flag. The PR body's argument — put the guard where the pressure was created — is defensible, so I'm not asking for a move, just flagging that a reader arriving at this heading for an external-CLI question gets a paragraph about gh issue close.
|
Both points taken; pushed in bf73b02. The placement note is correct and I should have caught it — the On the section-fit tension you flagged without asking for a move: I agree it's real and I'm leaving it as-is for now. A reader arriving at "Verifying external-tool behavior" for a |
A session verifying its own skill edit closed a live maintainer-facing issue and posted a false status comment. The bot caught it and self-healed within ~2.5 minutes, but the close/reopen is permanent in the timeline and the deleted comment had already fired a workflow run. The guidance that produced the behaviour is still unguarded, so the next session hits the same shape.
What happened
Run 31257558939 (
tend-mention,repository_dispatch) was answering a review question on #898 — a change toplugins/tend-ci-runner/skills/nightly/SKILL.md. To verify the edit it extracted a code block from that file by fence ordinal and ran it:The third
bashfence in that file is not the step-3 recipe it was aiming at. It is the drift-issue closer:Issue #822 (
tend check: configuration drift on max-sixty/tend) matched. It was closed at12:38:56Zwith the commenttend check now passes.— false; the check still fails. The session noticed, reopened at12:39:04Z, deleted the comment, and left a correction on the thread plus a flag in its PR reply. Its own words from the session log: "My mistake — I ran an extracted block without reading it first."What did not recover:
#822's timeline still showsclosed by tend-agent/reopened by tend-agent, and the deleted comment had already firedissue_comment, so run 31257716552 exists and always will.Root cause
Two things compose, and only the second is a slip:
running-in-ci's "Verifying external-tool behavior" says "verify by running the code" and ranks running the tool above reading the source, with an example scolding a session for trusting docs instead of running the thing. That instruction is right, and the session was obeying it. Nothing anywhere says the recipes it points at includegh issue close,gh pr comment, andgit push, or that "run it" needs to mean something different when the recipe writes. Every adopter'snightlyskill ships that same closer, so this is a bundled gap, not a tend-local one.The change
One paragraph plus a two-line example in the section that created the pressure. It names the write surface, kills positional extraction, and gives the read-half recipe. No new section, no reorganisation.
Gate assessment
0 reopened issueunder negative signals, so there is no historical count to add to.running-in-ciPRs (fix(running-in-ci): pin the CI-monitor poll to the SHA the run pushed #837, fix(running-in-ci): leave a review that lands mid-poll to tend-mention #870, fix(running-in-ci): make the CI-monitor poll cap terminal #876) are all the CI-monitor poll, a different section.Verified, not inferred
#822's events API shows the close and the reopen both retained; run31257716552fired from the since-deleted comment and is still listed.bash /tmp/step3.share read verbatim from the session log'stool_useentries, not reconstructed from the bot's summary.awkagainstnightly/SKILL.mdat03f8e0d.Evidence log: https://gist.github.com/e08f6e62d6478163cb425a75648eb7e4