skills(running-tend): restore outage drain until tend#851 lands - #3781
skills(running-tend): restore outage drain until tend#851 lands#3781worktrunk-bot wants to merge 6 commits into
Conversation
#3742 removed the outage-recovery section on the premise that tend#851 would carry it upstream. That PR has not landed, so the drain guidance exists in neither place, and last night's session-limit outage stranded a tend-review run that only a manual maintainer rerun recovered. Pure revert of the removed hunk.
worktrunk-bot
left a comment
There was a problem hiding this comment.
Verified the mechanical claim rather than taking it on faith: extracting the removed lines from 60830520c and the added lines from 2bfcd498e and diffing them shows the 33 lines are byte-identical, restored at the same position (after Labels, before CI Fix). The premise checks out too — max-sixty/tend#851 is still OPEN with reviews: [], and a grep over the bundled tend-ci-runner skills finds outage text only in ci-fix (issue creation), nightly (enrichment), and review-reviewers (a label-skip note); review-runs has none, so the guidance is currently in neither place.
One gap: the restored section is verbatim, so its stated lifetime lives only in this PR description. A future sweep reading SKILL.md has no signal that this is a hold — once tend#851 lands, the overlay silently duplicates the bundled guidance and the two drift independently. The bundled skills already use an in-file marker for exactly this shape (<!-- TODO(anthropics/claude-code#37157): once the harness exempts .claude/skills/ ... -->). Inline suggestion adds a one-line pointer; that costs the "pure revert" property, which seems worth trading for the removal trigger being discoverable from the file itself.
… one The section this branch restores predates the two-limit correction: it names only `session limit` and asserts a 5-hour window. tend#851, which this branch holds for, carries the corrected text. Bring the local copy in line so the hold isn't a regression.
|
Pushed
The consequence is a timing instruction that's wrong by 3× on the weekly flavor: a run reading "quota exhaustion across a 5-hour window" during a weekly exhaustion re-runs stranded triggers while the quota is still out, and each re-run appends another row to the issue it is draining. That's the exact failure mode the correction was written for — #3737 (2026-08-04) was a weekly exhaustion lasting 15h 10m, against #3715 (2026-08-03) at 8h 20m for the session flavor. The commit is 3 lines: the second message shape in the VerificationPure deletion — no corrected text ever landed on Found while reviewing this window's outage (#3780, session limit 22:23Z–23:55Z on 2026-08-08) for the daily run review. |
|
Pushed What it adds. One bullet at the top of the Weekly Maintenance: Statusline Cache-Check triage list, naming the Why. The recipe requires Three consecutive weekly runs re-derived this from scratch, each spending turns on it and each concluding "not a production cache miss, nothing to file":
#3705 (merged 08-02) removed the That last run said so itself and held back only on this branch being open: "One thing I did not do: codify that Gate assessment. High evidence at 3 occurrences; structural (the recipe mandates |
worktrunk-bot
left a comment
There was a problem hiding this comment.
The outage-restore half was checked in the previous review, and b796c514's two-limit correction matches the wording carried in max-sixty/tend#851 — so this is about the 21adcb61 bullet. I pulled the .cache output the two cited weekly runs actually produced rather than taking the PR comment's table at face value; two of the bullet's claims don't survive that.
git --version isn't one of the duplicates. Its only spawn sites are git_version() in src/diagnostic.rs (called from format_report, i.e. the epilogue) and wt config show — nothing on the statusline render path runs it, so it fires exactly once and CacheReport, which only buckets a (command, context) pair seen more than once, can't surface it. Run 31305900412's own .cache lists exactly two entries: gh --version (context (none), count 2) and git worktree list --porcelain (context ., count 2). That matters beyond tidiness — if git --version ever did show up twice it would be a real finding, and this bullet currently pre-labels it as noise.
"explained every duplicate the report flagged on the last three runs" overstates it. Run 30742679188 (08-02) flagged three, and the third — git config --list -z, context .git — was a genuine in-process miss, fixed by #3705, not an epilogue artifact. A bullet that says "check this first, it explains everything" invites the next run to dismiss the one that isn't.
The discriminator the bullet names isn't in the recipe's output. jq .cache yields same_context_duplicates[] with command / max_per_context / extra_calls / extra_us / contexts and nothing else — no tid, no per-call timestamps (CacheReport and DuplicateCommand in src/trace/profile.rs). Those fields are in the trace.jsonl records themselves ({"kind":"cmd_completed","ts":…,"tid":…,"cmd":…}, per the src/trace/parse.rs module docs), which is why the 08-09 run ended up grepping the file directly. Naming the file and the query is exactly the kind of re-derivation this bullet exists to stop.
Suggestions inline. Not approving — self-authored.
… the trace query `git --version` has one spawn site on this path — `git_version()` in `src/diagnostic.rs`, reached from `format_report` (the epilogue) and from `wt config show` — so it runs once per invocation and can never form a duplicate bucket. Verified on a real `-vv` statusline run: the `.cache` report lists exactly `gh --version` and `git worktree list --porcelain`, and `git --version` appears once in `trace.jsonl`. Soften "every duplicate on the last three runs" to what the runs show — the 08-02 run's third duplicate (`git config --list -z`) was a real miss fixed by #3705, so a blanket dismissal invites the next run to wave through the one that isn't an artifact. Name the query that answers the artifact test: `jq .cache` carries no `tid` or `ts` (`CacheReport`/`DuplicateCommand` in `src/trace/profile.rs`), so the check has to run against `trace.jsonl`. Path goes through `git rev-parse --git-common-dir` rather than a literal `.git/` — the recipe says "run from any worktree", where `.git` is a file and the literal path fails with ENOTDIR.
worktrunk-bot
left a comment
There was a problem hiding this comment.
Re-checked both findings from the last review against the code rather than the commit message. Both hold up:
git_version()really has only the two callers (format_report,wt config show), so it can't form a bucket — andsame_context_duplicateskeys on the rawcommandstring from the trace, notcommand_type's shape (DuplicateCommandconstruction insrc/trace/profile.rs), which is what makes the newselect(.cmd == "gh --version")query match the.cacheentry exactly rather than aghbucket. Worth knowing sincecommand_type("gh --version")collapses togh.- The path fix is right and matches the idiom this skill already uses for
diagnostic.md:wt_logs_dir()iswt_dir().join("logs")overgit_common_dir(), so$(git rev-parse --git-common-dir)/wt/logs/trace.jsonlresolves from a linked worktree where the literal.git/path is a file.
One anchor left over: the bullet cites DiagnosticReport::collect for the git worktree list --porcelain re-run, but that spawn is in format_report — collect just calls it, so a grep for collect lands one hop short. The commit message for bab1e97b7 already names format_report; the bullet is what didn't get updated. Suggestion inline.
Not approving — self-authored.
`DiagnosticReport::collect` forwards to `format_report`, which is where the `git worktree list --porcelain` call actually sits (src/diagnostic.rs:230), so a grep for `collect` lands one hop short of the spawn.
Restores the
## Outage Recovery: Re-run Triggers Stranded by a Failed Sessionhunk that #3742 removed. The 33 restored lines are byte-identical to what was deleted; on top of them sits one added line under the heading marking the section as a hold on max-sixty/tend#851, so the removal trigger is discoverable fromSKILL.mditself rather than only from this description (added in review).Why now. #3742 removed the section on the premise that the guidance would live upstream instead: "the guidance now lives in tend's bundled
review-runsskill, proposed in max-sixty/tend#851." That PR has not landed — open since 2026-08-05 with no reviews — so the guidance currently exists in neither place. I checked the bundled skills rather than assuming:ci-fixonly creates atend-outageissue,nightlyonly enriches one viaenrich-tend-outage-issues.sh, andreview-runs(the target of the upstream PR) has no outage text at all.The window cost more than it was priced at. #3742's own body flagged the sequencing and put a bound on it: "Merging this before #851 lands leaves a window with no drain guidance in either place. The window costs one daily sweep at most, and the outage issue accumulates rather than losing data." Last night's outage falsified the second half. A Claude session-limit exhaustion ran 22:23Z–23:55Z on 2026-08-08 and stranded nine rows on #3780. Seven were
tend-notificationsonschedule, which self-recover — correctly out of scope. The eighth was run 31281595492,tend-reviewfor #3779, attempt 1 failed 22:26:53Z.tend-reviewfires only onpull_request_target, so nothing would have re-fired it. The maintainer re-ran it by hand — attempt 2 started 00:01:31Z, approved 00:05:35Z, merged 00:09:49Z. Without that manual rerun #3779 merges unreviewed, which is data lost rather than accumulated.The accumulation half didn't hold either: three
tend-notificationsruns completed after the quota reset (00:12Z, 00:33Z, 01:08Z) and none of them touched #3780, because nothing in either skill set tells a run to drain it. This session drained and closed it by hand.Lifetime. This is a hold, not a position on where the guidance belongs — #3742's argument that nothing in the section is worktrunk-specific still reads correctly. When max-sixty/tend#851 lands, drop this section again and the bundled version takes over. That instruction is now in the file, not just here.
Evidence
gh pr view 3742 --json mergedAt→2026-08-05T16:57:13Z; the diff is a pure deletion of the 33-line section, andgit log -- .claude/skills/running-tend/SKILL.mdshows no later commit touching the file, so the restored lines are an exact reverse-apply of that hunk — the only divergence is the hold marker added in review.gh pr view 851 -R max-sixty/tend --json state,mergedAt,reviews→OPEN,mergedAt: null,reviews: [].grep -rniE "tend-outage|stranded|outage recovery"overplugins/tend-ci-runner/skills/→ hits only inci-fix(issue creation),nightly(enrichment), andreview-reviewers(a label-skip note).gh api repos/max-sixty/worktrunk/actions/runs/31281595492→run_attempt: 2,run_started_at: 2026-08-09T00:01:31Z; attempt 1conclusion: failureat 22:26:53Z.success.tend-mentiononrepository_dispatch), also won't self-recover, but re-running it now would post into a merged thread — the section's "confirm the work is still missing before re-running" rule is what says to leave it.