From 2bfcd498e23f9a79053857593635577e998389ae Mon Sep 17 00:00:00 2001 From: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com> Date: Sun, 9 Aug 2026 01:57:46 +0000 Subject: [PATCH 1/6] skills(running-tend): restore outage drain until tend#851 lands #3742 removed the outage-recovery section on the premise that tend#851 would carry it upstream. That PR has not landed, so the drain guidance exists in neither place, and last night's session-limit outage stranded a tend-review run that only a manual maintainer rerun recovered. Pure revert of the removed hunk. --- .claude/skills/running-tend/SKILL.md | 33 ++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 2126d7964..1e521a4e2 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -83,6 +83,39 @@ Artifact paths: `-home-runner-work-worktrunk-worktrunk/.jsonl` - `automated-fix` — fix PRs from triage and ci-fix workflows - `nightly-cleanup` — nightly sweep issues and PRs +## Outage Recovery: Re-run Triggers Stranded by a Failed Session + +When `claude -p` exits non-zero the harness opens (or appends to) a `tend-outage`-labelled **"Bot temporarily unavailable"** issue, one table row per failure with the run link and the triggering `#N`. It records the failure; **nothing re-runs it.** `tend-review` fires only on `pull_request_target`, so a PR whose one review attempt died is silently never reviewed until someone happens to push again. + +Drain the open outage issue as part of the daily `review-runs` sweep: + +```bash +# Empty on most days — no open outage issue means nothing stranded; skip the rest. +OUTAGE=$(gh issue list --state open --label tend-outage --json number --jq '.[0].number // empty') +gh issue view "$OUTAGE" --json body,comments --jq '.body, .comments[].body' \ + | grep -oE 'runs/[0-9]+|\| #[0-9]+' +``` + +Only event-triggered runs strand — `tend-nightly` and `tend-notifications` recover on their next cron tick, so leave those alone. For a `tend-review` / `tend-mention` / `tend-triage` run, confirm the work is still missing before re-running; a later push often re-triggers the workflow on its own, and re-running a job that already happened burns quota for nothing: + +```bash +gh pr view --json state,headRefOid,reviews \ + --jq '{state, headRefOid, bot: ([.reviews[] | select(.author.login == "worktrunk-bot")] | length)}' +gh run rerun --failed +``` + +Re-running the bot's own failed workflow is restorative, not destructive — no maintainer approval needed. Close the issue once every row is drained (`gh issue close "$OUTAGE"`): the harness only auto-closes duplicates from a create-create race, never the surviving issue, so one left open makes tomorrow's sweep re-check the same rows and folds the next outage into a stale incident. + +**Diagnose the cause before calling it an outage.** The issue body says only "The bot failed to process a request". tend's nightly enrichment posts a per-run comment carrying each failed job's failure annotation, which is the cheapest first look — but for this class it reads `claude -p exited non-zero (exit=1) — see the session-logs artifact` (or `claude -p turn ended in failure (…) — rate limit, auth, max turns, or server error` when the CLI exits 0 with an error result), and neither says whether it was quota, auth, or a server error. It's also absent whenever `tend-nightly` was itself one of the stranded runs. To narrow it, read the session log: a subscription session-limit exhaustion surfaces as a `` assistant message: + +```bash +gh run download --pattern '*session-logs*' --dir /tmp/outage +jq -r 'select(.type == "assistant") | .message.content[]?.text // empty' /tmp/outage/*/*/*.jsonl +# → You've hit your session limit · resets 8:30am (UTC) +``` + +A cluster of these is quota exhaustion across a 5-hour window, not a bug — don't open a fix PR. Record the window's spend in the tracking issue and re-run the stranded triggers once the window resets. + ## CI Fix: Prefer Rerun for Transient Infrastructure Failures Before opening a `fix/ci-*` PR, classify the failure: From 818db4698562cc6c7a8a6c4aad7c70e8c98c9e01 Mon Sep 17 00:00:00 2001 From: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com> Date: Sun, 9 Aug 2026 02:04:26 +0000 Subject: [PATCH 2/6] skills(running-tend): mark the outage section as a hold on tend#851 --- .claude/skills/running-tend/SKILL.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 1e521a4e2..de833eb41 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -85,6 +85,8 @@ Artifact paths: `-home-runner-work-worktrunk-worktrunk/.jsonl` ## Outage Recovery: Re-run Triggers Stranded by a Failed Session +_Temporary hold: [max-sixty/tend#851](https://github.com/max-sixty/tend/pull/851) moves this into the bundled `review-runs` skill. Drop this section once that lands._ + When `claude -p` exits non-zero the harness opens (or appends to) a `tend-outage`-labelled **"Bot temporarily unavailable"** issue, one table row per failure with the run link and the triggering `#N`. It records the failure; **nothing re-runs it.** `tend-review` fires only on `pull_request_target`, so a PR whose one review attempt died is silently never reviewed until someone happens to push again. Drain the open outage issue as part of the daily `review-runs` sweep: From b796c514abe305342e1066299f08ea8db73e20b3 Mon Sep 17 00:00:00 2001 From: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com> Date: Sun, 9 Aug 2026 08:17:36 +0000 Subject: [PATCH 3/6] skills(running-tend): restore both quota limits, not just the session one The section this branch restores predates the two-limit correction: it names only `session limit` and asserts a 5-hour window. tend#851, which this branch holds for, carries the corrected text. Bring the local copy in line so the hold isn't a regression. --- .claude/skills/running-tend/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index de833eb41..666332bfc 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -108,15 +108,16 @@ gh run rerun --failed Re-running the bot's own failed workflow is restorative, not destructive — no maintainer approval needed. Close the issue once every row is drained (`gh issue close "$OUTAGE"`): the harness only auto-closes duplicates from a create-create race, never the surviving issue, so one left open makes tomorrow's sweep re-check the same rows and folds the next outage into a stale incident. -**Diagnose the cause before calling it an outage.** The issue body says only "The bot failed to process a request". tend's nightly enrichment posts a per-run comment carrying each failed job's failure annotation, which is the cheapest first look — but for this class it reads `claude -p exited non-zero (exit=1) — see the session-logs artifact` (or `claude -p turn ended in failure (…) — rate limit, auth, max turns, or server error` when the CLI exits 0 with an error result), and neither says whether it was quota, auth, or a server error. It's also absent whenever `tend-nightly` was itself one of the stranded runs. To narrow it, read the session log: a subscription session-limit exhaustion surfaces as a `` assistant message: +**Diagnose the cause before calling it an outage.** The issue body says only "The bot failed to process a request". tend's nightly enrichment posts a per-run comment carrying each failed job's failure annotation, which is the cheapest first look — but for this class it reads `claude -p exited non-zero (exit=1) — see the session-logs artifact` (or `claude -p turn ended in failure (…) — rate limit, auth, max turns, or server error` when the CLI exits 0 with an error result), and neither says whether it was quota, auth, or a server error. It's also absent whenever `tend-nightly` was itself one of the stranded runs. To narrow it, read the session log: a subscription quota exhaustion surfaces as a `` assistant message: ```bash gh run download --pattern '*session-logs*' --dir /tmp/outage jq -r 'select(.type == "assistant") | .message.content[]?.text // empty' /tmp/outage/*/*/*.jsonl # → You've hit your session limit · resets 8:30am (UTC) +# → You've hit your weekly limit · resets 12am (UTC) ``` -A cluster of these is quota exhaustion across a 5-hour window, not a bug — don't open a fix PR. Record the window's spend in the tracking issue and re-run the stranded triggers once the window resets. +A cluster of these is quota exhaustion, not a bug — don't open a fix PR. The two limits reset on different clocks, so read the reset off the message rather than assuming the shorter session window; a weekly exhaustion can strand most of a day. Record the window's spend in the tracking issue, and gate the re-runs on a later run that already succeeded rather than on an assumed clock — re-running inside a still-exhausted window just appends another row to the issue you are draining. ## CI Fix: Prefer Rerun for Transient Infrastructure Failures From 21adcb61b9e9301fe333f0e8bb1307444504ea4c Mon Sep 17 00:00:00 2001 From: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com> Date: Mon, 10 Aug 2026 08:33:49 +0000 Subject: [PATCH 4/6] skills(running-tend): name the -vv artifacts the cache-check keeps re-deriving --- .claude/skills/running-tend/SKILL.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 666332bfc..3f7cfa8d7 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -365,6 +365,17 @@ cargo run --release -- config state logs profile --format=json | jq .cache The `.cache` report flags commands invoked more than once with the same context. Triage each duplicate: +- **`-vv` epilogue artifact** — check this first; it has explained every + duplicate the report flagged on the last three runs. The `-vv` this recipe + requires writes the diagnostic bundle *after* the render, and assembling it + re-runs three commands the render already made: `git --version` and + `git worktree list --porcelain` (`DiagnosticReport::collect` in + `src/diagnostic.rs`), plus `gh --version` for the gist hint + (`is_gh_installed`). All three go through `Cmd`, so they land in the same + `trace.jsonl` the `.cache` report reads and pair with the render's own + calls. They run on the main thread after the render's worker threads finish, + so a differing thread id and a timestamp past the render's end identify + them. Nothing to file. - **Legitimate** (different cwd, different ref form that can't be normalized, intentional double-call across phases) — note in the response and move on. - **Cache miss** (same logical operation should hit cache but doesn't) — From bab1e97b753659a6af1febd5baeb24e9487ce3a7 Mon Sep 17 00:00:00 2001 From: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com> Date: Mon, 10 Aug 2026 08:47:26 +0000 Subject: [PATCH 5/6] skills(running-tend): drop git --version from the epilogue list, name the trace query MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `git --version` has one spawn site on this path — `git_version()` in `src/diagnostic.rs`, reached from `format_report` (the epilogue) and from `wt config show` — so it runs once per invocation and can never form a duplicate bucket. Verified on a real `-vv` statusline run: the `.cache` report lists exactly `gh --version` and `git worktree list --porcelain`, and `git --version` appears once in `trace.jsonl`. Soften "every duplicate on the last three runs" to what the runs show — the 08-02 run's third duplicate (`git config --list -z`) was a real miss fixed by #3705, so a blanket dismissal invites the next run to wave through the one that isn't an artifact. Name the query that answers the artifact test: `jq .cache` carries no `tid` or `ts` (`CacheReport`/`DuplicateCommand` in `src/trace/profile.rs`), so the check has to run against `trace.jsonl`. Path goes through `git rev-parse --git-common-dir` rather than a literal `.git/` — the recipe says "run from any worktree", where `.git` is a file and the literal path fails with ENOTDIR. --- .claude/skills/running-tend/SKILL.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 3f7cfa8d7..8001ed2ea 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -365,17 +365,20 @@ cargo run --release -- config state logs profile --format=json | jq .cache The `.cache` report flags commands invoked more than once with the same context. Triage each duplicate: -- **`-vv` epilogue artifact** — check this first; it has explained every - duplicate the report flagged on the last three runs. The `-vv` this recipe - requires writes the diagnostic bundle *after* the render, and assembling it - re-runs three commands the render already made: `git --version` and +- **`-vv` epilogue artifact** — check this first; it explained every duplicate + the report flagged on the most recent run, and all but one (`git config + --list -z`, a real miss fixed in #3705) on the run before. The `-vv` this + recipe requires writes the diagnostic bundle *after* the render, and + assembling it re-runs two commands the render already made: `git worktree list --porcelain` (`DiagnosticReport::collect` in - `src/diagnostic.rs`), plus `gh --version` for the gist hint - (`is_gh_installed`). All three go through `Cmd`, so they land in the same + `src/diagnostic.rs`) and `gh --version` for the gist hint + (`is_gh_installed`). Both go through `Cmd`, so they land in the same `trace.jsonl` the `.cache` report reads and pair with the render's own - calls. They run on the main thread after the render's worker threads finish, - so a differing thread id and a timestamp past the render's end identify - them. Nothing to file. + calls. `jq .cache` carries only commands, counts and durations, so confirm + against the trace itself: `jq -c 'select(.cmd == "gh --version") | {tid, ts}' + "$(git rev-parse --git-common-dir)/wt/logs/trace.jsonl"`. The epilogue runs + on the main thread after the render's worker threads finish, so a differing + `tid` and a `ts` past the render's end identify it. Nothing to file. - **Legitimate** (different cwd, different ref form that can't be normalized, intentional double-call across phases) — note in the response and move on. - **Cache miss** (same logical operation should hit cache but doesn't) — From 9d79f92f6598f14ca794ae96352c87b553261561 Mon Sep 17 00:00:00 2001 From: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com> Date: Mon, 10 Aug 2026 09:01:44 +0000 Subject: [PATCH 6/6] skills(running-tend): anchor the worktree-list spawn at format_report `DiagnosticReport::collect` forwards to `format_report`, which is where the `git worktree list --porcelain` call actually sits (src/diagnostic.rs:230), so a grep for `collect` lands one hop short of the spawn. --- .claude/skills/running-tend/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 8001ed2ea..917df94fc 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -370,7 +370,7 @@ Triage each duplicate: --list -z`, a real miss fixed in #3705) on the run before. The `-vv` this recipe requires writes the diagnostic bundle *after* the render, and assembling it re-runs two commands the render already made: - `git worktree list --porcelain` (`DiagnosticReport::collect` in + `git worktree list --porcelain` (`DiagnosticReport::format_report` in `src/diagnostic.rs`) and `gh --version` for the gist hint (`is_gh_installed`). Both go through `Cmd`, so they land in the same `trace.jsonl` the `.cache` report reads and pair with the render's own