From 3d856c619c887821e0853e2e33ec9e950e275f4f Mon Sep 17 00:00:00 2001 From: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com> Date: Wed, 5 Aug 2026 08:51:18 +0000 Subject: [PATCH 1/2] skills(running-tend): name both subscription limits in outage diagnosis --- .claude/skills/running-tend/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 1e521a4e2..3a2698d43 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -106,15 +106,16 @@ gh run rerun --failed Re-running the bot's own failed workflow is restorative, not destructive — no maintainer approval needed. Close the issue once every row is drained (`gh issue close "$OUTAGE"`): the harness only auto-closes duplicates from a create-create race, never the surviving issue, so one left open makes tomorrow's sweep re-check the same rows and folds the next outage into a stale incident. -**Diagnose the cause before calling it an outage.** The issue body says only "The bot failed to process a request". tend's nightly enrichment posts a per-run comment carrying each failed job's failure annotation, which is the cheapest first look — but for this class it reads `claude -p exited non-zero (exit=1) — see the session-logs artifact` (or `claude -p turn ended in failure (…) — rate limit, auth, max turns, or server error` when the CLI exits 0 with an error result), and neither says whether it was quota, auth, or a server error. It's also absent whenever `tend-nightly` was itself one of the stranded runs. To narrow it, read the session log: a subscription session-limit exhaustion surfaces as a `` assistant message: +**Diagnose the cause before calling it an outage.** The issue body says only "The bot failed to process a request". tend's nightly enrichment posts a per-run comment carrying each failed job's failure annotation, which is the cheapest first look — but for this class it reads `claude -p exited non-zero (exit=1) — see the session-logs artifact` (or `claude -p turn ended in failure (…) — rate limit, auth, max turns, or server error` when the CLI exits 0 with an error result), and neither says whether it was quota, auth, or a server error. It's also absent whenever `tend-nightly` was itself one of the stranded runs. To narrow it, read the session log: a subscription limit exhaustion surfaces as a `` assistant message: ```bash gh run download --pattern '*session-logs*' --dir /tmp/outage jq -r 'select(.type == "assistant") | .message.content[]?.text // empty' /tmp/outage/*/*/*.jsonl # → You've hit your session limit · resets 8:30am (UTC) +# → You've hit your weekly limit · resets 12am (UTC) ``` -A cluster of these is quota exhaustion across a 5-hour window, not a bug — don't open a fix PR. Record the window's spend in the tracking issue and re-run the stranded triggers once the window resets. +A cluster of these is quota exhaustion, not a bug — don't open a fix PR. Two limits produce it and they reset differently, so read the reset off the message rather than assuming the 5-hour session window; a weekly exhaustion can strand most of a day. Record the window's spend in the tracking issue, and re-run the stranded triggers only once a later run has completed cleanly — re-running into a still-exhausted quota just refills the outage issue. ## CI Fix: Prefer Rerun for Transient Infrastructure Failures From 911e84f500a0789c53aa196c3dd2999ad7028353 Mon Sep 17 00:00:00 2001 From: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com> Date: Wed, 5 Aug 2026 16:20:10 +0000 Subject: [PATCH 2/2] skills(running-tend): move outage recovery upstream into tend --- .claude/skills/running-tend/SKILL.md | 34 ---------------------------- 1 file changed, 34 deletions(-) diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 3a2698d43..2126d7964 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -83,40 +83,6 @@ Artifact paths: `-home-runner-work-worktrunk-worktrunk/.jsonl` - `automated-fix` — fix PRs from triage and ci-fix workflows - `nightly-cleanup` — nightly sweep issues and PRs -## Outage Recovery: Re-run Triggers Stranded by a Failed Session - -When `claude -p` exits non-zero the harness opens (or appends to) a `tend-outage`-labelled **"Bot temporarily unavailable"** issue, one table row per failure with the run link and the triggering `#N`. It records the failure; **nothing re-runs it.** `tend-review` fires only on `pull_request_target`, so a PR whose one review attempt died is silently never reviewed until someone happens to push again. - -Drain the open outage issue as part of the daily `review-runs` sweep: - -```bash -# Empty on most days — no open outage issue means nothing stranded; skip the rest. -OUTAGE=$(gh issue list --state open --label tend-outage --json number --jq '.[0].number // empty') -gh issue view "$OUTAGE" --json body,comments --jq '.body, .comments[].body' \ - | grep -oE 'runs/[0-9]+|\| #[0-9]+' -``` - -Only event-triggered runs strand — `tend-nightly` and `tend-notifications` recover on their next cron tick, so leave those alone. For a `tend-review` / `tend-mention` / `tend-triage` run, confirm the work is still missing before re-running; a later push often re-triggers the workflow on its own, and re-running a job that already happened burns quota for nothing: - -```bash -gh pr view --json state,headRefOid,reviews \ - --jq '{state, headRefOid, bot: ([.reviews[] | select(.author.login == "worktrunk-bot")] | length)}' -gh run rerun --failed -``` - -Re-running the bot's own failed workflow is restorative, not destructive — no maintainer approval needed. Close the issue once every row is drained (`gh issue close "$OUTAGE"`): the harness only auto-closes duplicates from a create-create race, never the surviving issue, so one left open makes tomorrow's sweep re-check the same rows and folds the next outage into a stale incident. - -**Diagnose the cause before calling it an outage.** The issue body says only "The bot failed to process a request". tend's nightly enrichment posts a per-run comment carrying each failed job's failure annotation, which is the cheapest first look — but for this class it reads `claude -p exited non-zero (exit=1) — see the session-logs artifact` (or `claude -p turn ended in failure (…) — rate limit, auth, max turns, or server error` when the CLI exits 0 with an error result), and neither says whether it was quota, auth, or a server error. It's also absent whenever `tend-nightly` was itself one of the stranded runs. To narrow it, read the session log: a subscription limit exhaustion surfaces as a `` assistant message: - -```bash -gh run download --pattern '*session-logs*' --dir /tmp/outage -jq -r 'select(.type == "assistant") | .message.content[]?.text // empty' /tmp/outage/*/*/*.jsonl -# → You've hit your session limit · resets 8:30am (UTC) -# → You've hit your weekly limit · resets 12am (UTC) -``` - -A cluster of these is quota exhaustion, not a bug — don't open a fix PR. Two limits produce it and they reset differently, so read the reset off the message rather than assuming the 5-hour session window; a weekly exhaustion can strand most of a day. Record the window's spend in the tracking issue, and re-run the stranded triggers only once a later run has completed cleanly — re-running into a still-exhausted quota just refills the outage issue. - ## CI Fix: Prefer Rerun for Transient Infrastructure Failures Before opening a `fix/ci-*` PR, classify the failure: