diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md index 2126d7964..917df94fc 100644 --- a/.claude/skills/running-tend/SKILL.md +++ b/.claude/skills/running-tend/SKILL.md @@ -83,6 +83,42 @@ Artifact paths: `-home-runner-work-worktrunk-worktrunk/.jsonl` - `automated-fix` — fix PRs from triage and ci-fix workflows - `nightly-cleanup` — nightly sweep issues and PRs +## Outage Recovery: Re-run Triggers Stranded by a Failed Session + +_Temporary hold: [max-sixty/tend#851](https://github.com/max-sixty/tend/pull/851) moves this into the bundled `review-runs` skill. Drop this section once that lands._ + +When `claude -p` exits non-zero the harness opens (or appends to) a `tend-outage`-labelled **"Bot temporarily unavailable"** issue, one table row per failure with the run link and the triggering `#N`. It records the failure; **nothing re-runs it.** `tend-review` fires only on `pull_request_target`, so a PR whose one review attempt died is silently never reviewed until someone happens to push again. + +Drain the open outage issue as part of the daily `review-runs` sweep: + +```bash +# Empty on most days — no open outage issue means nothing stranded; skip the rest. +OUTAGE=$(gh issue list --state open --label tend-outage --json number --jq '.[0].number // empty') +gh issue view "$OUTAGE" --json body,comments --jq '.body, .comments[].body' \ + | grep -oE 'runs/[0-9]+|\| #[0-9]+' +``` + +Only event-triggered runs strand — `tend-nightly` and `tend-notifications` recover on their next cron tick, so leave those alone. For a `tend-review` / `tend-mention` / `tend-triage` run, confirm the work is still missing before re-running; a later push often re-triggers the workflow on its own, and re-running a job that already happened burns quota for nothing: + +```bash +gh pr view --json state,headRefOid,reviews \ + --jq '{state, headRefOid, bot: ([.reviews[] | select(.author.login == "worktrunk-bot")] | length)}' +gh run rerun --failed +``` + +Re-running the bot's own failed workflow is restorative, not destructive — no maintainer approval needed. Close the issue once every row is drained (`gh issue close "$OUTAGE"`): the harness only auto-closes duplicates from a create-create race, never the surviving issue, so one left open makes tomorrow's sweep re-check the same rows and folds the next outage into a stale incident. + +**Diagnose the cause before calling it an outage.** The issue body says only "The bot failed to process a request". tend's nightly enrichment posts a per-run comment carrying each failed job's failure annotation, which is the cheapest first look — but for this class it reads `claude -p exited non-zero (exit=1) — see the session-logs artifact` (or `claude -p turn ended in failure (…) — rate limit, auth, max turns, or server error` when the CLI exits 0 with an error result), and neither says whether it was quota, auth, or a server error. It's also absent whenever `tend-nightly` was itself one of the stranded runs. To narrow it, read the session log: a subscription quota exhaustion surfaces as a `` assistant message: + +```bash +gh run download --pattern '*session-logs*' --dir /tmp/outage +jq -r 'select(.type == "assistant") | .message.content[]?.text // empty' /tmp/outage/*/*/*.jsonl +# → You've hit your session limit · resets 8:30am (UTC) +# → You've hit your weekly limit · resets 12am (UTC) +``` + +A cluster of these is quota exhaustion, not a bug — don't open a fix PR. The two limits reset on different clocks, so read the reset off the message rather than assuming the shorter session window; a weekly exhaustion can strand most of a day. Record the window's spend in the tracking issue, and gate the re-runs on a later run that already succeeded rather than on an assumed clock — re-running inside a still-exhausted window just appends another row to the issue you are draining. + ## CI Fix: Prefer Rerun for Transient Infrastructure Failures Before opening a `fix/ci-*` PR, classify the failure: @@ -329,6 +365,20 @@ cargo run --release -- config state logs profile --format=json | jq .cache The `.cache` report flags commands invoked more than once with the same context. Triage each duplicate: +- **`-vv` epilogue artifact** — check this first; it explained every duplicate + the report flagged on the most recent run, and all but one (`git config + --list -z`, a real miss fixed in #3705) on the run before. The `-vv` this + recipe requires writes the diagnostic bundle *after* the render, and + assembling it re-runs two commands the render already made: + `git worktree list --porcelain` (`DiagnosticReport::format_report` in + `src/diagnostic.rs`) and `gh --version` for the gist hint + (`is_gh_installed`). Both go through `Cmd`, so they land in the same + `trace.jsonl` the `.cache` report reads and pair with the render's own + calls. `jq .cache` carries only commands, counts and durations, so confirm + against the trace itself: `jq -c 'select(.cmd == "gh --version") | {tid, ts}' + "$(git rev-parse --git-common-dir)/wt/logs/trace.jsonl"`. The epilogue runs + on the main thread after the render's worker threads finish, so a differing + `tid` and a `ts` past the render's end identify it. Nothing to file. - **Legitimate** (different cwd, different ref form that can't be normalized, intentional double-call across phases) — note in the response and move on. - **Cache miss** (same logical operation should hit cache but doesn't) —