Skip to content

skills(running-tend): move outage recovery upstream into tend #3742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
max-sixty merged 2 commits into from
Aug 5, 2026
Merged

skills(running-tend): move outage recovery upstream into tend #3742

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3d856c6
skills(running-tend): name both subscription limits in outage diagnosis
worktrunk-bot Aug 5, 2026
911e84f
skills(running-tend): move outage recovery upstream into tend
worktrunk-bot Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 0 additions & 33 deletions .claude/skills/running-tend/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,39 +83,6 @@ Artifact paths: `-home-runner-work-worktrunk-worktrunk/<session-id>.jsonl`
- `automated-fix` — fix PRs from triage and ci-fix workflows
- `nightly-cleanup` — nightly sweep issues and PRs

## Outage Recovery: Re-run Triggers Stranded by a Failed Session

When `claude -p` exits non-zero the harness opens (or appends to) a `tend-outage`-labelled **"Bot temporarily unavailable"** issue, one table row per failure with the run link and the triggering `#N`. It records the failure; **nothing re-runs it.** `tend-review` fires only on `pull_request_target`, so a PR whose one review attempt died is silently never reviewed until someone happens to push again.

Drain the open outage issue as part of the daily `review-runs` sweep:

```bash
# Empty on most days — no open outage issue means nothing stranded; skip the rest.
OUTAGE=$(gh issue list --state open --label tend-outage --json number --jq '.[0].number // empty')
gh issue view "$OUTAGE" --json body,comments --jq '.body, .comments[].body' \
| grep -oE 'runs/[0-9]+|\| #[0-9]+'
```

Only event-triggered runs strand — `tend-nightly` and `tend-notifications` recover on their next cron tick, so leave those alone. For a `tend-review` / `tend-mention` / `tend-triage` run, confirm the work is still missing before re-running; a later push often re-triggers the workflow on its own, and re-running a job that already happened burns quota for nothing:

```bash
gh pr view <n> --json state,headRefOid,reviews \
--jq '{state, headRefOid, bot: ([.reviews[] | select(.author.login == "worktrunk-bot")] | length)}'
gh run rerun <run-id> --failed
```

Re-running the bot's own failed workflow is restorative, not destructive — no maintainer approval needed. Close the issue once every row is drained (`gh issue close "$OUTAGE"`): the harness only auto-closes duplicates from a create-create race, never the surviving issue, so one left open makes tomorrow's sweep re-check the same rows and folds the next outage into a stale incident.

**Diagnose the cause before calling it an outage.** The issue body says only "The bot failed to process a request". tend's nightly enrichment posts a per-run comment carrying each failed job's failure annotation, which is the cheapest first look — but for this class it reads `claude -p exited non-zero (exit=1) — see the session-logs artifact` (or `claude -p turn ended in failure (…) — rate limit, auth, max turns, or server error` when the CLI exits 0 with an error result), and neither says whether it was quota, auth, or a server error. It's also absent whenever `tend-nightly` was itself one of the stranded runs. To narrow it, read the session log: a subscription session-limit exhaustion surfaces as a `<synthetic>` assistant message:

```bash
gh run download <run-id> --pattern '*session-logs*' --dir /tmp/outage
jq -r 'select(.type == "assistant") | .message.content[]?.text // empty' /tmp/outage/*/*/*.jsonl
# → You've hit your session limit · resets 8:30am (UTC)
```

A cluster of these is quota exhaustion across a 5-hour window, not a bug — don't open a fix PR. Record the window's spend in the tracking issue and re-run the stranded triggers once the window resets.

## CI Fix: Prefer Rerun for Transient Infrastructure Failures

Before opening a `fix/ci-*` PR, classify the failure:
Expand Down
Loading