Skip to content

fix(list-recent-runs): recover the window on the last successful run, and stop the fetch limit re-truncating it - #838

Open
tend-agent wants to merge 3 commits into
mainfrom
hourly/review-30962483562-numbagg
Open

fix(list-recent-runs): recover the window on the last successful run, and stop the fetch limit re-truncating it#838
tend-agent wants to merge 3 commits into
mainfrom
hourly/review-30962483562-numbagg

Conversation

@tend-agent

@tend-agent tend-agent commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

list-recent-runs.sh's dropped-tick recovery anchors the window floor on the previous completed run of the analyzing workflow. completed is a status, not a conclusion, so a run that failed before its agent produced any analysis advances the anchor exactly as if it had covered its hour. After an outage the next green run therefore resumes one tick back and reports an all-clear for a window nothing ever looked at — and the gap is unrecoverable, because the following run's floor advances past it too.

This is not hypothetical: it just consumed ~15 hours of review-reviewers coverage.

What happened

Every review-reviewers run from 30897445507 (09:41Z) through 30959115609 (23:10Z) failed — 16 consecutive runs, all five matrix legs, claude -p exiting 1 after ~4s. The result event in every leg's session log is identical: is_error: true, num_turns: 1, result: "You've hit your weekly limit · resets 12am (UTC)", with token-usage.json all zeros. The agent never started, so none of those runs analyzed anything. (The outage itself is already tracked by #831 and the annotation-legibility work in #816 / #818 — this PR is only about the window the outage swallowed.)

The last run that did analyze anything is 30893072876 at 08:42Z, which is also the last entry in the numbagg/numbagg evidence gist. At this run's tick (23:47Z intended), the recovery query picked {"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609} and floored the window at 22:47Z, so list-recent-runs.sh returned exactly one row: a tend-notifications pre-check no-op.

Everything numbagg-bot actually did in the swallowed window sat outside that floor — PRs #719 and #720, issue #721, three reviews and four inline comments on #719, and comments on #716, #721 and #1. That is the densest window numbagg has had in days, and per the skill's Step 1 ("If empty, record the run as all-clear … then skip to Step 6") it would have been recorded as a quiet hour. I only found it by diffing the run list against the gist's last recorded boundary by hand.

The change

Anchor on the previous successful run rather than any completed one, filtering server-side with --status success (the flag takes conclusions as well as statuses), and when the existing 6h cap clamps the recovered floor, say so on stderr so the caller records a coverage gap instead of a false all-clear. A partially-failed matrix run counts as a failure here, which only ever widens the window — overlap is re-offered work the caller dedups against its own evidence log, whereas a gap is silently unanalyzed.

Widening the window then exposed a second truncation one layer down, so the fetch loop moves with it. It passed --limit 50 while a recovered window spans up to 8h, and gh run list returns newest-first — so a workflow over the limit silently drops its oldest runs, which are exactly the ones in the gap the anchor just reached back for. On max-sixty/tend, tend-mention alone produces 57 runs in an 8h window. The limit is now 200, and a workflow returning exactly the limit warns rather than truncating in silence.

Two smaller pieces: the anchor query routes through gh_retry and exits non-zero instead of 2>/dev/null || true, so a transient API error can no longer masquerade as "no successful run" and emit a confident warning naming a cause that didn't happen; and review-reviewers/SKILL.md, which still said "if empty, record all-clear" with nothing about the stderr warnings, now treats any WARNING: as a coverage gap — without that, an agent following the skill literally would print the warning and record an all-clear anyway.

When every tick fires and succeeds the anchor is the previous tick, prev_intended == intended - 3600 == COMPLETED_AFTER, and the comparison is a no-op — output is byte-identical to today's on the healthy path.

Verification against live data

Same tick, same repo, before vs. after:

# anchor query — before (--limit 10, no conclusion filter)
{"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609}

# anchor query — after (--limit 50, conclusion == "success")
{"conclusion":"success","createdAt":"2026-08-04T08:42:12Z","databaseId":30893072876}

TARGET_REPO=numbagg/numbagg ./list-recent-runs.sh after the change returns 5 runs instead of 1, preceded by:

WARNING: the last successful 'review-reviewers' run started 2026-08-04T08:42:12Z, more than 6h back. Window floored at 2026-08-04T17:47:00Z; runs that completed before it are NOT in this list. Record a coverage gap, not an all-clear.

The recovered anchor (08:42:12Z) matches the gist's last recorded coverage boundary exactly. Also exercised: the no-successful-run branch (warns, floors at the cap, exits 0) and the unset-GITHUB_WORKFLOW branch (recovery skipped, unchanged). shellcheck clean; bash -n clean. Also exercised after the review follow-ups: the fetch-limit truncation warning (via a copy with RUN_LIMIT=5) and the anchor-query failure path (exits 1 rather than degrading). See the follow-up comment for the full branch table.

Gates

Evidence log: https://gist.github.com/19b5ab297bb7ac7e1e9a44d595ccde0f

`--status completed` includes failures, so a run that died before its agent
produced any analysis advanced the recovery anchor as if it had covered its
hour. After an outage the next green run resumed one tick back and reported an
all-clear for a window nothing had looked at.

Anchor on the previous successful run, scan deeper than one cron period (during
an outage the recent completed runs are all failures), and announce on stderr
when the 6h cap clamps the recovered floor so the caller records a coverage gap
instead of a false all-clear.
@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 5, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things on the recovery path, plus a smaller note. Reviewing my own PR, so this is a COMMENT.

The widened window isn't matched by the per-workflow fetch limit. The recovery now routinely pushes COMPLETED_AFTER to intended - 6h on exactly the outage path this PR targets, which makes CREATED_SINCE reach intended - 8h. But the fetch loop that actually collects the runs — the gh_retry gh run list ... --created ">=${CREATED_SINCE}" call guarded by refusing to report a partial run list, outside this diff — still passes --limit 50. gh run list returns newest-first, so more than 50 runs in that window silently drops the oldest ones: precisely the runs sitting in the recovered gap the anchor just reached back for. Measured just now on max-sixty/tend, which is itself a matrix leg in review-reviewers.yaml: tend-mention alone produced 48 runs in the last 8h, i.e. one workflow at the cap before the other seven are counted. Unlike the 6h clamp, this truncation prints nothing, so the caller records the all-clear this PR exists to prevent. gh paginates past 100 (--limit 150 returns 150 here), so raising the limit is cheap; warning when a workflow comes back with exactly --limit rows would turn a silent truncation into a visible one. Happy to push that as a commit here if you'd rather it land together.

gh run list can do the conclusion filter server-side — inline suggestion below. --status success is accepted and returns the anchor directly: I ran it against review-reviewers and got run 30893072876 at 2026-08-04T08:42:12Z, the same anchor the "after" block in the description shows. That drops the client-side conclusion filter and, more usefully, the scan-depth heuristic with it — the "50 completed runs might all be failures" reasoning stops applying, and an outage longer than 50 runs can no longer defeat the anchor. It does need the comment paragraph about the scan reaching past one cron period trimmed, and the else branch's wording changed from "among the last 50 completed ones" to something like "at all".

Smaller: the anchor query keeps 2>/dev/null || true, so a transient gh failure is indistinguishable from a genuine "no successful run" — and now it produces a confidently-worded warning naming a cause that didn't happen, plus a 6h window. The script's other gh calls deliberately route through gh_retry and exit non-zero rather than degrade, on the same reasoning this PR is applying.

Comment thread plugins/tend-ci-runner/scripts/list-recent-runs.sh Outdated
…ndow

Review follow-ups on the recovery path:

- The fetch loop still passed `--limit 50` while recovery routinely widens
  the window to 8h. `gh run list` is newest-first, so a workflow over the
  limit silently drops its *oldest* runs — exactly the ones in the gap the
  anchor reached back for, restoring the false all-clear this PR exists to
  prevent. `tend-mention` alone produced 57 runs in an 8h window. Raise to
  200 and warn when a workflow returns exactly the limit, so a truncation
  is visible rather than silent.
- Filter the anchor query server-side with `--status success` (it accepts
  conclusions as well as statuses). The scan-depth heuristic goes away with
  it, so an outage longer than the scan can no longer bury the anchor.
- Route the anchor query through `gh_retry` and fail loud. It kept
  `2>/dev/null || true`, so a transient API error was indistinguishable
  from "no successful run" and produced a confident warning naming a cause
  that hadn't happened.
- The consuming skill still said "if empty, record all-clear", which
  contradicts every warning above; note that a WARNING means a coverage
  gap regardless of whether the list came back empty.
@tend-agent

Copy link
Copy Markdown
Collaborator Author

All three taken, pushed as fb26376. The fetch-limit one was the important catch — it would have restored the exact failure this PR exists to prevent, one layer down.

Fetch limit. Reproduced the measurement: over an 8h window on max-sixty/tend, tend-mention alone returns 57 runs, so the old --limit 50 was already truncating one workflow before the other seven were counted. Since gh run list is newest-first, the dropped rows are the oldest — i.e. the recovered gap itself, so recovery would reach back for a window and then hand back a list with that window cut off. Raised to 200 and added a warning when a workflow comes back with exactly the limit. That warns rather than exits: unlike a failed fetch, the rows in hand are still worth analyzing — the caller just can't read the list as complete.

Anchor query. Both applied as suggested — see the inline reply.

gh_retry on the anchor. Agreed, and it was worse with this PR than before it: the old code degraded to a silent one-hour window, whereas the new one degrades to a confidently-worded warning naming a cause that didn't happen. Now routed through gh_retry, exiting 1 with refusing to guess a floor that would read as a false all-clear. A genuine empty result still takes the no-anchor branch — --jq … // empty exits 0 with empty stdout, so the two cases are now distinguishable.

One thing the review didn't raise, folded in because it's the same false all-clear: the consuming skill still said "If empty, record the run as all-clear … then skip to Step 6", with nothing about the stderr warnings. An agent following it literally would print WARNING: … Record a coverage gap and then record an all-clear anyway. review-reviewers/SKILL.md now says a WARNING: means a coverage gap regardless of whether the list came back empty.

Branch verification (live, this checkout)

shellcheck and bash -n clean; pre-commit clean on both files.

The anchor is recent again as of this run, so the recovery path needs the just-succeeded run excluded to reproduce — that's GITHUB_RUN_ID=30962483562 below.

Branch Invocation Result
Recovery + 6h clamp GITHUB_RUN_ID=30962483562, TARGET_REPO=numbagg/numbagg anchors 2026-08-04T08:42:12Z, warns, returns 5 rows — matches the description's "after"
Healthy path same, GITHUB_RUN_ID=0 anchor is the 00:11Z run, no warning, no widening
Truncation copy with RUN_LIMIT=5, TARGET_REPO=max-sixty/tend warns per capped workflow (tend-mention, tend-review, tend-notifications)
No anchor at all stubbed gh returning empty for the anchor query warns, floors at cap, exits 0
Anchor query fails GITHUB_WORKFLOW=does-not-exist ERROR: … refusing to guess a floor, exits 1

At the real limit of 200, nothing on max-sixty/tend comes close over an 8h window — the busiest workflow is the 57 above.

@tend-agent tend-agent changed the title fix(list-recent-runs): anchor window recovery on the last successful run fix(list-recent-runs): recover the window on the last successful run, and stop the fetch limit re-truncating it Aug 5, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One finding on the anchor query, otherwise the follow-up commit holds up. Reviewing my own PR, so this is a COMMENT.

I exercised each branch against this repo rather than reading them: the normal path returns 55 runs with no warnings; --status success is a valid conclusion filter (it's in the flag's own value list) and an existing workflow with zero matching runs returns [] at exit 0, so the "no successful run" branch is reachable rather than shadowed by the new exit 1; the clamp branch fires correctly on tend-weekly (last success 2026-08-02, floored at 2026-08-04T17:47:00Z); and forcing RUN_LIMIT=5 produces the truncation warning per workflow. The limit raise is well sized — tend-mention returned 80 runs over an 8h window just now, so 50 was truncating by a third and 200 leaves real headroom.

Comment thread plugins/tend-ci-runner/scripts/list-recent-runs.sh Outdated
A `workflow_dispatch` run has no `.schedule` in its event payload, so it
takes the non-cron branch and covers a now-anchored 1h window rather than a
tiled one. Anchoring on it therefore floors the next scheduled run just past
a dispatch that never covered the gap — and unlike the clamp, it warns about
nothing, because the anchor looks recent. Dispatching the workflow by hand
to check on a fix mid-outage is the natural thing to do and is exactly when
that bites, so it's the same class of hole as anchoring on a failure.

Filter the anchor query with `--event schedule`. No-op today (no dispatch
runs on record for `review-reviewers`, which does declare
`workflow_dispatch:`), so it only bites the case above.
@tend-agent

Copy link
Copy Markdown
Collaborator Author

Second independent occurrence, from a different cause — reproduced live on this tick, so recording it here rather than opening a divergent PR.

The motivating case was the weekly-limit outage: claude -p exiting 1 after the agent never started. Today's is infrastructure. GitHub opened a critical Actions incident at 2026-08-06T15:22:49Z (components: Actions, Pages) that is still unresolved as I write this, and five consecutive review-reviewers runs died in it — 31119653960 (16:23Z) failed at Set up job with Failed to resolve action download info. / Service Unavailable, and 31122877870, 31125649607, 31126978811 and 31128259981 each had init-tracking cancelled with zero steps and the annotation The job was not acquired by Runner of type hosted even after multiple attempts, which skipped the whole matrix. Different mechanism, identical consequence: five runs that analyzed nothing, each advancing the anchor exactly as if they had.

The anchor query on this tick, run both ways against live data:

# current: --status completed, --limit 10
{"conclusion":"failure","createdAt":"2026-08-06T21:22:19Z","databaseId":31128259981}

# this PR: --status success
{"conclusion":"success","createdAt":"2026-08-06T15:32:18Z","databaseId":31116304881}

The failed 21:22Z run floors the window at 20:47Z, so list-recent-runs.sh returned 2 rows. Six tend-notifications runs on max-sixty/cargo-affected completed between 16:12Z and 19:22Z — four of them cancelled with the same runner-acquisition annotation, two still stuck in status: waiting five hours after creation with pending_deployments showing wait_timer: 0 and no reviewers — and none of them were in any leg's window. 31116304881 is also the last entry in this target's evidence gist, so the recovered anchor again matches the gist's last coverage boundary exactly.

Worth noting what the cap does here, since this run exercises the branch the PR added rather than the happy path: 15:32:18Z is more than 6h before the 22:47Z tick, so prev_intended clamps to 16:47Z and the WARNING: fires. That is the design working — the leg records a coverage gap instead of a false all-clear — but the real gap is ~8h (last successful analysis 15:45Z, this tick 23:45Z), and today's incident has now been open 8.5h, so a 6h cap under-covers an incident-length outage by a couple of hours in practice. Not asking for a change; just the datapoint, since the cap was picked before there was a case that exceeded it.

Substantive impact this time was nil, which is only luck: max-sixty/cargo-affected saw zero mutations of any kind across the whole 15:45Z–23:45Z window (issues?state=all&sort=updated, repo-level issues/comments and pulls/comments, and commits?since= all returned empty), so the swallowed window held nothing to analyze. The numbagg case showed what it costs when the window isn't idle.

Evidence log: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb

@tend-agent

Copy link
Copy Markdown
Collaborator Author

The max-sixty/worktrunk leg of the same tick hit this too, and it supplies the one thing the comment above says it was missing: "Substantive impact this time was nil, which is only luck… the swallowed window held nothing to analyze." On worktrunk it wasn't idle. The window the failed-run anchor swallowed (15:35Z, the last recorded coverage boundary, → the 20:47Z floor) contained the day's only public bot artefact — a 6.9 KB comment on max-sixty/worktrunk#3757 at 16:40Z, whose six external claims and six links I verified live only because this leg backfilled by hand — plus the tend-triage run for that issue dying in Set up job, and six more runs cancelled or stuck waiting. Reported as a one-hour all-clear, all of that would have gone unexamined.

That also puts a number on the cap question raised above without needing a change: with --status success the anchor is 31116304881 (15:32Z), clamped to 16:47Z, so four of the five lost hours come back and the WARNING flags the fifth. Under-covering by an hour while saying so is a different failure than covering five hours silently. Third datapoint for the --event schedule half, too: 31119653960 has been sitting in waiting since 16:23Z, which is exactly the state that invites a hand-dispatch to check on things mid-incident.

Evidence log: https://gist.github.com/a88c03f4d0c3fb1791060ff3dd97d1c4

max-sixty pushed a commit that referenced this pull request Aug 7, 2026
…s hit (#887)

`token-report.sh` fetches each workflow's runs with `--limit 100`. `gh
run list` returns newest-first and stops there silently, so on a
workflow busier than 100 runs in the window the report drops the oldest
ones and totals them at zero — with nothing in the output saying so.

## Measured on this repo

Current 24 h window:

```
$ gh run list --workflow tend-mention --created ">=$SINCE" --status completed --json databaseId --limit 100 | jq length
100
$ gh run list --workflow tend-mention --created ">=$SINCE" --status completed --json databaseId --limit 400 | jq length
116
```

16 of 116 `tend-mention` runs (14%) are outside today's report, and
their tokens are simply absent from the totals. `tend-review` returns 44
at both limits, so it isn't affected today — the shortfall is per
workflow and moves with whichever one is chattiest.

This matters because the report's output is the fleet cost figure
`review-runs` records in its evidence log every day, and #801's entries
have been reading it as a complete accounting. Under-reporting is also
the direction that hides a problem: a workflow that suddenly runs hot is
exactly the one that crosses 100 and starts having its excess dropped.

## Change

Three things, all small:

- **`--limit 1000`.** The limit is per workflow, not per report, so it
only has to clear the busiest one. 500 was the first draft and it was
already underfoot: at this script's own documented 168 h default,
`tend-mention` returns **497** today, so a default-argument call would
have started tripping the new warning within a day. Narrowing the
documented default instead would have moved that cost out of sight
rather than removed it. 1000 is the ceiling rather than a comfort margin
— the Actions runs endpoint stops paginating there whatever
`total_count` says, so anything larger is unreachable *and* puts the
truncation guard beyond what the fetch can ever return, i.e. buys no
runs and costs the warning. It is also the value that makes `-ge` trip
exactly at the ceiling.
- **Warn on an exact hit.** A count landing on the limit is the only
symptom of truncation visible without re-querying `.total_count`, so the
loop says so on stderr rather than trusting it.
- **Warn on a failed fetch.** The original line swallowed any `gh run
list` error into `[]` via `|| echo`, which the truncation guard reads as
"0 runs, not truncated" — so an API blip removed an entire workflow from
the totals with no marker at all. That is the same silent under-report
at full strength, and strictly worse than the tail-drop this PR started
out fixing. Branching on the exit status covers it. Warning rather than
exiting, unlike the sibling in `list-recent-runs.sh`, because this
script has no `gh_retry` behind it and a bare `exit 1` would make one
blip fatal to a report that is otherwise still useful.

Raising a limit alone would only move the cliff. The two warnings are
what make the next crossing — from either direction — visible instead of
silent. The residual this doesn't fix: at exactly 1000 the report is
still truncated, just no longer silently. Getting the full set past the
ceiling needs `.total_count` off the API or a narrower window per fetch,
both more than this PR is for — and a loud partial beats a silent one.

Verified all three branches against the API: `tend-review` → 52 runs,
silent; a nonexistent workflow → the fetch warning; `tend-mention` at
`--limit 100` → exactly 100, the truncation warning. The ceiling and the
guard's reachability at the new constant, measured on this repo:

```
$ gh api ".../actions/workflows/250047576/runs?status=completed&per_page=100&page=10" --jq '.workflow_runs | length'
100
$ gh api ".../actions/workflows/250047576/runs?status=completed&per_page=100&page=11" --jq '.workflow_runs | length'
0
$ gh run list --workflow tend-mention --created ">=2000-01-01T00:00:00Z" --status completed --json databaseId --limit 2000 | jq length
1000
$ gh run list --workflow tend-mention --created ">=2000-01-01T00:00:00Z" --status completed --json databaseId --limit 1000 | jq length
1000
```

`total_count` for that workflow is 3265, so the 1000 is the endpoint's
ceiling and not the window running out. The last line is the guard
firing condition met at `RUN_LIMIT=1000` — unreachable at 2000. Patched
script runs clean end to end (`token-report.sh 2 "review-"`, exit 0, 48
runs, no warnings), and `shellcheck` is clean.

## Provenance and scope

Found by the review on #886 — that PR fixes the same silent-truncation
shape in `review-runs`' Step 1 census (30 of 110 runs, a 68-minute view
of a 24-hour window), and the reviewer measured this adjacent case in
Step 2 while checking it. Kept separate because it's a different file
and a different fetcher.

#838 is doing the equivalent work for `list-recent-runs.sh` — raise the
bound, warn at the boundary — so this is the third instance of one
pattern rather than a new idea. Not a dedup hit: different script,
different call, no overlap in the diff.

## Gate assessment

- **Evidence level: High.** Reproduced directly against the API, twice,
at two limits. Structural — `gh run list` truncates at the limit
deterministically, no decision point.
- **Change type: targeted fix** — one constant and a four-line guard.
Normal Gate 1 bar, cleared.
- **Verified**: every branch exercised against the live API, and the
patched script run end to end.
- **Revised twice after review** on this PR. Round one caught the
swallowed-fetch path on the line being edited and measured 500 against
the 168 h default; both folded in as a second commit. Round two caught
that the replacement constant, 2000, sat above the API's 1000-result
pagination ceiling and so made the truncation guard dead code — the
defect this PR fixes, relocated. Third commit caps at 1000 and names the
ceiling as the reason, so the next raise hits the explanation first; a
fourth carries that framing into the runtime warning, which had called
1000 "the fetch limit" — a tunable-sounding phrase inviting the same
bump — and now names it as the API's pagination ceiling and points at
narrowing `HOURS`, the lever that does work.

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant