Skip to content

fix(token-report): raise the per-workflow run limit, and say when it's hit - #887

Merged
max-sixty merged 4 commits into
mainfrom
daily/review-runs-tokenlimit-31160618137
Aug 7, 2026
Merged

fix(token-report): raise the per-workflow run limit, and say when it's hit#887
max-sixty merged 4 commits into
mainfrom
daily/review-runs-tokenlimit-31160618137

Conversation

@tend-agent

@tend-agent tend-agent commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

token-report.sh fetches each workflow's runs with --limit 100. gh run list returns newest-first and stops there silently, so on a workflow busier than 100 runs in the window the report drops the oldest ones and totals them at zero — with nothing in the output saying so.

Measured on this repo

Current 24 h window:

$ gh run list --workflow tend-mention --created ">=$SINCE" --status completed --json databaseId --limit 100 | jq length
100
$ gh run list --workflow tend-mention --created ">=$SINCE" --status completed --json databaseId --limit 400 | jq length
116

16 of 116 tend-mention runs (14%) are outside today's report, and their tokens are simply absent from the totals. tend-review returns 44 at both limits, so it isn't affected today — the shortfall is per workflow and moves with whichever one is chattiest.

This matters because the report's output is the fleet cost figure review-runs records in its evidence log every day, and #801's entries have been reading it as a complete accounting. Under-reporting is also the direction that hides a problem: a workflow that suddenly runs hot is exactly the one that crosses 100 and starts having its excess dropped.

Change

Three things, all small:

  • --limit 1000. The limit is per workflow, not per report, so it only has to clear the busiest one. 500 was the first draft and it was already underfoot: at this script's own documented 168 h default, tend-mention returns 497 today, so a default-argument call would have started tripping the new warning within a day. Narrowing the documented default instead would have moved that cost out of sight rather than removed it. 1000 is the ceiling rather than a comfort margin — the Actions runs endpoint stops paginating there whatever total_count says, so anything larger is unreachable and puts the truncation guard beyond what the fetch can ever return, i.e. buys no runs and costs the warning. It is also the value that makes -ge trip exactly at the ceiling.
  • Warn on an exact hit. A count landing on the limit is the only symptom of truncation visible without re-querying .total_count, so the loop says so on stderr rather than trusting it.
  • Warn on a failed fetch. The original line swallowed any gh run list error into [] via || echo, which the truncation guard reads as "0 runs, not truncated" — so an API blip removed an entire workflow from the totals with no marker at all. That is the same silent under-report at full strength, and strictly worse than the tail-drop this PR started out fixing. Branching on the exit status covers it. Warning rather than exiting, unlike the sibling in list-recent-runs.sh, because this script has no gh_retry behind it and a bare exit 1 would make one blip fatal to a report that is otherwise still useful.

Raising a limit alone would only move the cliff. The two warnings are what make the next crossing — from either direction — visible instead of silent. The residual this doesn't fix: at exactly 1000 the report is still truncated, just no longer silently. Getting the full set past the ceiling needs .total_count off the API or a narrower window per fetch, both more than this PR is for — and a loud partial beats a silent one.

Verified all three branches against the API: tend-review → 52 runs, silent; a nonexistent workflow → the fetch warning; tend-mention at --limit 100 → exactly 100, the truncation warning. The ceiling and the guard's reachability at the new constant, measured on this repo:

$ gh api ".../actions/workflows/250047576/runs?status=completed&per_page=100&page=10" --jq '.workflow_runs | length'
100
$ gh api ".../actions/workflows/250047576/runs?status=completed&per_page=100&page=11" --jq '.workflow_runs | length'
0
$ gh run list --workflow tend-mention --created ">=2000-01-01T00:00:00Z" --status completed --json databaseId --limit 2000 | jq length
1000
$ gh run list --workflow tend-mention --created ">=2000-01-01T00:00:00Z" --status completed --json databaseId --limit 1000 | jq length
1000

total_count for that workflow is 3265, so the 1000 is the endpoint's ceiling and not the window running out. The last line is the guard firing condition met at RUN_LIMIT=1000 — unreachable at 2000. Patched script runs clean end to end (token-report.sh 2 "review-", exit 0, 48 runs, no warnings), and shellcheck is clean.

Provenance and scope

Found by the review on #886 — that PR fixes the same silent-truncation shape in review-runs' Step 1 census (30 of 110 runs, a 68-minute view of a 24-hour window), and the reviewer measured this adjacent case in Step 2 while checking it. Kept separate because it's a different file and a different fetcher.

#838 is doing the equivalent work for list-recent-runs.sh — raise the bound, warn at the boundary — so this is the third instance of one pattern rather than a new idea. Not a dedup hit: different script, different call, no overlap in the diff.

Gate assessment

  • Evidence level: High. Reproduced directly against the API, twice, at two limits. Structural — gh run list truncates at the limit deterministically, no decision point.
  • Change type: targeted fix — one constant and a four-line guard. Normal Gate 1 bar, cleared.
  • Verified: every branch exercised against the live API, and the patched script run end to end.
  • Revised twice after review on this PR. Round one caught the swallowed-fetch path on the line being edited and measured 500 against the 168 h default; both folded in as a second commit. Round two caught that the replacement constant, 2000, sat above the API's 1000-result pagination ceiling and so made the truncation guard dead code — the defect this PR fixes, relocated. Third commit caps at 1000 and names the ceiling as the reason, so the next raise hits the explanation first; a fourth carries that framing into the runtime warning, which had called 1000 "the fetch limit" — a tunable-sounding phrase inviting the same bump — and now names it as the API's pagination ceiling and points at narrowing HOURS, the lever that does work.

…s hit

gh run list returns newest-first and stops at --limit silently, so a
workflow busier than 100 runs in the window has its oldest runs dropped
and their tokens counted as zero. Measured: tend-mention returns 100
against a true 116 over the current 24h window, so 14% of the runs are
missing from a total that review-runs records as a complete accounting.

Raise the per-workflow limit to 500 and warn when a workflow comes back
exactly at it — a count landing on the limit is the only symptom of
truncation visible without re-querying total_count, and raising the
limit alone would just move the cliff.

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing as the author — submitting as a comment, not an approval.

The measurement holds up: tend-mention returns exactly 100 at the old limit and 130 at 500 for the 24 h window the callers actually use, so the truncation is real and the fix addresses it.

Two things.

The fetch-failure path is the same bug, on the line being edited. 2>/dev/null || echo "[]" turns any gh run list error into a zero-run workflow, and the new warning can't see it — a length of 0 is not an exact-limit hit, so a transient API error on the chattiest workflow drops all of its runs and the totals still print as a complete accounting. That's strictly worse than the truncation this PR fixes (100% of one workflow rather than its tail), and it's the direction the PR body itself names as the one that hides a problem. list-recent-runs.sh already refuses this: echo "ERROR: 'gh run list' for workflow '$wf' failed after retries — refusing to report a partial run list" >&2; exit 1. Inline suggestion warns rather than exits, matching the posture this PR already chose for truncation; exiting 1 to match the sibling is the defensible alternative, but token-report.sh has no gh_retry, so a bare exit 1 would be more brittle than the sibling's.

500 leaves no headroom at the script's own default lookback. HOURS defaults to 168 per the usage header. At that window today:

$ gh run list --workflow tend-mention --created ">=$(date -u -d '168 hours ago' +%Y-%m-%dT%H:%M:%SZ)" --status completed --json databaseId --limit 900 | jq length
495

Five short of the new limit, so a default-argument invocation trips the new warning within about a day. Every caller in the repo passes 24, so nothing breaks now — but "raising a limit alone would only move the cliff" applies to this number too, and at 168 h the cliff is already underfoot. The per-run gh run download loop is the reason a bigger number isn't free: it's sequential, one network round-trip per run, so 500 runs is a real cost at that lookback rather than the no-op the listing call is. Worth either raising the constant with that in view, or narrowing the documented default to the window the script is actually sized for.

Also folded into the suggestion: -ge rather than -eq, matching #838's version of the same guard.

Not a dedup concern — #838 (list-recent-runs.sh) and #886 (review-runs census) are the same pattern in different files, as the body says. #873 does touch this file, but in disjoint hunks.

Comment thread plugins/tend-ci-runner/scripts/token-report.sh Outdated
… the 168h default

Review on this PR found the guard incomplete in two ways.

A failed `gh run list` resolved to `[]` via `|| echo`, which the new
truncation guard reads as "0 runs, not truncated" — so an API blip drops
a whole workflow from the totals with no marker, the same silent
under-report at full strength. Branch on the exit status and warn.

500 also left no headroom at the script's own documented 168h default:
tend-mention returns 497 over that window today, so a default-argument
call would trip the truncation warning within a day. The limit only
bounds the listing call, which pages internally, so headroom is free —
the per-run download loop costs what the window holds either way.

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing as the author — comment, not an approval.

Both points from the previous round landed as written, and the failed-fetch branch is right. One problem with the new constant.

RUN_LIMIT=2000 puts the truncation guard above the API's ceiling, so it can never fire. The Actions runs endpoint stops paginating at 1000 results regardless of total_count, so gh run list --limit 2000 returns at most 1000 and [ "$(… | jq 'length')" -ge "$RUN_LIMIT" ] is unreachable. Measured on this repo with the script's own call shape:

$ gh api ".../actions/workflows/250047576/runs?status=completed&per_page=1" --jq .total_count
3256
$ gh api ".../runs?status=completed&per_page=100&page=10" --jq '.workflow_runs | length'
100
$ gh api ".../runs?status=completed&per_page=100&page=11" --jq '.workflow_runs | length'
0
$ gh run list --workflow tend-mention --created ">=2000-01-01T00:00:00Z" --status completed \
    --json databaseId,conclusion,createdAt,name --limit 2000 | jq length
1000

That reintroduces the exact defect this PR fixes, just relocated: a workflow past 1000 runs in the window drops the excess and the guard stays silent, which is the "raising a limit alone would only move the cliff" failure the body argues against — except now the marker that was supposed to make the next crossing visible is dead code. 1000 is both the largest reachable value and the one that makes -ge fire exactly at the ceiling, so the report says so instead of quietly under-reporting.

The "headroom here is free" rationale in the comment goes with it: above 1000 headroom isn't free, it's unreachable, and buying it costs the warning. Second suggestion rewrites those four lines to name the ceiling as the reason for the constant, so the next person raising it hits the explanation first.

Worth noting the residual this doesn't fix — at exactly 1000 the report is still truncated, just no longer silently. Getting the full set needs .total_count off the API or a narrower window per fetch; both are more than this PR is for, and a loud partial beats a silent one.

Nothing else in the diff. if ! runs=$(…) is set -e-safe as an if condition, unique_by sorts internally so the merge is fine, and gh stops paging once the window is exhausted, so the larger constant costs nothing on a normal run.

Comment thread plugins/tend-ci-runner/scripts/token-report.sh Outdated
Comment thread plugins/tend-ci-runner/scripts/token-report.sh Outdated
The Actions runs endpoint stops paginating at 1000 whatever total_count
says, so RUN_LIMIT=2000 made the truncation guard unreachable — the
larger constant bought no runs and cost the warning. 1000 is the largest
reachable value and trips the guard exactly at the ceiling.
… limit won't help

The truncation warning called 1000 'the fetch limit', which reads as a
tunable and invites exactly the 2000 bump the previous commit reverted.
Name it as the Actions API's pagination ceiling and point at the lever
that does work — narrowing HOURS.

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing as the author — comment, not an approval (GitHub blocks self-approval).

The round-two concern is addressed: RUN_LIMIT=1000 puts the guard exactly at the endpoint's ceiling, so -ge is reachable rather than dead code, and the comment now names the ceiling as the reason for the constant. Both threads resolved. The new warning text points at HOURS, which is the script's own first positional argument, so the remedy it names is the one a reader can act on.

Nothing further from me on the incremental — no new findings.

@max-sixty
max-sixty merged commit b184c4b into main Aug 7, 2026
9 checks passed
@max-sixty
max-sixty deleted the daily/review-runs-tokenlimit-31160618137 branch August 7, 2026 22:57
max-sixty pushed a commit that referenced this pull request Aug 7, 2026
…889)

Step 1 of `review-runs` enumerates the fleet with a hard-coded single
prefix, while Step 2 documents the opposite — that non-`tend-*`
workflows using the tend action are in scope and their prefixes come
from the repo's `running-tend` skill. The two steps therefore disagree
about what the fleet is, and Step 1 wins: a workflow outside the `tend-`
prefix is never classified for duration, never near-timeout-checked, and
never reaches Step 3's log analysis. Nothing in the output marks the
omission.

## Measured on this repo

Over a 24 h window (`SINCE=2026-08-06T08:42:07Z`), `review-reviewers` —
which runs the tend composite action but is not named `tend-*`, and
which tend's own `running-tend` overlay already lists as an extra prefix
for Step 2 — has 21 completed runs that Step 1 never sees:

```
$ gh api repos/max-sixty/tend/actions/workflows --jq '.workflows[] | select(.name | startswith("tend-")) | .name' | grep -c '^review-reviewers$'
0
$ gh api "repos/max-sixty/tend/actions/workflows/250009605/runs?created=>=$SINCE&status=completed&per_page=1" --jq '.total_count'
21
```

## Change

Replace the inline `startswith("tend-")` with a `PREFIXES` array
defaulting to `("tend-")`, matched as an anchored alternation. Step 2's
sentence now points at the same list rather than describing a parallel
one, so a single repo-level source drives both steps.

Behaviour is unchanged for an adopter with no extra prefixes — the
default array reproduces the old filter exactly. Running the edited Step
1 block verbatim:

```
# as written, default PREFIXES=("tend-")
tend-review 30, tend-mention 30, tend-notifications 26, tend-ci-fix 9, tend-triage 6, tend-nightly 1

# with tend's running-tend prefix list, PREFIXES=("tend-" "review-")
tend-review 30, tend-mention 30, tend-notifications 26, review-reviewers 20, tend-ci-fix 9, tend-triage 6, tend-nightly 1
```

The `30`s in that output are #886's separate bug (the unpaginated
endpoint capping at a page), still live on `main` — this change doesn't
address it and doesn't depend on it.

## Scope and conflict note

This is part 2 of #888. Part 1 of that issue — `token-report.sh` capping
the same window at `--limit 100` — is already fixed by #887, so nothing
here touches that script.

#886 edits the same two `gh api` lines in this block to add
`--paginate`. The changes are independent in intent but overlap
textually, so whichever lands second needs a trivial rebase; the two
edits compose (a `PREFIXES`-driven filter on a paginated fetch).

Not included: an explicit prefix-list line in tend's own
`.claude/skills/running-tend/SKILL.md`. Its "Usage analysis" section
already names `review-` as the extra prefix, so this repo's Step 1
resolves correctly today; adding a dedicated line there is a separate
overlay concern.

## Gate assessment

- **Structural.** The filter is fixed in the recipe text, so it excludes
identically on every run. No decision point.
- **Evidence: High.** Exclusion reproduced directly against the API, and
the fix verified by executing the edited block verbatim at both prefix
lists.
- **Change type: targeted fix** — one code block plus one
cross-reference sentence, no new sections.

---
Refs #888


## Follow-up commits

`3d4fcda` — review on this PR found the same disagreement one paragraph
later: the near-timeout instruction resolved a workflow's
`timeout-minutes` by globbing `.github/workflows/tend-*.yaml`, which
doesn't match the very workflow the widened census now admits. It now
reads the workflow's own file. `f823494` reworded the `PREFIXES` comment
from rationale into an instruction.

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-runs Findings from review-runs skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants