Skip to content

feat(rate-limit): let a maintainer approve past the spike limit - #874

Merged
max-sixty merged 11 commits into
mainfrom
pr-851-debug
Aug 6, 2026
Merged

feat(rate-limit): let a maintainer approve past the spike limit#874
max-sixty merged 11 commits into
mainfrom
pr-851-debug

Conversation

@max-sixty

@max-sixty max-sixty commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Problem

The spike rate limit is a circuit breaker with no reset: once tripped, the only cure is the UTC rollover. A legitimate nightly sweep — six PRs filed between 00:20 and 00:24 — pushed the day's count to 16 against a limit of 15, and every tend-review run from 11:47 UTC onward died at the preflight. Six hours with no reviews, and no signal anywhere a person would look: a failed preflight writes an annotation onto a job nobody opens. #851's own review was one of the ones stranded.

Two separable gaps: no notice that the bot had stopped, and no reset short of waiting for midnight.

Solution

On a spike trip the preflight files or reopens a tend-rate-limit issue listing the runs it refused. Opening it is the notice. Closing it is the approval, and doubles the ceiling for the rest of the UTC day; each further close doubles again, so the breaker re-arms after use rather than switching off.

A doubling rather than a flat increment because the ceiling it lifts is itself proportional — 10 + 2 * daily_avg is 15 here but 40 on a repo filing 15 a day, where a flat bump would mean closing the issue all afternoon. At the formula's floor the two coincide (10 doubles to 20), so they diverge only where a flat bump stops working.

The burst limit stays non-resumable: ten PRs in twenty minutes is a loop, not a busy day, and there is nothing there to wave through. ABORT is set true at three points and never cleared, so an approval cannot release a burst trip — and a burst trip files no issue at all, since one offering to lift the ceiling would promise a recovery that closing it cannot deliver.

Approval is a check, not an instruction the bot could be talked out of. The preflight counts only closes whose actor is neither the bot nor a GitHub App. GitHub admits only an issue's author or a triage/write collaborator to close it, and the bot is the author, so what remains is exactly the maintainers — no allowlist to keep.

Three things pin that down, because the bot holds issues: write and could otherwise arrange an approval for itself:

  • The anchor is an issue the preflight filed — lowest-numbered, authored by the bot, under the pause title. A label alone pins neither author nor title, so without this, labelling any issue would nominate it.
  • An approval must postdate the label going on. Moving the label onto an issue closed earlier would otherwise import that close as an approval. On a real pause issue the label is applied at creation, so nothing genuine is excluded.
  • The actor is matched by numeric id, and Apps are excluded by type. A rename doesn't shake the first; github-actions[bot], which a workflow's own GITHUB_TOKEN acts as, is caught by the second.

Identity comes from the credential rather than from config. The preflight runs as the bot, so gh api user yields the login the counts need and the id the approval filter compares. The configured name fed both before, and every way it could be wrong failed open: stale after a rename, the counting queries match nothing and the limit never trips at all; empty, the approval filter reduces to "login is not the empty string" and the bot's own close approves it. The preflight now publishes both facts for the rest of the job, replacing the Resolve bot ID step that re-derived the id from the configured name.

Why the bookkeeping labels moved into the generator

The rate-limit issue collects one comment per refused run, and tend-mention skipped only tend-outage — so each row would have re-triggered mention, whose handle job trips the same limit and appends another row. That is the self-sustaining loop the existing guard exists to stop, reachable again through a label it did not know about.

BOOKKEEPING_LABELS in workflows.py behind a not_bookkeeping() macro gives triage and mention one list, so the next such label reaches both at once. Rendered as contains(…) == false, since a value opening with ! is a tag indicator to YAML and only parses inside a block scalar. The worker keeps its own copy for now — unifying it needs a story for issues already filed without the label.

Testing

Ten cases in test_shared_steps.py cover the ceiling arithmetic, day-scoping, both actor exclusions, the rename case, the refusal when identity can't be read, and reopen-rather-than-duplicate. The fake gh runs the script's own --jq expression against a fixture with real jq, because that filter is the behaviour under test — a fake that pre-filtered would assert nothing.

Verified by mutation rather than by passing. Each of these reddens exactly the case named and nothing else: deleting the two actor conditions (bot, rename, App), putting the filter back on login (rename), dropping the label-ordering rule (relabel), dropping the title filter or the --author scope (foreign anchor), reverting the shared predicate to label-only (foreign anchor and reconciler), and removing the day floor (yesterday's approval).

run_issue_matching is the single definition of which issues belong to a record — label, author, title. The anchor lookup and the duplicate reconciler are the same call, so they cannot come to disagree about which issue is real.

Not yet exercised against real GitHub. No run has filed a pause issue, reopened one, or been approved end to end. tend-agent/tend-integration exists for that and would be worth a pass before this is relied on.

Deferred

TODO.md records the automatic re-runner for work a trip refused, blocked on confirming tend's re-runs work at all — building one over a broken re-run path would move the symptom from "my review never came back" to "the recovery workflow ran and my review still never came back". Recovery meanwhile is one gh run rerun <id> --failed per row in the issue's table.

Also open: an approval cannot be revoked, since reopening doesn't remove the close event. Counting only closes since the most recent reopen by a person would give a "stop" that works inside the day; left out here because it adds complexity to a security-critical script.

This was written by Claude Code on behalf of max-sixty

max-sixty and others added 8 commits August 5, 2026 14:58
The spike limit had no reset: once tripped, the only cure was the UTC
rollover. A legitimate nightly sweep tripped it and the repo stopped
reviewing for six hours, with no signal anywhere but a red check on
unrelated PRs.

On a trip the preflight now files or reopens a `tend-rate-limit` issue
listing the runs it refused. Opening it is the notice. Closing it is the
approval, and doubles the ceiling for the rest of the UTC day; each further
close doubles again, so the breaker re-arms after use rather than switching
off for the day.

Approval is a check rather than an instruction the bot could be talked out
of: only closes by a non-bot actor count. `login != BOT` excludes the tend
bot, which is a normal user account; `type != "Bot"` excludes GitHub Apps,
`github-actions[bot]` above all. GitHub admits only the author or a
triage/write collaborator to close an issue, and the bot is the author, so
what remains is exactly the maintainers — no allowlist.

A doubling rather than a flat bump because the ceiling it lifts is itself
proportional (`10 + 2 * daily_avg`); at the formula's floor the two
coincide. The burst limit stays non-resumable: ten PRs in twenty minutes is
a loop, not a busy day.

The bot's own bookkeeping issues stay in the count. A flood of them is
itself a plausible runaway, so excluding them would blind the metric to one
of the shapes it exists to catch.

Extracts the issue machinery both paths now need into
`shared/steps/lib/run-issue.sh` rather than copying the create-create dedup
race. `run_issue_canonical` picks the lowest-numbered issue on a label,
matching the reconciler's keeper rule — `gh issue list` orders newest-first,
so taking `.[0]` could return a duplicate the reconcile had already closed.
The two callers keep opposite policies on a closed issue: an outage issue
closed as resolved must not swallow the next incident, while the rate-limit
issue is one long-lived record whose closes are the approvals.

Refused runs still don't retry on their own; TODO.md carries that, blocked
on confirming tend's re-runs work at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in `report-failure.sh`, between this branch's extraction of the
REF block into `lib/run-issue.sh` and #823's rewrite of that same block on
main. Resolved by keeping the extraction and porting #823 into the shared
function: the `repository_dispatch` branch reading `client_payload.pr`, the
`workflow_run` branch linking the upstream run id, and `// empty` plus
`${VAR:+…}` on every extraction so a missing field renders as N/A rather than
`#null`.

Taking either side alone would have lost something — this branch's side drops
#823's fix, main's side drops the extraction the rate-limit path needs. The
merged form gives the rate-limit issue's rows the same trigger handling, which
matters most for exactly the case #823 named: a relayed review, which nothing
retries on its own.

Verified against #823's own recorded case table, reproduced through the shared
function.
The preflight's approval logic shipped with nothing but shellcheck behind it.
Main has since grown `test_shared_steps.py`, which is the right home, so the
security property gets a real test rather than only the live-API spot checks
it was written against.

Seven cases: under the limit nothing is looked up or filed; over it with no
approval the run is refused and the issue filed; one close by a person takes
the ceiling from 15 to 30; a close by the bot, by a GitHub App, or by anyone
yesterday does not; and past the doubled ceiling the existing issue is
reopened rather than duplicated.

The fake `gh` runs the script's own `--jq` expression against a fixture with
real jq, rather than returning a pre-filtered list. That filter *is* the
behaviour under test — which closes count — so a fake that pre-filtered would
assert nothing. Confirmed by deleting the two actor conditions and watching
exactly the bot and App cases fail.

`date` and `sleep` are shimmed too: the script is written for the runners' GNU
date, which macOS lacks, and its jitter would otherwise add up to 30s a test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nightly sweep closes bot-opened issues that reported a transient
condition once it resolves. A `tend-rate-limit` issue matches that
description exactly — the bot opened it, and by the next sweep the UTC
rollover has cleared the trip — so the sweep would close it.

The security property is unaffected: the preflight counts only closes by a
person, so a close by the bot lifts nothing. What it costs is the notice.
Closing the issue clears an alarm a maintainer may not have seen, and the
alarm is half the point of filing it.

Skips by label rather than by the "Do not close manually" body sentinel the
recurring trackers use. That sentinel would read as an instruction to the
maintainer not to do the one thing the issue is asking for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…name

The bot account is an ordinary user account rather than a GitHub App, so the
`type != "Bot"` exclusion does nothing for it — that one covers
`github-actions[bot]` and other Apps. Keeping the bot from approving itself
rested entirely on identifying it, and both ways it could fail did so open,
counting the bot's own close as a person's.

Matching on login broke under a rename: the account keeps closing its own
issues, the configured name no longer matches, and every close reads as an
approval. Now matched on the numeric id, which a rename doesn't change. The
action already resolves the same id a few steps later, for the same reason.

An empty BOT_NAME was worse — the filter reduced to "login is not the empty
string", true of everyone. The preflight now stops with an error rather than
running a check that has quietly reversed. Failing to resolve the id leaves
approvals at zero, which refuses the run.

Both new tests were confirmed to fail against the old logic: the rename case
goes red when the filter is put back on login, and only that case.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… config

`BOT_NAME` fed both the approval filter and all four counting queries, and
every way it could be wrong failed open. Stale after a rename, the searches
match nothing and the limit never trips at all; empty, the approval filter
reduces to "login is not the empty string" and the bot's own close approves
it. Both are silent.

The preflight runs as the bot, so the authenticated user is the bot by
definition. One `gh api user` yields the login the counts need and the id the
approval filter compares, with nothing to misconfigure and nothing to go
stale. Failing to read it now stops the run rather than proceeding on a guess.

Drops the resolve-by-name call the trip path had added, and `BOT_NAME` from
the step's environment in both actions. The input stays — later steps use it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…work

The rate-limit issue collects one comment per refused run, and `tend-mention`
skipped only `tend-outage` — so each row would have re-triggered mention,
whose handle job trips the same limit and appends another row. That is the
self-sustaining loop the existing guard was written to stop, reachable again
through a label it did not know about.

The label list moves to `BOOKKEEPING_LABELS` in workflows.py behind a
`not_bookkeeping()` macro, so triage and mention read one list and the next
such label reaches both at once. The worker's own copy gains the label too;
unifying that one with the generator's needs a story for issues already filed
without it, so it stays a separate list for now. Rendered as
`contains(…) == false`, since a value opening with `!` is a tag indicator to
YAML and only parses inside a block scalar.

Identity is resolved once. The preflight publishes the login and id it reads
off the token, and the "Resolve bot ID" step — which re-derived the id from
the configured name, the source the previous commit showed fails open — is
gone from both actions.

Smaller cuts, all from the same pass: `/events?per_page=100` instead of
`/timeline`, which carries the four fields the filter reads and excludes the
comments this issue accumulates forever; the jitter and its re-list now run
only when there is no issue to append to, saving ~15s on every refused run
after the first; `run_issue_create_and_reconcile` returns the keeper it
already computed rather than leaving the caller to re-list for it; bodies go
in on stdin rather than through /tmp files; and the `APPROVALS` clamp is gone
— it guarded an unreachable state whose unguarded outcome was already to
refuse.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three things worth a decision before this lands.

The approval anchor is something the bot can move. run_issue_canonical "$PAUSE_LABEL" all takes the lowest-numbered issue carrying tend-rate-limit in any state, with no constraint on author or title — and every generated workflow grants issues: write (the permissions() macro in macros.yaml.j2 emits issues: write), so the agent can put that label on any issue in the repo. Label one a maintainer already closed earlier today and the preflight reads that close as an approval: actor.id != BOT_ID, actor.type is "User", created_at is today — every condition satisfied, ceiling doubles, and no pause issue was ever closed. The same thing happens by accident if a maintainer labels an old issue for tracking, which then outranks the real pause issue on the lowest-number rule. Impact is bounded (one doubling per human close on whatever issue got anchored), but docs/security-model.md states the property without qualification — "The bot cannot approve itself even if a prompt injection tells it to". Constraining the pause lookup to an issue the preflight itself filed — author is the bot, title is PAUSE_TITLE — and requiring the close to postdate that issue's creation would close both the deliberate and the accidental path, and keeps the "no allowlist" property.

#857 addresses the same gap by a different route. It changes the Report failure gate to failure() && steps.security.outcome != 'failure' and cites this exact incident — "a rate-limit preflight abort held every workflow on this repo for a full UTC day without filing a thing". With both merged, a spike trip files a tend-rate-limit issue and a tend-outage issue, and that outage create counts into the next run's TODAY_POSTS. They also both rewrite report-failure.sh's header block and touch the same regions of both action files, so whichever lands second needs a rebase either way — worth deciding now whether #857 should carve the rate-limit abort back out.

bot_name is now dead as an action input. After this change inputs.bot_name has no reference left in either claude/action.yaml or codex/action.yaml, but it stays required: true in both and macros.yaml.j2 still renders bot_name: <<cfg.bot_name>> into every agent_step. Given the repo's no-backward-compatibility rule this reads as a half cut-over: dropping the input from both actions and the bot_name: line from the with: block would finish it. (The config key itself stays — the templates use cfg.bot_name in a dozen other places.)

Smaller: when the burst limit trips in the same run as the spike, the pause block still runs — it reopens the issue and appends a row, so a maintainer's close gets undone by the next refused run while the thing actually refusing the run is the non-resumable burst check, which closing the issue cannot lift. The issue body promises the opposite ("Closing this issue approves the volume and doubles the ceiling"). Gating the block on ABORT still being false, or naming the burst trip in the appended row, keeps the notice honest.

Testing looks good — running the script's real --jq against a fixture is the right call for a filter that is the security property, and the mutation check described in the PR body is the verification I'd have asked for.

Comment thread shared/steps/rate-limit-preflight.sh Outdated
Review found the approval anchor was bot-movable, which defeated the whole
property. The lookup took the lowest-numbered issue carrying the label in any
state, with no author or title constraint, and every generated workflow grants
`issues: write` — so labelling an issue a maintainer had closed earlier today
imported that close as an approval, doubling the ceiling with no pause issue
ever closed. The same by accident if a maintainer labelled an old issue, which
then outranked the real one on lowest-number.

Two constraints, for two different failures. The anchor is now the
lowest-numbered issue the *bot authored* under the pause title, so an
arbitrary issue cannot become it. And an approval must be a close that
postdates the label going on, so moving the label onto an already-closed issue
imports nothing — on a real pause issue the label is applied at creation, so
this excludes nothing genuine.

Also from the review: a burst trip no longer files or appends to the issue.
The burst limit is deliberately not resumable, so an issue offering to double
the ceiling promised a recovery that closing it cannot deliver; the burst
annotation is the honest signal, and the spike is still reported.

And the fix CI caught: `BOOKKEEPING_LABELS` gained the label without its test
fixture, which the worker's own test deliberately couples to.

The reviewer's remaining point — that `bot_name` is now an unread action input
— is left as is. Removing it drops the bot's name from every generated
workflow, which three tests assert an adopter can see, and that is a wider
decision than this change should make. Its description now says what it is
for and why the steps don't read it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The anchor fix reads right, and it holds up the way the PR body describes — deleting the label-ordering rule reddens exactly the relabel case. Two gaps in what the tests pin, and one place the tightening didn't reach.

Neither half of the new anchor is pinned by a test. pause_issue narrows the lookup three ways — the label, --author "$BOT", and title == PAUSE_TITLE — but only the label-ordering rule has a mutation-killing test. Dropping the title select(…) from the --jq, and dropping --author "$BOT" from the gh issue list line, each leave all 14 green. That's the headline property of this commit, and it's the rule the fake gh can't catch by accident: emit runs the script's real --jq against PAUSE_ISSUES_JSON, but every fixture _approve writes already carries PAUSE_TITLE, and the fake ignores --author entirely. The case suggested below fails under both mutations and passes unmutated.

The day-scoping test now passes for a reason other than its name. _approve defaults labelled_at to 2026-01-02T08:00:00Z while _closed_event(…, day="2026-01-01") closes at 2026-01-01T09:00:00Z, so that close is excluded by the label-ordering rule rather than by the ${TODAY}T00:00:00Z floor. Replacing the floor with SINCE="" keeps all 14 green. Moving the label back a day restores the kill and leaves the relabel test as the only cover for the ordering rule.

The reconciler still treats the label as the identity. pause_issue demotes the label to a finding aid — author and title are what pin the issue — but run_issue_create_and_reconcile picks its keeper from gh issue list --label "$label" --state open with no author or title constraint. If any lower-numbered open issue carries tend-rate-limit (a maintainer tracking one, or the bot putting it there), the first trip files the pause issue and the reconciler closes it again as a duplicate of that issue, so the ::error:: points at something holding none of the refused-run rows and a maintainer's unrelated issue collects a wrong duplicate comment. Recovery still works — the next run finds the real, bot-closed and therefore unapproved issue through pause_issue and reopens it — but the notice ends up split across two issues. Reconciling on the same author/title predicate would keep the two rules agreeing about which issue is the real one.

Comment thread generator/tests/test_shared_steps.py Outdated
Comment thread generator/tests/test_shared_steps.py
Review found the reconciler still selecting its keeper by label alone, while
the anchor lookup had been tightened to author and title. Any lower-numbered
open issue carrying the label — a maintainer's, or one the bot put the label
on — would outrank the freshly filed pause issue, and the reconciler would
close the real record as that issue's duplicate. Recovery still worked, since
the next run finds the bot-closed and therefore unapproved issue, but the
notice ended up split across two issues and a maintainer's unrelated issue
collected a duplicate comment.

Both now go through `run_issue_canonical`, which takes the label, the author
and the title. The outage path passes its own title and gains the same
narrowing, which is right for the same reason: it filed those issues too.

Two test gaps the review also named, both real — the anchor's author and title
halves had no mutation-killing test, and the day-scoping test was passing
because the label-ordering rule excluded its close before the day floor could,
so the floor could be deleted with the suite still green. Dating the label back
a day puts the floor under test, and a foreign-titled fixture covers the
anchor.

All three now verified by mutation: dropping the title filter, the `--author`
scope, or the day floor each reddens exactly one test, and a different one in
each case. The author check asserts on the `--state all` lookup specifically —
matching any `issue list` call passed while the reconciler's own list still
carried the flag.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reconciler now agrees with the anchor, but nothing holds it there. Reverting run_issue_create_and_reconcile's list to its pre-commit form — gh issue list --label "$label" --state open --json number --jq 'sort_by(.number) | .[].number' — leaves all 15 green, and so does dropping either half on its own. That is the change this commit exists to make, and the comment tying the two lookups together (“Same predicate as run_issue_canonical”) is a promise the suite does not keep: the predicate lives in two textual copies kept in sync by hand, which is the drift a test is for. The case below fails under all three mutations and passes unmutated.

A shared run_issue_matching that both call would close it from the other side instead, if one copy appeals more than one test.

Comment thread generator/tests/test_shared_steps.py
The last commit made the reconciler and the lookup agree by writing the same
predicate twice, with a comment asserting they matched. Review pointed out
that nothing held them there: reverting the reconciler's list to its label-only
form, or dropping either half, left the suite green — and hand-synced copies
are the drift a test exists for.

Taking the offered alternative rather than the test alone: `run_issue_matching`
is now the one definition, and `run_issue_canonical` is its first line. The
reconciler calls it too, so there is no second copy to keep in step.

The suggested test lands as well, because it pins something the extraction
does not — that no issue the preflight never filed is closed as a duplicate.
It reddens under all three mutations the review named.

`run_issue_canonical` slices the first line in the shell rather than piping to
`head`, which under `pipefail` can take the script down when `head` closes the
pipe first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants