feat(rate-limit): let a maintainer approve past the spike limit - #874
Conversation
The spike limit had no reset: once tripped, the only cure was the UTC rollover. A legitimate nightly sweep tripped it and the repo stopped reviewing for six hours, with no signal anywhere but a red check on unrelated PRs. On a trip the preflight now files or reopens a `tend-rate-limit` issue listing the runs it refused. Opening it is the notice. Closing it is the approval, and doubles the ceiling for the rest of the UTC day; each further close doubles again, so the breaker re-arms after use rather than switching off for the day. Approval is a check rather than an instruction the bot could be talked out of: only closes by a non-bot actor count. `login != BOT` excludes the tend bot, which is a normal user account; `type != "Bot"` excludes GitHub Apps, `github-actions[bot]` above all. GitHub admits only the author or a triage/write collaborator to close an issue, and the bot is the author, so what remains is exactly the maintainers — no allowlist. A doubling rather than a flat bump because the ceiling it lifts is itself proportional (`10 + 2 * daily_avg`); at the formula's floor the two coincide. The burst limit stays non-resumable: ten PRs in twenty minutes is a loop, not a busy day. The bot's own bookkeeping issues stay in the count. A flood of them is itself a plausible runaway, so excluding them would blind the metric to one of the shapes it exists to catch. Extracts the issue machinery both paths now need into `shared/steps/lib/run-issue.sh` rather than copying the create-create dedup race. `run_issue_canonical` picks the lowest-numbered issue on a label, matching the reconciler's keeper rule — `gh issue list` orders newest-first, so taking `.[0]` could return a duplicate the reconcile had already closed. The two callers keep opposite policies on a closed issue: an outage issue closed as resolved must not swallow the next incident, while the rate-limit issue is one long-lived record whose closes are the approvals. Refused runs still don't retry on their own; TODO.md carries that, blocked on confirming tend's re-runs work at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in `report-failure.sh`, between this branch's extraction of the REF block into `lib/run-issue.sh` and #823's rewrite of that same block on main. Resolved by keeping the extraction and porting #823 into the shared function: the `repository_dispatch` branch reading `client_payload.pr`, the `workflow_run` branch linking the upstream run id, and `// empty` plus `${VAR:+…}` on every extraction so a missing field renders as N/A rather than `#null`. Taking either side alone would have lost something — this branch's side drops #823's fix, main's side drops the extraction the rate-limit path needs. The merged form gives the rate-limit issue's rows the same trigger handling, which matters most for exactly the case #823 named: a relayed review, which nothing retries on its own. Verified against #823's own recorded case table, reproduced through the shared function.
The preflight's approval logic shipped with nothing but shellcheck behind it. Main has since grown `test_shared_steps.py`, which is the right home, so the security property gets a real test rather than only the live-API spot checks it was written against. Seven cases: under the limit nothing is looked up or filed; over it with no approval the run is refused and the issue filed; one close by a person takes the ceiling from 15 to 30; a close by the bot, by a GitHub App, or by anyone yesterday does not; and past the doubled ceiling the existing issue is reopened rather than duplicated. The fake `gh` runs the script's own `--jq` expression against a fixture with real jq, rather than returning a pre-filtered list. That filter *is* the behaviour under test — which closes count — so a fake that pre-filtered would assert nothing. Confirmed by deleting the two actor conditions and watching exactly the bot and App cases fail. `date` and `sleep` are shimmed too: the script is written for the runners' GNU date, which macOS lacks, and its jitter would otherwise add up to 30s a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nightly sweep closes bot-opened issues that reported a transient condition once it resolves. A `tend-rate-limit` issue matches that description exactly — the bot opened it, and by the next sweep the UTC rollover has cleared the trip — so the sweep would close it. The security property is unaffected: the preflight counts only closes by a person, so a close by the bot lifts nothing. What it costs is the notice. Closing the issue clears an alarm a maintainer may not have seen, and the alarm is half the point of filing it. Skips by label rather than by the "Do not close manually" body sentinel the recurring trackers use. That sentinel would read as an instruction to the maintainer not to do the one thing the issue is asking for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…name The bot account is an ordinary user account rather than a GitHub App, so the `type != "Bot"` exclusion does nothing for it — that one covers `github-actions[bot]` and other Apps. Keeping the bot from approving itself rested entirely on identifying it, and both ways it could fail did so open, counting the bot's own close as a person's. Matching on login broke under a rename: the account keeps closing its own issues, the configured name no longer matches, and every close reads as an approval. Now matched on the numeric id, which a rename doesn't change. The action already resolves the same id a few steps later, for the same reason. An empty BOT_NAME was worse — the filter reduced to "login is not the empty string", true of everyone. The preflight now stops with an error rather than running a check that has quietly reversed. Failing to resolve the id leaves approvals at zero, which refuses the run. Both new tests were confirmed to fail against the old logic: the rename case goes red when the filter is put back on login, and only that case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… config `BOT_NAME` fed both the approval filter and all four counting queries, and every way it could be wrong failed open. Stale after a rename, the searches match nothing and the limit never trips at all; empty, the approval filter reduces to "login is not the empty string" and the bot's own close approves it. Both are silent. The preflight runs as the bot, so the authenticated user is the bot by definition. One `gh api user` yields the login the counts need and the id the approval filter compares, with nothing to misconfigure and nothing to go stale. Failing to read it now stops the run rather than proceeding on a guess. Drops the resolve-by-name call the trip path had added, and `BOT_NAME` from the step's environment in both actions. The input stays — later steps use it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…work The rate-limit issue collects one comment per refused run, and `tend-mention` skipped only `tend-outage` — so each row would have re-triggered mention, whose handle job trips the same limit and appends another row. That is the self-sustaining loop the existing guard was written to stop, reachable again through a label it did not know about. The label list moves to `BOOKKEEPING_LABELS` in workflows.py behind a `not_bookkeeping()` macro, so triage and mention read one list and the next such label reaches both at once. The worker's own copy gains the label too; unifying that one with the generator's needs a story for issues already filed without it, so it stays a separate list for now. Rendered as `contains(…) == false`, since a value opening with `!` is a tag indicator to YAML and only parses inside a block scalar. Identity is resolved once. The preflight publishes the login and id it reads off the token, and the "Resolve bot ID" step — which re-derived the id from the configured name, the source the previous commit showed fails open — is gone from both actions. Smaller cuts, all from the same pass: `/events?per_page=100` instead of `/timeline`, which carries the four fields the filter reads and excludes the comments this issue accumulates forever; the jitter and its re-list now run only when there is no issue to append to, saving ~15s on every refused run after the first; `run_issue_create_and_reconcile` returns the keeper it already computed rather than leaving the caller to re-list for it; bodies go in on stdin rather than through /tmp files; and the `APPROVALS` clamp is gone — it guarded an unreachable state whose unguarded outcome was already to refuse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tend-agent
left a comment
There was a problem hiding this comment.
Three things worth a decision before this lands.
The approval anchor is something the bot can move. run_issue_canonical "$PAUSE_LABEL" all takes the lowest-numbered issue carrying tend-rate-limit in any state, with no constraint on author or title — and every generated workflow grants issues: write (the permissions() macro in macros.yaml.j2 emits issues: write), so the agent can put that label on any issue in the repo. Label one a maintainer already closed earlier today and the preflight reads that close as an approval: actor.id != BOT_ID, actor.type is "User", created_at is today — every condition satisfied, ceiling doubles, and no pause issue was ever closed. The same thing happens by accident if a maintainer labels an old issue for tracking, which then outranks the real pause issue on the lowest-number rule. Impact is bounded (one doubling per human close on whatever issue got anchored), but docs/security-model.md states the property without qualification — "The bot cannot approve itself even if a prompt injection tells it to". Constraining the pause lookup to an issue the preflight itself filed — author is the bot, title is PAUSE_TITLE — and requiring the close to postdate that issue's creation would close both the deliberate and the accidental path, and keeps the "no allowlist" property.
#857 addresses the same gap by a different route. It changes the Report failure gate to failure() && steps.security.outcome != 'failure' and cites this exact incident — "a rate-limit preflight abort held every workflow on this repo for a full UTC day without filing a thing". With both merged, a spike trip files a tend-rate-limit issue and a tend-outage issue, and that outage create counts into the next run's TODAY_POSTS. They also both rewrite report-failure.sh's header block and touch the same regions of both action files, so whichever lands second needs a rebase either way — worth deciding now whether #857 should carve the rate-limit abort back out.
bot_name is now dead as an action input. After this change inputs.bot_name has no reference left in either claude/action.yaml or codex/action.yaml, but it stays required: true in both and macros.yaml.j2 still renders bot_name: <<cfg.bot_name>> into every agent_step. Given the repo's no-backward-compatibility rule this reads as a half cut-over: dropping the input from both actions and the bot_name: line from the with: block would finish it. (The config key itself stays — the templates use cfg.bot_name in a dozen other places.)
Smaller: when the burst limit trips in the same run as the spike, the pause block still runs — it reopens the issue and appends a row, so a maintainer's close gets undone by the next refused run while the thing actually refusing the run is the non-resumable burst check, which closing the issue cannot lift. The issue body promises the opposite ("Closing this issue approves the volume and doubles the ceiling"). Gating the block on ABORT still being false, or naming the burst trip in the appended row, keeps the notice honest.
Testing looks good — running the script's real --jq against a fixture is the right call for a filter that is the security property, and the mutation check described in the PR body is the verification I'd have asked for.
Review found the approval anchor was bot-movable, which defeated the whole property. The lookup took the lowest-numbered issue carrying the label in any state, with no author or title constraint, and every generated workflow grants `issues: write` — so labelling an issue a maintainer had closed earlier today imported that close as an approval, doubling the ceiling with no pause issue ever closed. The same by accident if a maintainer labelled an old issue, which then outranked the real one on lowest-number. Two constraints, for two different failures. The anchor is now the lowest-numbered issue the *bot authored* under the pause title, so an arbitrary issue cannot become it. And an approval must be a close that postdates the label going on, so moving the label onto an already-closed issue imports nothing — on a real pause issue the label is applied at creation, so this excludes nothing genuine. Also from the review: a burst trip no longer files or appends to the issue. The burst limit is deliberately not resumable, so an issue offering to double the ceiling promised a recovery that closing it cannot deliver; the burst annotation is the honest signal, and the spike is still reported. And the fix CI caught: `BOOKKEEPING_LABELS` gained the label without its test fixture, which the worker's own test deliberately couples to. The reviewer's remaining point — that `bot_name` is now an unread action input — is left as is. Removing it drops the bot's name from every generated workflow, which three tests assert an adopter can see, and that is a wider decision than this change should make. Its description now says what it is for and why the steps don't read it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tend-agent
left a comment
There was a problem hiding this comment.
The anchor fix reads right, and it holds up the way the PR body describes — deleting the label-ordering rule reddens exactly the relabel case. Two gaps in what the tests pin, and one place the tightening didn't reach.
Neither half of the new anchor is pinned by a test. pause_issue narrows the lookup three ways — the label, --author "$BOT", and title == PAUSE_TITLE — but only the label-ordering rule has a mutation-killing test. Dropping the title select(…) from the --jq, and dropping --author "$BOT" from the gh issue list line, each leave all 14 green. That's the headline property of this commit, and it's the rule the fake gh can't catch by accident: emit runs the script's real --jq against PAUSE_ISSUES_JSON, but every fixture _approve writes already carries PAUSE_TITLE, and the fake ignores --author entirely. The case suggested below fails under both mutations and passes unmutated.
The day-scoping test now passes for a reason other than its name. _approve defaults labelled_at to 2026-01-02T08:00:00Z while _closed_event(…, day="2026-01-01") closes at 2026-01-01T09:00:00Z, so that close is excluded by the label-ordering rule rather than by the ${TODAY}T00:00:00Z floor. Replacing the floor with SINCE="" keeps all 14 green. Moving the label back a day restores the kill and leaves the relabel test as the only cover for the ordering rule.
The reconciler still treats the label as the identity. pause_issue demotes the label to a finding aid — author and title are what pin the issue — but run_issue_create_and_reconcile picks its keeper from gh issue list --label "$label" --state open with no author or title constraint. If any lower-numbered open issue carries tend-rate-limit (a maintainer tracking one, or the bot putting it there), the first trip files the pause issue and the reconciler closes it again as a duplicate of that issue, so the ::error:: points at something holding none of the refused-run rows and a maintainer's unrelated issue collects a wrong duplicate comment. Recovery still works — the next run finds the real, bot-closed and therefore unapproved issue through pause_issue and reopens it — but the notice ends up split across two issues. Reconciling on the same author/title predicate would keep the two rules agreeing about which issue is the real one.
Review found the reconciler still selecting its keeper by label alone, while the anchor lookup had been tightened to author and title. Any lower-numbered open issue carrying the label — a maintainer's, or one the bot put the label on — would outrank the freshly filed pause issue, and the reconciler would close the real record as that issue's duplicate. Recovery still worked, since the next run finds the bot-closed and therefore unapproved issue, but the notice ended up split across two issues and a maintainer's unrelated issue collected a duplicate comment. Both now go through `run_issue_canonical`, which takes the label, the author and the title. The outage path passes its own title and gains the same narrowing, which is right for the same reason: it filed those issues too. Two test gaps the review also named, both real — the anchor's author and title halves had no mutation-killing test, and the day-scoping test was passing because the label-ordering rule excluded its close before the day floor could, so the floor could be deleted with the suite still green. Dating the label back a day puts the floor under test, and a foreign-titled fixture covers the anchor. All three now verified by mutation: dropping the title filter, the `--author` scope, or the day floor each reddens exactly one test, and a different one in each case. The author check asserts on the `--state all` lookup specifically — matching any `issue list` call passed while the reconciler's own list still carried the flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tend-agent
left a comment
There was a problem hiding this comment.
The reconciler now agrees with the anchor, but nothing holds it there. Reverting run_issue_create_and_reconcile's list to its pre-commit form — gh issue list --label "$label" --state open --json number --jq 'sort_by(.number) | .[].number' — leaves all 15 green, and so does dropping either half on its own. That is the change this commit exists to make, and the comment tying the two lookups together (“Same predicate as run_issue_canonical”) is a promise the suite does not keep: the predicate lives in two textual copies kept in sync by hand, which is the drift a test is for. The case below fails under all three mutations and passes unmutated.
A shared run_issue_matching that both call would close it from the other side instead, if one copy appeals more than one test.
The last commit made the reconciler and the lookup agree by writing the same predicate twice, with a comment asserting they matched. Review pointed out that nothing held them there: reverting the reconciler's list to its label-only form, or dropping either half, left the suite green — and hand-synced copies are the drift a test exists for. Taking the offered alternative rather than the test alone: `run_issue_matching` is now the one definition, and `run_issue_canonical` is its first line. The reconciler calls it too, so there is no second copy to keep in step. The suggested test lands as well, because it pins something the extraction does not — that no issue the preflight never filed is closed as a duplicate. It reddens under all three mutations the review named. `run_issue_canonical` slices the first line in the shell rather than piping to `head`, which under `pipefail` can take the script down when `head` closes the pipe first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Problem
The spike rate limit is a circuit breaker with no reset: once tripped, the only cure is the UTC rollover. A legitimate nightly sweep — six PRs filed between 00:20 and 00:24 — pushed the day's count to 16 against a limit of 15, and every
tend-reviewrun from 11:47 UTC onward died at the preflight. Six hours with no reviews, and no signal anywhere a person would look: a failed preflight writes an annotation onto a job nobody opens. #851's own review was one of the ones stranded.Two separable gaps: no notice that the bot had stopped, and no reset short of waiting for midnight.
Solution
On a spike trip the preflight files or reopens a
tend-rate-limitissue listing the runs it refused. Opening it is the notice. Closing it is the approval, and doubles the ceiling for the rest of the UTC day; each further close doubles again, so the breaker re-arms after use rather than switching off.A doubling rather than a flat increment because the ceiling it lifts is itself proportional —
10 + 2 * daily_avgis 15 here but 40 on a repo filing 15 a day, where a flat bump would mean closing the issue all afternoon. At the formula's floor the two coincide (10 doubles to 20), so they diverge only where a flat bump stops working.The burst limit stays non-resumable: ten PRs in twenty minutes is a loop, not a busy day, and there is nothing there to wave through.
ABORTis set true at three points and never cleared, so an approval cannot release a burst trip — and a burst trip files no issue at all, since one offering to lift the ceiling would promise a recovery that closing it cannot deliver.Approval is a check, not an instruction the bot could be talked out of. The preflight counts only closes whose actor is neither the bot nor a GitHub App. GitHub admits only an issue's author or a triage/write collaborator to close it, and the bot is the author, so what remains is exactly the maintainers — no allowlist to keep.
Three things pin that down, because the bot holds
issues: writeand could otherwise arrange an approval for itself:github-actions[bot], which a workflow's ownGITHUB_TOKENacts as, is caught by the second.Identity comes from the credential rather than from config. The preflight runs as the bot, so
gh api useryields the login the counts need and the id the approval filter compares. The configured name fed both before, and every way it could be wrong failed open: stale after a rename, the counting queries match nothing and the limit never trips at all; empty, the approval filter reduces to "login is not the empty string" and the bot's own close approves it. The preflight now publishes both facts for the rest of the job, replacing theResolve bot IDstep that re-derived the id from the configured name.Why the bookkeeping labels moved into the generator
The rate-limit issue collects one comment per refused run, and
tend-mentionskipped onlytend-outage— so each row would have re-triggered mention, whose handle job trips the same limit and appends another row. That is the self-sustaining loop the existing guard exists to stop, reachable again through a label it did not know about.BOOKKEEPING_LABELSinworkflows.pybehind anot_bookkeeping()macro gives triage and mention one list, so the next such label reaches both at once. Rendered ascontains(…) == false, since a value opening with!is a tag indicator to YAML and only parses inside a block scalar. The worker keeps its own copy for now — unifying it needs a story for issues already filed without the label.Testing
Ten cases in
test_shared_steps.pycover the ceiling arithmetic, day-scoping, both actor exclusions, the rename case, the refusal when identity can't be read, and reopen-rather-than-duplicate. The fakeghruns the script's own--jqexpression against a fixture with real jq, because that filter is the behaviour under test — a fake that pre-filtered would assert nothing.Verified by mutation rather than by passing. Each of these reddens exactly the case named and nothing else: deleting the two actor conditions (bot, rename, App), putting the filter back on login (rename), dropping the label-ordering rule (relabel), dropping the title filter or the
--authorscope (foreign anchor), reverting the shared predicate to label-only (foreign anchor and reconciler), and removing the day floor (yesterday's approval).run_issue_matchingis the single definition of which issues belong to a record — label, author, title. The anchor lookup and the duplicate reconciler are the same call, so they cannot come to disagree about which issue is real.Not yet exercised against real GitHub. No run has filed a pause issue, reopened one, or been approved end to end.
tend-agent/tend-integrationexists for that and would be worth a pass before this is relied on.Deferred
TODO.mdrecords the automatic re-runner for work a trip refused, blocked on confirming tend's re-runs work at all — building one over a broken re-run path would move the symptom from "my review never came back" to "the recovery workflow ran and my review still never came back". Recovery meanwhile is onegh run rerun <id> --failedper row in the issue's table.Also open: an approval cannot be revoked, since reopening doesn't remove the close event. Counting only closes since the most recent reopen by a person would give a "stop" that works inside the day; left out here because it adds complexity to a security-critical script.