fix(rate-limit): pause creation on a daily spike instead of failing the run - #856
fix(rate-limit): pause creation on a daily spike instead of failing the run#856tend-agent wants to merge 4 commits into
Conversation
|
Added regression tests for the three tiers in 66f8e84 — this run's
One correction for the body: Worth noting alongside: the outage is invisible to |
tend-agent
left a comment
tend-agent
left a comment
There was a problem hiding this comment.
Splitting the daily guard by failure shape is the right call, and keeping the two abort tiers in pure shell keeps the injection-resistant floor intact. The tests exercise all four paths against a stubbed gh and pass locally (7 passed in generator/tests/test_shared_steps.py). Three things on the new tiers, one of which I think is load-bearing.
The main one: both tiers scale off the same 6-day baseline, so the fix does the least exactly where the diagnosed failure mode is worst. hard = 10 + P and spike = 10 + P/3 converge as P shrinks — at P = 0 they are both 10, so the pause tier is unreachable and the run hard-aborts at 11 items, identical to pre-PR behaviour. At P = 1 and P = 2 the pause band is one and two items wide. The PR's own diagnosis is that a depressed baseline is what caused 2026-08-05 (07-30 through 08-02 sat inside the quota outage), and a longer outage — or a genuinely quiet repo, or a fresh adopter on install day — drives P toward 0. Flooring the gap would keep the pause tier reachable regardless of baseline. Detail inline.
The second is about who the pause directive can actually be followed by. Inline on the note text.
Tier arithmetic across baselines
baseline=0 spike=10 hard=10 <- pause tier unreachable
baseline=1 spike=10 hard=11
baseline=2 spike=10 hard=12
baseline=3 spike=11 hard=13
baseline=6 spike=12 hard=16
baseline=17 spike=15 hard=27 <- the 2026-08-05 case
baseline=30 spike=20 hard=40
…o put its work Three review points on the tiering. Both limits scaled off the same 6-day baseline (`10 + P` and `10 + P/3`), so they converged as the baseline fell and coincided at 10 when it reached 0 — the pause tier was unreachable and the run hard-aborted on the 11th item, exactly the behaviour this change exists to replace. A depressed baseline is the diagnosed trigger, so floor the gap at 10. It binds at baselines below 15 and leaves the 2026-08-05 case (spike 15, hard 27) untouched. The pause directive told the agent to describe what it would have filed "in a comment on the triggering thread", but the workflows whose entire deliverable is a new issue or PR are the scheduled and workflow_run ones, which have no thread. The note now names /tmp/claude/step-summary.md for those, which the Claude action already copies into the job summary; add the same step to the Codex action, where two bundled skills were already writing there into nothing. The pause note was also exported when a burst check had already set ABORT, so a run that exits 1 could still export a directive for an agent that never starts.
tend-agent
left a comment
tend-agent
left a comment
There was a problem hiding this comment.
The floor and the ABORT guard both check out — at a zero baseline the pause band is 11–20 with the hard tier at 20, and a burst abort now exports nothing. Tests pass locally (10 passed in generator/tests/test_shared_steps.py). One gap in the new threadless-run destination, inline.
…gent to The pause directive pointed threadless runs at /tmp/claude/step-summary.md, but nothing created /tmp/claude — the two skills already using the path each run their own `mkdir -p`. Claude's Write tool creates missing parents, so the directive worked there; Codex writes through the shell, where the missing directory is a hard failure, and the copy step added in the previous commit would then find nothing to read on exactly the runs it exists for. Create it deterministically rather than relying on the model: in setup-sandbox.sh as the sandbox user (/tmp is 1777, so a runner-owned directory would be unwritable by the agent), and in the Run Codex step as the runner user. Drop "with the Write tool" from the directive, which is Claude-specific text staged into Codex's AGENTS.md.
|
Corroborating evidence from the Both write-ups attribute the trip to a supply-side artifact — the 07-30…08-02 quota outage suppressing the baseline to 17. That holds, but it only explains half the ratio. Attributing all 16 items by branch prefix:
The largest single block is Two things follow. It settles open question 1 in #855 against the The ratio has no ceiling, because its two halves are independent. The gate's baseline is The forward-looking caveat, which I don't think changes the design: replaying this window against the new tiers gives 16 vs spike 15, hard 27 — a pause, correctly. But 6 of the 6 paused creations would have been triage PRs fixing consumer-reported bugs, so on an inbound-heavy day the pause note's fallback path (describe what you would have filed in a thread comment) becomes triage's normal outcome rather than a rare one. That path is fine for a scheduled run writing to the job summary; for triage the deliverable lands as a comment on an issue whose reporter is a bot in another repo, and nothing re-drives it into a PR once the day rolls over. Worth a look at whether the pause should defer creation to the next run rather than substitute a comment for it — as a follow-up, not a change to this PR. Method
|
|
Tried to merge What landed5b30d22 — "feat(rate-limit): let a maintainer approve past the spike limit" (#874) — rewrote the same spike branch of
So the notice-and-reset half of this PR's problem statement is now covered. The other half is not: #874 still refuses reviews, mention replies, triage comments, and CI fixes on a spike trip — work that creates nothing and so cannot contribute to the count being enforced — until a person notices the issue and closes it. That gap is real and this PR is the only thing that addresses it. Which hunks need human judgment
RecommendationClose this PR, and if the "don't drop work that creates nothing" gap is still wanted, take it as a small change on top of #874's script rather than a merge of this branch — the shared code underneath it has moved far enough that the diff is a rewrite either way. Two hunks here are worth salvaging first, because they are independent of the tier design and fix a live bug on main:
Neither touches the rate-limit script and both merge cleanly today.
|
) ## Problem `review-reviewers` matrixes over five target repos ([`review-reviewers.yaml`](https://github.com/max-sixty/tend/blob/f3e309a475af03147ce6290104c85baafe6ad9f5/.github/workflows/review-reviewers.yaml)), and all five legs of a tick share one `$GITHUB_RUN_ID`. Step 5 of the skill hands every leg the same branch recipe: > **PR** (default): Branch `hourly/review-$GITHUB_RUN_ID`, fix, commit, push, … So the moment two legs of the same tick both find something worth a PR, they contend for one ref. The loser's `git push` is rejected as a non-fast-forward and it has to recover mid-session; the ordering that hurts more is the other one, where the second leg pushes *before* the first opens its PR and the two unrelated concerns land on a single branch — the atomic-PR rule broken by the branch name, not by any judgement the agent made. This just happened on run 31058673934: the `max-sixty/cargo-affected` leg's push to `hourly/review-31058673934` was rejected against a sibling leg's commit, which by then was #856. It recovered by suffixing the target name (#857), and an earlier tick shows the same improvisation baked into `hourly/review-30962483562-numbagg` — legs have been working around the recipe rather than following it. The run ID is non-unique along a second axis too: the same section caps a leg at two PRs (`**Limit to at most 2 PRs per run**`), and a leg that acts on two findings computes one name for both — same collision, one job instead of two. ## Fix Put both discriminators in the recipe: `hourly/review-$GITHUB_RUN_ID-<target-repo-name>-<topic-slug>`. The target name is unique across the matrix by construction, the topic slug separates a single leg's two PRs, and the branch stays greppable back to its run. Scoped to this skill. `review-runs` carries the same `$GITHUB_RUN_ID`-only shape, but `tend-review-runs.yaml` has no matrix, so a run there is a single job and the name can't collide — no change needed and none made. **Overlaps #845 on this line.** That PR rewrites the same recipe to rename the prefix (`hourly/` → `review-reviewers/`) as part of its cadence change, and keeps `$GITHUB_RUN_ID` as the whole discriminator — so it ships the collision under a new name. The two are orthogonal in substance and conflict only textually: whichever lands second wants `<prefix>/review-$GITHUB_RUN_ID-<target-repo-name>-<topic-slug>`, taking the prefix from #845 and the suffixes from here. ## Gate assessment - **Evidence level**: High — the collision is deterministic given two acting legs, and there are two independent traces of it (this run's rejected push, and a prior tick's hand-suffixed branch). The one-leg-two-PRs axis is structural rather than traced. - **Structural, not stochastic**: no decision point. Every leg is told to compute the same name from the same variable; replayed ten times it collides ten times. - **Change type**: targeted fix — one line of the recipe. - **Passes both gates.** Evidence log: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb --------- Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
…nst tend before filing upstream (#891) ## Problem Two dedup blocks were blind in two different ways, and the cited duplicate needed both fixed. **State filter.** `review-runs` Step 5 and `review-reviewers` Step 4 both deduped against PRs with `gh pr list --state open`. A merged PR is never returned by that query, so a finding whose fix already landed reads as undeduped and gets filed again. `running-in-ci`'s PR-creation dedup recheck already gets this right ("with `--state all` so closed and merged siblings show up"); these two recipes contradicted it. **Repo scope.** `review-runs` is a generated workflow ([`generator/src/tend/config.py:24`](https://github.com/max-sixty/tend/blob/f65f49f/generator/src/tend/config.py#L24) lists it in the enabled set), so it runs in each adopter's checkout and an unqualified `gh pr list` returns *the adopter's* PRs. Step 6 routes bundled-skill defects upstream to tend, but neither Step 5 nor any of `running-in-ci`'s dedup recipes — all local-repo — told the agent to dedup in the target repo before filing there. `--state all` alone does not close this: the adopter's PR list never contained the upstream fix at any state. `review-reviewers` is unaffected by the second half. It runs in `max-sixty/tend` and files onto tend, so its unqualified `gh pr list` already resolves to the right repo; only the state filter was wrong there. This bites hardest on tend specifically, because of the pinning model: adopters call `max-sixty/tend/<harness>@X.Y.Z`, so a merged skill fix stays dormant on their repos until the next release tags. The bug keeps reproducing after the fix merges — which is exactly the window in which the analysis legs are looking at it, and exactly when the dedup queries are blind to the fix. ## What happened `max-sixty/cargo-affected`'s `tend-review-runs` run [31160677649](https://github.com/max-sixty/cargo-affected/actions/runs/31160677649) (08:11:33Z → 08:21:41Z) hit the `| last` evidence-log mis-selection: it appended ~12 KB of run evidence into the nightly's unrelated comment on target [#73](max-sixty/cargo-affected#73), noticed on its post-verify read, restored comment `5188771252`, and re-appended to the real log `5150650688`. Good recovery. It then filed [#883](#883) upstream, whose "Proposed fix" is a `## Run ` heading predicate on the comment selector. [#875](#875) merged that exact fix at 07:34:40Z — 46 minutes before the issue was filed — as `test("^## Run [0-9]")` on the same selector, in the same file. #883 is a duplicate of a merged PR. The run made three dedup queries before filing (`gh issue list --state all --search "tracking issue comment append"`, a broader `gh issue list --state all` title regex, and a final `gh issue list --state open` recheck). All three were `gh issue list`, which never returns PRs — and all three ran against `max-sixty/cargo-affected`. Even had it run Step 5's PR line verbatim, it would not have returned #875, for both reasons: the state filter excluded merged PRs, and the query's repo was the adopter's, not tend's. ## The fix - Both skills: `gh pr list --state open` → `--state all`, projecting `state,mergedAt` so a merged hit is legible. - `review-runs` only: add the cross-repo pair (`gh pr list`/`gh issue list --repo max-sixty/tend --state all`) so a finding heading upstream under Step 6 is deduped against tend first. - `review-runs` only: the pinning note is scoped to the upstream repo, since in that skill the reader is the adopter and the local `gh pr list` above it has nothing to do with pinned refs. `review-reviewers` keeps the original wording, where tend is the reader and "on adopters" is the correct direction. Both added commands were run against this repo to confirm they parse and return the expected shape. ## Gate assessment - **Evidence level**: High. **Occurrences: 1** direct, verified end to end (session log, both dedup query sets, #875's merge time and diff, #883's body). - **Structural, not stochastic.** `gh pr list --state open` deterministically cannot return a merged PR, and a query scoped to the adopter's repo deterministically cannot return a tend PR; replayed ten times it misses #875 ten times. There is no decision point. - **Change type**: targeted fix — query lines plus one sentence naming the pinning consequence. It brings both recipes into line with a rule the same plugin already states in `running-in-ci`, rather than introducing new policy. - **Why act at one occurrence**: the six-PR batch merged at 07:34:40Z ([#875](#875), [#834](#834), [#868](#868), [#818](#818), [#877](#877), [#858](#858)) is all unreleased, so six distinct bugs remain observable on every adopter until the next release. The first analysis leg after that batch produced the first duplicate. The exposure is six-wide and standing, not one-off. ## Not covered by No open PR touches either dedup block. [#845](https://github.com/max-sixty/tend/pull/845)/[#850](https://github.com/max-sixty/tend/pull/850)/[#838](https://github.com/max-sixty/tend/pull/838) are the run-window cluster; [#849](#849), [#856](#856), [#857](#857), [#864](#864), [#869](#869), [#876](#876), [#809](#809), [#837](#837), [#821](#821), [#836](#836) are elsewhere. #883 stays open as a maintainer call — it is redundant with #875 but the option-2 half of its body (finish the gist migration for `review-runs`) is not. Evidence: https://gist.github.com/dca23a6e6a0d8cae2665944ba31676fb --------- Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
|
Left conflicted on purpose — this one isn't a mechanical resolution.
The two answers to #855 are genuinely different: this PR keeps the run alive with creation paused ( Not rebasing it blind. If the composed shape is the one you want, say so and I'll rebase onto the current tier and rewrite the notice accordingly; if What conflicts
The non-conflicting half of this branch ( |
2 participants
On 2026-08-05 the bot went dark on its own repo for twelve hours. Every
tend-*andreview-reviewersrun from 11:47Z to the end of the UTC day died in the same shell step, before the agent started, with the same annotation:35 runs failed that way —
tend-notifications×19,review-reviewers×12 (each fanning out to 5 matrix legs, so 60 failed jobs),tend-review×3,tend-mention×1. The dropped work was real, not just monitoring: three pull requests never got reviewed and one human comment never got answered. The first casualty was run 31002862308 at 11:47:16Z — thetend-reviewfired by the creation of #850, which was itself item 16.Why it fired
shared/steps/rate-limit-preflight.shcompares today's issue+PR count against a 6-day baseline:spike_limit = 10 + past_posts / 3. The bot's per-day creation counts leading in:Baseline 17 → limit 15. The 16th item tripped it, and because the count only resets at UTC midnight, it stayed tripped for the rest of the day. Nothing was wrong with those 16 items; they are the ordinary output of the review/review-runs/nightly workflows, and most are still open for review. The low baseline is itself an artifact — 07-30 through 08-02 fall inside the model weekly-quota outage that reset at 2026-08-05T00:00Z, so the guard was calibrated on outage days and then fired on the recovery day. A quiet week makes the next busy day fail.
What's wrong with the response, not the threshold
The guard exists to stop runaway issue/PR creation, and hard-aborting is the right response to a loop. But it aborts every invocation, including work that creates nothing and so cannot contribute to the count it is enforcing — reviews, mention replies, triage comments, CI fixes, and the read-only analysis workflows. Raising the constant would only move the cliff.
So this splits the response by which failure shape is actually present:
max(10 + past_posts, spike_limit + 10)— a whole baseline week's output in one day, floored 10 above the spike tier10 + past_posts / 3On a spike-tier trip the script now emits a warning and writes
TEND_CREATION_PAUSED_NOTEtoGITHUB_ENV;compose-system-prompt.sh(Claude) and theStage AGENTS.mdstep (Codex) append it to the agent's prompt, telling it not to open a new issue or PR this run and to describe what it would have filed in a thread comment instead. Everything else proceeds. Replaying 2026-08-05 against the new tiers: 16 vs spike 15, hard 27 → creation paused, and the three reviews and one mention reply happen.The honest tradeoff: the spike tier becomes a prompt directive, which a prompt-injected session could disregard where an
exit 1could not. That is why it sits below a hard ceiling rather than replacing one — the two abort tiers are still pure shell, still unskippable, and still catch both the fast loop (burst) and the slow one (hard).docs/security-model.mdandREADME.mdare updated to describe the tiers as they now are.Verification
generator/tests/test_shared_steps.pycovers the tiers against a stubbedgh(7 tests):TEND_CREATION_PAUSED_NOTEwrittenBoth limits scale off the same baseline, so without the floor they converge as it falls and coincide at 10 when it reaches 0 — the pause tier would be unreachable and the run would hard-abort on the 11th item, exactly the pre-PR behaviour. A depressed baseline is the diagnosed trigger, so the floor keeps a pause band at least 10 wide at any baseline. It binds below a baseline of 15 and leaves the 2026-08-05 case (spike 15, hard 27) unchanged.
On a spike-tier trip a run with no triggering thread — the scheduled and
workflow_runworkflows, whose whole deliverable is a new issue or PR — is told to write what it would have filed to/tmp/claude/step-summary.md, which a later step copies into the job summary, so the work isn't discarded with the runner.claude/action.yamlalready had that step and two bundled skills already used it;codex/action.yamldid not, so it's added here.compose-system-prompt.shwas run with the note set and appends it after the adopter'ssystem_prompt_append. The Codex step body parses and passesbash -n; shellcheck is clean on both shared scripts.Gate assessment