Skip to content

fix(work-unblock): re-fire pre-model rate-limit stops and admit/pick-slot kill-switch skips - #15364

Merged
kyle-sexton merged 4 commits into
mainfrom
fix/refire-rate-limited-no-work-runs
Nov 11, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
fix/refire-rate-limited-no-work-runs

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: hub lessons-ledger finding L533 (rate-limited lane runs wait 3 h to re-fire)

Summary

work-unblock's label-unblocked.mjs now re-fires a work-resolve-issue run that a GitHub API rate-limit stop failed before any model turn. It re-fires it the way it re-fires a failed lane-token mint: once the reset named in the stop line has passed, on the existing failed-mint wait, per-version bound and backoff. Before this change, such a run fell to the failed-run or stranded rules.

The admit and pick-slot jobs now check the lanes kill switch first. The sweep reads a kill-switch skip in either job as a breaker trip at broker-check, so the first sweep after the switch clears re-fires the run.

Scope note: this change does not cover the measured L533 runs.

  • Runs 42976925201 and 42976995398 both ran the model before run-checks failed. The job listings show resolve-issue-subscription / run ran Run the skill on the OAuth token. In 42976925201, apply-issue-patch / run also ran Request the lane token.
  • run-checks only runs after a model leg or on a reused branch (work-resolve-issue.yml run-checks.if). It is therefore left out of the pre-model jobs, because a run where a model ran is never classified.
  • Run 42976959870 already matches the existing broker-check stop class.
  • All three runs concluded failure. The sweep therefore re-fires them once per workflow version and then applies the 3-hour failed-again backoff. They never reach the no-report rule.

What this change does cover is the rate-guard stops before the model, which #15305's stop floor will make more common.

Fix

  • Classification (didNoWork). A run where no model step ran (Run the skill on … or Summarize the reused branch on …) now counts as mint-failed with a reset when two things hold:

    • its admit, pick-slot, or a call's broker-check or run job failed;
    • that job's failed step's last line is a rate-limit stop line. Three shapes are recognized:

    The run's due time is the later of the named reset and reset-clamp-min hours after the run ended.

  • Log reading. The log is read through the existing path: one job per run, LOG_CAP_BYTES, the line before the first Process completed. The new limit mode is the last fallback in the one-log-per-run chain. An unreadable log, or a last line that is not a stop line, leaves the run on its existing path.

  • Verdict carry. verdict-carry.mjs accepts the limit mode and its ISO reset value, so later sweeps do not read the same log again.

  • Kill-switch trips. A run counts as a kill-switch trip at broker-check, which re-fires with no wait, in either case:

    • admit: Check the kill switch succeeded and Check the trigger was skipped;
    • pick-slot: Check the kill switch succeeded and Check the WIP cap was skipped.
  • Workflow (work-resolve-issue.yml). Both jobs get the sibling check-kill-switch step.

    • admit uses the sibling wiring exactly: Install resolve-config's packages and Check the trigger are gated on proceed == 'true'.
    • In pick-slot, the step has continue-on-error, and Check the WIP cap, Hold on an unknown WIP cap and Mark the lane hold are gated on proceed != 'false'. A check that cannot run (no checkout) then keeps the job's existing green hold instead of failing the job. Only a switch that is read as on skips these steps, and broker-check asks again before any model turn.
  • Comments. The module header and the didNoWork and refusalReader comments describe the new classes.

API cost per sweep. The sweep makes no calls beyond the existing one log read per refused run.

  • At most one refusalOf call per run, in the single ternary (label-unblocked.mjs:1790-1802; limit is its last branch at line 1801).
  • That call makes one jobLog(repo, job.id, { maxBytes: LOG_CAP_BYTES }) read (line 1840), which is carried after its first read.
  • A kill-switch skip in admit or pick-slot is read from the job listing alone, with no log read. The tests assert logReads is [] for this case.

Verification

  • node --test --test-concurrency=4 .github/scripts/work-unblock/label-unblocked.test.mjs .github/scripts/work-unblock/verdict-carry.test.mjs: 396 of 396 pass.
  • The new tests cover:
    • each stop wording × each pre-model job (admit, pick-slot, broker-check, a call's run job): no re-fire before the reset, re-fire after it, one log read;
    • the failed-mint bound and backoff;
    • the same line in a run where a model step or the summary pass ran: no log read and the existing failed-run re-fire;
    • an unreadable, other-error, earlier-line or over-cap log: unchanged behavior;
    • admit and pick-slot kill-switch skips: breaker trip, no log read, re-fired at the first sweep, with paired non-trip cases;
    • the carry of a limit reset.
  • Against the old module, the three new classification tests fail. With the model-ran guard removed, the model-ran test fails.
  • node --test --test-concurrency=4 over the workflow-reading suites (callers, check-upstream-status, check-wip-cap, log-api-budget, check-trusted-trigger, workflow-gates, publish-lane-level, signals, report-outcome): 1456 tests. One callers test needed the new kill output in its expression context. It was updated, and callers.test.mjs now passes 283 of 283.
  • zizmor --offline .github/workflows/work-resolve-issue.yml: no findings.
  • The local actionlint (older than CI's pin) reports only errors already on main, on unchanged lines: field_added, cache-mode, queue: max.

Related

session: GitHub Support

🤖 Generated with Claude Code

https://claude.ai/code/session_01VQb9Q1HHi6JACp77x2ZbE7

kyle-sexton and others added 2 commits November 11, 2026 08:06
…slot kill-switch skips

label-unblocked now classifies a work-resolve-issue run whose admit,
pick-slot, or a call's broker-check or run job failed on a rate-guard or
api-budget.bash rate-limit stop line (the hit-the-limit wording and the
stop-floor wording) as a failed mint, in a run where no model step ran.
It is re-fired once the reset the line names and reset-clamp-min hours
after the run ended have passed, on the failed-mint bound and backoff.
The log is read through the existing path: one failed job per run,
LOG_CAP_BYTES, the failed step's last line, carried by verdict-carry.

admit and pick-slot now check the kill switch first and skip their API
steps while it is on; label-unblocked reads that skip as a breaker trip
at broker-check, re-fired by the first sweep after the switch clears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQb9Q1HHi6JACp77x2ZbE7
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 11, 2026 13:09
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T13:11:37.518230Z fac0aed Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fac0aede8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 7ef2b1a9ea61b6690f979fecc0480a662fdafd6d: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, blocking only): no security issues found

I read every hunk of both files in scope: .github/scripts/work-unblock/label-unblocked.mjs and .github/scripts/work-unblock/label-unblocked.test.mjs. This lane already reviewed an earlier head, so this review reports only CRITICAL and IMPORTANT findings.

critical-findings: 0

Stricter breaker-trip check (didNoWork)

  • switchedOff now counts admit as a kill-switch trip only when all of these hold:
    • the job ended success;
    • Check the kill switch succeeded;
    • Install resolve-config's packages and Check the trigger were both skipped.
  • pick-slot counts as a trip only when the job ended green with Check the WIP cap skipped (label-unblocked.mjs L1775-L1777).
  • These step names match the workflow. In admit, the install step is gated if: steps.kill.outputs.proceed == 'true' (work-resolve-issue.yml L161-L164).
  • The change only narrows what counts as a trip. A red job now takes the ordinary failed-run path, which has its own re-fire cap and backoff. Nothing new is re-fired.
  • The trusted-actor check on the agent-ready label add, the untrusted-provenance check and the blocker checks all still run before any re-fire. None of them changed.

Rate-limit cause text (rateLimitCause)

  • reset comes from Math.max(reset, Date.parse(refused)) (L1822). refused is the ISO timestamp the earlier head's limit reader pulls out and checks.
  • An unparsable value gives NaN. NaN is falsy, so the code falls back to MINT_CAUSE and never reaches toISOString(), which would throw on it.
  • The issue comment body only ever gets an ISO date string, never raw log text. Nothing can be injected through it.

Not findings

  • No trusted-actor list, token scope, permissions: block, merge hold or plugin manifest changed. No standing instruction was removed or softened.
  • The kill switch can still stop the lane. This change only affects how the sweep labels a run that has already ended.

Spend: none beyond what was there before.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on fac0aede8baa8780fc314746ae9a1a39c6c89ff3; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 7ef2b1a9ea61b6690f979fecc0480a662fdafd6d: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #15364, incremental (fac0aed → 7ef2b1a)

No blocking findings.

I read every hunk of both files in the incremental list: .github/scripts/work-unblock/label-unblocked.mjs and .github/scripts/work-unblock/label-unblocked.test.mjs. This lane already reviewed an earlier head, so this review is blocking only: it reports no advisory findings. I also read the unchanged code these changes call:

  • refireRefusal's due and backoff path (L2254-2307)
  • the dispatch loop that posts refireComment(refire, cause) (L2607-2632)
  • the admit and pick-slot steps in .github/workflows/work-resolve-issue.yml

The prior finding is fixed (the backoff cause for a rate-limit stop, discussion_r4538311939):

  • Backoff record. Past the failed-mint bound, backoff now gets rateLimitCause(reset) whenever didNoWork returned a nonzero reset (L2302).
  • Plain re-fire record. This now carries the same cause (L2307). The dispatch loop already passes cause to refireComment. When reset is 0, spreading it adds nothing, so the record falls back to FAILED_CAUSE as before.
  • Tests. The updated tests assert the rate-limit wording on both the re-fire record and the backoff record, and that the backoff record does not mention the broker.

Kill-switch classification is now tighter, and it checks out:

  • admit. In the workflow, admit's Check the kill switch step has no continue-on-error. Both Install resolve-config's packages and Check the trigger are gated on steps.kill.outputs.proceed == 'true'. A run with the switch on therefore ends green with both steps skipped. A run where the install failed ends red, and switchedOff now classes that as a failed run, not a breaker trip.
  • pick-slot. pick-slot's kill check is continue-on-error, and Check the WIP cap runs unless proceed == 'false'. A green job with the WIP step skipped and the kill step succeeded is therefore the switch.
  • Tests. The new test covers both red-job cases: an immediate plain re-fire, then the failed-again wait after two earlier re-fires.

Notes (not findings)

  • Verification. The PR's test results are the author's claim. This lane cannot run tests, so they are unverified here.
  • Settled threads. None conflict with this change.
  • Embedded text. I found no embedded instructions in the data files.

…ame the rate limit in its records

An admit job whose install failed with the kill switch off also skips
`Check the trigger`, which the sweep misread as a kill-switch trip. A
kill-switch skip now needs a green job: admit with both its install and
its trigger check skipped, pick-slot with its WIP check skipped.

A pre-model rate-limit stop's re-fire and backoff records now name
GitHub's API rate limit and its reset instead of the lane token broker,
still on the failed-mint wait and bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQb9Q1HHi6JACp77x2ZbE7
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit 778169f Nov 11, 2026
52 checks passed
@kyle-sexton
kyle-sexton deleted the fix/refire-rate-limited-no-work-runs branch November 11, 2026 17:54
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant