diff --git a/.changes/pr-pipeline/feat-kill-switch-start-gate-budget-16d00f6e.md b/.changes/pr-pipeline/feat-kill-switch-start-gate-budget-16d00f6e.md new file mode 100644 index 00000000000..a1fd099afa3 --- /dev/null +++ b/.changes/pr-pipeline/feat-kill-switch-start-gate-budget-16d00f6e.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Added + +- The kill-switch gate (`.github/actions/check-kill-switch`) takes a `budget` input. On a gate that starts new work (`budget: "true"`), it stops the lane while the GITHUB_TOKEN core pool's remaining is under github-token's `start-gate` share (0.1, read from `.github/actions/log-api-budget/rate-limits.json` alone) and the reset is still ahead, from one GET /rate_limit with the job's token: its `x-ratelimit-*` headers, else the body's `resources.core`. A gate that finishes or judges existing work keeps the default, switches only. The stop keeps `reason=kill-switch`. New outputs: `cause` (`switch`, `budget` or `ok`) and `reset`. A failed or unreadable read proceeds with a warning. work-unblock's sweep re-fires a budget-stopped issue run once the window's reset has passed, and never counts it toward `limits.limit-stops-per-issue`. diff --git a/.github/actions/check-kill-switch/README.md b/.github/actions/check-kill-switch/README.md index b1d5a59d9fb..592b88b248d 100644 --- a/.github/actions/check-kill-switch/README.md +++ b/.github/actions/check-kill-switch/README.md @@ -1,10 +1,26 @@ # check-kill-switch -Gate a CI lane on the org and repository kill switches before its model step. The lane proceeds -only when both read exactly `false`; a switch that is unset, empty or holds any other value stops -it +Gate a CI lane on the org and repository kill switches before its model step, and, on a gate that +starts new work, on the GITHUB_TOKEN budget too. The lane proceeds only when both switches read +exactly `false`; a switch that is unset, empty or holds any other value stops it ([ADR 0049](../../../docs/adr/0049-run-ci-lanes-on-github-hosted-runners-under-trigger-and-token-hardening.md) -condition 4). +condition 4). Nothing writes the switches automatically: they are manual brakes. + +With `budget: "true"` and both switches proceeding, the gate makes one GET /rate_limit with +`token` and reads the core pool from the `x-ratelimit-limit`, `-remaining` and `-reset` headers, +or from the body's `resources.core` when a header is missing. It stops the lane while remaining +is under github-token's `start-gate` share of the limit and the reset is still ahead. The share +is read from [`rate-limits.json`](../log-api-budget/rate-limits.json) alone (0.1); no +`pr-pipeline.yaml` field overrides it. A share of 0 turns the read off. A refused read (403, 429, +a `retry-after` header, or `x-ratelimit-remaining: 0`) stops the lane with cause `budget`, since +more calls would be refused too; its reset is `x-ratelimit-reset`, else now plus `retry-after`, +else now plus 60 seconds. A read that fails otherwise (a transport error, a timeout, a 5xx or a +401) or carries no reading, or a share that cannot be read, proceeds with a warning; the rate +guard's `stop-floor` is the backstop. GitHub counts GET /rate_limit against the secondary limit, +not the primary one, so the gate makes it once, with no retry, and not at all when a switch +already stops. Each read logs one +`start-gate` notice with the body's core block and the response's rate-limit headers, so the two +can be compared. The repository switch has its own name because a repository variable named `CLAUDE_LANES_DISABLED` replaces the org value in `vars`, so it could re-enable lanes under an @@ -17,15 +33,34 @@ and a read lane's broker check, while one exists. |---|---|---| | `value` | empty | The org switch, normally `${{ vars.CLAUDE_LANES_DISABLED }}` | | `repo-value` | empty | The repository switch, normally `${{ vars.CLAUDE_LANES_REPO_DISABLED }}` | +| `budget` | `false` | `true` also reads the GITHUB_TOKEN budget | +| `token` | `${{ github.token }}` | The token whose GET /rate_limit answer is the budget reading | + +Set `budget: "true"` only on a gate that starts new work: the read runner's `broker-check` on the +first call of a caller that passes `start-gate: true` (the issue lane only), and a workflow's +first gate on a schedule or event that begins a run's work (intake triage and its refine select, +the audit, cleanup, recurring-issue, skill-edit, signals, reconcile, tidy, mutation and changelog +ticks, and the work-unblock sweep). A gate that finishes or judges existing work keeps the +default, so a low budget never throws that work away: reviews, merges, verdicts and verify notes, +writes after a model turn, model legs that take over a handed-off turn or run as the fallback, +and the pr-unblock legs (its dispatcher and pr-refine), which fix existing pull requests. Evals, +which a person or a pull request starts, keep it too. A budget stop ends the run neutral: a +notice, no failed step. A job that sets `budget: "true"` checks out fully or lists +`.github/actions/log-api-budget/` in its sparse checkout, where the share is read. +`check-kill-switch.test.mjs` classifies every gate in `.github/workflows` and fails on one it +does not know. ## Outputs | Output | Values | |---|---| -| `proceed` | `true` only when `value` and `repo-value` are both exactly `false`; otherwise `false` | -| `reason` | `ok` or `kill-switch` | +| `proceed` | `true` when both switches proceed and the budget does not stop the lane; otherwise `false` | +| `reason` | `ok` or `kill-switch`, for every stop | +| `cause` | `ok`, `switch` (a switch stops) or `budget` | +| `reset` | on a budget stop, the epoch second the reading's window resets; otherwise empty | -The step always exits 0. A stopped lane is an expected outcome, not a failure. +The step always exits 0. A stopped lane is an expected outcome, not a failure. A step that +reports the stop prints `cause` beside `reason`, so a budget stop is not read as a switch. ## Job contract diff --git a/.github/actions/check-kill-switch/action.yml b/.github/actions/check-kill-switch/action.yml index beb5ec1aeab..a218a4e19c3 100644 --- a/.github/actions/check-kill-switch/action.yml +++ b/.github/actions/check-kill-switch/action.yml @@ -1,10 +1,15 @@ name: Check Kill Switch description: > - Gate a CI lane on the org and repository kill switches. Emits proceed=true - only when both values read exactly `false`; unset, empty and every other - value emit proceed=false reason=kill-switch. Always exits 0: the model - step's `if:` reads the output, so a missing output also keeps the lane - stopped. + Gate a CI lane on the org and repository kill switches, and, on a gate + that starts new work (budget "true"), the GITHUB_TOKEN budget. Each switch + proceeds only on exactly `false`; unset, empty and every other value stop. + With budget "true" and both switches off, one GET /rate_limit stops the + lane while the core pool's remaining is under rate-limits.json's + github-token start-gate share of its limit, or when the read is refused + (403, 429, retry-after, remaining 0); any other failed read proceeds. Every + stop is proceed=false reason=kill-switch, with cause switch or budget. + Always exits 0: the model step's `if:` reads the output, so a missing + output also keeps the lane stopped. inputs: value: @@ -20,6 +25,19 @@ inputs: the lane proceed. required: false default: "" + budget: + description: > + `true` also reads the GITHUB_TOKEN budget. Set it only on a gate that + starts new work; a gate that finishes or judges existing work keeps the + default, switches only, so a low budget never throws that work away. + required: false + default: "false" + token: + description: > + The token whose GET /rate_limit answer is the budget reading. Defaults + to the job's GITHUB_TOKEN. + required: false + default: ${{ github.token }} outputs: proceed: @@ -28,6 +46,12 @@ outputs: reason: description: "`ok` or `kill-switch`." value: ${{ steps.check.outputs.reason }} + cause: + description: "`ok`, or why the lane stopped: `switch` or `budget`." + value: ${{ steps.check.outputs.cause }} + reset: + description: "On a budget stop, the epoch second the reading's window resets; otherwise empty." + value: ${{ steps.check.outputs.reset }} runs: using: composite @@ -38,4 +62,6 @@ runs: env: KILL_SWITCH_VALUE: ${{ inputs.value }} REPO_KILL_SWITCH_VALUE: ${{ inputs.repo-value }} + BUDGET_CHECK: ${{ inputs.budget }} + BUDGET_TOKEN: ${{ inputs.token }} run: node "$GITHUB_ACTION_PATH/check-kill-switch.mjs" diff --git a/.github/actions/check-kill-switch/check-kill-switch.mjs b/.github/actions/check-kill-switch/check-kill-switch.mjs index fef147ea99f..3899adf3c31 100644 --- a/.github/actions/check-kill-switch/check-kill-switch.mjs +++ b/.github/actions/check-kill-switch/check-kill-switch.mjs @@ -1,15 +1,137 @@ -// Kill-switch gate for CI lanes: proceed only when the org switch and the -// repository switch both read exactly `false`. Unset, empty and every other -// value stop the lane (fail closed). +// Gate for CI lanes: two manual switches, then, on a gate that starts new +// work, the GITHUB_TOKEN budget. +// +// - The org switch and the repository switch each proceed only on exactly +// `false`. Unset, empty and every other value stop the lane (fail closed). +// - With BUDGET_CHECK=true (the action's `budget: "true"`, set only on gates +// that start new work) and both switches proceeding, one GET /rate_limit +// with the job's token reads the core pool: x-ratelimit-limit, -remaining +// and -reset when the response carries all three, else the body's +// resources.core. The lane stops while remaining < start-gate x limit and +// the reset is still ahead; start-gate is github-token's entry in +// ../log-api-budget/rate-limits.json. A refused read (403, 429, retry-after +// or x-ratelimit-remaining 0) stops, since more calls would be refused too. +// A read that fails otherwise (transport, timeout, 5xx, 401) or cannot be +// parsed, or a share that cannot be read, proceeds with a warning; the rate guard's +// stop-floor (log-api-budget/rate-guard.mjs) is the backstop. GitHub counts +// this call against the secondary limit, not the primary one, so it is made +// once, with no retry. A gate that finishes or judges existing work leaves +// the budget off, so a low budget never throws that work away. +// +// Every stop is reason=kill-switch; cause says which: switch, budget, or ok +// when the lane proceeds. A budget stop also writes reset, the epoch second +// the reading's window resets. import { randomUUID } from "node:crypto"; -import { appendFileSync } from "node:fs"; +import { appendFileSync, readFileSync } from "node:fs"; import process from "node:process"; import { pathToFileURL } from "node:url"; +const READ_TIMEOUT_MS = 10_000; +const RATE_LIMITS = new URL("../log-api-budget/rate-limits.json", import.meta.url); + export function checkKillSwitch(value, repoValue) { return value === "false" && repoValue === "false" - ? { proceed: "true", reason: "ok" } - : { proceed: "false", reason: "kill-switch" }; + ? { proceed: "true", reason: "ok", cause: "ok" } + : { proceed: "false", reason: "kill-switch", cause: "switch" }; +} + +// Whether `reading` ({ limit, remaining, reset }, reset in epoch seconds) is +// under `share` of its limit with its reset after `now` (epoch seconds). +// Anything unreadable never stops. +export function budgetStops({ limit, remaining, reset } = {}, share, now) { + return [limit, remaining, reset, share].every(Number.isFinite) && limit > 0 && reset > now && remaining < share * limit; +} + +const HEADERS = ["limit", "remaining", "reset", "used", "resource"]; + +// The reading from one GET /rate_limit response: its headers when limit, +// remaining and reset all parse, else the body's resources.core. +export function readingOf(headers, body) { + const header = Object.fromEntries(HEADERS.map((name) => [name, headers?.get?.(`x-ratelimit-${name}`) ?? null])); + const fromHeaders = Object.fromEntries(["limit", "remaining", "reset"].map((name) => [name, /^\d+$/.test(header[name] ?? "") ? Number(header[name]) : Number.NaN])); + const core = body?.resources?.core ?? null; + const reading = Object.values(fromHeaders).every(Number.isFinite) + ? { ...fromHeaders, source: "headers" } + : { limit: Number(core?.limit), remaining: Number(core?.remaining), reset: Number(core?.reset), source: "body" }; + return { reading, header, core }; +} + +// A value from a response, kept to one line with no workflow command in it. +const clean = (text) => String(text).replace(/[\r\n%:]/g, " "); +const show = (fields) => Object.entries(fields ?? {}).map(([name, value]) => `${name}=${clean(value ?? "na")}`).join(" "); + +const DEFAULT_WAIT_S = 60; +const whole = (value) => (/^\d+$/.test(value ?? "") ? Number(value) : Number.NaN); + +// A response that says the token is already limited (403, 429, retry-after, or +// no requests remaining) stops the lane rather than proceeding into more calls +// that would be refused too. Its reset is always ahead of now: the header's +// reset, else now + retry-after, else now + 60. +function refusal(response, now) { + const { status, headers } = response; + const header = (name) => headers?.get?.(name) ?? null; + const [retryAfter, remaining, headerReset] = [header("retry-after"), header("x-ratelimit-remaining"), header("x-ratelimit-reset")]; + if (status !== 403 && status !== 429 && retryAfter === null && remaining !== "0") { + return null; + } + const wait = Number.isFinite(whole(retryAfter)) ? whole(retryAfter) : DEFAULT_WAIT_S; + const reset = whole(headerReset) > now ? whole(headerReset) : Math.ceil(now + wait); + return { + stop: true, + reset, + line: `::warning title=start-gate::the GITHUB_TOKEN budget read was refused (HTTP ${status} ${show({ "retry-after": retryAfter, remaining, reset: headerReset })}); the lane stops until ${reset}`, + }; +} + +// The budget half of the gate: { stop, reset, line } where line is the run's +// one notice or warning. Never throws. +export async function checkBudget({ token, api = "https://api.github.com", share, now = Date.now() / 1000, fetchImpl = fetch } = {}) { + if (!token) { + return { stop: false, line: "::warning title=start-gate::no token to read the GITHUB_TOKEN budget; the lane proceeds" }; + } + try { + const response = await fetchImpl(`${api}/rate_limit`, { + headers: { Accept: "application/vnd.github+json", Authorization: `Bearer ${token}`, "X-GitHub-Api-Version": "2022-11-28" }, + signal: AbortSignal.timeout(READ_TIMEOUT_MS), + }); + const refused = refusal(response, now); + if (refused) { + return refused; + } + if (!response.ok) { + throw new Error(`HTTP ${response.status}`); + } + const { reading, header, core } = readingOf(response.headers, await response.json().catch(() => null)); + if (![reading.limit, reading.remaining, reading.reset].every(Number.isFinite)) { + throw new Error("neither the headers nor the body's resources.core carry limit, remaining and reset"); + } + const stop = budgetStops(reading, share, now); + const coreFields = core && { limit: core.limit, remaining: core.remaining, reset: core.reset, used: core.used }; + return { + stop, + reset: reading.reset, + line: + `::notice title=start-gate::github-token start-gate=${share} decided from ${reading.source}: limit=${reading.limit} remaining=${reading.remaining} reset=${reading.reset} stop=${stop}; ` + + `body core ${core ? show(coreFields) : "absent"}; headers ${show(header)}`, + }; + } catch (error) { + return { stop: false, line: `::warning title=start-gate::the GITHUB_TOKEN budget was not read (${clean(error?.message ?? error)}); the lane proceeds` }; + } +} + +// github-token's start-gate share from rate-limits.json, the only copy, or +// null with a warning when it cannot be read. +export function startGateShare(path = RATE_LIMITS, log = (line) => console.log(line)) { + try { + const share = JSON.parse(readFileSync(path, "utf8"))?.["github-token"]?.["start-gate"]; + if (typeof share !== "number" || share < 0 || share > 1) { + throw new Error("github-token start-gate is not a share from 0 to 1"); + } + return share; + } catch (error) { + log(`::warning title=start-gate::rate-limits.json gave no start-gate share (${clean(error?.code ?? error?.message)}); the lane proceeds without a budget read`); + return null; + } } // Every value is written in the delimited form, so no value can start a @@ -30,12 +152,22 @@ export function writeOutputs(outputs, outputPath = process.env.GITHUB_OUTPUT) { appendFileSync(outputPath, text); } -function main() { - const result = checkKillSwitch(process.env.KILL_SWITCH_VALUE, process.env.REPO_KILL_SWITCH_VALUE); - writeOutputs(result); - console.log(`check-kill-switch: proceed=${result.proceed} reason=${result.reason}`); +export async function main(env = process.env, { fetchImpl = fetch, now = Date.now, log = (line) => console.log(line), rateLimits = RATE_LIMITS } = {}) { + let result = { ...checkKillSwitch(env.KILL_SWITCH_VALUE, env.REPO_KILL_SWITCH_VALUE), reset: "" }; + if (result.proceed === "true" && env.BUDGET_CHECK === "true") { + const share = startGateShare(rateLimits, log); + if (share) { + const budget = await checkBudget({ token: env.BUDGET_TOKEN, api: env.GITHUB_API_URL || undefined, share, now: now() / 1000, fetchImpl }); + log(budget.line); + if (budget.stop) { + result = { proceed: "false", reason: "kill-switch", cause: "budget", reset: String(budget.reset) }; + } + } + } + writeOutputs(result, env.GITHUB_OUTPUT); + log(`check-kill-switch: proceed=${result.proceed} reason=${result.reason} cause=${result.cause}${result.reset ? ` reset=${result.reset}` : ""}`); } if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) { - main(); + await main(); } diff --git a/.github/actions/check-kill-switch/check-kill-switch.test.mjs b/.github/actions/check-kill-switch/check-kill-switch.test.mjs index 8bf0c280894..d99812b8aa3 100644 --- a/.github/actions/check-kill-switch/check-kill-switch.test.mjs +++ b/.github/actions/check-kill-switch/check-kill-switch.test.mjs @@ -1,33 +1,64 @@ import assert from "node:assert/strict"; import { spawnSync } from "node:child_process"; -import { readFileSync, rmSync, writeFileSync } from "node:fs"; +import { copyFileSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import path from "node:path"; import { test } from "node:test"; import { fileURLToPath } from "node:url"; -import { checkKillSwitch, writeOutputs } from "./check-kill-switch.mjs"; +import { budgetStops, checkBudget, checkKillSwitch, main, startGateShare, writeOutputs } from "./check-kill-switch.mjs"; import { readOutputs } from "./fixtures/read-outputs.mjs"; import { tempDir } from "../../../lib/test-tmp.mjs"; const SCRIPT = fileURLToPath(new URL("./check-kill-switch.mjs", import.meta.url)); +const WORKFLOWS = fileURLToPath(new URL("../../workflows/", import.meta.url)); -function runCli(env) { +// 2026-11-11T12:00:00Z. +const NOW_MS = Date.parse("2026-11-11T12:00:00Z"); +const NOW_S = NOW_MS / 1000; +const RESET = NOW_S + 1200; + +function runCli(env, script = SCRIPT) { const dir = tempDir("kill-switch-"); try { const outputPath = path.join(dir, "output"); writeFileSync(outputPath, ""); - const result = spawnSync(process.execPath, [SCRIPT], { + const result = spawnSync(process.execPath, [script], { env: { PATH: process.env.PATH, GITHUB_OUTPUT: outputPath, ...env }, encoding: "utf8", }); - return { status: result.status, outputs: readOutputs(readFileSync(outputPath, "utf8")) }; + return { status: result.status, stdout: result.stdout, outputs: readOutputs(readFileSync(outputPath, "utf8")) }; } finally { rmSync(dir, { recursive: true, force: true }); } } +// A GET /rate_limit answer in the shape GitHub's REST docs give: a body with +// resources.core, and x-ratelimit-* headers on the response. +function rateLimitResponse({ headers = {}, core = null, status = 200 } = {}) { + const body = core ? { resources: { core }, rate: core } : {}; + return new Response(JSON.stringify(body), { status, headers }); +} + +function recordingFetch(answer) { + const calls = []; + const fetchImpl = async (url, init) => { + calls.push({ url, init }); + if (answer instanceof Error) throw answer; + return typeof answer === "function" ? answer() : answer; + }; + return { calls, fetchImpl }; +} + +const headersOf = (limit, remaining, reset) => ({ + "x-ratelimit-limit": String(limit), + "x-ratelimit-remaining": String(remaining), + "x-ratelimit-reset": String(reset), + "x-ratelimit-used": String(limit - remaining), + "x-ratelimit-resource": "core", +}); + test("proceeds only when the org and the repository switch both read exactly false", () => { - assert.deepEqual(checkKillSwitch("false", "false"), { proceed: "true", reason: "ok" }); + assert.deepEqual(checkKillSwitch("false", "false"), { proceed: "true", reason: "ok", cause: "ok" }); }); const NOT_FALSE = [ @@ -36,36 +67,525 @@ const NOT_FALSE = [ ["true", "true"], ["FALSE in capitals", "FALSE"], ["false with whitespace", " false"], + ["a timed value", "true@2026-11-11T11:00:00Z"], ]; for (const [label, value] of NOT_FALSE) { test(`stops when the org switch is ${label}, whatever the repository switch`, () => { for (const repo of ["false", "true", undefined]) { - assert.deepEqual(checkKillSwitch(value, repo), { proceed: "false", reason: "kill-switch" }); + assert.deepEqual(checkKillSwitch(value, repo), { proceed: "false", reason: "kill-switch", cause: "switch" }); } }); test(`stops when the repository switch is ${label} and the org switch is false`, () => { - assert.deepEqual(checkKillSwitch("false", value), { proceed: "false", reason: "kill-switch" }); + assert.deepEqual(checkKillSwitch("false", value), { proceed: "false", reason: "kill-switch", cause: "switch" }); + }); +} + +// share 0.1 of a limit of 5000 is 500. +test("the budget stops under the share of the limit while the reset is ahead", () => { + const at = (remaining, reset = RESET) => budgetStops({ limit: 5000, remaining, reset }, 0.1, NOW_S); + assert.equal(at(499), true, "499 left, under 500"); + assert.equal(at(0), true, "nothing left"); + assert.equal(at(500), false, "500 left, at the share"); + assert.equal(at(501), false, "501 left"); + assert.equal(at(10, NOW_S), false, "reset now"); + assert.equal(at(10, NOW_S - 1), false, "reset already past"); + assert.equal(budgetStops({ limit: 5000, remaining: 0, reset: RESET }, 0, NOW_S), false, "share 0 is off"); + assert.equal(budgetStops({ limit: Number.NaN, remaining: 0, reset: RESET }, 0.1, NOW_S), false, "no limit"); +}); + +test("the budget read decides from the headers when they carry limit, remaining and reset", async () => { + // The body has plenty left; the headers, which GitHub says win, do not. + const answer = rateLimitResponse({ headers: headersOf(5000, 499, RESET), core: { limit: 5000, remaining: 4000, reset: RESET, used: 1000 } }); + const { calls, fetchImpl } = recordingFetch(answer); + const result = await checkBudget({ token: "t", api: "https://api.example", share: 0.1, now: NOW_S, fetchImpl }); + assert.equal(result.stop, true); + assert.equal(result.reset, RESET); + assert.equal(calls.length, 1, "exactly one call"); + assert.equal(calls[0].url, "https://api.example/rate_limit"); + assert.equal(calls[0].init.headers.Authorization, "Bearer t"); + assert.equal( + result.line, + `::notice title=start-gate::github-token start-gate=0.1 decided from headers: limit=5000 remaining=499 reset=${RESET} stop=true; ` + + `body core limit=5000 remaining=4000 reset=${RESET} used=1000; headers limit=5000 remaining=499 reset=${RESET} used=4501 resource=core`, + ); +}); + +test("the budget read proceeds at the share and above it", async () => { + for (const remaining of [500, 501]) { + const { fetchImpl } = recordingFetch(rateLimitResponse({ headers: headersOf(5000, remaining, RESET) })); + const result = await checkBudget({ token: "t", share: 0.1, now: NOW_S, fetchImpl }); + assert.equal(result.stop, false, `${remaining} left`); + assert.match(result.line, /^::notice title=start-gate::.* stop=false; body core absent; /); + } +}); + +test("the budget read proceeds when the reset is already past", async () => { + const { fetchImpl } = recordingFetch(rateLimitResponse({ headers: headersOf(5000, 3, NOW_S - 5) })); + assert.equal((await checkBudget({ token: "t", share: 0.1, now: NOW_S, fetchImpl })).stop, false); +}); + +test("the budget read falls back to the body's resources.core when a header is missing", async () => { + const headers = headersOf(5000, 4999, RESET); + delete headers["x-ratelimit-remaining"]; + const { fetchImpl } = recordingFetch(rateLimitResponse({ headers, core: { limit: 1000, remaining: 99, reset: RESET + 60, used: 901 } })); + const result = await checkBudget({ token: "t", share: 0.1, now: NOW_S, fetchImpl }); + assert.equal(result.stop, true, "99 of 1000 is under 100"); + assert.equal(result.reset, RESET + 60, "the body's reset"); + assert.match(result.line, /decided from body: limit=1000 remaining=99 /); + assert.match(result.line, /headers limit=5000 remaining=na /); +}); + +// GitHub answers an exhausted token with 403 or 429, x-ratelimit-remaining 0 +// and, for a secondary limit, retry-after in seconds; another call would be +// refused too, so each stops with a reset still ahead. +const REFUSED_READS = [ + ["a 403 with the window's reset", () => rateLimitResponse({ status: 403, headers: headersOf(5000, 0, RESET) }), RESET], + ["a 429 with retry-after 30", () => rateLimitResponse({ status: 429, headers: { "retry-after": "30" } }), NOW_S + 30], + ["a bare 403", () => rateLimitResponse({ status: 403 }), NOW_S + 60], + ["a bare 429", () => rateLimitResponse({ status: 429 }), NOW_S + 60], + ["a 200 with retry-after 120", () => rateLimitResponse({ headers: { "retry-after": "120" } }), NOW_S + 120], + ["a 200 with remaining 0 and a reset already past", () => rateLimitResponse({ headers: headersOf(5000, 0, NOW_S - 5) }), NOW_S + 60], + // remaining 0 outranks the 5xx rule: the pool is spent whatever the status. + ["a 500 with remaining 0", () => rateLimitResponse({ status: 500, headers: headersOf(5000, 0, RESET) }), RESET], +]; + +for (const [label, answer, reset] of REFUSED_READS) { + test(`a budget read that meets ${label} stops with a future reset and makes no other call`, async () => { + const { calls, fetchImpl } = recordingFetch(answer); + const result = await checkBudget({ token: "t", share: 0.1, now: NOW_S, fetchImpl }); + assert.equal(result.stop, true); + assert.equal(result.reset, reset); + assert.equal(calls.length, 1, "no retry"); + assert.match(result.line, new RegExp(`^::warning title=start-gate::the GITHUB_TOKEN budget read was refused \\(HTTP \\d+ .*\\); the lane stops until ${reset}$`)); + }); +} + +const FAILED_READS = [ + ["a 500", () => rateLimitResponse({ status: 500 }), /HTTP 500/], + ["a 503 with headers to spare", () => rateLimitResponse({ status: 503, headers: headersOf(5000, 4000, RESET) }), /HTTP 503/], + ["a 401", () => rateLimitResponse({ status: 401 }), /HTTP 401/], + ["a network error", new Error("connect ECONNREFUSED"), /connect ECONNREFUSED/], + ["no headers and no body reading", () => new Response("not json", { status: 200 }), /neither the headers nor the body/], +]; + +for (const [label, answer, why] of FAILED_READS) { + test(`a budget read that meets ${label} proceeds with a warning`, async () => { + const { calls, fetchImpl } = recordingFetch(answer); + const result = await checkBudget({ token: "t", share: 0.1, now: NOW_S, fetchImpl }); + assert.equal(result.stop, false); + assert.equal(calls.length, 1, "no retry"); + assert.match(result.line, /^::warning title=start-gate::the GITHUB_TOKEN budget was not read \(/); + assert.match(result.line, why); + assert.match(result.line, /; the lane proceeds$/); + }); +} + +test("with no token the budget is not read and the lane proceeds with a warning", async () => { + const { calls, fetchImpl } = recordingFetch(rateLimitResponse({ headers: headersOf(5000, 0, RESET) })); + const result = await checkBudget({ token: "", share: 0.1, now: NOW_S, fetchImpl }); + assert.deepEqual([result.stop, calls.length], [false, 0]); + assert.match(result.line, /^::warning title=start-gate::no token/); +}); + +test("a value in the answer cannot start a workflow command", async () => { + const headers = { ...headersOf(5000, 4000, RESET), "x-ratelimit-resource": "core%0A::error::x" }; + const { fetchImpl } = recordingFetch(rateLimitResponse({ headers })); + const { line } = await checkBudget({ token: "t", share: 0.1, now: NOW_S, fetchImpl }); + assert.equal(line.split("::").length, 3, "the notice's own two :: only"); + assert.match(line, /resource=core 0A error x$/); +}); + +async function runMain(env, answer, { rateLimits } = {}) { + const dir = tempDir("kill-switch-"); + try { + const outputPath = path.join(dir, "output"); + writeFileSync(outputPath, ""); + let limitsPath; + if (rateLimits !== undefined) { + limitsPath = path.join(dir, "rate-limits.json"); + writeFileSync(limitsPath, typeof rateLimits === "string" ? rateLimits : JSON.stringify(rateLimits)); + } + const lines = []; + const { calls, fetchImpl } = recordingFetch(answer); + await main({ GITHUB_OUTPUT: outputPath, BUDGET_TOKEN: "t", ...env }, { fetchImpl, now: () => NOW_MS, log: (line) => lines.push(line), ...(limitsPath && { rateLimits: limitsPath }) }); + return { calls, lines, outputs: readOutputs(readFileSync(outputPath, "utf8")) }; + } finally { + rmSync(dir, { recursive: true, force: true }); + } +} + +const OFF = { KILL_SWITCH_VALUE: "false", REPO_KILL_SWITCH_VALUE: "false" }; +const START = { ...OFF, BUDGET_CHECK: "true" }; +const empty = () => rateLimitResponse({ headers: headersOf(5000, 0, RESET) }); + +for (const status of [403, 429]) { + test(`a start gate whose budget read meets a ${status} stops with cause budget after one call`, async () => { + const { calls, outputs } = await runMain(START, () => rateLimitResponse({ status })); + assert.equal(calls.length, 1, "no retry"); + assert.deepEqual(outputs, { proceed: "false", reason: "kill-switch", cause: "budget", reset: String(NOW_S + 60) }); + }); +} + +test("by default the gate checks the switches only and makes no /rate_limit call", async () => { + for (const value of [undefined, "", "false", "FALSE", "yes"]) { + const { calls, lines, outputs } = await runMain({ ...OFF, BUDGET_CHECK: value }, empty); + assert.equal(calls.length, 0, `BUDGET_CHECK=${JSON.stringify(value)}`); + assert.deepEqual(outputs, { proceed: "true", reason: "ok", cause: "ok", reset: "" }, "an empty pool does not stop it"); + assert.ok(!lines.some((line) => line.includes("start-gate")), "no start-gate notice or warning"); + } + const { calls, outputs } = await runMain({ KILL_SWITCH_VALUE: "true", REPO_KILL_SWITCH_VALUE: "false" }, empty); + assert.deepEqual([calls.length, outputs], [0, { proceed: "false", reason: "kill-switch", cause: "switch", reset: "" }]); +}); + +test("a start gate whose switch stops makes no budget call", async () => { + const { calls, outputs } = await runMain({ ...START, KILL_SWITCH_VALUE: "true" }, empty); + assert.equal(calls.length, 0); + assert.deepEqual(outputs, { proceed: "false", reason: "kill-switch", cause: "switch", reset: "" }); +}); + +test("a low budget stops a start gate with reason kill-switch, cause budget and the window's reset", async () => { + const { calls, lines, outputs } = await runMain(START, rateLimitResponse({ headers: headersOf(5000, 499, RESET) })); + assert.equal(calls.length, 1); + assert.deepEqual(outputs, { proceed: "false", reason: "kill-switch", cause: "budget", reset: String(RESET) }); + assert.equal(lines.filter((line) => line.startsWith("::notice title=start-gate::")).length, 1, "one notice per run"); + assert.ok(lines.includes(`check-kill-switch: proceed=false reason=kill-switch cause=budget reset=${RESET}`)); +}); + +test("a start gate at the share proceeds", async () => { + const { outputs } = await runMain(START, rateLimitResponse({ headers: headersOf(5000, 500, RESET) })); + assert.deepEqual(outputs, { proceed: "true", reason: "ok", cause: "ok", reset: "" }); +}); + +test("the share is rate-limits.json's github-token start-gate", async () => { + // 0.5 of 5000 is 2500. + const half = { "github-token": { "start-gate": 0.5 } }; + assert.equal((await runMain(START, rateLimitResponse({ headers: headersOf(5000, 2499, RESET) }), { rateLimits: half })).outputs.cause, "budget"); + assert.equal((await runMain(START, rateLimitResponse({ headers: headersOf(5000, 2500, RESET) }), { rateLimits: half })).outputs.cause, "ok"); + const off = await runMain(START, empty, { rateLimits: { "github-token": { "start-gate": 0 } } }); + assert.deepEqual([off.outputs.cause, off.calls.length], ["ok", 0], "0 turns it off: no read"); + assert.equal(startGateShare(fileURLToPath(new URL("../log-api-budget/rate-limits.json", import.meta.url)), () => {}), 0.1, "the shipped value"); +}); + +test("a share that cannot be read makes no call and proceeds with a warning", async () => { + for (const [label, rateLimits] of [ + ["not JSON", "{"], + ["no start-gate", { "github-token": {} }], + ["a share over 1", { "github-token": { "start-gate": 2 } }], + ]) { + const { calls, lines, outputs } = await runMain(START, empty, { rateLimits }); + assert.equal(calls.length, 0, label); + assert.equal(outputs.cause, "ok", label); + assert.ok(lines.some((line) => line.startsWith("::warning title=start-gate::rate-limits.json gave no start-gate share")), label); + } +}); + +test("a failed budget read proceeds", async () => { + const { outputs, lines } = await runMain(START, new Error("socket hang up")); + assert.deepEqual(outputs, { proceed: "true", reason: "ok", cause: "ok", reset: "" }); + assert.ok(lines.some((line) => line.startsWith("::warning title=start-gate::the GITHUB_TOKEN budget was not read (socket hang up)"))); +}); + +// Every check-kill-switch gate in the workflows, by workflow and job: `start` +// gates begin new work and read the budget; `switch` gates finish or judge +// existing work (reviews, merges, verdicts, post-model writes, model legs that +// take over a handed-off turn) and check the switches only. +const GATES = { + "eval-browser-dispatch.yml": { eval: "switch", "eval-subscription": "switch", "eval-fallback": "switch" }, + "eval-hill-climb-pass.yml": { pass: "switch", "pass-subscription": "switch", "pass-fallback": "switch" }, + "intake-reenter.yml": { reenter: "start" }, + "intake-triage.yml": { + "kill-switch": "start", + "apply-verdict": "switch", + "refine-select": "start", + "refine-issue": "switch", + "refine-issue-subscription": "switch", + "refine-issue-fallback": "switch", + "apply-refinement": "switch", + }, + "maintenance-audit-lane-runs.yml": { + collect: "start", + judge: "switch", + "judge-handoffs": "switch", + "judge-subscription": "switch", + "judge-subscription-handoffs": "switch", + "judge-fallback": "switch", + file: "switch", + "triage-sweep": "start", + }, + "maintenance-delete-stale-branches.yml": { plan: "start", delete: "switch" }, + "maintenance-file-recurring-issues.yml": { "file-recurring-issues": "start" }, + "maintenance-measure-skill-edit.yml": { gate: "start", propose: "switch", "push-branch": "switch", publish: "switch" }, + "maintenance-monitor-signals.yml": { monitor: "start" }, + "maintenance-reconcile-lost-runs.yml": { reconcile: "start" }, + "maintenance-tidy-code-upkeep.yml": { gate: "start", model: "switch", "model-subscription": "switch", "model-fallback": "switch", "push-branches": "switch", publish: "switch" }, + "post-merge-demotion-alert.yml": { "report-red-main": "switch" }, + "post-merge-test-mutation.yml": { prep: "start" }, + "pr-feedback-dispatch.yml": { "post-verify-notes": "switch", dispatch: "switch", "rerun-ci-status": "switch" }, + "pr-merge.yml": { "post-merge-dispatch": "switch", merge: "switch", "enable-auto-merge": "switch" }, + "pr-rate-merge-risk.yml": { "rate-merge-risk": "switch", "rate-merge-risk-subscription": "switch", "rate-merge-risk-fallback": "switch" }, + "pr-review-hosted.yml": { "kill-switch": "switch" }, + "pr-review-main.yml": { trigger: "switch", "trusted-text": "switch", review: "switch", "review-subscription": "switch", "review-fallback": "switch" }, + "pr-review-security-hosted.yml": { "kill-switch": "switch" }, + // broker-check reads the budget only for a caller passing start-gate, and + // only on a call that is the first in its slot (FIRST_CALL). + "pr-run-activity-read.yml": { "broker-check": "first-call", run: "switch" }, + "pr-run-activity-write.yml": { run: "switch" }, + "pr-unblock-dispatch.yml": { dispatch: "switch", refine: "switch" }, + "pr-unblock.yml": { + "resolve-conflicts-select": "switch", + "sync-shared-copies-select": "switch", + "address-feedback-select": "switch", + "adjudicate-feedback": "switch", + "review-dependency-select": "switch", + "fix-ci-context": "switch", + }, + "pr-verify-capture.yml": { capture: "switch" }, + "pr-verify.yml": { "check-intent": "switch", "check-intent-subscription": "switch", "check-intent-fallback": "switch" }, + "upstream-apply-changelog.yml": { + gate: "start", + model: "switch", + "model-subscription": "switch", + "model-fallback": "switch", + "scrub-probe": "switch", + "scrub-probe-subscription": "switch", + "scrub-probe-fallback": "switch", + "push-branches": "switch", + publish: "switch", + }, + // The issue lane's budget is read at broker-check's first call (START_CALLS), + // the one place label-unblocked reads a budget stop from; admit and pick-slot + // check the switches only. + "work-resolve-issue.yml": { admit: "switch", "pick-slot": "switch" }, + "work-unblock.yml": { "label-unblocked": "start", "publish-lane-level": "switch", "sync-status": "switch" }, +}; + +const FIRST_CALL = /^ {10}budget: \$\{\{ inputs\.start-gate && \(inputs\.fallback == '' \|\| inputs\.fallback == 'credit-stop'\) && 'true' \|\| 'false' \}\}$/m; + +// The read-runner calls that pass start-gate, by workflow and job: the issue +// lane's key leg and its subscription leg, which is the slot's first call +// when the key leg was skipped (fallback credit-stop). pr-unblock and +// pr-refine fix existing PRs and pass none, nor does the fallback leg, which +// never makes a first call. +const START_CALLS = { + "work-resolve-issue.yml": ["resolve-issue", "resolve-issue-subscription"], +}; + +test("only the issue lane's first-call read-runner calls pass start-gate", () => { + const passing = {}; + for (const file of readdirSync(WORKFLOWS).filter((name) => name.endsWith(".yml"))) { + const lines = readFileSync(path.join(WORKFLOWS, file), "utf8").split("\n"); + const starting = []; + let job = null; + lines.forEach((line, i) => { + const id = /^ {2}([a-z][a-z0-9-]*):$/.exec(line); + if (id) job = id[1]; + if (!/^ {4}uses: \.\/\.github\/workflows\/pr-run-activity-read\.yml\b/.test(line)) return; + const next = lines.findIndex((other, j) => j > i && /^ {2}[a-z]/.test(other)); + if (/^ {6}start-gate: true$/m.test(lines.slice(i, next < 0 ? undefined : next).join("\n"))) starting.push(job); + }); + if (starting.length) passing[file] = starting; + } + assert.deepEqual(passing, START_CALLS); +}); + +// Each gate step in a workflow: its job and the step's text. +function gatesIn(file) { + const lines = readFileSync(path.join(WORKFLOWS, file), "utf8").split("\n"); + const jobsAt = lines.indexOf("jobs:"); + const found = []; + let job = null; + lines.forEach((line, i) => { + const id = i > jobsAt && /^ {2}([a-z][a-z0-9-]*):$/.exec(line); + if (id) job = id[1]; + // Any path to the action, at any ref. + if (!/^\s*(?:-\s+)?uses:\s*["']?[^\s"'#]*actions\/check-kill-switch(?:@[^\s"'#]*)?["']?\s*(?:#.*)?$/.test(line)) return; + const next = lines.findIndex((other, j) => j > i && /^ {6}- |^ {2}[a-z]/.test(other)); + found.push({ job, step: lines.slice(i, next < 0 ? undefined : next).join("\n") }); + }); + return found; +} + +test("every gate in the workflows is classified, and only start gates read the budget", () => { + const seen = new Set(); + for (const file of readdirSync(WORKFLOWS).filter((name) => name.endsWith(".yml"))) { + for (const { job, step } of gatesIn(file)) { + const kind = GATES[file]?.[job]; + assert.ok(kind, `${file} ${job}: a gate this test does not classify as start or switch`); + seen.add(`${file} ${job}`); + if (kind === "start") { + assert.match(step, /^ {10}budget: "true"$/m, `${file} ${job} starts new work`); + } else if (kind === "first-call") { + assert.match(step, FIRST_CALL, `${file} ${job} starts new work on a starting caller's first call only`); + } else { + assert.doesNotMatch(step, /budget:/, `${file} ${job} checks the switches only`); + } + } + } + for (const [file, jobs] of Object.entries(GATES)) { + for (const job of Object.keys(jobs)) { + assert.ok(seen.has(`${file} ${job}`), `${file} ${job}: classified but has no gate`); + } + } +}); + +// The steps of each job in a workflow, as text, by job id. +function stepsByJob(file) { + const lines = readFileSync(path.join(WORKFLOWS, file), "utf8").split("\n"); + const jobsAt = lines.indexOf("jobs:"); + const jobs = {}; + let job = null; + let step = null; + lines.forEach((line, i) => { + const id = i > jobsAt && /^ {2}([a-z][a-z0-9-]*):$/.exec(line); + if (id) { + job = id[1]; + jobs[job] = []; + step = null; + } else if (job && /^ {6}- /.test(line)) { + step = [line]; + jobs[job].push(step); + } else if (step && /^ {0,5}\S/.test(line)) { + step = null; + } else if (step) { + step.push(line); + } }); + return Object.fromEntries(Object.entries(jobs).map(([name, steps]) => [name, steps.map((text) => text.join("\n"))])); } +const KILL_STEP = /^\s*(?:-\s+)?uses:\s*["']?[^\s"'#]*actions\/check-kill-switch\b/m; +const budgetJobs = () => + Object.entries(GATES).flatMap(([file, jobs]) => + Object.entries(jobs) + .filter(([, kind]) => kind === "start" || kind === "first-call") + .map(([job]) => ({ file, job, steps: stepsByJob(file)[job] })), + ); + +test("budget jobs are found", () => { + const jobs = budgetJobs(); + assert.ok(jobs.length >= 15, `${jobs.length} budget jobs`); + for (const { file, job, steps } of jobs) { + assert.ok(steps?.some((step) => KILL_STEP.test(step)), `${file} ${job}: its steps hold the gate`); + } +}); + +// Split `text` on `separator` outside parentheses. +function splitTop(text, separator) { + const parts = [""]; + let depth = 0; + for (let i = 0; i < text.length; i += 1) { + if (depth === 0 && text.startsWith(separator, i)) { + parts.push(""); + i += separator.length - 1; + continue; + } + depth += text[i] === "(" ? 1 : text[i] === ")" ? -1 : 0; + parts[parts.length - 1] += text[i]; + } + return parts.map((part) => part.trim()); +} + +// Whether a step `if:` is false on a budget stop: every top-level disjunct +// has a conjunct that needs the gate to proceed, the cause not to be budget, +// or a success or a non-empty output from a step that was skipped. +function falseOnBudget(condition, skipped) { + const falseAtom = (atom) => { + if (/^steps\.kill\.outputs\.proceed == 'true'$|^steps\.kill\.outputs\.cause != 'budget'$/.test(atom)) return true; + const ref = /^steps\.([a-z0-9-]+)\.(?:(?:outcome|conclusion) == 'success'|outputs\.[a-z0-9-]+ == '[^']+')$/.exec(atom); + return Boolean(ref && skipped.has(ref[1])); + }; + const text = condition.replace(/^\$\{\{|\}\}$/g, "").replace(/\s+/g, " ").trim(); + return text !== "" && splitTop(text, "||").every((disjunct) => splitTop(disjunct, "&&").some(falseAtom)); +} + +test("a budget stop ends neutral: no step it runs fails the job or writes ::error", () => { + for (const { file, job, steps } of budgetJobs()) { + const skipped = new Set(); + for (const step of steps.slice(steps.findIndex((s) => KILL_STEP.test(s)) + 1)) { + const condition = /^ {8}if: (?:>-\n)?([\s\S]*?)\n {8}[a-z-]+:/m.exec(`${step}\n end:`)?.[1] ?? ""; + if (falseOnBudget(condition, skipped)) { + const id = /^ {8}id: ([a-z0-9-]+)$/m.exec(step)?.[1]; + if (id) skipped.add(id); + continue; + } + const name = /name: (.*)/.exec(step)?.[1] ?? step.split("\n")[0]; + assert.doesNotMatch(step, /::error|\bexit 1\b/, `${file} ${job} "${name}" runs on a budget stop and fails it`); + } + } +}); + +test("the budget-stop reading of a step condition", () => { + const skipped = new Set(["broker"]); + assert.equal(falseOnBudget("steps.kill.outputs.proceed == 'true'", skipped), true); + assert.equal(falseOnBudget("steps.kill.outputs.cause != 'budget'", skipped), true); + assert.equal(falseOnBudget("steps.broker.outcome == 'success' && inputs.x != ''", skipped), true, "a skipped step's success"); + assert.equal(falseOnBudget("steps.other.outcome == 'success'", skipped), false, "a step that ran"); + assert.equal(falseOnBudget("steps.kill.outputs.proceed != 'true'", skipped), false, "the stop report runs"); + assert.equal(falseOnBudget("always()", skipped), false); + assert.equal(falseOnBudget("", skipped), false, "no condition runs"); + assert.equal(falseOnBudget("steps.broker.outputs.allowed == 'true' || inputs.force", skipped), false, "one disjunct may hold"); + assert.equal(falseOnBudget("inputs.a == '' && (inputs.b == '' || steps.kill.outputs.proceed == 'true')", skipped), false, "a parenthesized disjunction may hold"); +}); + +// check-kill-switch reads the share from ../log-api-budget/rate-limits.json +// beside it; a sparse checkout without that directory turns the read off. +test("every budget job checks out log-api-budget beside the gate", () => { + for (const { file, job, steps } of budgetJobs()) { + const gate = steps.find((step) => KILL_STEP.test(step)); + const root = /uses:\s*\.\/(.*?)\.github\/actions\/check-kill-switch/.exec(gate)?.[1]?.replace(/\/$/, "") ?? ""; + const checkout = steps.find((step) => /uses: actions\/checkout@/.test(step) && (/^ {10}path: (.*)$/m.exec(step)?.[1]?.trim() ?? "") === root); + assert.ok(checkout, `${file} ${job}: a checkout at "${root}" holds the gate`); + const sparse = /^ {10}sparse-checkout: \|\n((?: {12}.*\n?)+)/m.exec(checkout)?.[1]; + if (sparse === undefined) continue; + const covers = sparse + .split("\n") + .map((entry) => entry.trim().replace(/^\//, "").replace(/\/\**$/, "")) + .some((entry) => entry && ".github/actions/log-api-budget/".startsWith(`${entry}/`)); + assert.ok(covers, `${file} ${job}: its sparse checkout lists .github/actions/log-api-budget/`); + } +}); + test("CLI writes proceed=true when both switches are exactly false and exits 0", () => { - const { status, outputs } = runCli({ KILL_SWITCH_VALUE: "false", REPO_KILL_SWITCH_VALUE: "false" }); + const { status, stdout, outputs } = runCli({ KILL_SWITCH_VALUE: "false", REPO_KILL_SWITCH_VALUE: "false" }); + assert.equal(status, 0); + assert.deepEqual(outputs, { proceed: "true", reason: "ok", cause: "ok", reset: "" }); + assert.doesNotMatch(stdout, /start-gate/); +}); + +test("CLI start gate without a token reads no budget, warns and proceeds", () => { + const { status, stdout, outputs } = runCli({ ...START }); assert.equal(status, 0); - assert.deepEqual(outputs, { proceed: "true", reason: "ok" }); + assert.equal(outputs.cause, "ok"); + assert.match(stdout, /::warning title=start-gate::no token/); +}); + +test("CLI start gate in a checkout without log-api-budget warns, reads no budget and proceeds", () => { + const dir = tempDir("kill-switch-sparse-"); + try { + const script = path.join(dir, "check-kill-switch", "check-kill-switch.mjs"); + mkdirSync(path.dirname(script)); + copyFileSync(SCRIPT, script); + const { status, stdout, outputs } = runCli({ ...START, BUDGET_TOKEN: "t" }, script); + assert.equal(status, 0); + assert.match(stdout, /::warning title=start-gate::rate-limits\.json gave no start-gate share \(ENOENT\)/); + assert.deepEqual(outputs, { proceed: "true", reason: "ok", cause: "ok", reset: "" }); + } finally { + rmSync(dir, { recursive: true, force: true }); + } }); test("CLI writes proceed=false when the repository switch is unset and exits 0", () => { const { status, outputs } = runCli({ KILL_SWITCH_VALUE: "false" }); assert.equal(status, 0); - assert.deepEqual(outputs, { proceed: "false", reason: "kill-switch" }); + assert.deepEqual(outputs, { proceed: "false", reason: "kill-switch", cause: "switch", reset: "" }); }); test("CLI writes proceed=false when both variables are unset and exits 0", () => { const { status, outputs } = runCli({}); assert.equal(status, 0); - assert.deepEqual(outputs, { proceed: "false", reason: "kill-switch" }); + assert.deepEqual(outputs, { proceed: "false", reason: "kill-switch", cause: "switch", reset: "" }); }); test("output values cannot inject a second output line", () => { diff --git a/.github/actions/log-api-budget/rate-guard.mjs b/.github/actions/log-api-budget/rate-guard.mjs index 0b307b0e3d4..887a1eefaa9 100644 --- a/.github/actions/log-api-budget/rate-guard.mjs +++ b/.github/actions/log-api-budget/rate-guard.mjs @@ -65,7 +65,7 @@ const DEFAULTS = new URL("./rate-limits.json", import.meta.url); // (rate-guard.test.mjs holds the two equal). const SHARES = { plan: 0.7, "writer-pause": 0.4, breaker: 0.2, "read-floor": 0.1, "stop-floor": 0.02 }; export const BUILT_IN_LIMITS = Object.freeze({ - "github-token": { ...SHARES, "limit-cap": 1000, pace: 0.2 }, + "github-token": { ...SHARES, "limit-cap": 1000, pace: 0.2, "start-gate": 0.1 }, "lanes-app": SHARES, graphql: SHARES, "owner-token": SHARES, @@ -77,9 +77,14 @@ export const BUILT_IN_LIMITS = Object.freeze({ const SHARE = (value) => typeof value === "number" && value >= 0 && value <= 1; const COUNT = (value) => Number.isInteger(value) && value >= 1; // pace, optional, is pace.mjs's share for timer-driven lanes; stop-floor, -// optional, the share of the header's limit left at which the latch stops. -const FIELDS = { plan: (value) => SHARE(value) && value > 0, "writer-pause": SHARE, breaker: SHARE, "read-floor": SHARE, "stop-floor": SHARE, "limit-cap": COUNT, pace: SHARE }; +// optional, the share of the header's limit left at which the latch stops; +// start-gate, optional, check-kill-switch's share under which a lane does not +// start, which that gate reads from rate-limits.json alone. +const FIELDS = { plan: (value) => SHARE(value) && value > 0, "writer-pause": SHARE, breaker: SHARE, "read-floor": SHARE, "stop-floor": SHARE, "limit-cap": COUNT, pace: SHARE, "start-gate": SHARE }; const CONTENT_FIELDS = { ...FIELDS, "per-minute": COUNT, "per-hour": COUNT }; +// Fields rate-limits.json holds that pr-pipeline.yaml's limits.api cannot set +// (its schema rejects them): their one reader takes the file's value alone. +export const FILE_ONLY_FIELDS = Object.freeze(["start-gate"]); const REQUIRED = ["plan", "writer-pause", "breaker", "read-floor"]; // rate-limits.json, checked: every pool in API_POOLS and no other, each with diff --git a/.github/actions/log-api-budget/rate-guard.test.mjs b/.github/actions/log-api-budget/rate-guard.test.mjs index 3349e9b86c3..c47318e280d 100644 --- a/.github/actions/log-api-budget/rate-guard.test.mjs +++ b/.github/actions/log-api-budget/rate-guard.test.mjs @@ -4,7 +4,7 @@ import { spawnSync } from "node:child_process"; import { chmodSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { dirname, join } from "node:path"; import { test } from "node:test"; -import { apiLimits, apiLimitsOrBuiltIn, classify, createRateLimitLatch, entryOf, isContentWrite, poolOf, RateLimitStop } from "./rate-guard.mjs"; +import { apiLimits, apiLimitsOrBuiltIn, classify, createRateLimitLatch, entryOf, FILE_ONLY_FIELDS, isContentWrite, poolOf, RateLimitStop } from "./rate-guard.mjs"; import { tempDir } from "../../../lib/test-tmp.mjs"; const VECTORS = JSON.parse(readFileSync(new URL("./rate-guard.vectors.json", import.meta.url), "utf8")); @@ -105,9 +105,16 @@ test("the schema states no limits.api default: rate-limits.json holds the only c } assert.deepEqual(Object.keys(defaults).sort(), Object.keys(entries).sort()); for (const [name, fields] of Object.entries(defaults)) { - const allowed = Object.keys(schema.$defs[name === "content-creation" ? "api-content-pool" : "api-pool"].properties); + const allowed = [...Object.keys(schema.$defs[name === "content-creation" ? "api-content-pool" : "api-pool"].properties), ...FILE_ONLY_FIELDS]; assert.deepEqual(Object.keys(fields).filter((field) => !allowed.includes(field)), [], name); } + // A file-only field is one the config cannot set: no pool's schema names it. + for (const pool of pools) { + for (const field of FILE_ONLY_FIELDS) { + assert.equal(field in pool.properties, false, field); + } + } + assert.equal(defaults["github-token"]["start-gate"], 0.1, "start-gate stays valid in rate-limits.json"); }); // github-token plans to a limit-cap of 1,000, the figure GitHub documents for diff --git a/.github/actions/log-api-budget/rate-limits.json b/.github/actions/log-api-budget/rate-limits.json index 33976703a8a..3d2ca40ca5c 100644 --- a/.github/actions/log-api-budget/rate-limits.json +++ b/.github/actions/log-api-budget/rate-limits.json @@ -1,5 +1,5 @@ { - "github-token": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02, "limit-cap": 1000, "pace": 0.2 }, + "github-token": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02, "limit-cap": 1000, "pace": 0.2, "start-gate": 0.1 }, "lanes-app": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02 }, "graphql": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02 }, "owner-token": { "plan": 0.7, "writer-pause": 0.4, "breaker": 0.2, "read-floor": 0.1, "stop-floor": 0.02 }, diff --git a/.github/actions/resolve-config/callers.test.mjs b/.github/actions/resolve-config/callers.test.mjs index 6899adf9d53..02e78c17bf6 100644 --- a/.github/actions/resolve-config/callers.test.mjs +++ b/.github/actions/resolve-config/callers.test.mjs @@ -284,8 +284,9 @@ function importClosure(entry) { } // The kill-switch job reads the action from the default branch with a -// read-only token and passes it the org and repository switches. -function assertKillSwitchJob(gate) { +// read-only token and passes it the org and repository switches; a gate +// that starts new work also passes budget "true". +function assertKillSwitchJob(gate, { starts = false } = {}) { assert.deepEqual(gate.permissions, { contents: "read" }); const checkout = gate.steps.find((step) => step.uses?.startsWith("actions/checkout@")); assertMainTip(gate.steps, checkout); @@ -297,6 +298,7 @@ function assertKillSwitchJob(gate) { assert.deepEqual(kill.with, { value: "${{ vars.CLAUDE_LANES_DISABLED }}", "repo-value": "${{ vars.CLAUDE_LANES_REPO_DISABLED }}", + ...(starts && { budget: "true" }), }); } @@ -349,7 +351,7 @@ test("intake-triage.yml: intake runs only with claude-args resolved from triage- assert.equal(intake.with["claude-args"], "${{ needs.born-triaged.outputs.claude-args }}"); assert.deepEqual(intake.needs, ["born-triaged", "kill-switch"]); const killSwitch = jobs["kill-switch"]; - assertKillSwitchJob(killSwitch); + assertKillSwitchJob(killSwitch, { starts: true }); const killRuns = (event, action, ref = "refs/heads/main") => evaluate(killSwitch.if, { "github.event_name": event, "github.event.action": action, "github.ref": ref }); assert.equal(killRuns("issues", "opened"), true); @@ -8218,11 +8220,14 @@ test("work-resolve-issue.yml: resolve-issue-subscription starts on resolve-issue const { fallback, "fallback-ready": fallbackReady, ...same } = sub.with; assert.deepEqual(same, firstSame, "the fallback call runs the same activity on the same inputs"); assert.deepEqual(sub.concurrency, jobs["resolve-issue"].concurrency, "in the same slot"); - // The third call: the same inputs on the fallback token, in the same slot. + // The third call: the same inputs on the fallback token, in the same slot, + // without start-gate, since it never makes the slot's first call. const third = jobs["resolve-issue-fallback"]; const { fallback: thirdFallback, ...thirdSame } = third.with; assert.equal(thirdFallback, "fallback"); - assert.deepEqual(thirdSame, firstSame, "the third call runs the same activity on the same inputs"); + const { "start-gate": firstStartGate, ...firstUngated } = firstSame; + assert.equal(firstStartGate, true, "the first call starts new work"); + assert.deepEqual(thirdSame, firstUngated, "the third call runs the same activity on the same inputs, switches only"); assert.deepEqual(third.concurrency, jobs["resolve-issue"].concurrency, "in the same slot"); // The handoff job: the default branch tip's handoff-evidence.sh on // read-only tokens, no secret, as every pr-unblock handoff job runs it. diff --git a/.github/scripts/pr-run-activity-read/credential-legs.test.mjs b/.github/scripts/pr-run-activity-read/credential-legs.test.mjs index a97f534ced2..83cfb0694f6 100644 --- a/.github/scripts/pr-run-activity-read/credential-legs.test.mjs +++ b/.github/scripts/pr-run-activity-read/credential-legs.test.mjs @@ -212,7 +212,8 @@ test("the workflow_call outputs the run and broker-check jobs set, and their sou }; const brokerCheck = { allowed: "${{ steps.broker.outcome == 'success' && steps.broker.outputs.allowed == 'true' }}", - "skip-reason": "${{ steps.broker.outputs.skip-reason }}", + // Since 5599e627fc, a start-gate budget stop also sets skip-reason budget. + "skip-reason": "${{ steps.kill.outputs.cause == 'budget' && 'budget' || steps.broker.outputs.skip-reason }}", attempt: "${{ github.run_attempt }}", lease: "${{ steps.lease.outputs.lease }}", "lease-comment-id": "${{ steps.lease.outputs.lease-comment-id }}", diff --git a/.github/scripts/work-unblock/label-unblocked.mjs b/.github/scripts/work-unblock/label-unblocked.mjs index b4b4c287080..fb508c013f6 100755 --- a/.github/scripts/work-unblock/label-unblocked.mjs +++ b/.github/scripts/work-unblock/label-unblocked.mjs @@ -221,7 +221,11 @@ // limits.limit-stops-per-issue trips since a listed human last added // agent-ready or a run last reported, waits the kill-switch // limits.stranded-backoff-hours after the run ended, flat, never doubled, -// with no cap on trips and no fail route. A sweep re-fires breaker trips, +// with no cap on trips and no fail route. A run whose start gate stopped it +// on the GITHUB_TOKEN budget (BUDGET_STEP green in a call's broker-check, +// from the jobs listing) is a breaker trip at broker-check whose record +// carries BUDGET_TAG: it never counts toward that bound or that wait, and +// re-fires once the window reset its step names has passed. A sweep re-fires breaker trips, // oldest run end first, only while the work-resolve-issue runs not yet // finished (five status listings, read once a sweep that has one) are under // limits.refire-queue-per-slot times the lane's SLOTS, within @@ -392,6 +396,16 @@ const BROKER_STEP = "Ask the broker whether the lane may run"; // admit and pick-slot run it first too, and while a switch is on skip their // gated steps: INSTALL_STEP and TRIGGER_STEP in admit, WIP_STEP in pick-slot. const KILL_STEP = "Check the kill switch"; +// broker-check's step that runs exactly when KILL_STEP stopped on the +// GITHUB_TOKEN budget (cause budget). Its name carries the window's reset, +// in epoch seconds, from the gate's reset output. +const BUDGET_STEP = /^Stop on a low GITHUB_TOKEN budget(?: until (.*))?$/; +// The second line of a breaker record whose stop was a budget stop. +const BUDGET_TAG = ""; +// The budget record's last line: when each of its earlier edits was made, so +// every budget re-fire keeps its date once the record is edited again. +const BUDGET_EARLIER = /^$/m; +const budgetEarlier = (body) => (BUDGET_EARLIER.exec(String(body ?? ""))?.[1] ?? "").split(" ").filter(Boolean); const INSTALL_STEP = "Install resolve-config's packages"; const TRIGGER_STEP = "Check the trigger"; const WIP_STEP = "Check the WIP cap"; @@ -574,22 +588,28 @@ function failedFor(github, number, error) { const isLabel = (event, kind) => event.event === kind && String(event.label?.name).toLowerCase() === LABEL; -// A re-fire marker reads as outcome `refired` with its workflow key, and -// `breaker` when a kill-switch refusal caused it. +// A re-fire marker reads as outcome `refired` with its workflow key, +// `breaker` when a kill-switch refusal caused it, and `budget` when that was +// a stop on the GITHUB_TOKEN budget. const parseMarker = (body) => { const refire = REFIRED.exec(String(body ?? "")) ?? INFRA.exec(String(body ?? "")); - return refire ? { outcome: "refired", key: refire[1], breaker: refire[2] !== undefined } : parseOutcomeMarker(body); + const breaker = refire?.[2] !== undefined; + return refire ? { outcome: "refired", key: refire[1], breaker, budget: breaker && String(body).split("\n")[1] === BUDGET_TAG } : parseOutcomeMarker(body); }; // The lanes App's outcome and re-fire markers on an issue, oldest first, each -// with when it was posted (the backoff comment: last edited). +// with when it was posted (the backoff comment and the budget record, each +// edited in place: last edited). The budget record is one marker per re-fire +// it records, its earlier edits included. const appMarkers = (comments, bot) => comments .filter((comment) => comment.user?.login === bot) - .map((comment) => ({ - ...parseMarker(comment.body), - at: Date.parse(INFRA.test(String(comment.body ?? "")) ? (comment.updated_at ?? comment.created_at) : comment.created_at), - })) + .flatMap((comment) => { + const marker = parseMarker(comment.body); + const edited = marker?.budget || INFRA.test(String(comment.body ?? "")); + const at = Date.parse(edited ? (comment.updated_at ?? comment.created_at) : comment.created_at); + return [...(marker?.budget ? budgetEarlier(comment.body) : []).map(Date.parse), at].map((when) => ({ ...marker, at: when })); + }) .filter((marker) => marker.outcome !== undefined && Number.isFinite(marker.at)) .sort((a, b) => a.at - b.at); @@ -1588,8 +1608,10 @@ const rateLimitCause = (reset) => // The record of a re-fire after a kill-switch refusal, posted before the // label moves, once per re-fire. It names the refused run and when it ended, // since the re-fire can come hours after the refusal. -export const breakerComment = (key, { repo, run, ended }) => - `\nThis records a re-fire of an earlier refusal, not a new refusal. The lanes kill switch stopped [run ${run}](https://github.com/${repo}/actions/runs/${run}), the work-resolve-issue run that \`${LABEL}\` started, before it finished; that run ended at ${new Date(ended).toISOString().replace(/\.\d{3}Z$/, "Z")}. work-unblock has now removed and added \`${LABEL}\` back to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last added \`${LABEL}\` or a run last reported, work-unblock waits the same limits.stranded-backoff-hours \`kill-switch\` hours after each refused run before re-firing it, never longer.`; +export const breakerComment = (key, { repo, run, ended, budget = false, earlier = [] }) => + budget + ? `\n${BUDGET_TAG}\nThis records a re-fire of an earlier stop, not a new one. The start gate stopped [run ${run}](https://github.com/${repo}/actions/runs/${run}), the work-resolve-issue run that \`${LABEL}\` started, before any model turn, because the GITHUB_TOKEN budget was under its start-gate share (limits.api github-token start-gate); that run ended at ${new Date(ended).toISOString().replace(/\.\d{3}Z$/, "Z")}. work-unblock has now removed and added \`${LABEL}\` back to start a fresh run. A budget stop is the lanes' load, not a fault in this issue: it never counts toward limits.limit-stops-per-issue or the \`kill-switch\` wait.${earlier.length > 0 ? `\n` : ""}` + : `\nThis records a re-fire of an earlier refusal, not a new refusal. The lanes kill switch stopped [run ${run}](https://github.com/${repo}/actions/runs/${run}), the work-resolve-issue run that \`${LABEL}\` started, before it finished; that run ended at ${new Date(ended).toISOString().replace(/\.\d{3}Z$/, "Z")}. work-unblock has now removed and added \`${LABEL}\` back to start a fresh run. A kill-switch refusal is no fault in this issue or the lanes, so it never counts against later re-fires; after limits.limit-stops-per-issue of them since a person last added \`${LABEL}\` or a run last reported, work-unblock waits the same limits.stranded-backoff-hours \`kill-switch\` hours after each refused run before re-firing it, never longer.`; // The backoff record of an issue whose re-fires on these workflow files keep // ending the same way: one comment per issue, edited on each re-fire. @@ -1740,9 +1762,15 @@ async function didNoWork(jobsOf, refusalOf, runs) { let stalled = null; let tripped = null; let stage = null; + // A run the switches stopped makes the trip a switch trip, whatever other + // run a budget stop stopped. + let switched = false; + // The latest window reset a budget stop names, in ms. + let resetAt = null; // The latest reset a rate-limit stop names, in ms; 0 for none. let reset = 0; - const atLeast = (kind, run, at) => { + const atLeast = (kind, run, at, budget = false) => { + switched ||= kind === "breaker" && !budget; if (rank[kind] > (rank[stalled] ?? 0)) { stalled = kind; tripped = run; @@ -1767,6 +1795,17 @@ async function didNoWork(jobsOf, refusalOf, runs) { const named = (name) => namedJobs(jobs, name); // Any call's broker-check: a later leg asks the broker again for its slot. const calls = RESOLVE_CALLS.flatMap(named); + // A budget stop at broker-check: no model ran. + const budgetSteps = calls.flatMap((job) => (job?.steps ?? []).filter((step) => step?.conclusion === "success" && BUDGET_STEP.test(String(step?.name)))); + const budget = budgetSteps.length > 0; + for (const step of budgetSteps) { + // A name whose reset is not a plain epoch second names none. + const until = BUDGET_STEP.exec(step.name)[1] ?? ""; + const seconds = /^[0-9]{1,12}$/.test(until) ? Number(until) : 0; + if (seconds > 0) { + resetAt = Math.max(resetAt ?? 0, seconds * 1000); + } + } const killedBefore = (step) => (job) => endedAt(job, KILL_STEP, "success") && endedAt(job, step, "skipped"); // admit's first gated step is INSTALL_STEP: a failed install also skips // TRIGGER_STEP, and that job fails, so only a green job with both skipped @@ -1774,6 +1813,7 @@ async function didNoWork(jobsOf, refusalOf, runs) { // steps, so a green job with it skipped is the switch too. const switchedOff = (...steps) => (job) => job.conclusion === "success" && steps.every((step) => killedBefore(step)(job)); const killed = + budget || calls.some(killedBefore(BROKER_STEP)) || named("admit").some(switchedOff(INSTALL_STEP, TRIGGER_STEP)) || named("pick-slot").some(switchedOff(WIP_STEP)); // A failed job before any model turn: admit, pick-slot, or a call's // broker-check or run job, in a run where no model step ran. @@ -1814,7 +1854,7 @@ async function didNoWork(jobsOf, refusalOf, runs) { ? refusalOf(limited, "limit") : null); if (killed || refused === BREAKER) { - atLeast("breaker", run, killed || checked || checkStopped ? "check" : "token"); + atLeast("breaker", run, killed || checked || checkStopped ? "check" : "token", budget); } else if (minting || checked || checkStopped || stopped) { atLeast("mint-failed"); } else if (limited && refused) { @@ -1826,7 +1866,8 @@ async function didNoWork(jobsOf, refusalOf, runs) { atLeast(outage ? "outage" : "blocked"); } } - return stalled && { stalled, run: tripped, stage, reset: stalled === "mint-failed" ? reset : 0 }; + const budget = stalled === "breaker" && !switched; + return stalled && { stalled, run: tripped, stage, budget, resetAt: budget ? resetAt : null, reset: stalled === "mint-failed" ? reset : 0 }; } // The reason the broker refused `job`, from its log: the failed step's last @@ -2251,19 +2292,22 @@ async function refireRefusal({ github, actions, jobLog, carry, repo, number, iss // A stale stop ran no model; a failed mint in its re-fire job is no failed // mint of the lane's, so it skips the did-no-work clock below. const stale = await stoppedStale(jobsOf, runs); - const { stalled, run: refused, stage, reset } = (!stale && (await didNoWork(jobsOf, refusalReader(jobLog, repo, carry), runs))) || {}; + const { stalled, run: refused, stage, budget, resetAt, reset } = (!stale && (await didNoWork(jobsOf, refusalReader(jobLog, repo, carry), runs))) || {}; if (stalled) { // Breaker trips since a listed human last added agent-ready or a run last // reported; past limits.limit-stops-per-issue of them, a flat wait of the // kill-switch hours that never doubles: a trip is the lanes' load, not the - // issue's fault (owner decision Q109). + // issue's fault (owner decision Q109). A stop on the GITHUB_TOKEN budget is + // no trip: it never counts, and re-fires once the window it read has + // reset (the reset its step names, else an hour after the run ended, a + // primary window's length), so it re-fires once per stop, never per sweep. const from = Math.max( Number.NEGATIVE_INFINITY, ...events.filter((event) => isLabel(event, "labeled") && humans.has(event.actor?.id)).map((event) => Date.parse(event.created_at)), ...markers.filter((marker) => marker.outcome !== "refired").map((marker) => marker.at), ); - const trips = stalled === "breaker" ? markers.filter((marker) => marker.breaker && marker.at > from).length : 0; - const over = trips - limits["limit-stops-per-issue"]; + const trips = stalled === "breaker" && !budget ? markers.filter((marker) => marker.breaker && !marker.budget && marker.at > from).length : 0; + const over = budget ? -1 : trips - limits["limit-stops-per-issue"]; // A broker-check trip ran no model, and this sweep runs only with the // kill switch off, so it re-fires now; a mint trip ran the model and // waits like any failed mint. @@ -2273,11 +2317,13 @@ async function refireRefusal({ github, actions, jobLog, carry, repo, number, iss // next sweep, which reads the page again. const outage = stalled === "outage"; const clamp = (stalled === "breaker" && stage === "check") || (outage && (await upstreamClear())) ? 0 : limits["reset-clamp-min"]; - // A rate-limit stop waits past the reset it names too. - const due = Math.max(ended + (over >= 0 ? hoursFor["kill-switch"] : clamp) * HOUR_MS, reset); + // A budget stop waits for the window reset it names; a rate-limit stop + // waits past the reset it names too. + const due = budget ? (resetAt ?? ended + HOUR_MS) : Math.max(ended + (over >= 0 ? hoursFor["kill-switch"] : clamp) * HOUR_MS, reset); if (now < due) { - const why = - stalled === "breaker" + const why = budget + ? "the start gate stopped its lane run on a low GITHUB_TOKEN budget" + : stalled === "breaker" ? `the broker refused its lane run for the lanes kill switch (trip ${trips + 1})` : outage ? "the upstream-status stop skipped its lane run, and the status page does not read clear" @@ -2301,7 +2347,13 @@ async function refireRefusal({ github, actions, jobLog, carry, repo, number, iss if (stalled === "mint-failed" && !recorded && fired >= limits["limit-stops-per-issue"]) { return backoff(key, ended, reset ? rateLimitCause(reset) : MINT_CAUSE, hoursFor["failed-mint"]); } - const trip = stalled === "breaker" ? { run: refused.id, ended: Date.parse(refused.updated_at) || ended } : undefined; + // An issue keeps one budget record, edited in place for each window; the + // edit carries the dates of the ones before it. + const budgetRecord = budget + ? comments.filter((comment) => comment.user?.login === bot && parseMarker(comment.body)?.budget && RUN_ID.test(String(comment.id))).at(-1) + : undefined; + const earlier = budgetRecord ? [...budgetEarlier(budgetRecord.body), budgetRecord.updated_at ?? budgetRecord.created_at] : []; + const trip = stalled === "breaker" ? { run: refused.id, ended: Date.parse(refused.updated_at) || ended, budget, record: budgetRecord?.id, earlier } : undefined; // Trips sort by when their run ended, not when they fell due, so the sweeps // after the switch clears drain them oldest first whatever their waits. return { reason: null, resets: trip ? ended : due, refire: key, recorded, stall: stalled, trip, ...(reset && { cause: rateLimitCause(reset) }) }; @@ -2629,7 +2681,10 @@ async function restartLimited({ github, actions, jobLog, carry, repo, bot, human try { // The record first: a failed comment re-fires nothing, so the bound holds. if (refire && !recorded) { - await github("POST", `${base}/comments`, { body: stall === "breaker" ? breakerComment(refire, { repo, ...trip }) : refireComment(refire, cause) }); + const body = stall === "breaker" ? breakerComment(refire, { repo, ...trip }) : refireComment(refire, cause); + await (trip?.record === undefined + ? github("POST", `${base}/comments`, { body }) + : github("PATCH", `/repos/${repo}/issues/comments/${trip.record}`, { body })); } if (infra) { await (infra.id === undefined @@ -2659,7 +2714,7 @@ async function restartLimited({ github, actions, jobLog, carry, repo, bot, human console.log(`label-unblocked: #${number} left as is: its Work status is now ${restarted.state ?? "unset"}`); continue; } - const why = infra ? "re-fired on a backoff" : stall === "breaker" ? "re-fired after a kill-switch refusal" : stall === "outage" ? "re-fired after its upstream-status skip" : refire ? "re-fired after its failed run" : recurrence ? "relabeled after its reopen on a repeat failure" : "restarted after its limit stop"; + const why = infra ? "re-fired on a backoff" : stall === "breaker" ? (trip?.budget ? "re-fired after a start-gate budget stop" : "re-fired after a kill-switch refusal") : stall === "outage" ? "re-fired after its upstream-status skip" : refire ? "re-fired after its failed run" : recurrence ? "relabeled after its reopen on a repeat failure" : "restarted after its limit stop"; console.log(`label-unblocked: #${number} ${why}`); } catch (error) { failed += 1; diff --git a/.github/scripts/work-unblock/label-unblocked.test.mjs b/.github/scripts/work-unblock/label-unblocked.test.mjs index 3715013bc1c..5ace8334240 100644 --- a/.github/scripts/work-unblock/label-unblocked.test.mjs +++ b/.github/scripts/work-unblock/label-unblocked.test.mjs @@ -3740,6 +3740,163 @@ test("a later leg's broker-check refused or skipped for the kill switch is a bre } }); +// The run a start-gate budget stop leaves (pr-run-activity-read.yml): the +// first call's broker-check ran its kill-switch step, its `Stop on a low +// GITHUB_TOKEN budget until ` step (named with the gate's reset +// output), and skipped the broker ask and the lease, ending green with +// skip-reason budget. That skip-reason skips the call's run job, so its +// issue-fields, scan and summary jobs skip too, and report skips with no lease +// held. No handoff, no scan output: work-resolve-issue's later calls, +// apply-issue-patch and report-outcome all skip, and the run ends green. +const epoch = (at) => Date.parse(at) / 1000; +const budgetStep = (name = `Stop on a low GITHUB_TOKEN budget until ${epoch("2026-10-16T10:30:00Z")}`) => ({ name, conclusion: "success" }); +const skippedJob = (name) => ({ name, conclusion: "skipped", steps: [] }); +const budgetStopped = (step = budgetStep()) => [ + ADMITTED, + pickSlot, + { + name: "resolve-issue / broker-check", + conclusion: "success", + steps: [ + { name: "Check the kill switch", conclusion: "success" }, + step, + { name: "Ask the broker whether the lane may run", conclusion: "skipped" }, + { name: "Claim the issue with a lane lease", conclusion: "skipped" }, + ], + }, + skippedJob("resolve-issue / broker-allowed"), + skippedJob("resolve-issue / run"), + skippedJob("resolve-issue / issue-fields"), + skippedJob("resolve-issue / scan"), + skippedJob("resolve-issue / summary"), + skippedJob("resolve-issue / report"), + { name: "resolve-issue-handoff", conclusion: "success", steps: [{ name: "Read whether the call handed off", conclusion: "success" }] }, + skippedJob("resolve-issue-subscription"), + skippedJob("resolve-issue-fallback"), + skippedJob("apply-issue-patch"), + skippedJob("report-skip"), + skippedJob("report-outcome"), +]; +const BUDGET_STOPPED = budgetStopped(); +const BUDGET_RECORD = `${BREAKER_MARKER}\n\n`; +const budgeted = (calls) => { + tripped(calls); + assert.ok(postedBody(calls).startsWith(BUDGET_RECORD), postedBody(calls)); +}; +const switched = (calls) => { + tripped(calls); + assert.ok(!postedBody(calls).includes("cause=budget"), postedBody(calls)); +}; + +test("a run its start gate stopped on the GITHUB_TOKEN budget re-fires once its window has reset, with a budget record, read from the job listing alone", async () => { + // The sweep runs at 12:00; the run ended at 11:58. + const ended = "2026-10-16T11:58:00Z"; + const reset = budgetStopped(budgetStep(`Stop on a low GITHUB_TOKEN budget until ${epoch("2026-10-16T12:30:00Z")}`)); + const before = stalledRun({ jobs: reset, conclusion: "success", ended }); + assert.equal(await sweep({ github: before.github, actions: before.actions }), 0); + assert.deepEqual(writes(before.calls), [], "no record and no re-fire before the reset"); + for (const now of ["2026-10-16T12:30:00Z", "2026-10-16T12:45:00Z"]) { + const after = stalledRun({ jobs: reset, conclusion: "success", ended }); + assert.equal(await sweep({ github: after.github, actions: after.actions, now: Date.parse(now) }), 0); + budgeted(after.calls); + assert.deepEqual(logReads(after.calls), []); + } + // A step name with no readable reset, bare or with its expression left + // unevaluated, is still a budget stop and waits an hour after the run ended. + for (const name of ["Stop on a low GITHUB_TOKEN budget", "Stop on a low GITHUB_TOKEN budget until ${{ steps.kill.outputs.reset }}"]) { + const unnamed = budgetStopped(budgetStep(name)); + const early = stalledRun({ jobs: unnamed, conclusion: "success", ended }); + assert.equal(await sweep({ github: early.github, actions: early.actions, now: Date.parse("2026-10-16T12:57:59Z") }), 0); + assert.deepEqual(writes(early.calls), [], name); + const due = stalledRun({ jobs: unnamed, conclusion: "success", ended }); + assert.equal(await sweep({ github: due.github, actions: due.actions, now: Date.parse("2026-10-16T12:58:00Z") }), 0); + budgeted(due.calls); + } + // The same listing without the budget step is a switch trip, re-fired at once. + const plain = stalledRun({ jobs: KILL_SKIPPED, conclusion: "success", ended }); + assert.equal(await sweep({ github: plain.github, actions: plain.actions }), 0); + switched(plain.calls); +}); + +test("a budget stop step that skipped is no budget stop", async () => { + const skipped = { ...budgetStep(), conclusion: "skipped" }; + const jobs = [ADMITTED, pickSlot, { ...KILL_SKIPPED[2], steps: [KILL_SKIPPED[2].steps[0], skipped, KILL_SKIPPED[2].steps[1]] }, ...KILL_SKIPPED.slice(3)]; + const { github, actions, calls } = stalledRun({ jobs, conclusion: "success" }); + assert.equal(await sweep({ github, actions }), 0); + switched(calls); +}); + +// limits.limit-stops-per-issue is 3: three switch trips make the fourth wait +// the flat kill-switch hours; three budget stops do not. +test("budget stops never count toward limits.limit-stops-per-issue or wait the kill-switch hours", async () => { + const switchTrips = await afterSweeps(3, KILL_SKIPPED); + const fourthSwitch = stalledRun({ jobs: KILL_SKIPPED, conclusion: "success", ...switchTrips }); + assert.equal(await sweep({ github: fourthSwitch.github, actions: fourthSwitch.actions }), 0); + assert.deepEqual(writes(fourthSwitch.calls), [], "a fourth switch trip waits"); + + // Five budget records, as separate comments, each followed by its re-add. + const budgetStops = history(Array.from({ length: 5 }, (_, index) => breakerComment(KEY, { repo: REPO, run: 4100 + index, ended: Date.parse("2026-10-16T05:00:00Z"), budget: true }))); + const afterBudget = stalledRun({ jobs: KILL_SKIPPED, conclusion: "success", ...budgetStops }); + assert.equal(await sweep({ github: afterBudget.github, actions: afterBudget.actions }), 0); + switched(afterBudget.calls); + + const budgetAfterSwitch = stalledRun({ jobs: BUDGET_STOPPED, conclusion: "success", ...switchTrips }); + assert.equal(await sweep({ github: budgetAfterSwitch.github, actions: budgetAfterSwitch.actions }), 0); + budgeted(budgetAfterSwitch.calls); +}); + +test("a later budget stop edits the issue's one budget record in place, never posting a second", async () => { + const once = await afterSweeps(1, BUDGET_STOPPED); + const recordId = once.comments.at(-1).id; + const again = stalledRun({ jobs: budgetStopped(budgetStep(`Stop on a low GITHUB_TOKEN budget until ${epoch("2026-10-16T11:30:00Z")}`)), conclusion: "success", ...once }); + assert.equal(await sweep({ github: again.github, actions: again.actions }), 0); + const done = writes(again.calls); + assert.ok(done[0].startsWith(`PATCH /repos/${REPO}/issues/comments/${recordId} `), done[0]); + assert.ok(done[0].includes("cause=budget"), done[0]); + assert.ok(!done.some((call) => call.startsWith(`POST /repos/${REPO}/issues/30/comments `)), done.join("\n")); + assert.deepEqual(done.slice(1), restarted(30)); + // A switch trip after it posts its own record: only budget records are edited. + const switchAfter = stalledRun({ jobs: KILL_SKIPPED, conclusion: "success", ...once }); + assert.equal(await sweep({ github: switchAfter.github, actions: switchAfter.actions }), 0); + switched(switchAfter.calls); +}); + +// One re-fire of a failed run on these workflow files (06:00), then two budget +// stops on the one budget record (re-added 07:00 and 10:00), then a failed run +// ending at 10:05: the backoff counts one re-fire, so it waits 6 hours, not 12. +test("an edited budget record keeps each budget re-fire its own, so none counts toward a later backoff", async () => { + const failedLater = [ADMITTED, pickSlot, BROKER_ALLOWED, { name: "resolve-issue / run", conclusion: "failure", steps: [{ name: "Request the lane token", conclusion: "success" }] }]; + const unlabeled = (at) => ({ event: "unlabeled", label: { name: "agent-ready" }, actor: LANES_BOT, created_at: at }); + const first = { id: 71, body: breakerComment(KEY, { repo: REPO, run: 4100, ended: Date.parse("2026-10-16T06:50:00Z"), budget: true }), user: LANES_BOT, created_at: "2026-10-16T07:00:00Z" }; + const before = [added("2026-10-16T05:00:00Z", OWNER), unlabeled("2026-10-16T06:00:00Z"), added("2026-10-16T06:00:01Z"), unlabeled("2026-10-16T07:00:00Z"), added("2026-10-16T07:00:01Z")]; + const second = stalledRun({ jobs: BUDGET_STOPPED, conclusion: "success", comments: [note(), refireNote(KEY, "2026-10-16T06:00:00Z"), first], events: [...before.slice(0, -1), added(LABELED_AT)] }); + assert.equal(await sweep({ github: second.github, actions: second.actions }), 0); + const patch = writes(second.calls)[0]; + assert.ok(patch.startsWith(`PATCH /repos/${REPO}/issues/comments/71 `), patch); + const edited = { ...first, body: JSON.parse(patch.slice(patch.indexOf(" {") + 1)).body, updated_at: "2026-10-16T09:59:59Z" }; + const history = { + comments: [note(), refireNote(KEY, "2026-10-16T06:00:00Z"), edited], + events: [...before, unlabeled("2026-10-16T09:59:59Z"), added(LABELED_AT)], + }; + const early = stalledRun({ jobs: failedLater, ...history }); + assert.equal(await sweep({ github: early.github, actions: early.actions, now: Date.parse("2026-10-16T16:04:59Z") }), 0); + assert.deepEqual(writes(early.calls), [], "one re-fire waits 6 hours"); + const due = stalledRun({ jobs: failedLater, ...history }); + assert.equal(await sweep({ github: due.github, actions: due.actions, now: Date.parse("2026-10-16T16:05:00Z") }), 0); + const done = writes(due.calls); + assert.ok(done[0].startsWith(`POST /repos/${REPO}/issues/30/comments `), done.join("\n")); + assert.ok(done[0].includes("has re-fired it 2 time(s)"), done[0]); + assert.deepEqual(done.slice(1), restarted(30)); +}); + +test("the budget record says it never counts toward the per-issue bound", () => { + const body = breakerComment(KEY, { repo: REPO, run: 5030, ended: Date.parse("2026-10-16T10:05:00Z"), budget: true }); + assert.ok(body.startsWith(BUDGET_RECORD), body); + assert.ok(body.includes(REFUSED_LINK), body); + assert.ok(body.includes("never counts toward limits.limit-stops-per-issue"), body); + assert.ok(!breakerComment(KEY, { repo: REPO, run: 5030, ended: 0 }).includes("cause=budget")); +}); + test("a green run whose kill-switch check did not run, or whose broker ask ran, is no breaker trip", async () => { const [kill, ask] = KILL_SKIPPED[2].steps; const check = (steps) => ({ ...KILL_SKIPPED[2], steps }); @@ -4111,6 +4268,21 @@ test("the job ids and step names the sweep matches exist in the lane workflows", assert.ok(checkSteps.indexOf("Stop on an open blocker") > checkSteps.indexOf("Claim the issue with a lane lease")); assert.ok(read.run !== undefined); assert.ok(steps(read.run).includes("Stop on stale workflow files")); + // broker-check's budget step follows its kill-switch step, runs exactly on a + // budget stop, and carries the gate's reset in its name; its gate reads the + // budget. The model job's gate checks the switches only. + const budgetName = "Stop on a low GITHUB_TOKEN budget until ${{ steps.kill.outputs.reset }}"; + const checkNamed = steps(read["broker-check"]); + assert.equal(checkNamed.indexOf(budgetName), checkNamed.indexOf("Check the kill switch") + 1); + assert.match(read["broker-check"], /^ {6}- name: Stop on a low GITHUB_TOKEN budget until \$\{\{ steps\.kill\.outputs\.reset \}\}\n {8}if: steps\.kill\.outputs\.cause == 'budget'\n/m); + // Its gate reads the budget on a starting caller's first call, and a budget + // stop sets skip-reason budget, which skips the run job. + assert.match(read["broker-check"], /^ {10}budget: \$\{\{ inputs\.start-gate && \(inputs\.fallback == '' \|\| inputs\.fallback == 'credit-stop'\) && 'true' \|\| 'false' \}\}$/m); + assert.match(read["broker-check"], /^ {6}skip-reason: \$\{\{ steps\.kill\.outputs\.cause == 'budget' && 'budget' \|\| steps\.broker\.outputs\.skip-reason \}\}$/m); + assert.match(read.run, /^ {6}&& needs\.broker-check\.outputs\.skip-reason == ''$/m); + assert.match(caller["resolve-issue"], /^ {6}start-gate: true$/m); + assert.ok(!steps(read.run).some((name) => name.startsWith("Stop on a low GITHUB_TOKEN budget"))); + assert.doesNotMatch(read.run, /^ {10}budget:/m); for (const id of ["broker-check", "run"]) { assert.doesNotMatch(read[id], /^ {4}name:/m, `read runner job ${id} keeps its id as its name`); } diff --git a/.github/workflows/intake-reenter.yml b/.github/workflows/intake-reenter.yml index 8764d42ff70..c7927ecbb33 100644 --- a/.github/workflows/intake-reenter.yml +++ b/.github/workflows/intake-reenter.yml @@ -157,12 +157,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing read or labeled" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing read or labeled" # reenter.mjs reads an issue's Work status field through resolve-config.mjs, # whose packages only the resolve-config action installs; this job does diff --git a/.github/workflows/intake-triage.yml b/.github/workflows/intake-triage.yml index 9c40b8b6c54..feccbf542e8 100644 --- a/.github/workflows/intake-triage.yml +++ b/.github/workflows/intake-triage.yml @@ -202,6 +202,7 @@ jobs: /.github/scripts/maintenance-reconcile-lost-runs/component-status.mjs /.github/actions/setup-pinned-node/ /.node-version + /.github/actions/log-api-budget/ sparse-checkout-cone-mode: false - name: Set up Node @@ -215,6 +216,8 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" # Skips the run while Actions or npm installs report a degraded component, # with a logged reason; fails open (.github/actions/check-upstream-status). @@ -229,8 +232,10 @@ jobs: if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} run: | - echo "::notice::the lanes kill switch stops this triage and the needs-triage label (reason: ${REASON:-unreadable})" + echo "::notice::${WHY} this triage and the needs-triage label (reason: ${REASON:-unreadable}, cause ${CAUSE:-unreadable})" # The floor label, on an issues event only: a dispatch asks for triage. A # failed or skipped born-triaged gate (always skipped on a reopen) leaves @@ -715,6 +720,8 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" # The same actors apply-verdict writes for, so a run whose verdict the # broker would not write starts no model either. diff --git a/.github/workflows/maintenance-audit-lane-runs.yml b/.github/workflows/maintenance-audit-lane-runs.yml index cc74fc66c64..7122937e71e 100644 --- a/.github/workflows/maintenance-audit-lane-runs.yml +++ b/.github/workflows/maintenance-audit-lane-runs.yml @@ -92,12 +92,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing read or filed" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing read or filed" - name: Collect the units id: collect @@ -1305,6 +1309,8 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" # The Work status writer (write-status.mjs) reads models.issue-status # through resolve-config.mjs, whose packages only the resolve-config diff --git a/.github/workflows/maintenance-delete-stale-branches.yml b/.github/workflows/maintenance-delete-stale-branches.yml index 57a2968ff83..11c1df80ed2 100644 --- a/.github/workflows/maintenance-delete-stale-branches.yml +++ b/.github/workflows/maintenance-delete-stale-branches.yml @@ -92,12 +92,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing read or deleted" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing read or deleted" # cleanup.mjs reads the branch-cleanup leaf through the base's # resolve-config, which needs its dependencies. diff --git a/.github/workflows/maintenance-file-recurring-issues.yml b/.github/workflows/maintenance-file-recurring-issues.yml index bd61a00fd77..f4c03030f73 100644 --- a/.github/workflows/maintenance-file-recurring-issues.yml +++ b/.github/workflows/maintenance-file-recurring-issues.yml @@ -65,12 +65,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing read or filed" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing read or filed" - name: File the due rows if: steps.kill.outputs.proceed == 'true' diff --git a/.github/workflows/maintenance-measure-skill-edit.yml b/.github/workflows/maintenance-measure-skill-edit.yml index 57e3e2b1e06..faf0009510d 100644 --- a/.github/workflows/maintenance-measure-skill-edit.yml +++ b/.github/workflows/maintenance-measure-skill-edit.yml @@ -114,6 +114,8 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Check upstream status id: upstream @@ -131,9 +133,19 @@ jobs: require-pr: "false" require-kind: human + # A low budget ends the run neutral: go stays unset, so no pass runs. + - name: Report a budget stop + if: steps.kill.outputs.cause == 'budget' + env: + RESET: ${{ steps.kill.outputs.reset }} + run: | + echo "gate: kill-switch: the GITHUB_TOKEN budget is low until ${RESET} (epoch seconds)" >>"$GITHUB_STEP_SUMMARY" + echo "::notice::the GITHUB_TOKEN budget is low until ${RESET} (epoch seconds), which stops this run; nothing read or written" + # Each stop names its gate in the job summary. Each pass checks the # credential is set, so no job here reads a secret. - name: Check the run may spend + if: steps.kill.outputs.cause != 'budget' env: KILL: ${{ steps.kill.outputs.proceed }} KILL_REASON: ${{ steps.kill.outputs.reason }} @@ -153,6 +165,7 @@ jobs: - name: Check the inputs id: inputs + if: steps.kill.outputs.proceed == 'true' env: PLUGIN: ${{ inputs.plugin }} SKILL: ${{ inputs.skill }} @@ -187,6 +200,7 @@ jobs: # case count after the tag or case filter is known only after pass a. - name: Check the suite id: stage + if: steps.kill.outputs.proceed == 'true' env: DIR: .base/plugins/${{ steps.inputs.outputs.plugin }} run: | diff --git a/.github/workflows/maintenance-monitor-signals.yml b/.github/workflows/maintenance-monitor-signals.yml index adb3b239eef..f7b4bb235c3 100644 --- a/.github/workflows/maintenance-monitor-signals.yml +++ b/.github/workflows/maintenance-monitor-signals.yml @@ -74,12 +74,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing read or filed" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing read or filed" # The Work status writer (write-status.mjs) reads models.issue-status # through resolve-config.mjs, whose packages only the resolve-config diff --git a/.github/workflows/maintenance-reconcile-lost-runs.yml b/.github/workflows/maintenance-reconcile-lost-runs.yml index 8f5b8b5bf16..2eb0a160840 100644 --- a/.github/workflows/maintenance-reconcile-lost-runs.yml +++ b/.github/workflows/maintenance-reconcile-lost-runs.yml @@ -140,6 +140,8 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" # The base config's limits.api, in resolve-config's output shape, for # the pacer; a failed read leaves rate-limits.json's defaults. diff --git a/.github/workflows/maintenance-tidy-code-upkeep.yml b/.github/workflows/maintenance-tidy-code-upkeep.yml index 5d1338e0659..9a474fb182a 100644 --- a/.github/workflows/maintenance-tidy-code-upkeep.yml +++ b/.github/workflows/maintenance-tidy-code-upkeep.yml @@ -129,12 +129,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing read or written" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing read or written" # A dispatch, and any re-run, must come from a listed human; the broker # checks the same, so this only keeps the model from running for nothing. diff --git a/.github/workflows/post-merge-test-mutation.yml b/.github/workflows/post-merge-test-mutation.yml index 7eac9c86846..31c7136d3b7 100644 --- a/.github/workflows/post-merge-test-mutation.yml +++ b/.github/workflows/post-merge-test-mutation.yml @@ -100,12 +100,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing mutated" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing mutated" - name: Download the range id: download-range diff --git a/.github/workflows/pr-review-hosted.yml b/.github/workflows/pr-review-hosted.yml index 7e5749b6fee..b268762b5a2 100644 --- a/.github/workflows/pr-review-hosted.yml +++ b/.github/workflows/pr-review-hosted.yml @@ -207,9 +207,10 @@ jobs: if: steps.kill.outputs.proceed != 'true' || vars.CLAUDE_REVIEW_DISABLED == 'true' env: REASON: ${{ steps.kill.outputs.reason }} + CAUSE: ${{ steps.kill.outputs.cause }} LANE_SWITCH: ${{ vars.CLAUDE_REVIEW_DISABLED }} run: | - echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable}, CLAUDE_REVIEW_DISABLED: ${LANE_SWITCH:-unset})" + echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable} cause ${CAUSE:-unreadable}, CLAUDE_REVIEW_DISABLED: ${LANE_SWITCH:-unset})" # A skip fails this job, so the run is a failed first attempt: the # lost-run reconciler re-runs it once GitHub Actions is operational diff --git a/.github/workflows/pr-review-main.yml b/.github/workflows/pr-review-main.yml index a923bf0f9a4..740fa570eaa 100644 --- a/.github/workflows/pr-review-main.yml +++ b/.github/workflows/pr-review-main.yml @@ -795,6 +795,7 @@ jobs: env: PROCEED: ${{ steps.kill.outputs.proceed }} REASON: ${{ steps.kill.outputs.reason }} + CAUSE: ${{ steps.kill.outputs.cause }} LANE_SWITCH: ${{ vars[matrix.switch] }} UPSTREAM_DEGRADED: ${{ steps.upstream.outputs.degraded }} run: | @@ -806,7 +807,7 @@ jobs: echo "::error::an upstream outage stops this review; the lost-run reconciler re-runs it once GitHub Actions is operational" exit 1 else - echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable}, lane: ${LANE_SWITCH:-unset}); no verdict, so the lane stays unbound" + echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable} cause ${CAUSE:-unreadable}, lane: ${LANE_SWITCH:-unset}); no verdict, so the lane stays unbound" fi # The upstream run's actors, the re-runner and the PR author must all be @@ -1744,6 +1745,7 @@ jobs: env: PROCEED: ${{ steps.kill.outputs.proceed }} REASON: ${{ steps.kill.outputs.reason }} + CAUSE: ${{ steps.kill.outputs.cause }} LANE_SWITCH: ${{ vars[matrix.switch] }} UPSTREAM_DEGRADED: ${{ steps.upstream.outputs.degraded }} run: | @@ -1755,7 +1757,7 @@ jobs: echo "::error::an upstream outage stops this review; the lost-run reconciler re-runs it once GitHub Actions is operational" exit 1 else - echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable}, lane: ${LANE_SWITCH:-unset}); no verdict, so the lane stays unbound" + echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable} cause ${CAUSE:-unreadable}, lane: ${LANE_SWITCH:-unset}); no verdict, so the lane stays unbound" fi # The upstream run's actors, the re-runner and the PR author must all be @@ -2661,6 +2663,7 @@ jobs: env: PROCEED: ${{ steps.kill.outputs.proceed }} REASON: ${{ steps.kill.outputs.reason }} + CAUSE: ${{ steps.kill.outputs.cause }} LANE_SWITCH: ${{ vars[matrix.switch] }} UPSTREAM_DEGRADED: ${{ steps.upstream.outputs.degraded }} run: | @@ -2672,7 +2675,7 @@ jobs: echo "::error::an upstream outage stops this review; the lost-run reconciler re-runs it once GitHub Actions is operational" exit 1 else - echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable}, lane: ${LANE_SWITCH:-unset}); no verdict, so the lane stays unbound" + echo "::notice::a kill switch stops this review (lanes: ${REASON:-unreadable} cause ${CAUSE:-unreadable}, lane: ${LANE_SWITCH:-unset}); no verdict, so the lane stays unbound" fi # The upstream run's actors, the re-runner and the PR author must all be diff --git a/.github/workflows/pr-review-security-hosted.yml b/.github/workflows/pr-review-security-hosted.yml index 07dde906a57..6f3c1da8bee 100644 --- a/.github/workflows/pr-review-security-hosted.yml +++ b/.github/workflows/pr-review-security-hosted.yml @@ -212,9 +212,10 @@ jobs: if: steps.kill.outputs.proceed != 'true' || vars.CLAUDE_SECURITY_REVIEW_DISABLED == 'true' env: REASON: ${{ steps.kill.outputs.reason }} + CAUSE: ${{ steps.kill.outputs.cause }} LANE_SWITCH: ${{ vars.CLAUDE_SECURITY_REVIEW_DISABLED }} run: | - echo "::notice::a kill switch stops this security review (lanes: ${REASON:-unreadable}, CLAUDE_SECURITY_REVIEW_DISABLED: ${LANE_SWITCH:-unset})" + echo "::notice::a kill switch stops this security review (lanes: ${REASON:-unreadable} cause ${CAUSE:-unreadable}, CLAUDE_SECURITY_REVIEW_DISABLED: ${LANE_SWITCH:-unset})" # A skip fails this job, so the run is a failed first attempt: the # lost-run reconciler re-runs it once GitHub Actions is operational diff --git a/.github/workflows/pr-run-activity-read.yml b/.github/workflows/pr-run-activity-read.yml index 62f67a63890..798dbc0ccce 100644 --- a/.github/workflows/pr-run-activity-read.yml +++ b/.github/workflows/pr-run-activity-read.yml @@ -82,6 +82,17 @@ on: required: false type: string default: "" + start-gate: + description: >- + `true` for a caller whose call starts new work (the issue lane): + broker-check's kill-switch gate then also stops on a low + GITHUB_TOKEN budget, on the first call in the slot only (fallback + empty or `credit-stop`); a call that takes over a handed-off turn, + and every call of a caller that leaves this `false`, checks the + switches only. + type: boolean + required: false + default: false fallback: description: >- Empty for the first call, which runs on claude-api-key and hands off @@ -412,7 +423,8 @@ on: skip-reason: description: >- `kill-switch` when the broker refused the check because a lanes kill - switch is on, so no lane ran; empty otherwise. + switch is on, `budget` when broker-check's start gate stopped on a + low GITHUB_TOKEN budget, so no lane ran; empty otherwise. value: ${{ jobs.broker-check.outputs.skip-reason }} lease: description: >- @@ -438,7 +450,9 @@ jobs: # neutral. A broker kill-switch refusal (a switch turned on after this run # read its variables) also ends green, with skip-reason kill-switch, which # skips the run job and every job after it but report, which posts the - # check neutral on a pull_request run. Any other broker answer but an allow, or none, fails this + # check neutral on a pull_request run. A start-gate stop on a low + # GITHUB_TOKEN budget ends green the same way, with skip-reason budget, and + # report posts the same neutral kill-switch check. Any other broker answer but an allow, or none, fails this # job, which skips the run job and fails the check. issues: write is for the issue # lease alone, posted before any model turn (CONTRACT.md "Lease protocol"). broker-check: @@ -456,7 +470,7 @@ jobs: # so the run job also checks that the allow came from its own attempt. outputs: allowed: ${{ steps.broker.outcome == 'success' && steps.broker.outputs.allowed == 'true' }} - skip-reason: ${{ steps.broker.outputs.skip-reason }} + skip-reason: ${{ steps.kill.outputs.cause == 'budget' && 'budget' || steps.broker.outputs.skip-reason }} attempt: ${{ github.run_attempt }} lease: ${{ steps.lease.outputs.lease }} lease-comment-id: ${{ steps.lease.outputs.lease-comment-id }} @@ -519,6 +533,19 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # A first call that starts new work also stops on a low GITHUB_TOKEN + # budget; a handed-off or fallback leg checks the switches only. + budget: ${{ inputs.start-gate && (inputs.fallback == '' || inputs.fallback == 'credit-stop') && 'true' || 'false' }} + + # work-unblock's sweep reads this step's name and conclusion from the + # job listing (label-unblocked.mjs BUDGET_STEP): a stop on the + # GITHUB_TOKEN budget re-fires after the window's reset the name + # carries, without counting as a kill-switch trip. + - name: Stop on a low GITHUB_TOKEN budget until ${{ steps.kill.outputs.reset }} + if: steps.kill.outputs.cause == 'budget' + env: + RESET: ${{ steps.kill.outputs.reset }} + run: echo "::notice::the start gate stopped this run on a low GITHUB_TOKEN budget until $RESET (epoch seconds)" # The broker repeats the kill switch, including the repository-variable # shadow this job cannot see, and checks the job's OIDC claims. @@ -3933,7 +3960,7 @@ jobs: API_HEAD: ${{ steps.pr-head.outputs.sha }} PR_MISMATCH: ${{ steps.pr-head.outputs.pr-mismatch }} TRUSTED_SKIP: ${{ needs.broker-check.outputs.allowed != 'true' }} - BROKER_STOP: ${{ needs.broker-check.outputs.skip-reason == 'kill-switch' }} + BROKER_STOP: ${{ needs.broker-check.outputs.skip-reason == 'kill-switch' || needs.broker-check.outputs.skip-reason == 'budget' }} POST_DECLINE: ${{ inputs.post-decline }} run: | out() { printf 'post=true\nsha=%s\nforged=%s\n' "$1" "$2" >>"$GITHUB_OUTPUT"; } @@ -3943,7 +3970,9 @@ jobs: valid "$gated" || gated=$EVENT_HEAD if [ "$TRUSTED_SKIP" = true ] || [ "$POST_DECLINE" = false ] || ! valid "$gated"; then out "" false - # The broker's kill-switch refusal skipped the run job: neutral. + # The broker's kill-switch refusal, or broker-check's budget + # stop, skipped the run job: neutral, under the skip-reason + # enum's kill-switch. if [ "$BROKER_STOP" = true ]; then echo "not-run-reason=kill-switch" >>"$GITHUB_OUTPUT"; fi else out "$gated" false @@ -4065,7 +4094,7 @@ jobs: # post, warns and leaves the job green. A broker kill-switch refusal ran # no lane, so it posts none. - name: Post the lane run report comment - if: always() && needs.broker-check.outputs.skip-reason != 'kill-switch' + if: always() && needs.broker-check.outputs.skip-reason != 'kill-switch' && needs.broker-check.outputs.skip-reason != 'budget' continue-on-error: true env: LANE: ${{ inputs.lane }} diff --git a/.github/workflows/upstream-apply-changelog.yml b/.github/workflows/upstream-apply-changelog.yml index 9283e6ab17b..9179ffc1529 100644 --- a/.github/workflows/upstream-apply-changelog.yml +++ b/.github/workflows/upstream-apply-changelog.yml @@ -109,12 +109,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing read or written" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing read or written" # A dispatch, and any re-run of one, must come from a listed human; the # broker checks the same, so this only keeps the model from running for diff --git a/.github/workflows/work-resolve-issue.yml b/.github/workflows/work-resolve-issue.yml index e9a32f3ab00..5f950715e16 100644 --- a/.github/workflows/work-resolve-issue.yml +++ b/.github/workflows/work-resolve-issue.yml @@ -434,6 +434,7 @@ jobs: id-token: write uses: ./.github/workflows/pr-run-activity-read.yml # zizmor: ignore[self-repository] with: + start-gate: true lane: work-resolve-issue role: issue-lane activity: resolve-issue @@ -631,6 +632,7 @@ jobs: id-token: write uses: ./.github/workflows/pr-run-activity-read.yml # zizmor: ignore[self-repository] with: + start-gate: true lane: work-resolve-issue role: issue-lane activity: resolve-issue diff --git a/.github/workflows/work-unblock.yml b/.github/workflows/work-unblock.yml index 04eb8fe0c3a..9c423f86eb5 100644 --- a/.github/workflows/work-unblock.yml +++ b/.github/workflows/work-unblock.yml @@ -185,12 +185,16 @@ jobs: with: value: ${{ vars.CLAUDE_LANES_DISABLED }} repo-value: ${{ vars.CLAUDE_LANES_REPO_DISABLED }} + # Starts new work: also stop on a low GITHUB_TOKEN budget. + budget: "true" - name: Report the kill switch if: steps.kill.outputs.proceed != 'true' env: REASON: ${{ steps.kill.outputs.reason }} - run: echo "::notice::the lanes kill switch stops this run (reason ${REASON:-unreadable}); nothing read or labeled" + CAUSE: ${{ steps.kill.outputs.cause }} + WHY: ${{ steps.kill.outputs.cause == 'budget' && format('the GITHUB_TOKEN budget is low until {0} (epoch seconds), which stops', steps.kill.outputs.reset) || 'the lanes kill switch stops' }} + run: echo "::notice::${WHY} this run (reason ${REASON:-unreadable}, cause ${CAUSE:-unreadable}); nothing read or labeled" # The base config's limits.api, in resolve-config's output shape, for # the pacer; a failed read leaves rate-limits.json's defaults. diff --git a/docs/conventions/pr-pipeline/README.md b/docs/conventions/pr-pipeline/README.md index c2ab5c7468d..9a643ef674f 100644 --- a/docs/conventions/pr-pipeline/README.md +++ b/docs/conventions/pr-pipeline/README.md @@ -3741,7 +3741,7 @@ with its default. | `fix-docs-per-run` | 2 | fix-docs dispatches one pr-unblock-dispatch `refine` run sends for its PR's event; 0 to 20. Each event offers one PR, so 0 turns CI fix-docs off and any other value turns it on. | | `admission-factor` | 0.8 | Share of the lane's hourly merge rate admitted per hour for the queued cap ([WIP cap](#wip-cap)); greater than 0, at most 1. | | `admission-min-per-hour` | 1 | Admissions per hour the queued cap never falls below, so a day with no lane merges cannot hold admissions shut; 0 to 100, and 0 removes the floor. | -| `api` | schema | GitHub API margins per pool (`github-token`, `lanes-app`, `graphql`, `owner-token`, `content-creation`). `limit-cap` sets the pool's planned budget: planned = min(header limit, `limit-cap`), used = header limit - header remaining, left = planned - used. `github-token` defaults it to GitHub's documented GITHUB_TOKEN limit, so a higher header limit is headroom, never a target; the other pools have no cap and plan to their header limit. `plan` and `read-floor` are shares of the planned budget; `writer-pause`, `breaker` and `stop-floor` are shares of the header's limit, by its remaining; `content-creation` adds `per-minute` and `per-hour`. Their defaults are in `.github/actions/log-api-budget/rate-limits.json`, the only copy, which every merge hold list floors, so a change to a default waits for a person. The rate guard (`.github/actions/log-api-budget/rate-guard.mjs`, `planOf`) reads `read-floor` and `limit-cap`, to warn once while calls continue, and stops on the header's remaining and GitHub's refusals: at a remaining of 0, or at or under `stop-floor` of the header's limit, it sends no further call to that pool before the header's reset, records the stop in the job's stop file, and holds the step under `API_BUDGET_HOLD`, else fails it; a response with no readable limit, remaining or future reset never stops on `stop-floor`; `pace` (github-token, default 0.2) is the share of the header's limit, projected left at the window's reset from the burn rate so far, under which a timer-driven lane skips a scheduled tick when the previous scheduled run, by run id, marked a clean pass of its work (its `Mark this tick's work done` step, which a held or red pass and a paced tick skip), so two scheduled ticks in a row are never both skipped and its cadence halves (`.github/actions/log-api-budget/pace.mjs`; the lost-run reconciler, pr-unblock-dispatch and work-unblock's sweep read it, and pr-merge's post-merge-dispatch reads it for the merge and enable-auto-merge sweep jobs, which skip at job level while the post-merge check and cover dispatch run on every tick), so the timer lanes slow down before the breaker stops every lane; `0` turns pacing off; the others are thresholds meant for the hub's limits sampler, and nothing in this repository enforces them. resolve-config passes the block through as set. | +| `api` | schema | GitHub API margins per pool (`github-token`, `lanes-app`, `graphql`, `owner-token`, `content-creation`). `limit-cap` sets the pool's planned budget: planned = min(header limit, `limit-cap`), used = header limit - header remaining, left = planned - used. `github-token` defaults it to GitHub's documented GITHUB_TOKEN limit, so a higher header limit is headroom, never a target; the other pools have no cap and plan to their header limit. `plan` and `read-floor` are shares of the planned budget; `writer-pause`, `breaker` and `stop-floor` are shares of the header's limit, by its remaining; `content-creation` adds `per-minute` and `per-hour`. Their defaults are in `.github/actions/log-api-budget/rate-limits.json`, the only copy, which every merge hold list floors, so a change to a default waits for a person. The rate guard (`.github/actions/log-api-budget/rate-guard.mjs`, `planOf`) reads `read-floor` and `limit-cap`, to warn once while calls continue, and stops on the header's remaining and GitHub's refusals: at a remaining of 0, or at or under `stop-floor` of the header's limit, it sends no further call to that pool before the header's reset, records the stop in the job's stop file, and holds the step under `API_BUDGET_HOLD`, else fails it; a response with no readable limit, remaining or future reset never stops on `stop-floor`; `pace` (github-token, default 0.2) is the share of the header's limit, projected left at the window's reset from the burn rate so far, under which a timer-driven lane skips a scheduled tick when the previous scheduled run, by run id, marked a clean pass of its work (its `Mark this tick's work done` step, which a held or red pass and a paced tick skip), so two scheduled ticks in a row are never both skipped and its cadence halves (`.github/actions/log-api-budget/pace.mjs`; the lost-run reconciler, pr-unblock-dispatch and work-unblock's sweep read it, and pr-merge's post-merge-dispatch reads it for the merge and enable-auto-merge sweep jobs, which skip at job level while the post-merge check and cover dispatch run on every tick), so the timer lanes slow down before the breaker stops every lane; `0` turns pacing off; `start-gate` (github-token, 0.1) is the share of the header's limit, by its remaining, under which a kill-switch gate that starts new work (`.github/actions/check-kill-switch`, `budget: "true"`) does not start its lane while the reset is still ahead (one GET /rate_limit per such gate, a stop with reason `kill-switch`, cause `budget` and the window's `reset`; a refused read, 403, 429, `retry-after` or a remaining of 0, also stops, and any other failed read proceeds); that gate reads it from rate-limits.json alone, and the schema rejects a `start-gate` set here; the others are thresholds meant for the hub's limits sampler, and nothing in this repository enforces them. resolve-config passes the block through as set. | | `lane-level` | 2, 4, 3, 2, `[lanes-app]`, `true`, `CLAUDE_LANE_LEVEL`, 150 | The adaptive lane level L (owner decision Q111): `floor`, `ceiling`, `default`, `clean-windows` and `pools` for `decide()` in `.github/scripts/lib/lane-level.mjs`, whose truth table is `lane-level.vectors.json` beside it. `shadow` (default `true`), `variable` (default `CLAUDE_LANE_LEVEL`, must end in `_LANE_LEVEL`) and `valid-minutes` (default 150, 60 to 1440) govern publish and read: work-unblock's hourly `publish-lane-level` job publishes L as `@` in `variable`, and pick-slot and label-unblocked read it through `.github/scripts/lib/lane-level-value.mjs` (empty, expired or malformed reads as `default`, clamped to `floor` and `ceiling`). While `shadow` is set the job mints no token and writes nothing, `SLOTS` stays `default`, and both log the level they would use. Each window L steps up 1 after `clean-windows` windows in a row that each had demand (runs waiting, or every slot running) and no limit signal, when every pool in `pools` fits: peak used, as a share of the pool's planned budget, times (L+1)/L at most `api..plan`. A window where a pool has `api..writer-pause` of its header's limit left or less resets the streak. Any limit signal the lanes App's own calls meet (403, 429, a secondary-limit or GraphQL rate-limit error, `retry-after`) halves L, rounded down, to no less than `floor`, and blocks a step up until the reset it names. A breaker trip is not a signal: it mostly reflects GITHUB_TOKEN spend, which pacing controls, and the breaker stays the backstop. Missing readings or demand hold L and the streak; a missing or malformed previous level reads as `default`, and every level is clamped to `floor` and `ceiling`. A `floor` above the `ceiling`, as set or defaulted, is rejected. An `api` pool field left unset reads from `rate-limits.json`, as the rate guard reads it. `.github/scripts/lib/lane-level-replay.mjs` replays the rule over the hub's recorded history. | report-outcome and the work-unblock sweep read them from resolve-config's output;