From d7da25c016cfd887cec43b82eed88c1ef1bc42b0 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 00:55:59 -0700 Subject: [PATCH 01/15] docs(release): write down how to curate release notes v1.3.0 was the first release whose notes were curated rather than generated, and the method was rediscovered from scratch. Both inputs mislead in ways worth naming once. operator-caveats-since.sh reports headings that CHANGED, not headings that are NEW. v1.3.0 listed two BREAKING headings and neither was a caveat for an upgrading operator: priorityTiers was already in v1.2.0, and capacityGate.mode's removed values had only ever existed on main. Repeating them unexamined would have sent operators after migrations they did not need. The section gives the one-line git test that tells the two apart. The commit log needs the inverse filter: keep what ships, drop ci/dogfood/test/ docs/build, and say you dropped them so 461 commits do not read as 57 user-facing changes. Keeping the trailing (#NNNN) costs nothing and GitHub auto-links it. Also recorded: promote by danger rather than by label (v1.3.0's most hazardous item carried no BREAKING heading), distinguish a guarded migration from a breaking change, caveat anything the validation run did not exercise, verify anchors by hand because make doc-links cannot see a notes file, and the chart-version form that makes a copy-pasted helm command fail. --- docs/operations/release.md | 80 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/docs/operations/release.md b/docs/operations/release.md index caf01258c..9092abeac 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -618,6 +618,86 @@ clobbers curated notes. If you want richer notes (highlights, upgrade caveats), --draft --notes-file …` — and the pipeline will leave your body untouched while still attaching the signed v2 CRD manifest asset. +#### Writing the curated notes + +A minor release accumulates more than a generated changelog can convey, and the +two inputs that feed it, `operator-caveats-since.sh` and the commit log, both +mislead in specific ways. What follows is the method, written after `v1.3.0`. + +**Notes answer "what is in it" and "what must I do". The docs answer "how" and +"why".** Every explanation that can live in `upgrade.md` or an `operations/` page +should, behind a link. `v1.3.0`'s first draft ran ~1000 words of prose. The +shipped version ran ~525 with nine links and lost nothing. Link a Highlight from +its bold lead, and link group headings rather than every line. + +**The caveats script reports headings that *changed*, not headings that are +*new*.** A section edited in this window is listed exactly like one added in it. +Test each before repeating it: + +```bash +git show :docs/operations/upgrade.md | grep -qF "### " \ + && echo "pre-existing" || echo "new in this window" +``` + +`v1.3.0` listed two `BREAKING` headings and **neither** was a caveat for an +upgrading operator: `priorityTiers` was already in `v1.2.0` (a pre-1.0 change), +and `capacityGate.mode`'s removed values had only ever existed on `main`. +Repeating them unexamined would have sent operators after migrations they did not +need. + +**Promote by danger, not by label.** The most hazardous item in `v1.3.0` carried +no `BREAKING` heading at all: the PriorityClass allowlist CRD apply, which affects +every install and whose rollback re-arms a cluster-wide outage. Read for +consequence, not for keywords. + +**Distinguish "breaking" from "guarded migration".** If skipping a step stops the +upgrade with a message naming the fix, it is a required migration and saying +"breaking change" overstates it. If a wrong path fails silently, say so loudly. +State which of the two you mean. + +**Enumerate from the commit log, filtered to what ships.** Conventional Commit +subjects are already terse diagnoses, so they need only the prefix and Q-ID +removed: + +```bash +git log --format='%s' ..HEAD \ + | grep -E '^(feat|fix)\((agc|gmc|proxy|worker|api|scaleset|chart|broker|wrapper|admission|provisioner|metrics)\)' +``` + +Exclude the `ci`, `dogfood`, `test`, `docs`, and `build` scopes, which ship in no +image or chart. Say in the notes that you excluded them, so a reader does not read 461 +commits as 57 user-visible changes. Keep the trailing `(#NNNN)`: GitHub auto-links +a bare `#NNNN` in a release body, so every line becomes traceable for free. + +**Fold long lists.** `
` renders on the Releases page and keeps +the top scannable. + +**Count what you list.** State a count in a `` and it will be wrong the +moment you curate the list. `v1.3.0`'s draft claimed 25 features and listed 23. + +**Caveat anything a validation run did not exercise.** The dogfood gate runs +against github.com, so it says nothing about GHES. A feature list that reads as +finished support overclaims. Check `docs/plan/archive/` for the feature's own +"what this will not verify" section before describing it. + +**Verify every link and anchor by hand.** `make doc-links` covers the repo, not a +notes file living outside it. Anchors are the usual failure: + +```bash +grep -E '^#{1,6} ' docs/operations/upgrade.md \ + | sed -E 's/^#+ //; s/[^a-zA-Z0-9 -]//g; s/ /-/g' | tr 'A-Z' 'a-z' +``` + +Pin doc links at `blob/vX.Y.Z/` so a reader of these notes gets that release's +instructions. They 404 until the tag is pushed, which is expected while the +Release is still a draft. + +**Watch the chart-version form.** Images are tagged `vX.Y.Z`, charts `X.Y.Z`. +A copy-pasteable `helm` command with a `v` in it fails. + +**Run the `deslop` skill over the draft before publishing.** Release notes are the +most-read prose the project ships. + ### 6. Chart version & metadata The `chart-publish` job sets the published chart's `version` and `appVersion` to From 38265251f5e97bc21f6f07e8fada401c7b4e3103 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 02:45:23 -0700 Subject: [PATCH 02/15] docs(release): make docs/releases/ the authoring source for release bodies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v1.3.0 notes were drafted in a scratch file and edited straight on the GitHub Release. By the time they were right they had been through a wrong count, a dead anchor, 46 forced line breaks, two mismatched PR numbers, and a GHES caveat that never said "GHES" — each caught by hand, none by review, because the text was never in a diff. Move them to docs/releases/v1.3.0.md, byte-identical to the published draft body, and republish the draft from that path. One file per stable tag, holding the release body verbatim; `gh release edit --notes-file` publishes it. These files are written for github.com's comment-flavour renderer, not for MkDocs: their GFM alerts would publish as literal text and their links already point into the versioned site. So docs/releases/ is excluded from every site version including dev, which means all four copies of that exclusion have to agree — mkdocs.yml, two `env:` blocks and one `export` in pages.yml, and docs-preview.sh. Missing the last would make `make docs-build` disagree with CI. Also corrects two claims in the method doc that the shipped notes contradict: they link the versioned docs site, not `blob/vX.Y.Z/` (18 such links, zero blob), and being in-repo does not get them link-checked, since check-doc-links skips external URLs by design and every link in a notes file is absolute. The hard-wrap check the method doc gave was void: `gh release view --json body` returns raw Markdown, which contains no `
` however badly wrapped. Replaced with a render through `gh api /markdown -f mode=gfm`, verified to distinguish the two (mode=markdown reports 0 on the same input). Against the real body: 0 `
`, 3 alert blocks, 6 folds, no literal `[!`. --- .github/workflows/pages.yml | 13 +- docs/README.md | 1 + docs/development/website.md | 9 + docs/operations/release.md | 68 +++++++- docs/releases/README.md | 45 +++++ docs/releases/v1.3.0.md | 324 +++++++++++++++++++++++++++++++++++ mkdocs.yml | 8 +- scripts/docs/docs-preview.sh | 4 +- 8 files changed, 462 insertions(+), 10 deletions(-) create mode 100644 docs/releases/README.md create mode 100644 docs/releases/v1.3.0.md diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index 183e083a5..722358ed2 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -97,7 +97,9 @@ jobs: # that breaks a plan/development page fails here rather than on `main`. - name: Build the dev scope (repo-internal docs included) env: - MKDOCS_EXCLUDE_DOCS: /README.md + # /releases/ stays excluded on dev too — those files are GitHub Release + # bodies, not site pages (mkdocs.yml documents why). + MKDOCS_EXCLUDE_DOCS: "/README.md\n/releases/\n" run: mkdocs build --strict --site-dir _site-dev # Push to main: re-run the PR gate against the merged tree. @@ -129,7 +131,9 @@ jobs: - run: mkdocs build --strict --site-dir _site - name: Build the dev scope (repo-internal docs included) env: - MKDOCS_EXCLUDE_DOCS: /README.md + # /releases/ stays excluded on dev too — those files are GitHub Release + # bodies, not site pages (mkdocs.yml documents why). + MKDOCS_EXCLUDE_DOCS: "/README.md\n/releases/\n" run: mkdocs build --strict --site-dir _site-dev # Push/dispatch: update the mike version tree on gh-pages and deploy it. @@ -322,8 +326,11 @@ jobs: # !ENV default only applies when the variable is ABSENT: a # conditional expression falling through to '' would publish the # internal docs on every release instead. + # + # /releases/ is excluded here too: GitHub Release bodies are authored + # for github.com's renderer and publish as literal text under MkDocs. if [[ "${VERSION}" == "dev" ]]; then - export MKDOCS_EXCLUDE_DOCS='/README.md' + export MKDOCS_EXCLUDE_DOCS=$'/README.md\n/releases/\n' fi # Links from the docs into the source tree resolve against the ref diff --git a/docs/README.md b/docs/README.md index 291beab84..f944d9b2d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,6 +21,7 @@ point see [DESIGN.md](../DESIGN.md). | [development/](development/README.md) | Developer workflow — building, testing, kind iteration, Go workspaces, code generation. | | [operations/](operations/README.md) | Operator guides — runbook, troubleshooting, observability, upgrades, tenant onboarding. | | [plan/](plan/README.md) | Implementation plans and audits. Authoritative ordering lives in [STATUS.md](STATUS.md). | +| [releases/](releases/README.md) | The curated GitHub Release body for each stable tag, verbatim. Authoring source, not a site page. | ## Find your path diff --git a/docs/development/website.md b/docs/development/website.md index a92fef295..3c7ecc116 100644 --- a/docs/development/website.md +++ b/docs/development/website.md @@ -402,6 +402,7 @@ Scope is **per version**, not per site (Q558): |---|---| | `stable` and every numbered release | Operator docs only: `docs/design/`, `docs/operations/`, `docs/index.md`, `why-gag.md`, `features.md`, `roadmap.md` | | `dev` | The above **plus** the repo-internal docs: `docs/STATUS.md` (the [backlog](#the-backlog-page)), `docs/plan/`, `docs/development/` (this file included), `docs/assets/`'s READMEs | +| *no version* | `docs/releases/` — see the trap below | A release is a frozen build, so a backlog published in one would be a snapshot stale from tag day. `dev` redeploys on every push to `main`, which is the only @@ -426,6 +427,14 @@ Two traps worth keeping in mind when editing that wiring: the list keeps that from surfacing as a build warning. Because it is never a site page, a doc that links to it must use the absolute `github.com` URL — a relative `../README.md` fails `mkdocs build --strict`. +- **`docs/releases/` is excluded on every version too**, and it is the one entry + that is not a repo-internal-vs-operator split: those files are GitHub Release + bodies, authored for github.com's comment-flavour renderer. Their GFM alerts + (`> [!WARNING]`) have no MkDocs equivalent and would publish as literal text, + and their links already point into the versioned site. The exclusion is spelled + out in **four** places that must agree — `mkdocs.yml`'s default, the two `env:` + blocks and the one `export` in `pages.yml`, and `scripts/docs/docs-preview.sh`. + Miss the last and `make docs-build` disagrees with CI. PR builds validate **both** scopes (`pages.yml`'s `build` job runs `mkdocs build` twice), so a PR that breaks a plan or development page fails there rather than on diff --git a/docs/operations/release.md b/docs/operations/release.md index 9092abeac..3f6cf9ee9 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -624,6 +624,37 @@ A minor release accumulates more than a generated changelog can convey, and the two inputs that feed it, `operator-caveats-since.sh` and the commit log, both mislead in specific ways. What follows is the method, written after `v1.3.0`. +**Author the notes in +[`docs/releases/`](https://github.com/actions-gateway/github-actions-gateway/tree/main/docs/releases), +not in a scratch file.** +One file per stable tag, `vX.Y.Z.md`, holding the release body verbatim — no +front matter, no heading above the first line. Publish from it: + +```bash +gh release edit vX.Y.Z --notes-file docs/releases/vX.Y.Z.md +``` + +`v1.3.0`'s notes were drafted under `tmp/` and edited straight on the Release. By +the time they were right they had been through a wrong count, a dead anchor, 46 +forced line breaks, two mismatched PR numbers, and a caveat that never said +"GHES" — every one caught by hand, none by review, and none of it reviewable +because the text was not in a diff. In-repo makes each fix a diff and each +published body reproducible from a commit. + +These files are excluded from the docs site on **every** version, dev included: +they are written for github.com's renderer, which the site is not. The exclusion +is spelled out in four places that must agree — `mkdocs.yml`, two `env:` blocks +and one `export` in [`pages.yml`](../../.github/workflows/pages.yml), and +`scripts/docs/docs-preview.sh`. + +**Past bodies are retrievable, so the previous release is your template:** + +```bash +gh release view vX.Y.Z --json body --jq .body +``` + +Use it to seed a new file when a tag predates this convention. + **Notes answer "what is in it" and "what must I do". The docs answer "how" and "why".** Every explanation that can live in `upgrade.md` or an `operations/` page should, behind a link. `v1.3.0`'s first draft ran ~1000 words of prose. The @@ -680,17 +711,44 @@ against github.com, so it says nothing about GHES. A feature list that reads as finished support overclaims. Check `docs/plan/archive/` for the feature's own "what this will not verify" section before describing it. -**Verify every link and anchor by hand.** `make doc-links` covers the repo, not a -notes file living outside it. Anchors are the usual failure: +**Link the versioned docs site, not `main` and not `blob/`.** A reader of these +notes should land on that release's instructions, and the site publishes a build +per stable tag: `https://actions-gateway.com/X.Y.Z/operations/…`. Mind the form — +the site drops the leading `v` exactly like the chart does. `v1.3.0` shipped 18 +such links. They 404 until the docs deploy for the tag completes, which is +expected while the Release is still a draft. + +**Verify every link and anchor by hand.** Being in `docs/releases/` does not get +them checked: `make doc-links` skips external URLs by design, and every link in a +notes file is absolute. Anchors are the usual failure — MkDocs and GitHub agree on +heading slugs, so deriving them from the source heading is reliable: ```bash grep -E '^#{1,6} ' docs/operations/upgrade.md \ | sed -E 's/^#+ //; s/[^a-zA-Z0-9 -]//g; s/ /-/g' | tr 'A-Z' 'a-z' ``` -Pin doc links at `blob/vX.Y.Z/` so a reader of these notes gets that release's -instructions. They 404 until the tag is pushed, which is expected while the -Release is still a draft. +**Do not hard-wrap.** GitHub renders a release body with comment-flavour GFM, +where a single newline becomes `
`. Keep every paragraph, blockquote, and list +continuation on one line. `v1.3.0`'s hard-wrapped draft rendered 46 of them. + +Check this against the **renderer**, not the source. `gh release view --json body` +returns the raw Markdown, which never contains `
` however badly it is wrapped, +so grepping that is a check that cannot fail. Render it the way GitHub will: + +```bash +gh api -X POST /markdown -f mode=gfm -f "text=$(cat docs/releases/vX.Y.Z.md)" \ + | grep -c '
' +``` + +`mode=gfm` is the comment flavour; `mode=markdown` is not, and reports 0 on a +hard-wrapped file. The same render confirms the rest of the GitHub-only markup +survived — expect one `markdown-alert-*` class per `> [!…]` block, one `
` +per fold, and no literal `[!` anywhere. + +**In-page anchors do not work in a release body.** Release-body headings carry no +`id`, so `[Upgrading](#upgrading)` is a dead link. Refer to a section by name in +bold instead. **Watch the chart-version form.** Images are tagged `vX.Y.Z`, charts `X.Y.Z`. A copy-pasteable `helm` command with a `v` in it fails. diff --git a/docs/releases/README.md b/docs/releases/README.md new file mode 100644 index 000000000..8d611f714 --- /dev/null +++ b/docs/releases/README.md @@ -0,0 +1,45 @@ +# Release notes + +One file per stable tag, `vX.Y.Z.md`, holding that release's **GitHub Release +body verbatim** — no front matter, no heading above the first line. The file is +the authoring source; the published body is a copy of it: + +```bash +gh release edit vX.Y.Z --notes-file docs/releases/vX.Y.Z.md +``` + +Curating a release body is a `publish.yml` step with its own method and its own +traps (what to promote, what the caveats script actually reports, why hard-wrapping +breaks the rendering). All of it lives in +[operations/release.md § Writing the curated notes](../operations/release.md#writing-the-curated-notes). + +| Release | Notes | +|---|---| +| v1.3.0 | [v1.3.0.md](v1.3.0.md) | + +Releases before `v1.3.0` predate this convention; their bodies live only on the +Releases page. Retrieve one with `gh release view vX.Y.Z --json body --jq .body`. + +## Not published to the docs site + +These files are written for github.com's **comment-flavour GFM** renderer, which +the MkDocs site is not. Two differences matter: + +- GFM alerts (`> [!WARNING]`, `> [!CAUTION]`, `> [!NOTE]`) render on github.com and + publish as literal text under MkDocs. +- A single newline inside a paragraph becomes `
` in a release body, so these + files are deliberately **not hard-wrapped**. Do not reflow them. Verify against + the renderer (`gh api -X POST /markdown -f mode=gfm …`), never against the raw + Markdown — the source has no `
` in it either way. + +`docs/releases/` is therefore excluded from every site version, `dev` included. +The exclusion is spelled out in four places that must agree — `mkdocs.yml`, two +`env:` blocks and one `export` in `.github/workflows/pages.yml`, and +`scripts/docs/docs-preview.sh`. Because the directory is never a site page, a doc +that links here must use the absolute `github.com` URL; a relative link fails +`mkdocs build --strict`. + +Their own links point into the **versioned docs site** +(`https://actions-gateway.com/X.Y.Z/…`, no leading `v`), so an operator reading +the notes for a release gets that release's instructions. `make doc-links` skips +external URLs by design, so those links are not checked — verify them by hand. diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md new file mode 100644 index 000000000..bbcb20db0 --- /dev/null +++ b/docs/releases/v1.3.0.md @@ -0,0 +1,324 @@ +Zero-idle self-hosted GitHub Actions runners for multi-tenant Kubernetes. + +> [!WARNING] +> **Two required upgrade steps, and one rollback that is not symmetric.** Read **Upgrading** below before deploying. Rolling back past the PriorityClass allowlist migration re-arms a cluster-wide apiserver outage, and upgrading correctly does not protect you from it. + +## Highlights + +**[Measured worker right-sizing](https://actions-gateway.com/1.3.0/operations/worker-rightsizing/).** The AGC sizes workers from what jobs actually use, not what the template asks for. Two opt-in profiles: `NodeShare` divides a node's allocatable capacity, `Throughput` derives from observed usage. Validated on GKE: a worker took a derived `1500m` where its templates asked for 2 and 3 CPU. + +**[A capacity gate that stops claiming jobs the cluster cannot place](https://actions-gateway.com/1.3.0/operations/troubleshooting/#runnerset-reports-workercapacitydeclined-the-gateway-stopped-claiming-jobs).** A `RunnerSet` refuses work it has nowhere to run instead of acquiring a job and stranding it. Whether the cluster can grow is now a gateway-level fact, not something each tenant asserts. + +**[Automatic re-run for interrupted workers](https://actions-gateway.com/1.3.0/operations/troubleshooting/#which-disruptions-auto-re-run-a-job-and-which-never-do).** Evicted, preempted, drained, and hand-deleted workers get their job re-run instead of vanishing. The eviction counter's new `cause` label tells the four apart. + +**[GitHub Enterprise Server gateways address their own appliance](https://actions-gateway.com/1.3.0/operations/troubleshooting/#a-ghes-tenants-traffic-never-reaches-the-appliance).** They never did: the GMC set no `GITHUB_API_BASE_URL`, so token exchange fell back to `api.github.com` and a GHES tenant failed before acquiring any job. A private CA in front of the appliance is now trusted via `spec.githubCABundleRef`. + +> [!NOTE] +> **Untested against a real GHES appliance.** The flow and the plumbing have test coverage. No GHES host has ever run it. + +## Upgrading + +Two steps, both guarded: skip either and the upgrade stops with a message naming the fix. Full procedure in [the upgrade guide](https://actions-gateway.com/1.3.0/operations/upgrade/), which also documents rollback per component: [GMC](https://actions-gateway.com/1.3.0/operations/upgrade/#gmc-rollback), [AGC](https://actions-gateway.com/1.3.0/operations/upgrade/#agc-rollback), [proxy](https://actions-gateway.com/1.3.0/operations/upgrade/#proxy-rollback), and [worker image](https://actions-gateway.com/1.3.0/operations/upgrade/#worker-image-rollback). + +**1. Apply the chart's CRDs.** Part of every upgrade of this chart, not a one-off: Helm installs the chart-root `crds/` directory on a fresh install only. + +```bash +helm show crds oci://ghcr.io/actions-gateway/charts/actions-gateway \ + --version 1.3.0 | kubectl apply -f - +``` + +Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. + +**2. If you set `priorityClassAllowlist.configMapName`, move those names into `allowedPriorityClasses`.** The allowlist is now a cluster-scoped `PriorityClassAllowlist` CR. It **fails closed**, so names you do not carry across start being rejected. + +> [!CAUTION] +> **Rolling back past step 2 re-arms a cluster-wide outage, and upgrading correctly does not protect you from it.** The allowlist moved off a `ConfigMap` because a ValidatingAdmissionPolicy `paramKind` on a core type stays broken for the life of a kube-apiserver process once its bindings go empty, which `helm uninstall` does. Read [Rolling back past this change](https://actions-gateway.com/1.3.0/operations/upgrade/#rolling-back-past-this-change-re-arms-the-outage-it-fixes) before reverting. + +Also worth a look before upgrading: worker quota accounting now counts native sidecars, `RuntimeClass` overhead, and storage, so a tight `ResourceQuota` may admit fewer workers. An `EgressProxy` pool drops the `app: actions-gateway-proxy` label and is recreated once. `v2alpha1` is deprecated and the apiserver warns on it. Each has its own section in [the upgrade guide](https://actions-gateway.com/1.3.0/operations/upgrade/). + +## Deprecations + +**Nothing is removed in this release.** Both notices below point at `v2.0.0`. + +**`v2alpha1` is deprecated**, and as of this release the apiserver warns on every apply. It stays served until `v2.0.0`, which removes it. Moving to `v2beta1` is a re-apply rather than a migration tool: [Before you upgrade past v2.0.0](https://actions-gateway.com/1.3.0/operations/v1alpha1-deprecation/#before-you-upgrade-past-v200). + +**`v1alpha1` and Classic are removed at `v2.0.0` too.** Tenants still on `v1alpha1` migrate with `gag-migrate`: [Migrating a tenant to v2](https://actions-gateway.com/1.3.0/operations/migration-v1-to-v2/#why-upgrade-to-v2). Unlike `v2alpha1`, `v1alpha1` emits **no** apiserver warning today, so nothing will remind you it is going away. + +## Everything since v1.2.0 + +461 commits. These are the ones that ship in the product. Build, CI, test, and docs work is left out. + +
+New features (23) + +**[Worker right-sizing](https://actions-gateway.com/1.3.0/operations/worker-rightsizing/)** + +- Per-`RunnerSet` worker usage sampling (#741) +- Measured sizing recommendations in `RunnerSet` status (#744) +- Opt-in sizing profiles applied at pod-build time (#746) +- Report a `Throughput` profile cancelled by an injected CPU limit (#956) + +**[Capacity-aware job intake](https://actions-gateway.com/1.3.0/operations/troubleshooting/#runnerset-reports-workercapacitydeclined-the-gateway-stopped-claiming-jobs)** + +- Opt-in capacity gate: refuse jobs the cluster cannot place (#902) +- Gate intake on the node autoscaler's own declination (#913) +- Refuse to claim jobs when namespace quota has no headroom (#793) +- Pre-claim quota rung ported to the scale-set tier (#868) + +**[Worker recovery](https://actions-gateway.com/1.3.0/operations/troubleshooting/#which-disruptions-auto-re-run-a-job-and-which-never-do)** + +- Recover preempted workers off `PreemptionByScheduler` (#972) +- Recover drained workers on the deletion mark (#1011) +- Eviction recovery ported to the scale-set tier (#851) +- Bound a worker orphaned while the AGC is down with a lifetime cap (#914) + +**[GitHub Enterprise Server](https://actions-gateway.com/1.3.0/operations/troubleshooting/#a-ghes-tenants-traffic-never-reaches-the-appliance)** (untested against a real appliance) + +- Trust a private CA in front of a GHES appliance (`spec.githubCABundleRef`) (#1137) + +**[Autoscaling and observability](https://actions-gateway.com/1.3.0/operations/observability-metrics/)** + +- Bring-your-own proxy autoscaler via `EgressProxy` `managedAutoscaling` (#758) +- Managed VPA opt-in for the GMC and per-gateway AGC (#761) +- Export `AGCAutoscalingUnavailable` as a Prometheus gauge (#763) +- Export `GitHubEgressIncomplete` as a gauge for fleet alerting (#1038) +- Count scale-set poll errors in the shared counter (#878) +- Alias `runner_set` onto `worker_pods_reaped_total` so scale-set series join (#1020) +- Attribute usage rows to the machine that measured them (#840) + +**[API](https://actions-gateway.com/1.3.0/operations/v1alpha1-deprecation/)** + +- Mark `v2alpha1` deprecated so the apiserver warns (#828) +- Rename `sizingRecommendation.windowStart` to `windowStartTime` (breaking, pre-GA) (#943) +- Exempt deletion-only updates from PriorityClass allowlist re-validation (#1017) + +
+ +
+Fixes (34) + +**Scale-set listener** + +- Deregister scale-set runner records on reap, and sweep unclaimed ones (#1052) +- Re-offer a scale-set job whose runner name will not register (#1048) +- Measure and fix the AGC restart queue replay (#1138) +- Flush the listener's pending message deletes on shutdown (#1148) +- Stop stranding a worker pod on a reclaimed job Secret (#1119) +- Reclaim the per-job JIT-config Secret (#727) +- Latch the capacity gate past the reap with a one-probe slot (#1000) +- One autoscaler loop's two verdicts must not gate intake (#949) + +**Shutdown and lifecycle** + +- Delete broker sessions reliably on `SIGTERM` (#739) +- Forward `SIGTERM` to `Runner.Worker` so cancelled jobs report (#747) +- Arm the worker termination relay before starting the child (#872) +- Drain in-flight `CONNECT` tunnels on proxy shutdown (#748) +- Linger for endpoint removal inside the drain budget (#755) +- Give a Running scale-set worker a reap deadline (#824) +- Delete the worker pod of a job the listener abandons (#1032) + +**Quota accounting** + +- Count native sidecars and `RuntimeClass` overhead in the worker footprint (#883) +- Count the storage keys in the worker footprint (#887) + +**Naming and identity** + +- Keep a truncated worker pod name valid and unique (#921) +- Bound the derived `RunnerGroup` name via one shared helper (#922) +- Keep v1 and v2 agent pools apart during coexistence (#915) +- Key a v2 proxy pool solely on its own identity label (#1117) +- A gateway-scoped AGC no longer reconciles v1 `RunnerGroups` (#1036) + +**GitHub Enterprise Server** (untested against a real appliance) + +- Make GHES gateways address their own appliance (#1034) +- Address `GITHUB_API_BASE_URL` for the re-run call (#973) +- Retry an evicted job's re-run until GitHub accepts it (#1010) + +**Admission and API** + +- Move the PriorityClass VAP `paramKind` off a core type (#958) +- Reject a `nodeShare.allocatable` declaring neither cpu nor memory (#946) +- Ratchet the v2beta1 single-`runnerLabel` rule onto the field (#847) +- Generate the missing `ClusterCapacity` deepcopy (#928) + +**GMC and proxy** + +- Let a rollout restart of a managed Deployment take effect (#1104) +- Derive the AGC `NO_PROXY` apiserver exemption from the cluster (#912) +- Own every namespaced v1 child so a stripped finalizer cannot leak them (#848) +- Fail AGC startup on an unreadable proxy CA cert (#1133) +- Read classic-tier run identity from the `github` context (#967) + +
+ +## API surface + +**One new CRD: [`PriorityClassAllowlist`](https://actions-gateway.com/1.3.0/operations/upgrade/#priorityclass-allowlist-configmap-to-cr)**, cluster-scoped, the only kind added since v1.2.0. It is also the only CRD the chart ships in its chart-root `crds/` directory, which Helm installs on a fresh install and **skips on every upgrade**. That is why applying CRDs is step 1 of Upgrading above, and why the upgrade hard-fails without it. + +The rest are new fields this tag publishes for the first time. There is no generated API reference yet, so the CRDs themselves are the authority: `kubectl explain`, or the signed `actions-gateway-crds-v2.yaml` attached below. + +
+New spec fields (10) + +`RunnerSet` + +- `spec.sizing.profile`: `Static`, `Binpack`, `Throughput`, or `NodeShare` (default `Static`) +- `spec.sizing.nodeShare.allocatable` / `.workersPerNode`: the NodeShare envelope +- `spec.sizing.minRequests` / `.maxRequests` / `.limitHeadroomPercent`: bounds on what a profile may derive +- `spec.capacityGate.mode`: `Off` or `Observe` (default `Off`) +- `spec.maxWorkerLifetime`: cap on a worker pod's lifetime + +`ActionsGateway` + +- `spec.clusterCapacity.nodeAutoscaling`: `Present` or `Absent` (default `Present`) +- `spec.agcAutoscaling.mode`: managed VPA for the AGC, `Off`, `Initial`, or `Recreate` +- `spec.githubCABundleRef.name`: ConfigMap holding a private CA for a GHES appliance + +`EgressProxy` + +- `spec.managedAutoscaling`: bring-your-own proxy autoscaler opt-in (default `true`) + +[`PriorityClassAllowlist`](https://actions-gateway.com/1.3.0/operations/security-operations/#priority-classes-the-allowed-priority-classes-allowlist) (new cluster-scoped kind) + +- `spec.allowedPriorityClasses`: the platform allowlist, replacing the watched ConfigMap + +
+ +
+New status fields (2) + +`RunnerSet.status` + +- `sizingProfileState`: `Active` or `AwaitingSamples` +- `sizingRecommendation[]`: per-container `observedPeak`, `observedP95`, `requests`, `limits`, `sampleCount`, `windowStartTime` + +
+ +
+New condition reasons (28) + +Reasons that gate or alert are documented with their remedies in [observability-metrics](https://actions-gateway.com/1.3.0/operations/observability-metrics/) and [troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/). + +**Sizing** + +- `SizingProfileActive` +- `SizingProfileOverridden` +- `SizingWithinRange` +- `SizingDrift` +- `SizingDriftDetected` +- `InsufficientSamples` +- `CPULimitInjected` +- `NoCPULimitInjected` + +**Capacity and job intake** + +- `CapacityAvailable` +- `WorkerCapacityDeclined` +- `WorkerCeilingReached` +- `ScaleUpDeclined` +- `GateModeUnsupported` +- `AwaitingProbe` +- `AwaitingWorkerPods` +- `JobsProvisioning` +- `JobProvisionStalled` +- `RunnerNameConflict` + +**AGC autoscaling** + +- `AGCAutoscalingActive` +- `AGCAutoscalingDisabled` +- `AGCAutoscalingUnavailable` +- `VPACRDNotInstalled` + +**GHES, egress, and CA trust** + +- `GitHubEgressAllowed` +- `GitHubEgressIncomplete` +- `ApplianceRangesRequired` +- `CABundleNotFound` +- `CABundleInvalid` + +**Lifecycle** + +- `GatewayTerminating` + +
+ +## Validation + +**This is the first release validated end to end on real infrastructure before shipping.** The pre-GA gate has existed since before v1.2.0, but no candidate had ever cleared it; `v1.3.0-rc.5` is the first to produce a verdict, and it gates the tag rather than advising it. + +What the run exercised on a live GKE cluster, against the signed release artifacts rather than a build: + +- The full e2e matrix on gateway-provisioned runners, **73/73 specs**, on Kata microVM-isolated workers ([run](https://github.com/actions-gateway/github-actions-gateway/actions/runs/30786972228)) +- Both sizing profiles actuating on real worker pods, with the derived value confirmed at the pod: `1500m` where the templates asked for 2 and 3 CPU +- The signed v2 CRD artifact downloaded, signature-verified, applied, and all five CRDs registered +- Teardown back to zero nodes + +**Every PR now upgrades from the last released chart, not just HEAD to HEAD** (#1008). It immediately caught step 1: #958 shipped a CRD into chart-root `crds/`, breaking every v1.2.0 upgrade while CI stayed green. + +This tag adds no product code beyond that candidate. + +**Full changelog:** [`v1.2.0...v1.3.0`](https://github.com/actions-gateway/github-actions-gateway/compare/v1.2.0...v1.3.0) + +## Project and tooling + +**Documentation is now versioned.** Since this release cycle the site publishes a build per stable tag, so these notes link `1.3.0` pages rather than whatever `main` says today, and an operator on an older release reads that release's docs. Older versions were seeded, so `1.0.0` through `1.3.0` are all live. + +The rest is contributor-facing, listed because a release is also a claim about how the project is built. + +
+Release, docs-site, CI, and tooling (26) + +**Documentation site** + +- Versioned docs via mike — stable default, dev opt-in (#768) +- Publish docs from stable release tags (#766) +- Derive the site announce bar from the git tags (#775) +- Gate the release on the docs-site announce bar naming the tag (#801) +- Publish the backlog on the dev docs site, filterable (#1059) +- Gate site-side link resolution, fix 13 site-only 404s (#1062) + +**Release process** + +- Release delta report + scope-ledger convention (#1058) +- Surface operator caveats in the pre-flight, don't rely on memory (#992) +- A status file and a sentinel for the release gate (#1161) +- Relay e2e spec progress into the release-validation gate (#1159) +- Make the release gate able to fail on a dead sizing profile (#867) +- Gate upgrades from the last released chart tag (#988) + +**Test and CI speed** + +- Show live spec progress and render the JUnit report (#1152) +- Report unit-run progress from the `go test -json` stream (#1162) +- Compile the e2e images' dependency tree once, and cache it across runs (#870) +- Overlap the e2e runner disk cleanup with job setup (#871) +- Cut the suite's serial tail (#874) +- Measure coverage in one workspace-wide invocation (#952) +- Run workspace unit tests as one multi-module invocation (#722) +- Change-scope local golangci-lint to the modules a diff affects (#724) + +**Repository tooling** + +- Allocate Q-IDs from a ref claim, not a counter line (#823) +- Merge driver resolving STATUS.md Queue rows by ID (#879) +- Repo-wide leftover conflict-marker gate (#732) +- Auto-rebase stranded Dependabot Go PRs (#827) +- Parse workflow YAML with a real parser, in a new devtools/ module (#1114) +- Add nolintlint + a ratcheted funlen gate (#771) + +
+ +## Verifying this release + +We sign every image and both charts. + +```bash +make verify-release VERSION=v1.3.0 +``` + +The signed v2 CRD manifest and `SHA256SUMS` ship as release assets with detached cosign bundles, and the publish workflow attests build provenance. diff --git a/mkdocs.yml b/mkdocs.yml index 3dfca76b7..9af7d478a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -19,8 +19,14 @@ copyright: Licensed under the Apache License 2.0. # a backlog in one would be a snapshot stale from tag day. Only the dev path in # .github/workflows/pages.yml sets it, and it keeps /README.md excluded on every # version: MkDocs drops it anyway as a conflict with the index.md landing page. +# +# /releases/ is excluded on EVERY version including dev, the one path here that is +# not repo-internal-vs-operator: those files are GitHub Release bodies, authored +# for github.com's renderer. Their GFM alerts (`> [!WARNING]`) have no MkDocs +# equivalent and would publish as literal text, and their links already point at +# the versioned site. The dev override in pages.yml carries this entry too. # See website.md § Publication scope. -exclude_docs: !ENV [MKDOCS_EXCLUDE_DOCS, "/README.md\n/STATUS.md\n/plan/\n/development/\n/assets/README.md\n/assets/fonts/README.md\n"] +exclude_docs: !ENV [MKDOCS_EXCLUDE_DOCS, "/README.md\n/STATUS.md\n/plan/\n/development/\n/releases/\n/assets/README.md\n/assets/fonts/README.md\n"] # Published but deliberately absent from the nav, which stays the operator-facing # table of contents on every version: the repo-internal tree the `dev` version diff --git a/scripts/docs/docs-preview.sh b/scripts/docs/docs-preview.sh index 0f1a77d88..af2d7cf3b 100755 --- a/scripts/docs/docs-preview.sh +++ b/scripts/docs/docs-preview.sh @@ -77,7 +77,9 @@ main() { # gate: the release scope excludes docs/plan/ and docs/development/, so a # break in those pages only ever surfaces in the dev scope. "${venv_dir}/bin/mkdocs" build --strict - MKDOCS_EXCLUDE_DOCS=/README.md \ + # Must match the dev override in pages.yml: /releases/ holds GitHub Release + # bodies, excluded from every version including dev (mkdocs.yml says why). + MKDOCS_EXCLUDE_DOCS=$'/README.md\n/releases/\n' \ "${venv_dir}/bin/mkdocs" build --strict --site-dir site-dev } From 5c936ddf7e3a98565032be1f3af689d66c6b0fa7 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 02:46:07 -0700 Subject: [PATCH 03/15] docs(status): file Q635, doc-links passes on a file it never scanned check-doc-links builds its scan list from `git ls-files -- '*.md'`, which is tracked-only, while the existence oracle deliberately includes untracked files "so a link to a brand-new file added in the same change resolves before it is staged". The asymmetry means links *to* a new doc resolve and links *from* it are never read. Hit while adding docs/releases/: the gate reported ok with an unchanged file count and one extra link, which reads exactly like a clean pass. CI never sees this because it runs on committed trees; the gap is local, pre-stage, and it makes a green local run mean less than it appears to. --- docs/STATUS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/STATUS.md b/docs/STATUS.md index de669d3fa..2a4e5ef84 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -60,6 +60,7 @@ Specific actionable items in priority order. Pick from the top; skip 🚫 items | Q630 | [A healthy quiet e2e leg is indistinguishable from a wedged one, so the stall report is unusable](../scripts/dogfood/release-sentinel.sh) | `dogfood` `tests` | 🔲 | S | Stall keys on status-file writes, which need relayed heartbeats, which need a fetchable job log. GitHub served `BlobNotFound` for a whole 30-min PASSED run: `hb: none`, false stalls, and re-firing instantly on relaunch. Consult run status instead. | | Q631 | [The release gate starves itself: routed CI and the e2e leg draw on one global CPU budget](plan/gke-dogfood.md#part-f--e2e-on-gke-kata-containers) | `dogfood` `debt` | 🔲 | S | Deploy routes CI to GAG, scaling `workers` to 8 (32 vCPU) out of the same `CPUS_ALL_REGIONS` the e2e pool then needs. Cost two rc.5 runs at the old limit of 32. Raised to 64, so 52 worst case now fits — but nothing caps the competition. | | Q623 | [`no-plan-refs-in-code` only scopes Go, but shell and workflow comments rot the same way](../scripts/docs/check-no-plan-refs-in-code.sh) | `ci` `debt` `retro` | 🔲 | S | Three shell files already carry `docs/plan/archive/` paths — they rotted and were repaired by hand. Extending needs care, not a wider glob: workflow `paths:` filters and `probe-investigations-cd.sh` (reads a plan file as data) are legitimate. | +| Q635 | [`doc-links` never reads a new doc's own links until it is staged, so it passes on a file it did not scan](../scripts/docs/check-doc-links.sh) | `ci` `debt` | 🔲 | S | `git ls-files -- '*.md'` lists tracked sources; the existence oracle deliberately adds untracked ones, so a link *to* a new file resolves while links *from* it are never read. Green, on a file count short by one. | | Q166 | [v2 API M4: cross-namespace EgressProxy sharing](plan/v2-api.md) | `security` `feature` | 🔲 | M | Demand fired 2026-08-01: `sharing.allowedNamespaces` is served v2beta1 API with no enforcement (#1079 sweep), so the beta contract would harden a dormant field. Next: M4 consent check, CA distribution, dual-side NetworkPolicy. | | Q408 | [Untrusted-PR egress posture for Kata workers](plan/q408-untrusted-pr-egress.md) | `security` | 🔲 | L | Demand fired 2026-07-31; plan written. Athens pattern (Q244) applied to images: per-upstream pull-through mirrors, then swap `e2e-open-egress` for a mirror-scoped NP. Next: Phase 0 egress inventory. | | Q606 | [A hard-killed AGC still replays an assignment it concluded but had not yet deleted](plan/q603-settle-delete-gap.md#the-residual-stated-precisely) | `bug` | 🔲 | M | Q603 closed the graceful half; a hard kill between `settle` and its `DELETE` remains, and no ordering can close it. Both candidate fixes need something first — a live measurement, or a [Q597](#Q597) decision — and the plan doc states which. | From 8898bee217a9a3b32181e016c63859cde6b1ed1b Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 02:52:01 -0700 Subject: [PATCH 04/15] docs(release): capture the rest of how the v1.3.0 notes were curated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The method doc recorded the first third of that work and stopped. Everything from "add a feature list" onward — the enumerations, the API surface, the receipts, the contributor section, the structure they all sit in — existed only in the session that produced them. Adds the section skeleton as a table, ordered by what a reader needs first, plus the rules behind the four sections whose content is not self-evident: - Deprecations, written even when nothing is removed, and stating whether the apiserver actually warns. v1.3.0 deprecated v2alpha1 (warns) alongside v1alpha1 (removal-slated, warns on nothing) — an asymmetry no reader can discover for themselves. - API surface, led by any new CRD, because the chart installs chart-root crds/ on a fresh install only; that is why "apply the CRDs" is upgrade step 1. - Validation as receipts: link the run, quote the counts, quote a value measured at the layer that matters. - Project and tooling, last, labelled not-user-facing. Also records three constraints found by hitting them: GFM alerts render as real callouts in a release body and carry the danger banner; the Releases index truncates behind "read more" and a fold is the only lever against it; and a caveat is read out of its heading's context, so it has to name its own subject ("Untested against a real GHES appliance", not "a real appliance"). Corrects the claim that the shipped notes ran ~525 words with nine links. They ran 2100 words and 25 links. The brevity rule applies to prose, not to the enumerations a reader actually searches — as written it prescribed cutting the notes to a quarter of what shipped. --- docs/operations/release.md | 84 +++++++++++++++++++++++++++++++++++--- 1 file changed, 79 insertions(+), 5 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 3f6cf9ee9..9e644da84 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -657,9 +657,70 @@ Use it to seed a new file when a tag predates this convention. **Notes answer "what is in it" and "what must I do". The docs answer "how" and "why".** Every explanation that can live in `upgrade.md` or an `operations/` page -should, behind a link. `v1.3.0`'s first draft ran ~1000 words of prose. The -shipped version ran ~525 with nine links and lost nothing. Link a Highlight from -its bold lead, and link group headings rather than every line. +should, behind a link. `v1.3.0`'s first draft ran ~1000 words of prose; cutting it +to links lost nothing. Link a Highlight from its bold lead, and link group +headings rather than every line. + +That rule shortens the *prose*. It does not shorten the notes: `v1.3.0` shipped +2100 words and 25 links, because enumerations kept being added — a feature list, a +fix list, the API surface, the condition reasons. Prose is what gets cut; lists are +what a reader actually searches. Fold the lists (below) and the length costs +nothing. + +##### The section skeleton + +`v1.3.0` arrived at this order after several passes. It is ordered by what a +reader needs first, not by what took the most work: + +| Section | Answers | Notes | +|---|---|---| +| *(one-line tagline)* | what this project is | for the reader who arrived from a search result | +| *(danger banner)* | is anything here going to hurt me | a GFM alert, above the fold; see below | +| **Highlights** | why upgrade | 3–5, each linked from its bold lead | +| **Upgrading** | what must I do | numbered steps; say which are guarded | +| **Deprecations** | what is going away | see below | +| **Everything since ``** | did my bug get fixed | folded lists with counts | +| **API surface** | what changed in the contract | see below | +| **Validation** | why should I believe you | receipts, not adjectives | +| **Project and tooling** | is this project healthy | contributor-facing; last for a reason | +| **Verifying this release** | how do I check the artifacts | the `make verify-release` line | + +**Lead with a danger banner, and make it an alert.** GFM alerts — `> [!WARNING]`, +`> [!CAUTION]`, `> [!NOTE]` — render as real coloured callouts in a release body +(verified; the render check below counts them). `v1.3.0` opens with a `[!WARNING]` +naming the two required upgrade steps and the asymmetric rollback, because those +are the only things that can hurt an operator who reads no further. Use `[!NOTE]` +for a scope caveat and `[!CAUTION]` for the one thing that is genuinely +destructive. Three alerts is a lot; more and none of them read as urgent. + +**Write a Deprecations section even when nothing is removed** — saying so is the +point, since "deprecation" reads as "removal" to a skimmer. For each notice give +the removal version, the migration path, and **whether the apiserver actually +warns**. `v1.3.0` deprecated `v2alpha1` (warns on every apply) alongside +`v1alpha1`, which is removal-slated and emits **no** warning at all — so nothing +reminds an operator it is going away. That asymmetry is exactly what a reader +cannot discover for themselves. + +**Give the API surface its own section, and lead it with any new CRD.** A new kind +is not a field: the chart installs chart-root `crds/` on a *fresh install only*, +so a new CRD is the reason "apply the CRDs" is step 1 of Upgrading, and the two +must cross-reference. Then fold the rest — new spec fields, new status fields, new +condition reasons — grouped by kind, counted like any other fold. `v1.3.0` listed +28 new condition reasons this way. Until a generated API reference exists, say +which artifact is authoritative (`kubectl explain`, or the signed CRD asset). + +**Validation is receipts, not adjectives.** "Thoroughly tested" is worth nothing. +Link the run, quote the counts, and quote a value measured at the layer that +matters: `v1.3.0` cites 73/73 specs with a run link, and a derived `1500m` observed +on the pod where the templates asked for 2 and 3 CPU. Ship the receipt wherever a +claim is made, not only in this section — a feature line that links its own PR is a +receipt too. + +**Keep a contributor-facing section, and put it last.** Release, CI, docs-site, and +tooling work ships in no image and no chart, so it does not belong in the change +lists. It still belongs in the notes: it is what a reader evaluating the project's +health is looking for. Fold it, label it as not user-facing, and let it sit below +everything an operator needs. **The caveats script reports headings that *changed*, not headings that are *new*.** A section edited in this window is listed exactly like one added in it. @@ -701,7 +762,11 @@ commits as 57 user-visible changes. Keep the trailing `(#NNNN)`: GitHub auto-lin a bare `#NNNN` in a release body, so every line becomes traceable for free. **Fold long lists.** `
` renders on the Releases page and keeps -the top scannable. +the top scannable. It is also the only lever against truncation: the Releases +*index* collapses a long body behind a **"read more"** link, and a fold counts as +its one summary line while collapsed. `v1.3.0` hit that limit and was folded back +under it — six folds, covering the two change lists, three API lists, and the +tooling list. If the index is truncating, the fix is another fold, not a cut. **Count what you list.** State a count in a `` and it will be wrong the moment you curate the list. `v1.3.0`'s draft claimed 25 features and listed 23. @@ -709,7 +774,16 @@ moment you curate the list. `v1.3.0`'s draft claimed 25 features and listed 23. **Caveat anything a validation run did not exercise.** The dogfood gate runs against github.com, so it says nothing about GHES. A feature list that reads as finished support overclaims. Check `docs/plan/archive/` for the feature's own -"what this will not verify" section before describing it. +"what this will not verify" section before describing it. Then ask whether an +unexercised feature belongs in Highlights at all — `v1.3.0` kept GHES there and +paid for it with a caveat in three places. + +**A caveat must survive being read alone.** Every line in a folded list is read +out of its heading's context — by search, by a linked anchor, by a skimmer. The +`v1.3.0` draft said "Untested against a real appliance" under a GHES heading, +which says nothing at all once the heading scrolls away; it shipped as "Untested +against a real **GHES** appliance". Name the subject inside the caveat, and repeat +the caveat at each place the feature is claimed rather than relying on proximity. **Link the versioned docs site, not `main` and not `blob/`.** A reader of these notes should land on that release's instructions, and the site publishes a build From 8ca640eb2a186cad3c16c01abe31c0dab3bb761f Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 02:59:06 -0700 Subject: [PATCH 05/15] docs(release): break the upgrade footnote into a list, and say why MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Upgrading section closed with three unrelated operator-facing changes — quota accounting counting more, an EgressProxy pool losing a label, v2alpha1 starting to warn — run together in one sentence-run directly below its two numbered steps. Three things a reader might act on individually, positioned and formatted as a footnote to the procedure above them. As a bulleted list with a bold lead each, the same words are scannable, and the deprecation line now points at the Deprecations section that owns it rather than restating it. Records the general rule in the method doc: prose is for framing, and anything a reader might need to act on individually gets its own line. Republished the draft body from the file and re-diffed: identical. Rendered through mode=gfm: 0
, 3 alerts, 6 folds, the new list a real
    . --- docs/operations/release.md | 8 ++++++++ docs/releases/v1.3.0.md | 6 +++++- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 9e644da84..aae944d22 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -761,6 +761,14 @@ image or chart. Say in the notes that you excluded them, so a reader does not re commits as 57 user-visible changes. Keep the trailing `(#NNNN)`: GitHub auto-links a bare `#NNNN` in a release body, so every line becomes traceable for free. +**One fact per line, especially next to a procedure.** Distinct operator-facing +changes run together into a paragraph read as background, and a paragraph sitting +under a numbered list reads as a footnote to it. `v1.3.0`'s Upgrading section +closed with three unrelated changes — quota accounting, a dropped proxy label, a +new apiserver warning — in one sentence-run below its two numbered steps; as a +bulleted list with a bold lead each, the same words are scannable. Prose is for +framing. Anything a reader might need to act on individually gets its own line. + **Fold long lists.** `
    ` renders on the Releases page and keeps the top scannable. It is also the only lever against truncation: the Releases *index* collapses a long body behind a **"read more"** link, and a fold counts as diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md index bbcb20db0..39aa1643e 100644 --- a/docs/releases/v1.3.0.md +++ b/docs/releases/v1.3.0.md @@ -34,7 +34,11 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. > [!CAUTION] > **Rolling back past step 2 re-arms a cluster-wide outage, and upgrading correctly does not protect you from it.** The allowlist moved off a `ConfigMap` because a ValidatingAdmissionPolicy `paramKind` on a core type stays broken for the life of a kube-apiserver process once its bindings go empty, which `helm uninstall` does. Read [Rolling back past this change](https://actions-gateway.com/1.3.0/operations/upgrade/#rolling-back-past-this-change-re-arms-the-outage-it-fixes) before reverting. -Also worth a look before upgrading: worker quota accounting now counts native sidecars, `RuntimeClass` overhead, and storage, so a tight `ResourceQuota` may admit fewer workers. An `EgressProxy` pool drops the `app: actions-gateway-proxy` label and is recreated once. `v2alpha1` is deprecated and the apiserver warns on it. Each has its own section in [the upgrade guide](https://actions-gateway.com/1.3.0/operations/upgrade/). +**Three changes that need no action, but will be visible.** Each has its own section in [the upgrade guide](https://actions-gateway.com/1.3.0/operations/upgrade/). + +- **Worker quota accounting now counts native sidecars, `RuntimeClass` overhead, and storage.** A tight `ResourceQuota` may admit fewer workers than it did. +- **An `EgressProxy` pool drops the `app: actions-gateway-proxy` label**, and is recreated once on upgrade. +- **`v2alpha1` now warns on every apply.** Deprecated, not removed — see **Deprecations** below. ## Deprecations From 09f26d4faecf4e83a90f971b5a44aedac580128b Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:03:08 -0700 Subject: [PATCH 06/15] docs(release): give the upgrade bullets their own sections and PRs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The three "will be visible" changes shared one link to the top of the upgrade guide, under a line claiming each had its own section. They do — so each bullet now links its own, and carries the PR that made the change. An operator lands on what to do rather than on a table of contents. Anchors were not hand-slugged. MkDocs emits them with punctuation, backticks and parenthesised clauses folded in ways a re-derived slug gets wrong, so they were read out of the built site, and every site URL in the notes was then resolved against site/ in one pass: 20 resolved, plus a deliberately bogus anchor the same run flagged. Without that planted failure a checker that resolves nothing is indistinguishable from a clean file. Replaces the method doc's hand-slugging recipe with that approach, and records the planted-failure requirement with it. --- docs/operations/release.md | 19 +++++++++++++------ docs/releases/v1.3.0.md | 8 ++++---- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index aae944d22..376a1a63a 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -800,16 +800,23 @@ the site drops the leading `v` exactly like the chart does. `v1.3.0` shipped 18 such links. They 404 until the docs deploy for the tag completes, which is expected while the Release is still a draft. -**Verify every link and anchor by hand.** Being in `docs/releases/` does not get -them checked: `make doc-links` skips external URLs by design, and every link in a -notes file is absolute. Anchors are the usual failure — MkDocs and GitHub agree on -heading slugs, so deriving them from the source heading is reliable: +**Verify every link and anchor.** Nothing checks them for you: `make doc-links` +skips external URLs by design, and every link in a notes file is absolute. Anchors +are the usual failure, and the built site is the authoritative oracle — read the +ids MkDocs actually emitted rather than re-deriving a slug by hand, which gets +punctuation, backticks, and parenthesised clauses wrong: ```bash -grep -E '^#{1,6} ' docs/operations/upgrade.md \ - | sed -E 's/^#+ //; s/[^a-zA-Z0-9 -]//g; s/ /-/g' | tr 'A-Z' 'a-z' +make docs-build +grep -oE 'id="[^"]*"' site/operations/upgrade/index.html | sed 's/id="//;s/"//' ``` +Better, resolve every link in the file against `site/` in one pass. Whatever you +write, **include a URL you know is broken** and confirm it is reported: a checker +that silently resolves nothing looks exactly like a clean file. `v1.3.0`'s notes +were verified this way — 20 site URLs resolved, plus one deliberately bogus +anchor that the same run flagged. + **Do not hard-wrap.** GitHub renders a release body with comment-flavour GFM, where a single newline becomes `
    `. Keep every paragraph, blockquote, and list continuation on one line. `v1.3.0`'s hard-wrapped draft rendered 46 of them. diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md index 39aa1643e..6f2295936 100644 --- a/docs/releases/v1.3.0.md +++ b/docs/releases/v1.3.0.md @@ -34,11 +34,11 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. > [!CAUTION] > **Rolling back past step 2 re-arms a cluster-wide outage, and upgrading correctly does not protect you from it.** The allowlist moved off a `ConfigMap` because a ValidatingAdmissionPolicy `paramKind` on a core type stays broken for the life of a kube-apiserver process once its bindings go empty, which `helm uninstall` does. Read [Rolling back past this change](https://actions-gateway.com/1.3.0/operations/upgrade/#rolling-back-past-this-change-re-arms-the-outage-it-fixes) before reverting. -**Three changes that need no action, but will be visible.** Each has its own section in [the upgrade guide](https://actions-gateway.com/1.3.0/operations/upgrade/). +**Three changes that need no action, but will be visible.** Each links its own section of the upgrade guide. -- **Worker quota accounting now counts native sidecars, `RuntimeClass` overhead, and storage.** A tight `ResourceQuota` may admit fewer workers than it did. -- **An `EgressProxy` pool drops the `app: actions-gateway-proxy` label**, and is recreated once on upgrade. -- **`v2alpha1` now warns on every apply.** Deprecated, not removed — see **Deprecations** below. +- **[Worker quota accounting now counts native sidecars, `RuntimeClass` overhead, and storage](https://actions-gateway.com/1.3.0/operations/upgrade/#worker-quota-accounting-now-counts-native-sidecars-runtimeclass-overhead-and-storage)** — a tight `ResourceQuota` may admit fewer workers than it did (#883, #887). +- **[An `EgressProxy` pool drops the `app: actions-gateway-proxy` label](https://actions-gateway.com/1.3.0/operations/upgrade/#non-breaking-an-egressproxy-pools-pods-drop-the-app-actions-gateway-proxy-label-its-pool-is-recreated-once)** — the pool is recreated once on upgrade (#1117). +- **[`v2alpha1` now warns on every apply](https://actions-gateway.com/1.3.0/operations/upgrade/#non-breaking-v2alpha1-is-deprecated-and-the-apiserver-now-warns)** — deprecated, not removed; see **Deprecations** below (#828). ## Deprecations From 6fc107eba775bbf8edf0846c4c430212d1866f54 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:03:14 -0700 Subject: [PATCH 07/15] docs(status): file Q636, nothing checks a release note's links MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/releases/ is in-repo now, but check-doc-links skips external URLs by design and every link in a notes file is absolute — so the one doc whose links are entirely unverifiable is the one published to every operator. site/ turns out to be a working oracle: resolving the v1.3.0 notes against the built HTML checked all 20 site URLs and flagged a planted bad anchor. Filed rather than built, since a gate needs somewhere to live and doc-links skipping external URLs is deliberate. --- docs/STATUS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/STATUS.md b/docs/STATUS.md index 2a4e5ef84..e653c3a6e 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -61,6 +61,7 @@ Specific actionable items in priority order. Pick from the top; skip 🚫 items | Q631 | [The release gate starves itself: routed CI and the e2e leg draw on one global CPU budget](plan/gke-dogfood.md#part-f--e2e-on-gke-kata-containers) | `dogfood` `debt` | 🔲 | S | Deploy routes CI to GAG, scaling `workers` to 8 (32 vCPU) out of the same `CPUS_ALL_REGIONS` the e2e pool then needs. Cost two rc.5 runs at the old limit of 32. Raised to 64, so 52 worst case now fits — but nothing caps the competition. | | Q623 | [`no-plan-refs-in-code` only scopes Go, but shell and workflow comments rot the same way](../scripts/docs/check-no-plan-refs-in-code.sh) | `ci` `debt` `retro` | 🔲 | S | Three shell files already carry `docs/plan/archive/` paths — they rotted and were repaired by hand. Extending needs care, not a wider glob: workflow `paths:` filters and `probe-investigations-cd.sh` (reads a plan file as data) are legitimate. | | Q635 | [`doc-links` never reads a new doc's own links until it is staged, so it passes on a file it did not scan](../scripts/docs/check-doc-links.sh) | `ci` `debt` | 🔲 | S | `git ls-files -- '*.md'` lists tracked sources; the existence oracle deliberately adds untracked ones, so a link *to* a new file resolves while links *from* it are never read. Green, on a file count short by one. | +| Q636 | [No gate checks a release note's links, and they are the one doc whose links are all absolute](releases/README.md) | `ci` `docs` | 🔲 | S | `docs/releases/` is now in-repo, but `check-doc-links` skips external URLs, so every anchor into the versioned site is unverified. `site/` is a working oracle — resolving the v1.3.0 notes against it caught a planted bad anchor. | | Q166 | [v2 API M4: cross-namespace EgressProxy sharing](plan/v2-api.md) | `security` `feature` | 🔲 | M | Demand fired 2026-08-01: `sharing.allowedNamespaces` is served v2beta1 API with no enforcement (#1079 sweep), so the beta contract would harden a dormant field. Next: M4 consent check, CA distribution, dual-side NetworkPolicy. | | Q408 | [Untrusted-PR egress posture for Kata workers](plan/q408-untrusted-pr-egress.md) | `security` | 🔲 | L | Demand fired 2026-07-31; plan written. Athens pattern (Q244) applied to images: per-upstream pull-through mirrors, then swap `e2e-open-egress` for a mirror-scoped NP. Next: Phase 0 egress inventory. | | Q606 | [A hard-killed AGC still replays an assignment it concluded but had not yet deleted](plan/q603-settle-delete-gap.md#the-residual-stated-precisely) | `bug` | 🔲 | M | Q603 closed the graceful half; a hard kill between `settle` and its `DELETE` remains, and no ordering can close it. Both candidate fixes need something first — a live measurement, or a [Q597](#Q597) decision — and the plan doc states which. | From b7979b0e88f97b6ea350bbef855a50aa38c60af8 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:19:23 -0700 Subject: [PATCH 08/15] docs(release): correct the validation claim, and complete the change lists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Critical review turned up five defects in the notes. rc.4 passed the gate the day before rc.5. The notes claimed no candidate had ever cleared it and that rc.5 was the first to return a verdict; the plan doc records otherwise. The true version is better and checkable: five candidates, three aborted, rc.4 passed without catching a live worker pod, rc.5 caught one. The commit-log scope filter dropped seven shipping fixes and nothing said so — two fix(scalesetlistener) (the pattern had `scaleset`, which does not match a longer scope), a compound fix(agc,gmc), three fix(migrate) for the shipped gag-migrate binary, and one fix(observability). Reconciling the residue found them: 57 kept of 132 feat/fix. Fixes 34 -> 41. "New spec fields (10)" listed thirteen fields in ten bullets, because three bullets group related ones. Every other fold counts the noun in its own label. The commit total was 461 and had already drifted to 462; replaced with a form that cannot go stale. Validation now says what the gate does not assert, rather than leaving the leg list to be read as full coverage. Method doc records each of these as a rule: reconcile what the scope filter drops, count the unit named in the label, and check the validation story against the plan doc rather than memory. --- docs/operations/release.md | 40 +++++++++++++++++++++++++++++++++----- docs/releases/v1.3.0.md | 26 ++++++++++++++++++------- 2 files changed, 54 insertions(+), 12 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 376a1a63a..36e1036ba 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -716,6 +716,14 @@ on the pod where the templates asked for 2 and 3 CPU. Ship the receipt wherever claim is made, not only in this section — a feature line that links its own PR is a receipt too. +**Check the validation story against the plan doc, not memory.** This is the +section a sceptical reader checks first, so a wrong detail here costs more than +anywhere else. `v1.3.0`'s draft claimed no candidate had ever cleared the gate and +that rc.5 was the first to return a verdict; the plan doc records rc.4 passing the +day before. The true version was better anyway — five candidates, three aborted, +rc.4 passed without catching a live worker pod, rc.5 caught one — and it is +checkable, which the flattering version was not. + **Keep a contributor-facing section, and put it last.** Release, CI, docs-site, and tooling work ships in no image and no chart, so it does not belong in the change lists. It still belongs in the notes: it is what a reader evaluating the project's @@ -753,12 +761,29 @@ removed: ```bash git log --format='%s' ..HEAD \ - | grep -E '^(feat|fix)\((agc|gmc|proxy|worker|api|scaleset|chart|broker|wrapper|admission|provisioner|metrics)\)' + | grep -E '^(feat|fix)\((agc|gmc|proxy|worker|api|scaleset|chart|broker|wrapper|admission|provisioner|metrics|scalesetlistener|migrate|observability)\)' +``` + +**That scope list is a guess, so reconcile what it dropped — never just run it.** +An allow-list silently omits any scope nobody thought of, and the result still +looks like a complete list. Print the residue and read every scope in it: + +```bash +git log --format='%s' ..HEAD | grep -E '^(feat|fix)' > /tmp/all +# ...run the filter above into /tmp/kept, then: +grep -vxF -f /tmp/kept /tmp/all | sed -E 's/^((feat|fix)\([^)]*\)).*/\1/' | sort | uniq -c | sort -rn ``` +`v1.3.0`'s first pass kept 57 of 132 and dropped **seven shipping fixes**: two +`fix(scalesetlistener)` (the pattern had `scaleset`, which does not match it), a +compound `fix(agc,gmc)`, three `fix(migrate)` for the shipped `gag-migrate` +binary, and one `fix(observability)`. Nothing in the output said so. The last +three scopes above were added only after that reconciliation — assume the list is +still incomplete for the next release. + Exclude the `ci`, `dogfood`, `test`, `docs`, and `build` scopes, which ship in no -image or chart. Say in the notes that you excluded them, so a reader does not read 461 -commits as 57 user-visible changes. Keep the trailing `(#NNNN)`: GitHub auto-links +image or chart. Say in the notes that you excluded them, so a reader does not read +hundreds of commits as the user-visible change count. Keep the trailing `(#NNNN)`: GitHub auto-links a bare `#NNNN` in a release body, so every line becomes traceable for free. **One fact per line, especially next to a procedure.** Distinct operator-facing @@ -776,8 +801,13 @@ its one summary line while collapsed. `v1.3.0` hit that limit and was folded bac under it — six folds, covering the two change lists, three API lists, and the tooling list. If the index is truncating, the fix is another fold, not a cut. -**Count what you list.** State a count in a `` and it will be wrong the -moment you curate the list. `v1.3.0`'s draft claimed 25 features and listed 23. +**Count what you list — and count the unit in the label.** State a count in a +`` and it will be wrong the moment you curate the list. `v1.3.0`'s draft +claimed 25 features and listed 23. Subtler: its "New spec fields (10)" had ten +*bullets* carrying thirteen *fields*, because three bullets grouped related ones +(`.minRequests` / `.maxRequests` / `.limitHeadroomPercent`). Every other fold +counted the noun in its own label; that one silently switched to bullets. Count +what the label says, then re-count after every edit — mechanically, not by eye. **Caveat anything a validation run did not exercise.** The dogfood gate runs against github.com, so it says nothing about GHES. A feature list that reads as diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md index 6f2295936..0d0192f43 100644 --- a/docs/releases/v1.3.0.md +++ b/docs/releases/v1.3.0.md @@ -7,7 +7,7 @@ Zero-idle self-hosted GitHub Actions runners for multi-tenant Kubernetes. **[Measured worker right-sizing](https://actions-gateway.com/1.3.0/operations/worker-rightsizing/).** The AGC sizes workers from what jobs actually use, not what the template asks for. Two opt-in profiles: `NodeShare` divides a node's allocatable capacity, `Throughput` derives from observed usage. Validated on GKE: a worker took a derived `1500m` where its templates asked for 2 and 3 CPU. -**[A capacity gate that stops claiming jobs the cluster cannot place](https://actions-gateway.com/1.3.0/operations/troubleshooting/#runnerset-reports-workercapacitydeclined-the-gateway-stopped-claiming-jobs).** A `RunnerSet` refuses work it has nowhere to run instead of acquiring a job and stranding it. Whether the cluster can grow is now a gateway-level fact, not something each tenant asserts. +**[A capacity gate that stops claiming jobs the cluster cannot place](https://actions-gateway.com/1.3.0/operations/troubleshooting/#runnerset-reports-workercapacitydeclined-the-gateway-stopped-claiming-jobs).** A `RunnerSet` refuses work it has nowhere to run instead of acquiring a job and stranding it. Whether the cluster can grow is now a gateway-level fact, not something each tenant asserts. Opt-in via `spec.capacityGate.mode`, default `Off`. **[Automatic re-run for interrupted workers](https://actions-gateway.com/1.3.0/operations/troubleshooting/#which-disruptions-auto-re-run-a-job-and-which-never-do).** Evicted, preempted, drained, and hand-deleted workers get their job re-run instead of vanishing. The eviction counter's new `cause` label tells the four apart. @@ -50,7 +50,7 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. ## Everything since v1.2.0 -461 commits. These are the ones that ship in the product. Build, CI, test, and docs work is left out. +Over 450 commits. These 64 are the ones that ship in the product; build, CI, test, and docs work is left out.
    New features (23) @@ -99,7 +99,7 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`.
    -Fixes (34) +Fixes (41) **Scale-set listener** @@ -111,6 +111,8 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. - Reclaim the per-job JIT-config Secret (#727) - Latch the capacity gate past the reap with a one-probe slot (#1000) - One autoscaler loop's two verdicts must not gate intake (#949) +- Give up on assignments GitHub no longer holds (#1108) +- Defer ceiling-blocked jobs instead of spinning on redelivery (#1105) **Shutdown and lifecycle** @@ -121,6 +123,7 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. - Linger for endpoint removal inside the drain budget (#755) - Give a Running scale-set worker a reap deadline (#824) - Delete the worker pod of a job the listener abandons (#1032) +- Reap tenant worker pods when a v2 gateway is deleted (#1140) **Quota accounting** @@ -147,6 +150,13 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. - Reject a `nodeShare.allocatable` declaring neither cpu nor memory (#946) - Ratchet the v2beta1 single-`runnerLabel` rule onto the field (#847) - Generate the missing `ClusterCapacity` deepcopy (#928) +- Dedupe apiserver deprecation warnings to one log line per process (#1021) + +**Tenant migration (`gag-migrate`)** + +- Dual-read the privileged grant across both label domains (#911) +- Retry a transiently unreachable webhook in `gag-migrate --apply` (#904) +- Make a Docker-in-Docker tenant actually migratable (#856) **GMC and proxy** @@ -165,7 +175,7 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. The rest are new fields this tag publishes for the first time. There is no generated API reference yet, so the CRDs themselves are the authority: `kubectl explain`, or the signed `actions-gateway-crds-v2.yaml` attached below.
    -New spec fields (10) +New spec fields (13) `RunnerSet` @@ -204,7 +214,7 @@ The rest are new fields this tag publishes for the first time. There is no gener
    New condition reasons (28) -Reasons that gate or alert are documented with their remedies in [observability-metrics](https://actions-gateway.com/1.3.0/operations/observability-metrics/) and [troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/). +Most are documented with their remedies in [observability-metrics](https://actions-gateway.com/1.3.0/operations/observability-metrics/) and [troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/). Five are not yet: `SizingWithinRange`, `SizingDriftDetected`, `InsufficientSamples`, `JobsProvisioning`, and `GitHubEgressAllowed`. Until the operator docs catch up, their godoc in `api/apiconditions/conditions.go` is the reference. **Sizing** @@ -253,7 +263,7 @@ Reasons that gate or alert are documented with their remedies in [observability- ## Validation -**This is the first release validated end to end on real infrastructure before shipping.** The pre-GA gate has existed since before v1.2.0, but no candidate had ever cleared it; `v1.3.0-rc.5` is the first to produce a verdict, and it gates the tag rather than advising it. +**This is the first release whose tag is gated by a passing end-to-end run on real infrastructure, rather than advised by one.** The gate predates `v1.2.0`. Getting a verdict out of it took five candidates: rc.1 through rc.3 aborted before finishing, rc.4 passed but never caught a live worker pod, and `v1.3.0-rc.5` passed with the pod-level check rc.4 missed. rc.5 needed its own verdict regardless — it carries product changes past rc.4, so rc.4's pass does not transfer. What the run exercised on a live GKE cluster, against the signed release artifacts rather than a build: @@ -262,7 +272,9 @@ What the run exercised on a live GKE cluster, against the signed release artifac - The signed v2 CRD artifact downloaded, signature-verified, applied, and all five CRDs registered - Teardown back to zero nodes -**Every PR now upgrades from the last released chart, not just HEAD to HEAD** (#1008). It immediately caught step 1: #958 shipped a CRD into chart-root `crds/`, breaking every v1.2.0 upgrade while CI stayed green. +That list is what the gate itself asserts. It does not assert the capacity gate, the namespace-quota rung, or anything GHES — the run targets github.com, and the gate builds an autoscaling cluster with headroom rather than a constrained one. Those rest on the unit, envtest, and e2e suites instead. + +**Every PR now upgrades from the last released chart, not just HEAD to HEAD** (#1008). It immediately caught what became step 1 above: #958 shipped a CRD into chart-root `crds/`, breaking every v1.2.0 upgrade while CI stayed green. This tag adds no product code beyond that candidate. From d9cf33c0704b72d3730ddad3b61fb2064c011559 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:19:31 -0700 Subject: [PATCH 09/15] docs(status): file Q637, the release gate never drives the constrained path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit validate-release.sh has zero references to capacityGate, nodeAutoscaling or quota; its legs are deploy, e2e, sizing and CRD smoke. It builds an autoscaling cluster with headroom, so the capacity gate and the namespace-quota rung — the paths that exist precisely for a constrained cluster — are never exercised on real infrastructure by the gate that decides a release ships. --- docs/STATUS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/STATUS.md b/docs/STATUS.md index e653c3a6e..e71b1e5a5 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -62,6 +62,7 @@ Specific actionable items in priority order. Pick from the top; skip 🚫 items | Q623 | [`no-plan-refs-in-code` only scopes Go, but shell and workflow comments rot the same way](../scripts/docs/check-no-plan-refs-in-code.sh) | `ci` `debt` `retro` | 🔲 | S | Three shell files already carry `docs/plan/archive/` paths — they rotted and were repaired by hand. Extending needs care, not a wider glob: workflow `paths:` filters and `probe-investigations-cd.sh` (reads a plan file as data) are legitimate. | | Q635 | [`doc-links` never reads a new doc's own links until it is staged, so it passes on a file it did not scan](../scripts/docs/check-doc-links.sh) | `ci` `debt` | 🔲 | S | `git ls-files -- '*.md'` lists tracked sources; the existence oracle deliberately adds untracked ones, so a link *to* a new file resolves while links *from* it are never read. Green, on a file count short by one. | | Q636 | [No gate checks a release note's links, and they are the one doc whose links are all absolute](releases/README.md) | `ci` `docs` | 🔲 | S | `docs/releases/` is now in-repo, but `check-doc-links` skips external URLs, so every anchor into the versioned site is unverified. `site/` is a working oracle — resolving the v1.3.0 notes against it caught a planted bad anchor. | +| Q637 | [The release gate asserts nothing about the capacity gate or the namespace-quota rung](../scripts/dogfood/validate-release.sh) | `dogfood` `tests` | 🔲 | M | `validate-release.sh` has zero references to `capacityGate`, `nodeAutoscaling` or quota; its legs are deploy, e2e, sizing, CRD smoke. It builds an autoscaling cluster with headroom, so the constrained path is never driven on real infra. | | Q166 | [v2 API M4: cross-namespace EgressProxy sharing](plan/v2-api.md) | `security` `feature` | 🔲 | M | Demand fired 2026-08-01: `sharing.allowedNamespaces` is served v2beta1 API with no enforcement (#1079 sweep), so the beta contract would harden a dormant field. Next: M4 consent check, CA distribution, dual-side NetworkPolicy. | | Q408 | [Untrusted-PR egress posture for Kata workers](plan/q408-untrusted-pr-egress.md) | `security` | 🔲 | L | Demand fired 2026-07-31; plan written. Athens pattern (Q244) applied to images: per-upstream pull-through mirrors, then swap `e2e-open-egress` for a mirror-scoped NP. Next: Phase 0 egress inventory. | | Q606 | [A hard-killed AGC still replays an assignment it concluded but had not yet deleted](plan/q603-settle-delete-gap.md#the-residual-stated-precisely) | `bug` | 🔲 | M | Q603 closed the graceful half; a hard kill between `settle` and its `DELETE` remains, and no ordering can close it. Both candidate fixes need something first — a live measurement, or a [Q597](#Q597) decision — and the plan doc states which. | From 9db14ca5007718e268e26de6caff27014a4cc3a4 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:23:39 -0700 Subject: [PATCH 10/15] docs(release): add a Security section and a testing/quality fold MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reconciling every commit since v1.2.0 against the PR refs the notes cite — 462 with a ref, 91 cited — left 371 uncited. Most is docs and harness work, correctly excluded. Three categories were not. Security had no section at all, which reads as an omission to the one reader scanning for it. The release carries a gRPC and x/net security bump (#749) and six routine dependency updates, no advisory, and three fixes that harden credential and trust handling without patching a reported vulnerability. Says so, and names the gates: govulncheck and Trivy on every PR with no path filter, re-run on every push to main. 52 test and 15 refactor commits were unrepresented. Thirteen are now folded in — the ones that changed what the suites can observe (a live cluster-autoscaler matcher, the Karpenter drift gate, day-2 helm upgrade over a live release, migration on GKE, quota-blocked redelivery, cosign's own download path) and the consolidation behind them, including the code-restraint sweep. Kata, DinD, workload identity, backlog tooling and perf were checked and need nothing: their shipping changes are already cited, or they are dogfood and test infrastructure that ships in no image. Method doc gains the Security row in the skeleton, and the rule that the section is written even when the answer is "no advisory". --- docs/operations/release.md | 8 ++++++++ docs/releases/v1.3.0.md | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/docs/operations/release.md b/docs/operations/release.md index 36e1036ba..fd33f6d75 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -683,8 +683,16 @@ reader needs first, not by what took the most work: | **API surface** | what changed in the contract | see below | | **Validation** | why should I believe you | receipts, not adjectives | | **Project and tooling** | is this project healthy | contributor-facing; last for a reason | +| **Security** | is there anything I must patch for | state it even when the answer is no | | **Verifying this release** | how do I check the artifacts | the `make verify-release` line | +**Say something about security even when there is nothing to report.** Silence +reads as an omission to the one reader scanning specifically for it. State +plainly that no advisory accompanies the release, then list what it does carry — +dependency security bumps, and any fix that hardens credential or trust handling +without patching a reported vulnerability. Name the scanning gates and when they +run. `v1.3.0` had no CVE and still warranted the section. + **Lead with a danger banner, and make it an alert.** GFM alerts — `> [!WARNING]`, `> [!CAUTION]`, `> [!NOTE]` — render as real coloured callouts in a release body (verified; the render check below counts them). `v1.3.0` opens with a `[!WARNING]` diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md index 0d0192f43..d7030ab03 100644 --- a/docs/releases/v1.3.0.md +++ b/docs/releases/v1.3.0.md @@ -329,6 +329,40 @@ The rest is contributor-facing, listed because a release is also a claim about h
    +
    +Testing and internal quality (13) + +Coverage that reaches real infrastructure rather than a fake, and the consolidation work behind it. 52 test and 15 refactor commits landed in this window; these are the ones that changed what the suites can observe. + +**Tests that drive a live cluster** + +- Assert the capacity gate's autoscaler matcher against a live cluster-autoscaler (#929) +- Extend the live-autoscaler drift gate to Karpenter (#1024) +- Gate day-2 `helm upgrade` over a live release (#935) +- Validate the v1→v2 migration live on GKE (#909) +- Prove a quota-blocked job redelivers to a sibling gateway (#1028) +- Scale-set disruption recovery under the chart's real RBAC, and the ordering fix it caught (#1018) +- Measure the node-drain path on both tiers (#895) +- Execute the cosign download path and `verify-release` itself (#1145) + +**Consolidation** + +- Code-restraint sweep: trim narrative comments, dedup helpers, drop dead weight (#1013) +- Merge the twin worker-quota evaluators into one shared core (#1022) +- Decompose the GMC `main()` and AGC `run()` god functions (#765) +- Unify the broker/scale-set error taxonomy in `githubapp/httpx` (#794) +- Share the version-neutral v2 API code, gate the identical remainder (#760) + +
    + +## Security + +**No security advisory accompanies this release**, and no fix here is embargoed or backported. What it carries is dependency maintenance: a security bump of gRPC to v1.82.1 and, in the build-tools module, `x/net` to v0.56.0 (#749), plus six routine dependency-group updates. + +Three fixes below harden handling of credentials and trust material rather than patching a reported vulnerability: an AGC that now fails startup on an unreadable proxy CA cert (#1133), and two that stop a per-job JIT-config `Secret` being stranded or reused (#727, #1119). + +Every pull request — with no path filter, deliberately — runs `govulncheck` across the workspace and a Trivy scan of the images, and `main` re-runs both on every push. + ## Verifying this release We sign every image and both charts. From ad82c5784760d216b109285d4cc5922ef2484920 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:26:18 -0700 Subject: [PATCH 11/15] docs(release): fold Validation's leg detail behind its own receipt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measuring collapsed height — each section's bytes with
    bodies excluded — put the body at 9.5k, with Validation third largest at 1781. Not an obvious candidate by eye: Upgrading and Highlights are both larger, and both have to stay open. Folding it needed care, because the four legs are the receipts and hiding receipts defeats the section. So the summary line carries the finding rather than a label: "73/73 e2e specs on Kata microVM workers, on live GKE — the four legs, and what none of them assert". A reader who never expands it has still seen the number. Validation drops to 902 chars collapsed, the body to 8638. Method doc records both: measure collapsed height to choose the next fold, and when the folded content is evidence, put the evidence in the summary — the exception to the count-in-the-summary convention. --- docs/operations/release.md | 17 +++++++++++++++-- docs/releases/v1.3.0.md | 11 ++++++++--- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index fd33f6d75..922b1229f 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -806,8 +806,21 @@ framing. Anything a reader might need to act on individually gets its own line. the top scannable. It is also the only lever against truncation: the Releases *index* collapses a long body behind a **"read more"** link, and a fold counts as its one summary line while collapsed. `v1.3.0` hit that limit and was folded back -under it — six folds, covering the two change lists, three API lists, and the -tooling list. If the index is truncating, the fix is another fold, not a cut. +under it — eight folds. If the index is truncating, the fix is another fold, not +a cut. + +**Pick the next fold by measuring collapsed height, not by eye.** Sum each +section's bytes with `
    …
    ` bodies excluded; the biggest sections +are rarely the ones that feel long. `v1.3.0` measured 9.5k collapsed, and the +third-largest section was Validation — not an obvious candidate, since Upgrading +and Highlights are larger but must both stay open. + +**When the content being folded is evidence, put the evidence in the ``.** +A fold whose summary reads "Validation details" hides the receipt; one that reads +`73/73 e2e specs on Kata microVM workers, on live GKE — the four legs, and what +none of them assert` *is* the receipt, and a reader who never expands it has +still seen the number. That is the exception to the count-in-the-summary +convention: enumerations carry a count, evidence carries the finding. **Count what you list — and count the unit in the label.** State a count in a `` and it will be wrong the moment you curate the list. `v1.3.0`'s draft diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md index d7030ab03..d94996787 100644 --- a/docs/releases/v1.3.0.md +++ b/docs/releases/v1.3.0.md @@ -265,14 +265,19 @@ Most are documented with their remedies in [observability-metrics](https://actio **This is the first release whose tag is gated by a passing end-to-end run on real infrastructure, rather than advised by one.** The gate predates `v1.2.0`. Getting a verdict out of it took five candidates: rc.1 through rc.3 aborted before finishing, rc.4 passed but never caught a live worker pod, and `v1.3.0-rc.5` passed with the pod-level check rc.4 missed. rc.5 needed its own verdict regardless — it carries product changes past rc.4, so rc.4's pass does not transfer. -What the run exercised on a live GKE cluster, against the signed release artifacts rather than a build: +
    +73/73 e2e specs on Kata microVM workers, on live GKE — the four legs, and what none of them assert + +Run against the signed release artifacts rather than a build ([run](https://github.com/actions-gateway/github-actions-gateway/actions/runs/30786972228)): -- The full e2e matrix on gateway-provisioned runners, **73/73 specs**, on Kata microVM-isolated workers ([run](https://github.com/actions-gateway/github-actions-gateway/actions/runs/30786972228)) +- The full e2e matrix on gateway-provisioned runners, **73/73 specs**, on Kata microVM-isolated workers - Both sizing profiles actuating on real worker pods, with the derived value confirmed at the pod: `1500m` where the templates asked for 2 and 3 CPU - The signed v2 CRD artifact downloaded, signature-verified, applied, and all five CRDs registered - Teardown back to zero nodes -That list is what the gate itself asserts. It does not assert the capacity gate, the namespace-quota rung, or anything GHES — the run targets github.com, and the gate builds an autoscaling cluster with headroom rather than a constrained one. Those rest on the unit, envtest, and e2e suites instead. +That is what the gate itself asserts. It does not assert the capacity gate, the namespace-quota rung, or anything GHES — the run targets github.com, and it builds an autoscaling cluster with headroom rather than a constrained one. Those rest on the unit, envtest, and e2e suites instead. + +
    **Every PR now upgrades from the last released chart, not just HEAD to HEAD** (#1008). It immediately caught what became step 1 above: #958 shipped a CRD into chart-root `crds/`, breaking every v1.2.0 upgrade while CI stayed green. From 43bd6a99b8f785c419789bb3eb9ca3b851cdada2 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:31:44 -0700 Subject: [PATCH 12/15] docs(release): add the metric surface, drop a non-product feature MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings, all from checking whether Validation and the testing fold should merge. They should not: Validation answers "why believe this build", operator-facing and about the artifact; the testing fold answers "is the project built well", contributor-facing and about the cycle. Merging drags one into the other's audience. What they needed was the cross-reference — Validation names the suites the gate does not exercise, and the fold lists the ones that grew to reach a live cluster. Checking that seam found #988 cited as "Gate upgrades from the last released chart tag" in the tooling fold. #988 is docs(plan): file and scope Q507 — the backlog row, not the gate. #1008 is the gate, already cited in Validation with more context. Removed the duplicate; tooling 26 -> 25. Verifying every shipped-fold entry touches a product path then found #840, listed as a feature, touching only claude-usage/ — the usage-tracking tooling. The `metrics` scope covers both that module and product metrics. It was the only one of 64; features 23 -> 22, shipped total 64 -> 63. Metrics themselves had no surface listing at all, though the condition reasons got one. Sixteen are added since v1.2.0 and none removed, so existing queries keep working; the fold reconciles both ways against a name diff of the two trees. The shipped Grafana dashboards were rebuilt around them and the PrometheusRule gains an egress alert — neither was mentioned. Section renamed to "API and metric surface" to match what it now covers. --- docs/operations/release.md | 2 +- docs/releases/v1.3.0.md | 51 ++++++++++++++++++++++++++++++++------ 2 files changed, 44 insertions(+), 9 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 922b1229f..2112da7e2 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -680,7 +680,7 @@ reader needs first, not by what took the most work: | **Upgrading** | what must I do | numbered steps; say which are guarded | | **Deprecations** | what is going away | see below | | **Everything since ``** | did my bug get fixed | folded lists with counts | -| **API surface** | what changed in the contract | see below | +| **API and metric surface** | what changed in the contract | CRDs *and* metrics; see below | | **Validation** | why should I believe you | receipts, not adjectives | | **Project and tooling** | is this project healthy | contributor-facing; last for a reason | | **Security** | is there anything I must patch for | state it even when the answer is no | diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md index d94996787..0feef2409 100644 --- a/docs/releases/v1.3.0.md +++ b/docs/releases/v1.3.0.md @@ -50,10 +50,10 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. ## Everything since v1.2.0 -Over 450 commits. These 64 are the ones that ship in the product; build, CI, test, and docs work is left out. +Over 450 commits. These 63 are the ones that ship in the product; build, CI, test, and docs work is left out.
    -New features (23) +New features (22) **[Worker right-sizing](https://actions-gateway.com/1.3.0/operations/worker-rightsizing/)** @@ -88,7 +88,6 @@ Over 450 commits. These 64 are the ones that ship in the product; build, CI, tes - Export `GitHubEgressIncomplete` as a gauge for fleet alerting (#1038) - Count scale-set poll errors in the shared counter (#878) - Alias `runner_set` onto `worker_pods_reaped_total` so scale-set series join (#1020) -- Attribute usage rows to the machine that measured them (#840) **[API](https://actions-gateway.com/1.3.0/operations/v1alpha1-deprecation/)** @@ -168,11 +167,13 @@ Over 450 commits. These 64 are the ones that ship in the product; build, CI, tes
    -## API surface +## API and metric surface **One new CRD: [`PriorityClassAllowlist`](https://actions-gateway.com/1.3.0/operations/upgrade/#priorityclass-allowlist-configmap-to-cr)**, cluster-scoped, the only kind added since v1.2.0. It is also the only CRD the chart ships in its chart-root `crds/` directory, which Helm installs on a fresh install and **skips on every upgrade**. That is why applying CRDs is step 1 of Upgrading above, and why the upgrade hard-fails without it. -The rest are new fields this tag publishes for the first time. There is no generated API reference yet, so the CRDs themselves are the authority: `kubectl explain`, or the signed `actions-gateway-crds-v2.yaml` attached below. +The rest are new fields and metrics this tag publishes for the first time. There is no generated API reference yet, so the CRDs themselves are the authority: `kubectl explain`, or the signed `actions-gateway-crds-v2.yaml` attached below. + +**Sixteen metrics are added and none removed**, so existing queries keep working. The shipped Grafana dashboards were rebuilt around them and the `PrometheusRule` gains an `ActionsGatewayGitHubEgressIncomplete` alert — both in [`deploy/monitoring/`](https://github.com/actions-gateway/github-actions-gateway/tree/v1.3.0/deploy/monitoring), described in [observability-dashboards](https://actions-gateway.com/1.3.0/operations/observability-dashboards/).
    New spec fields (13) @@ -211,6 +212,41 @@ The rest are new fields this tag publishes for the first time. There is no gener
    +
    +New metrics (16) + +All prefixed `actions_gateway_`, listed here without it. Their remedies and alerting guidance are in [observability-metrics](https://actions-gateway.com/1.3.0/operations/observability-metrics/). + +**Worker usage sampling** + +- `worker_usage_cpu_peak_cores` +- `worker_usage_memory_peak_bytes` +- `worker_usage_job_cpu_peak_cores` +- `worker_usage_job_memory_peak_bytes` +- `worker_usage_jobs_sampled_total` +- `worker_usage_jobs_unsampled_total` +- `worker_usage_poll_errors_total` + +**Scale-set capacity and intake** + +- `scaleset_advertised_capacity` +- `scaleset_capacity_withheld` +- `scaleset_jobs_deferred` +- `scaleset_jobs_abandoned_total` + +**Worker recovery** + +- `eviction_rerun_failures_total` +- `eviction_recovery_identity_unknown_total` + +**Gateway health** + +- `agc_autoscaling_unavailable` +- `github_egress_incomplete` +- `broker_session_leaks_total` + +
    +
    New condition reasons (28) @@ -275,7 +311,7 @@ Run against the signed release artifacts rather than a build ([run](https://gith - The signed v2 CRD artifact downloaded, signature-verified, applied, and all five CRDs registered - Teardown back to zero nodes -That is what the gate itself asserts. It does not assert the capacity gate, the namespace-quota rung, or anything GHES — the run targets github.com, and it builds an autoscaling cluster with headroom rather than a constrained one. Those rest on the unit, envtest, and e2e suites instead. +That is what the gate itself asserts. It does not assert the capacity gate, the namespace-quota rung, or anything GHES — the run targets github.com, and it builds an autoscaling cluster with headroom rather than a constrained one. Those rest on the unit, envtest, and e2e suites instead — several of which grew this cycle to reach a live cluster, listed under **Testing and internal quality** below.
    @@ -292,7 +328,7 @@ This tag adds no product code beyond that candidate. The rest is contributor-facing, listed because a release is also a claim about how the project is built.
    -Release, docs-site, CI, and tooling (26) +Release, docs-site, CI, and tooling (25) **Documentation site** @@ -310,7 +346,6 @@ The rest is contributor-facing, listed because a release is also a claim about h - A status file and a sentinel for the release gate (#1161) - Relay e2e spec progress into the release-validation gate (#1159) - Make the release gate able to fail on a dead sizing profile (#867) -- Gate upgrades from the last released chart tag (#988) **Test and CI speed** From 04b922c43c7f70b942c145a49b2ba4bf18b9d665 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:37:14 -0700 Subject: [PATCH 13/15] docs(release): add the Event-reason surface and the new operator guides MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Diffing docs/ from v1.2.0 to main found three things the notes had missed. Eight Kubernetes Event reasons are new and none removed — a third enumerable surface alongside the condition reasons and the metrics, and the one an operator meets first, since it is what `kubectl get events` prints. Six already have their own troubleshooting section. WorkerCapacityDeclined is both an event and a condition reason, and says so. Extracting them needed care: the same call takes a reason and an action, so ProvisionWorker and ApplyAGCAutoscaler read as reasons until each call site is checked. Two new operator guides were unlinked. resourcequota-sizing.md is 411 lines turning runner shapes into quota numbers — exactly what an operator needs after the accounting change the notes already flag, so it is linked from that bullet. node-shutdown-budgets.md is 337 lines on terminationGracePeriodSeconds being a request rather than a guarantee, with no feature of its own to hang off. troubleshooting.md gained 36 sections, most of them the remedy for a reason listed in one of the three surface folds. The notes linked the page without signalling that. Method doc records that a table of contents is not merely unnecessary but impossible: release-body headings carry no id — verified against the published v1.2.0 page, where the only ids belong to GitHub's own chrome — so an unlinked list would cost collapsed height to navigate nothing. Ten folds already read as the outline. --- docs/operations/release.md | 17 ++++++++++++++++- docs/releases/v1.3.0.md | 20 +++++++++++++++++++- 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 2112da7e2..2b597dd4f 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -888,7 +888,22 @@ per fold, and no literal `[!` anywhere. **In-page anchors do not work in a release body.** Release-body headings carry no `id`, so `[Upgrading](#upgrading)` is a dead link. Refer to a section by name in -bold instead. +bold instead. Verify on a published release rather than trusting this — the page +does emit ids, but only on GitHub's own chrome: + +```bash +curl -sS https://github.com///releases/tag/ \ + | grep -oE ']*>' | grep -c 'id=' +``` + +**So a table of contents is not available, and should not be faked.** An unlinked +list of section names is dead weight that costs collapsed height against the +truncation limit while navigating nothing. The folds already serve that role: a +collapsed `
    ` is a labelled one-line entry, so a body with ten folds +reads as an outline whether or not the reader expands any of them. Navigation +comes from section order and the danger banner, not from a ToC. (The in-repo copy +under `docs/releases/` does get GitHub's auto-generated file outline for free, +which is a second reason not to hand-roll one.) **Watch the chart-version form.** Images are tagged `vX.Y.Z`, charts `X.Y.Z`. A copy-pasteable `helm` command with a `v` in it fails. diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md index 0feef2409..82ac976fb 100644 --- a/docs/releases/v1.3.0.md +++ b/docs/releases/v1.3.0.md @@ -36,7 +36,7 @@ Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. **Three changes that need no action, but will be visible.** Each links its own section of the upgrade guide. -- **[Worker quota accounting now counts native sidecars, `RuntimeClass` overhead, and storage](https://actions-gateway.com/1.3.0/operations/upgrade/#worker-quota-accounting-now-counts-native-sidecars-runtimeclass-overhead-and-storage)** — a tight `ResourceQuota` may admit fewer workers than it did (#883, #887). +- **[Worker quota accounting now counts native sidecars, `RuntimeClass` overhead, and storage](https://actions-gateway.com/1.3.0/operations/upgrade/#worker-quota-accounting-now-counts-native-sidecars-runtimeclass-overhead-and-storage)** — a tight `ResourceQuota` may admit fewer workers than it did (#883, #887). New this release: [Sizing the platform-owned `ResourceQuota`](https://actions-gateway.com/1.3.0/operations/resourcequota-sizing/) turns runner shapes and concurrency ceilings into the numbers, so the first install is a calculation rather than a guess. - **[An `EgressProxy` pool drops the `app: actions-gateway-proxy` label](https://actions-gateway.com/1.3.0/operations/upgrade/#non-breaking-an-egressproxy-pools-pods-drop-the-app-actions-gateway-proxy-label-its-pool-is-recreated-once)** — the pool is recreated once on upgrade (#1117). - **[`v2alpha1` now warns on every apply](https://actions-gateway.com/1.3.0/operations/upgrade/#non-breaking-v2alpha1-is-deprecated-and-the-apiserver-now-warns)** — deprecated, not removed; see **Deprecations** below (#828). @@ -247,6 +247,22 @@ All prefixed `actions_gateway_`, listed here without it. Their remedies and aler
    +
    +New Kubernetes Event reasons (8) + +A surface separate from the condition reasons below — these land in `kubectl get events`, and six have their own section in [troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/). None was removed. + +- `WorkerPodLifetimeExceeded` — the lifetime cap killed a worker +- `WorkerPodOrphanedRunning` — a Running worker was reaped +- `WorkerPodCompletedPending` — a worker was reaped while Pending after its job finished +- `WorkerPodsReapedOnGatewayTeardown` — workers cleared as a gateway was deleted +- `WorkerDrainTimeout` — a drain did not finish in budget +- `WaitingForWorkerDrain` — deletion is blocked pending drain +- `AssignmentAbandoned` — the listener gave up an assignment +- `WorkerCapacityDeclined` — also a condition reason, listed below + +
    +
    New condition reasons (28) @@ -325,6 +341,8 @@ This tag adds no product code beyond that candidate. **Documentation is now versioned.** Since this release cycle the site publishes a build per stable tag, so these notes link `1.3.0` pages rather than whatever `main` says today, and an operator on an older release reads that release's docs. Older versions were seeded, so `1.0.0` through `1.3.0` are all live. +**Three operator guides are new**, and one grew by a third. [Worker right-sizing](https://actions-gateway.com/1.3.0/operations/worker-rightsizing/) and [Sizing the platform-owned `ResourceQuota`](https://actions-gateway.com/1.3.0/operations/resourcequota-sizing/) are linked from their features above. [Node shutdown budgets across platforms](https://actions-gateway.com/1.3.0/operations/node-shutdown-budgets/) is reference material with no feature of its own: `terminationGracePeriodSeconds` is a request, not a guarantee, and it collects what each platform actually grants — the proxy asks for 60s and gets 15 on a GKE Spot node. [Troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/) gained **36 sections**, most of them the remedy for a condition or event reason listed above. + The rest is contributor-facing, listed because a release is also a claim about how the project is built.
    From d521ece1aafa32b2ac48dd88aa9f158fad8052d0 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:43:26 -0700 Subject: [PATCH 14/15] docs(release): add the configuration surface the CRDs cannot show MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Diffing code from v1.2.0 to main found one operator-facing category with no representation at all: configuration. Four chart values (the GMC's managed VPA block, off by default), three environment tunables documented in the operator docs but nowhere else — PROXY_SHUTDOWN_DRAIN_TIMEOUT (45s), PROXY_SHUTDOWN_LINGER (10s), AGC_QUOTA_ADMISSION — and one renamed GMC flag, --priority-class-allowlist-configmap to --priority-class-allowlist-name, which matters only to an install passing GMC arguments directly. Nothing here is required; every item defaults to the previous behaviour. What the diff confirmed rather than changed is worth recording too. The 13 spec and 2 status fields match the CRD schemas at both tags exactly, and nothing was removed — the notes now say so, since "nothing removed" is what an operator is looking for. Both upgrade steps really are guarded: each is a Helm `fail` naming the fix, one for the CRD not being installed and one for priorityClassAllowlist.configMapName still being set. That values key survives as a tripwire, which is why a values diff shows no removal. Two extraction traps are recorded in the method doc, both hit here. A rename reads as a removal when the scope is one directory: env vars appeared to have 17 removals, every one of them code moving out of cmd/, and zero when re-run repo-wide. And adjacent string arguments read as the same thing — recordEvent(obj, type, reason, action) put ProvisionWorker and ApplyAGCAutoscaler through extraction as reasons until each call site was read. --- docs/operations/release.md | 18 ++++++++++++++++++ docs/releases/v1.3.0.md | 25 ++++++++++++++++++++++++- 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 2b597dd4f..3c5289058 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -709,6 +709,24 @@ warns**. `v1.3.0` deprecated `v2alpha1` (warns on every apply) alongside reminds an operator it is going away. That asymmetry is exactly what a reader cannot discover for themselves. +**Diff every surface an operator can see, not just the CRDs.** Each of these is +enumerable, and each hides in a different file, so a review that reads only the +Go diff misses most of them. `v1.3.0` shipped five: CRD fields, metric names, +Kubernetes Event reasons, condition reasons, and configuration (chart values, +env tunables, CLI flags). Diff each between the two tags mechanically rather than +reading the changelog for them — the Event reasons and the metrics had no +enumeration at all until they were diffed, and the notes had already been through +several reviews. + +Two traps. **A rename reads as a removal** when the extraction is scoped to one +directory: env vars first appeared to have 17 removals, all of which were code +moving out of `cmd/`; re-running repo-wide showed zero. **Adjacent string +arguments read as the same thing**: `recordEvent(obj, type, reason, action, …)` +puts a reason and an action side by side, so `ProvisionWorker` and +`ApplyAGCAutoscaler` both survived extraction as reasons until each call site was +checked. Always report "none removed" when it is true — operators are looking for +exactly that. + **Give the API surface its own section, and lead it with any new CRD.** A new kind is not a field: the chart installs chart-root `crds/` on a *fresh install only*, so a new CRD is the reason "apply the CRDs" is step 1 of Upgrading, and the two diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md index 82ac976fb..748d11a8e 100644 --- a/docs/releases/v1.3.0.md +++ b/docs/releases/v1.3.0.md @@ -171,7 +171,7 @@ Over 450 commits. These 63 are the ones that ship in the product; build, CI, tes **One new CRD: [`PriorityClassAllowlist`](https://actions-gateway.com/1.3.0/operations/upgrade/#priorityclass-allowlist-configmap-to-cr)**, cluster-scoped, the only kind added since v1.2.0. It is also the only CRD the chart ships in its chart-root `crds/` directory, which Helm installs on a fresh install and **skips on every upgrade**. That is why applying CRDs is step 1 of Upgrading above, and why the upgrade hard-fails without it. -The rest are new fields and metrics this tag publishes for the first time. There is no generated API reference yet, so the CRDs themselves are the authority: `kubectl explain`, or the signed `actions-gateway-crds-v2.yaml` attached below. +**Thirteen spec fields and two status fields are added, and nothing is removed** — verified against the CRD schemas at both tags, not assembled by hand. The rest are new fields, metrics, and settings this tag publishes for the first time. There is no generated API reference yet, so the CRDs themselves are the authority: `kubectl explain`, or the signed `actions-gateway-crds-v2.yaml` attached below. **Sixteen metrics are added and none removed**, so existing queries keep working. The shipped Grafana dashboards were rebuilt around them and the `PrometheusRule` gains an `ActionsGatewayGitHubEgressIncomplete` alert — both in [`deploy/monitoring/`](https://github.com/actions-gateway/github-actions-gateway/tree/v1.3.0/deploy/monitoring), described in [observability-dashboards](https://actions-gateway.com/1.3.0/operations/observability-dashboards/). @@ -212,6 +212,29 @@ The rest are new fields and metrics this tag publishes for the first time. There
    +
    +New configuration: 4 chart values, 3 tunables, 1 renamed flag + +Nothing here is required — every item defaults to the previous behaviour. Listed because none of it is discoverable from the CRDs. + +**Chart values** — the GMC's own managed VPA, off by default + +- `vpa.enabled` (default `false`) +- `vpa.updateMode` (default `"Off"`) +- `vpa.minAllowed` / `vpa.maxAllowed` (default `{}`) + +**Environment tunables** — documented with their defaults in [troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/) and [observability-metrics](https://actions-gateway.com/1.3.0/operations/observability-metrics/) + +- `PROXY_SHUTDOWN_DRAIN_TIMEOUT` (default `45s`) — the proxy's whole shutdown budget, covering endpoint linger then in-flight `CONNECT` drain +- `PROXY_SHUTDOWN_LINGER` (default `10s`, `-1s` to skip) — how long to wait for endpoint removal before draining +- `AGC_QUOTA_ADMISSION` (default on) — set `false` to turn the namespace-quota rung off AGC-wide + +**Renamed flag** — only affects an install that passes GMC arguments directly; the chart handles it + +- `--priority-class-allowlist-configmap` → `--priority-class-allowlist-name` (#958) + +
    +
    New metrics (16) From a6e241e559799a123f1707b9d3deb9db3ca4fe16 Mon Sep 17 00:00:00 2001 From: Karl Isenberg Date: Mon, 3 Aug 2026 03:46:32 -0700 Subject: [PATCH 15/15] docs(release): close five gaps in the curation method, add the check table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Auditing the method doc against what this session actually did found five techniques that caught real defects but were never written down. Two are citation checks. Passing the scope filter does not make a commit shippable — scopes are reused, and feat(metrics) covers both the product's Prometheus metrics and the claude-usage module, which is how a commit touching only claude-usage/ shipped as a feature. And a docs(plan) commit that files a Q-row reads exactly like the fix it describes, which is how #988 came to be cited under #1008's label. One is a caveat check. Understating coverage is as wrong as overstating it and feels safer, so it survives review: the draft said the capacity gate had unit and envtest coverage only, when a live-cluster-autoscaler test and 305 lines of quota-blocked e2e both exist. Grep every higher tier before writing "only tested at X", and prefer the narrow checkable claim. One is a source. docs/ deserves the same tag-to-tag diff as the code — a new operator page is the strongest signal of a capability the notes forgot, and it should be linked from the bullet it serves rather than a documentation inventory. The last is a table collecting every mechanical check in one place, with the two habits that make them mean anything: plant a known failure in whatever reports all-clear, and report the negative when it is true, since "none removed" is what an operator is scanning for. Also records why these files carry no title heading — the Releases page renders the tag name as its own h1, and stripping a heading at publish time would break the byte-identical property the files exist to guarantee. --- docs/operations/release.md | 72 +++++++++++++++++++++++++++++++++++++- docs/releases/README.md | 12 +++++-- 2 files changed, 81 insertions(+), 3 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 3c5289058..6460a821d 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -628,7 +628,9 @@ mislead in specific ways. What follows is the method, written after `v1.3.0`. [`docs/releases/`](https://github.com/actions-gateway/github-actions-gateway/tree/main/docs/releases), not in a scratch file.** One file per stable tag, `vX.Y.Z.md`, holding the release body verbatim — no -front matter, no heading above the first line. Publish from it: +front matter, and **no title heading** — the Releases page already renders the tag +name as the page's `

    `, so a `# vX.Y.Z` in the body duplicates it. Publish from +it: ```bash gh release edit vX.Y.Z --notes-file docs/releases/vX.Y.Z.md @@ -727,6 +729,22 @@ puts a reason and an action side by side, so `ProvisionWorker` and checked. Always report "none removed" when it is true — operators are looking for exactly that. +**Diff `docs/` as well, and link what is new from where it is actionable.** A new +operator page is the strongest signal of a capability the notes forgot, and a +heavily grown one shows where the release's real weight landed: + +```bash +git diff --name-status v..origin/main -- docs/operations/ | grep '^A' +git diff --stat v..origin/main -- docs/operations/ | sort -t'|' -k2 -rn | head +``` + +`v1.3.0` added three operator guides and grew `troubleshooting.md` by 36 +sections. Link a new guide from the bullet it serves rather than from a +documentation inventory — `resourcequota-sizing.md` belongs on the quota-accounting +upgrade note, where an operator hits the problem it solves. A guide with no +feature to attach to goes in the contributor-facing section with one line on why +it exists. + **Give the API surface its own section, and lead it with any new CRD.** A new kind is not a field: the chart installs chart-root `crds/` on a *fresh install only*, so a new CRD is the reason "apply the CRDs" is step 1 of Upgrading, and the two @@ -812,6 +830,27 @@ image or chart. Say in the notes that you excluded them, so a reader does not re hundreds of commits as the user-visible change count. Keep the trailing `(#NNNN)`: GitHub auto-links a bare `#NNNN` in a release body, so every line becomes traceable for free. +**Passing the scope filter does not make a commit shippable — check the paths.** +The filter matches a scope string, and scopes are reused. `v1.3.0` listed +"Attribute usage rows to the machine that measured them" as a feature; the commit +is `feat(metrics)` and touches only `claude-usage/`, the Claude Code usage +tooling, because that module and the product's Prometheus metrics share a scope. +Assert the paths for every cited PR instead of reading the subjects: + +```bash +git show --stat --format='' --name-only \ + | grep -qE '^(api|broker|cmd|githubapp|scaleset|charts)/' || echo "NOT PRODUCT" +``` + +It was one entry in 64, and no amount of re-reading the list would have found it. + +**Cite the commit that did the work, not the one that filed it.** A Q-numbered +backlog row and its implementation have near-identical subjects, so a `docs(plan)` +commit reads exactly like the fix. `v1.3.0` cited #988 — `docs(plan): file and +scope Q507` — under the label of #1008, the gate itself; a reader clicking through +would have landed on a planning row. Resolve each number to its title before +shipping, and look for the same work cited twice under two numbers. + **One fact per line, especially next to a procedure.** Distinct operator-facing changes run together into a paragraph read as background, and a paragraph sitting under a numbered list reads as a footnote to it. `v1.3.0`'s Upgrading section @@ -855,6 +894,15 @@ finished support overclaims. Check `docs/plan/archive/` for the feature's own unexercised feature belongs in Highlights at all — `v1.3.0` kept GHES there and paid for it with a caveat in three places. +**A caveat is a claim, so measure it before writing it.** Understating coverage +is as wrong as overstating it, and easier to do accidentally because it feels +safe. `v1.3.0`'s draft said the capacity gate had "unit and envtest coverage +only"; the repo has a live-cluster-autoscaler test for its matcher (#929) and 305 +lines of e2e proving a quota-blocked job redelivers (#1028). Before writing "only +tested at tier X", grep for the feature at every higher tier — and if the true +statement is narrower than the tidy one, ship the narrow statement. What survived +here was "the release gate does not *assert* it", which is checkable. + **A caveat must survive being read alone.** Every line in a folded list is read out of its heading's context — by search, by a linked anchor, by a skimmer. The `v1.3.0` draft said "Untested against a real appliance" under a GHES heading, @@ -929,6 +977,28 @@ A copy-pasteable `helm` command with a `v` in it fails. **Run the `deslop` skill over the draft before publishing.** Release notes are the most-read prose the project ships. +##### Before publishing: the mechanical checks + +Every rule above that can be checked by a machine, in the order they are cheapest +to run. None of these is a substitute for reading the notes — but each one caught +a defect in `v1.3.0` that several careful readings had not. + +| Check | How | What it catches | +|---|---|---| +| Fold counts | recount every `` against its bullets, counting the noun the label names | a count that drifted during curation | +| Enumerations | reconcile each surface fold against the tag-to-tag diff **both ways** | a name listed that no longer exists, or shipped and never listed | +| Citations resolve | `gh pr view --json title` for each `#NNNN` | a planning commit cited as the implementation | +| Citations ship | `git show --stat --name-only ` against product paths | a non-product commit listed as a feature | +| Anchors | resolve every site URL against a built `site/`, **with one planted bad anchor** | a heading that moved, and a checker that silently resolves nothing | +| Rendering | `gh api -X POST /markdown -f mode=gfm` | hard-wrapped `
    `s, alerts that did not render, literal `[!` | +| Published body | re-fetch and `diff` against the file | an edit made on the Release and not in the repo | + +Two habits make the difference. **Plant a known failure in anything that reports +"all clear"** — a checker with a broken query and a clean file produce identical +output. And **report the negative when it is true**: "16 metrics added, none +removed" and "13 spec fields, nothing removed" are what an operator is actually +scanning for, and neither is worth stating unless it was measured. + ### 6. Chart version & metadata The `chart-publish` job sets the published chart's `version` and `appVersion` to diff --git a/docs/releases/README.md b/docs/releases/README.md index 8d611f714..f4f970527 100644 --- a/docs/releases/README.md +++ b/docs/releases/README.md @@ -1,8 +1,16 @@ # Release notes One file per stable tag, `vX.Y.Z.md`, holding that release's **GitHub Release -body verbatim** — no front matter, no heading above the first line. The file is -the authoring source; the published body is a copy of it: +body verbatim** — no front matter, no heading above the first line. + +**No title heading**, specifically. The Releases page renders the tag name as the +page's own `

    `, so a `# vX.Y.Z` in the body is a duplicate title one line under +the real one. The file therefore opens on its first real sentence, and its +identity comes from the filename — which is also why stripping a heading at +publish time is not the answer: the published body has to stay byte-identical to +this file, or the drift these files exist to prevent comes back. + +The file is the authoring source; the published body is a copy of it: ```bash gh release edit vX.Y.Z --notes-file docs/releases/vX.Y.Z.md