diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index 183e083a5..722358ed2 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -97,7 +97,9 @@ jobs: # that breaks a plan/development page fails here rather than on `main`. - name: Build the dev scope (repo-internal docs included) env: - MKDOCS_EXCLUDE_DOCS: /README.md + # /releases/ stays excluded on dev too โ€” those files are GitHub Release + # bodies, not site pages (mkdocs.yml documents why). + MKDOCS_EXCLUDE_DOCS: "/README.md\n/releases/\n" run: mkdocs build --strict --site-dir _site-dev # Push to main: re-run the PR gate against the merged tree. @@ -129,7 +131,9 @@ jobs: - run: mkdocs build --strict --site-dir _site - name: Build the dev scope (repo-internal docs included) env: - MKDOCS_EXCLUDE_DOCS: /README.md + # /releases/ stays excluded on dev too โ€” those files are GitHub Release + # bodies, not site pages (mkdocs.yml documents why). + MKDOCS_EXCLUDE_DOCS: "/README.md\n/releases/\n" run: mkdocs build --strict --site-dir _site-dev # Push/dispatch: update the mike version tree on gh-pages and deploy it. @@ -322,8 +326,11 @@ jobs: # !ENV default only applies when the variable is ABSENT: a # conditional expression falling through to '' would publish the # internal docs on every release instead. + # + # /releases/ is excluded here too: GitHub Release bodies are authored + # for github.com's renderer and publish as literal text under MkDocs. if [[ "${VERSION}" == "dev" ]]; then - export MKDOCS_EXCLUDE_DOCS='/README.md' + export MKDOCS_EXCLUDE_DOCS=$'/README.md\n/releases/\n' fi # Links from the docs into the source tree resolve against the ref diff --git a/docs/README.md b/docs/README.md index 291beab84..f944d9b2d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,6 +21,7 @@ point see [DESIGN.md](../DESIGN.md). | [development/](development/README.md) | Developer workflow โ€” building, testing, kind iteration, Go workspaces, code generation. | | [operations/](operations/README.md) | Operator guides โ€” runbook, troubleshooting, observability, upgrades, tenant onboarding. | | [plan/](plan/README.md) | Implementation plans and audits. Authoritative ordering lives in [STATUS.md](STATUS.md). | +| [releases/](releases/README.md) | The curated GitHub Release body for each stable tag, verbatim. Authoring source, not a site page. | ## Find your path diff --git a/docs/STATUS.md b/docs/STATUS.md index de669d3fa..e71b1e5a5 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -60,6 +60,9 @@ Specific actionable items in priority order. Pick from the top; skip ๐Ÿšซ items | Q630 | [A healthy quiet e2e leg is indistinguishable from a wedged one, so the stall report is unusable](../scripts/dogfood/release-sentinel.sh) | `dogfood` `tests` | ๐Ÿ”ฒ | S | Stall keys on status-file writes, which need relayed heartbeats, which need a fetchable job log. GitHub served `BlobNotFound` for a whole 30-min PASSED run: `hb: none`, false stalls, and re-firing instantly on relaunch. Consult run status instead. | | Q631 | [The release gate starves itself: routed CI and the e2e leg draw on one global CPU budget](plan/gke-dogfood.md#part-f--e2e-on-gke-kata-containers) | `dogfood` `debt` | ๐Ÿ”ฒ | S | Deploy routes CI to GAG, scaling `workers` to 8 (32 vCPU) out of the same `CPUS_ALL_REGIONS` the e2e pool then needs. Cost two rc.5 runs at the old limit of 32. Raised to 64, so 52 worst case now fits โ€” but nothing caps the competition. | | Q623 | [`no-plan-refs-in-code` only scopes Go, but shell and workflow comments rot the same way](../scripts/docs/check-no-plan-refs-in-code.sh) | `ci` `debt` `retro` | ๐Ÿ”ฒ | S | Three shell files already carry `docs/plan/archive/` paths โ€” they rotted and were repaired by hand. Extending needs care, not a wider glob: workflow `paths:` filters and `probe-investigations-cd.sh` (reads a plan file as data) are legitimate. | +| Q635 | [`doc-links` never reads a new doc's own links until it is staged, so it passes on a file it did not scan](../scripts/docs/check-doc-links.sh) | `ci` `debt` | ๐Ÿ”ฒ | S | `git ls-files -- '*.md'` lists tracked sources; the existence oracle deliberately adds untracked ones, so a link *to* a new file resolves while links *from* it are never read. Green, on a file count short by one. | +| Q636 | [No gate checks a release note's links, and they are the one doc whose links are all absolute](releases/README.md) | `ci` `docs` | ๐Ÿ”ฒ | S | `docs/releases/` is now in-repo, but `check-doc-links` skips external URLs, so every anchor into the versioned site is unverified. `site/` is a working oracle โ€” resolving the v1.3.0 notes against it caught a planted bad anchor. | +| Q637 | [The release gate asserts nothing about the capacity gate or the namespace-quota rung](../scripts/dogfood/validate-release.sh) | `dogfood` `tests` | ๐Ÿ”ฒ | M | `validate-release.sh` has zero references to `capacityGate`, `nodeAutoscaling` or quota; its legs are deploy, e2e, sizing, CRD smoke. It builds an autoscaling cluster with headroom, so the constrained path is never driven on real infra. | | Q166 | [v2 API M4: cross-namespace EgressProxy sharing](plan/v2-api.md) | `security` `feature` | ๐Ÿ”ฒ | M | Demand fired 2026-08-01: `sharing.allowedNamespaces` is served v2beta1 API with no enforcement (#1079 sweep), so the beta contract would harden a dormant field. Next: M4 consent check, CA distribution, dual-side NetworkPolicy. | | Q408 | [Untrusted-PR egress posture for Kata workers](plan/q408-untrusted-pr-egress.md) | `security` | ๐Ÿ”ฒ | L | Demand fired 2026-07-31; plan written. Athens pattern (Q244) applied to images: per-upstream pull-through mirrors, then swap `e2e-open-egress` for a mirror-scoped NP. Next: Phase 0 egress inventory. | | Q606 | [A hard-killed AGC still replays an assignment it concluded but had not yet deleted](plan/q603-settle-delete-gap.md#the-residual-stated-precisely) | `bug` | ๐Ÿ”ฒ | M | Q603 closed the graceful half; a hard kill between `settle` and its `DELETE` remains, and no ordering can close it. Both candidate fixes need something first โ€” a live measurement, or a [Q597](#Q597) decision โ€” and the plan doc states which. | diff --git a/docs/development/website.md b/docs/development/website.md index a92fef295..3c7ecc116 100644 --- a/docs/development/website.md +++ b/docs/development/website.md @@ -402,6 +402,7 @@ Scope is **per version**, not per site (Q558): |---|---| | `stable` and every numbered release | Operator docs only: `docs/design/`, `docs/operations/`, `docs/index.md`, `why-gag.md`, `features.md`, `roadmap.md` | | `dev` | The above **plus** the repo-internal docs: `docs/STATUS.md` (the [backlog](#the-backlog-page)), `docs/plan/`, `docs/development/` (this file included), `docs/assets/`'s READMEs | +| *no version* | `docs/releases/` โ€” see the trap below | A release is a frozen build, so a backlog published in one would be a snapshot stale from tag day. `dev` redeploys on every push to `main`, which is the only @@ -426,6 +427,14 @@ Two traps worth keeping in mind when editing that wiring: the list keeps that from surfacing as a build warning. Because it is never a site page, a doc that links to it must use the absolute `github.com` URL โ€” a relative `../README.md` fails `mkdocs build --strict`. +- **`docs/releases/` is excluded on every version too**, and it is the one entry + that is not a repo-internal-vs-operator split: those files are GitHub Release + bodies, authored for github.com's comment-flavour renderer. Their GFM alerts + (`> [!WARNING]`) have no MkDocs equivalent and would publish as literal text, + and their links already point into the versioned site. The exclusion is spelled + out in **four** places that must agree โ€” `mkdocs.yml`'s default, the two `env:` + blocks and the one `export` in `pages.yml`, and `scripts/docs/docs-preview.sh`. + Miss the last and `make docs-build` disagrees with CI. PR builds validate **both** scopes (`pages.yml`'s `build` job runs `mkdocs build` twice), so a PR that breaks a plan or development page fails there rather than on diff --git a/docs/operations/release.md b/docs/operations/release.md index caf01258c..6460a821d 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -618,6 +618,387 @@ clobbers curated notes. If you want richer notes (highlights, upgrade caveats), --draft --notes-file โ€ฆ` โ€” and the pipeline will leave your body untouched while still attaching the signed v2 CRD manifest asset. +#### Writing the curated notes + +A minor release accumulates more than a generated changelog can convey, and the +two inputs that feed it, `operator-caveats-since.sh` and the commit log, both +mislead in specific ways. What follows is the method, written after `v1.3.0`. + +**Author the notes in +[`docs/releases/`](https://github.com/actions-gateway/github-actions-gateway/tree/main/docs/releases), +not in a scratch file.** +One file per stable tag, `vX.Y.Z.md`, holding the release body verbatim โ€” no +front matter, and **no title heading** โ€” the Releases page already renders the tag +name as the page's `

`, so a `# vX.Y.Z` in the body duplicates it. Publish from +it: + +```bash +gh release edit vX.Y.Z --notes-file docs/releases/vX.Y.Z.md +``` + +`v1.3.0`'s notes were drafted under `tmp/` and edited straight on the Release. By +the time they were right they had been through a wrong count, a dead anchor, 46 +forced line breaks, two mismatched PR numbers, and a caveat that never said +"GHES" โ€” every one caught by hand, none by review, and none of it reviewable +because the text was not in a diff. In-repo makes each fix a diff and each +published body reproducible from a commit. + +These files are excluded from the docs site on **every** version, dev included: +they are written for github.com's renderer, which the site is not. The exclusion +is spelled out in four places that must agree โ€” `mkdocs.yml`, two `env:` blocks +and one `export` in [`pages.yml`](../../.github/workflows/pages.yml), and +`scripts/docs/docs-preview.sh`. + +**Past bodies are retrievable, so the previous release is your template:** + +```bash +gh release view vX.Y.Z --json body --jq .body +``` + +Use it to seed a new file when a tag predates this convention. + +**Notes answer "what is in it" and "what must I do". The docs answer "how" and +"why".** Every explanation that can live in `upgrade.md` or an `operations/` page +should, behind a link. `v1.3.0`'s first draft ran ~1000 words of prose; cutting it +to links lost nothing. Link a Highlight from its bold lead, and link group +headings rather than every line. + +That rule shortens the *prose*. It does not shorten the notes: `v1.3.0` shipped +2100 words and 25 links, because enumerations kept being added โ€” a feature list, a +fix list, the API surface, the condition reasons. Prose is what gets cut; lists are +what a reader actually searches. Fold the lists (below) and the length costs +nothing. + +##### The section skeleton + +`v1.3.0` arrived at this order after several passes. It is ordered by what a +reader needs first, not by what took the most work: + +| Section | Answers | Notes | +|---|---|---| +| *(one-line tagline)* | what this project is | for the reader who arrived from a search result | +| *(danger banner)* | is anything here going to hurt me | a GFM alert, above the fold; see below | +| **Highlights** | why upgrade | 3โ€“5, each linked from its bold lead | +| **Upgrading** | what must I do | numbered steps; say which are guarded | +| **Deprecations** | what is going away | see below | +| **Everything since ``** | did my bug get fixed | folded lists with counts | +| **API and metric surface** | what changed in the contract | CRDs *and* metrics; see below | +| **Validation** | why should I believe you | receipts, not adjectives | +| **Project and tooling** | is this project healthy | contributor-facing; last for a reason | +| **Security** | is there anything I must patch for | state it even when the answer is no | +| **Verifying this release** | how do I check the artifacts | the `make verify-release` line | + +**Say something about security even when there is nothing to report.** Silence +reads as an omission to the one reader scanning specifically for it. State +plainly that no advisory accompanies the release, then list what it does carry โ€” +dependency security bumps, and any fix that hardens credential or trust handling +without patching a reported vulnerability. Name the scanning gates and when they +run. `v1.3.0` had no CVE and still warranted the section. + +**Lead with a danger banner, and make it an alert.** GFM alerts โ€” `> [!WARNING]`, +`> [!CAUTION]`, `> [!NOTE]` โ€” render as real coloured callouts in a release body +(verified; the render check below counts them). `v1.3.0` opens with a `[!WARNING]` +naming the two required upgrade steps and the asymmetric rollback, because those +are the only things that can hurt an operator who reads no further. Use `[!NOTE]` +for a scope caveat and `[!CAUTION]` for the one thing that is genuinely +destructive. Three alerts is a lot; more and none of them read as urgent. + +**Write a Deprecations section even when nothing is removed** โ€” saying so is the +point, since "deprecation" reads as "removal" to a skimmer. For each notice give +the removal version, the migration path, and **whether the apiserver actually +warns**. `v1.3.0` deprecated `v2alpha1` (warns on every apply) alongside +`v1alpha1`, which is removal-slated and emits **no** warning at all โ€” so nothing +reminds an operator it is going away. That asymmetry is exactly what a reader +cannot discover for themselves. + +**Diff every surface an operator can see, not just the CRDs.** Each of these is +enumerable, and each hides in a different file, so a review that reads only the +Go diff misses most of them. `v1.3.0` shipped five: CRD fields, metric names, +Kubernetes Event reasons, condition reasons, and configuration (chart values, +env tunables, CLI flags). Diff each between the two tags mechanically rather than +reading the changelog for them โ€” the Event reasons and the metrics had no +enumeration at all until they were diffed, and the notes had already been through +several reviews. + +Two traps. **A rename reads as a removal** when the extraction is scoped to one +directory: env vars first appeared to have 17 removals, all of which were code +moving out of `cmd/`; re-running repo-wide showed zero. **Adjacent string +arguments read as the same thing**: `recordEvent(obj, type, reason, action, โ€ฆ)` +puts a reason and an action side by side, so `ProvisionWorker` and +`ApplyAGCAutoscaler` both survived extraction as reasons until each call site was +checked. Always report "none removed" when it is true โ€” operators are looking for +exactly that. + +**Diff `docs/` as well, and link what is new from where it is actionable.** A new +operator page is the strongest signal of a capability the notes forgot, and a +heavily grown one shows where the release's real weight landed: + +```bash +git diff --name-status v..origin/main -- docs/operations/ | grep '^A' +git diff --stat v..origin/main -- docs/operations/ | sort -t'|' -k2 -rn | head +``` + +`v1.3.0` added three operator guides and grew `troubleshooting.md` by 36 +sections. Link a new guide from the bullet it serves rather than from a +documentation inventory โ€” `resourcequota-sizing.md` belongs on the quota-accounting +upgrade note, where an operator hits the problem it solves. A guide with no +feature to attach to goes in the contributor-facing section with one line on why +it exists. + +**Give the API surface its own section, and lead it with any new CRD.** A new kind +is not a field: the chart installs chart-root `crds/` on a *fresh install only*, +so a new CRD is the reason "apply the CRDs" is step 1 of Upgrading, and the two +must cross-reference. Then fold the rest โ€” new spec fields, new status fields, new +condition reasons โ€” grouped by kind, counted like any other fold. `v1.3.0` listed +28 new condition reasons this way. Until a generated API reference exists, say +which artifact is authoritative (`kubectl explain`, or the signed CRD asset). + +**Validation is receipts, not adjectives.** "Thoroughly tested" is worth nothing. +Link the run, quote the counts, and quote a value measured at the layer that +matters: `v1.3.0` cites 73/73 specs with a run link, and a derived `1500m` observed +on the pod where the templates asked for 2 and 3 CPU. Ship the receipt wherever a +claim is made, not only in this section โ€” a feature line that links its own PR is a +receipt too. + +**Check the validation story against the plan doc, not memory.** This is the +section a sceptical reader checks first, so a wrong detail here costs more than +anywhere else. `v1.3.0`'s draft claimed no candidate had ever cleared the gate and +that rc.5 was the first to return a verdict; the plan doc records rc.4 passing the +day before. The true version was better anyway โ€” five candidates, three aborted, +rc.4 passed without catching a live worker pod, rc.5 caught one โ€” and it is +checkable, which the flattering version was not. + +**Keep a contributor-facing section, and put it last.** Release, CI, docs-site, and +tooling work ships in no image and no chart, so it does not belong in the change +lists. It still belongs in the notes: it is what a reader evaluating the project's +health is looking for. Fold it, label it as not user-facing, and let it sit below +everything an operator needs. + +**The caveats script reports headings that *changed*, not headings that are +*new*.** A section edited in this window is listed exactly like one added in it. +Test each before repeating it: + +```bash +git show :docs/operations/upgrade.md | grep -qF "### " \ + && echo "pre-existing" || echo "new in this window" +``` + +`v1.3.0` listed two `BREAKING` headings and **neither** was a caveat for an +upgrading operator: `priorityTiers` was already in `v1.2.0` (a pre-1.0 change), +and `capacityGate.mode`'s removed values had only ever existed on `main`. +Repeating them unexamined would have sent operators after migrations they did not +need. + +**Promote by danger, not by label.** The most hazardous item in `v1.3.0` carried +no `BREAKING` heading at all: the PriorityClass allowlist CRD apply, which affects +every install and whose rollback re-arms a cluster-wide outage. Read for +consequence, not for keywords. + +**Distinguish "breaking" from "guarded migration".** If skipping a step stops the +upgrade with a message naming the fix, it is a required migration and saying +"breaking change" overstates it. If a wrong path fails silently, say so loudly. +State which of the two you mean. + +**Enumerate from the commit log, filtered to what ships.** Conventional Commit +subjects are already terse diagnoses, so they need only the prefix and Q-ID +removed: + +```bash +git log --format='%s' ..HEAD \ + | grep -E '^(feat|fix)\((agc|gmc|proxy|worker|api|scaleset|chart|broker|wrapper|admission|provisioner|metrics|scalesetlistener|migrate|observability)\)' +``` + +**That scope list is a guess, so reconcile what it dropped โ€” never just run it.** +An allow-list silently omits any scope nobody thought of, and the result still +looks like a complete list. Print the residue and read every scope in it: + +```bash +git log --format='%s' ..HEAD | grep -E '^(feat|fix)' > /tmp/all +# ...run the filter above into /tmp/kept, then: +grep -vxF -f /tmp/kept /tmp/all | sed -E 's/^((feat|fix)\([^)]*\)).*/\1/' | sort | uniq -c | sort -rn +``` + +`v1.3.0`'s first pass kept 57 of 132 and dropped **seven shipping fixes**: two +`fix(scalesetlistener)` (the pattern had `scaleset`, which does not match it), a +compound `fix(agc,gmc)`, three `fix(migrate)` for the shipped `gag-migrate` +binary, and one `fix(observability)`. Nothing in the output said so. The last +three scopes above were added only after that reconciliation โ€” assume the list is +still incomplete for the next release. + +Exclude the `ci`, `dogfood`, `test`, `docs`, and `build` scopes, which ship in no +image or chart. Say in the notes that you excluded them, so a reader does not read +hundreds of commits as the user-visible change count. Keep the trailing `(#NNNN)`: GitHub auto-links +a bare `#NNNN` in a release body, so every line becomes traceable for free. + +**Passing the scope filter does not make a commit shippable โ€” check the paths.** +The filter matches a scope string, and scopes are reused. `v1.3.0` listed +"Attribute usage rows to the machine that measured them" as a feature; the commit +is `feat(metrics)` and touches only `claude-usage/`, the Claude Code usage +tooling, because that module and the product's Prometheus metrics share a scope. +Assert the paths for every cited PR instead of reading the subjects: + +```bash +git show --stat --format='' --name-only \ + | grep -qE '^(api|broker|cmd|githubapp|scaleset|charts)/' || echo "NOT PRODUCT" +``` + +It was one entry in 64, and no amount of re-reading the list would have found it. + +**Cite the commit that did the work, not the one that filed it.** A Q-numbered +backlog row and its implementation have near-identical subjects, so a `docs(plan)` +commit reads exactly like the fix. `v1.3.0` cited #988 โ€” `docs(plan): file and +scope Q507` โ€” under the label of #1008, the gate itself; a reader clicking through +would have landed on a planning row. Resolve each number to its title before +shipping, and look for the same work cited twice under two numbers. + +**One fact per line, especially next to a procedure.** Distinct operator-facing +changes run together into a paragraph read as background, and a paragraph sitting +under a numbered list reads as a footnote to it. `v1.3.0`'s Upgrading section +closed with three unrelated changes โ€” quota accounting, a dropped proxy label, a +new apiserver warning โ€” in one sentence-run below its two numbered steps; as a +bulleted list with a bold lead each, the same words are scannable. Prose is for +framing. Anything a reader might need to act on individually gets its own line. + +**Fold long lists.** `
` renders on the Releases page and keeps +the top scannable. It is also the only lever against truncation: the Releases +*index* collapses a long body behind a **"read more"** link, and a fold counts as +its one summary line while collapsed. `v1.3.0` hit that limit and was folded back +under it โ€” eight folds. If the index is truncating, the fix is another fold, not +a cut. + +**Pick the next fold by measuring collapsed height, not by eye.** Sum each +section's bytes with `
โ€ฆ
` bodies excluded; the biggest sections +are rarely the ones that feel long. `v1.3.0` measured 9.5k collapsed, and the +third-largest section was Validation โ€” not an obvious candidate, since Upgrading +and Highlights are larger but must both stay open. + +**When the content being folded is evidence, put the evidence in the ``.** +A fold whose summary reads "Validation details" hides the receipt; one that reads +`73/73 e2e specs on Kata microVM workers, on live GKE โ€” the four legs, and what +none of them assert` *is* the receipt, and a reader who never expands it has +still seen the number. That is the exception to the count-in-the-summary +convention: enumerations carry a count, evidence carries the finding. + +**Count what you list โ€” and count the unit in the label.** State a count in a +`` and it will be wrong the moment you curate the list. `v1.3.0`'s draft +claimed 25 features and listed 23. Subtler: its "New spec fields (10)" had ten +*bullets* carrying thirteen *fields*, because three bullets grouped related ones +(`.minRequests` / `.maxRequests` / `.limitHeadroomPercent`). Every other fold +counted the noun in its own label; that one silently switched to bullets. Count +what the label says, then re-count after every edit โ€” mechanically, not by eye. + +**Caveat anything a validation run did not exercise.** The dogfood gate runs +against github.com, so it says nothing about GHES. A feature list that reads as +finished support overclaims. Check `docs/plan/archive/` for the feature's own +"what this will not verify" section before describing it. Then ask whether an +unexercised feature belongs in Highlights at all โ€” `v1.3.0` kept GHES there and +paid for it with a caveat in three places. + +**A caveat is a claim, so measure it before writing it.** Understating coverage +is as wrong as overstating it, and easier to do accidentally because it feels +safe. `v1.3.0`'s draft said the capacity gate had "unit and envtest coverage +only"; the repo has a live-cluster-autoscaler test for its matcher (#929) and 305 +lines of e2e proving a quota-blocked job redelivers (#1028). Before writing "only +tested at tier X", grep for the feature at every higher tier โ€” and if the true +statement is narrower than the tidy one, ship the narrow statement. What survived +here was "the release gate does not *assert* it", which is checkable. + +**A caveat must survive being read alone.** Every line in a folded list is read +out of its heading's context โ€” by search, by a linked anchor, by a skimmer. The +`v1.3.0` draft said "Untested against a real appliance" under a GHES heading, +which says nothing at all once the heading scrolls away; it shipped as "Untested +against a real **GHES** appliance". Name the subject inside the caveat, and repeat +the caveat at each place the feature is claimed rather than relying on proximity. + +**Link the versioned docs site, not `main` and not `blob/`.** A reader of these +notes should land on that release's instructions, and the site publishes a build +per stable tag: `https://actions-gateway.com/X.Y.Z/operations/โ€ฆ`. Mind the form โ€” +the site drops the leading `v` exactly like the chart does. `v1.3.0` shipped 18 +such links. They 404 until the docs deploy for the tag completes, which is +expected while the Release is still a draft. + +**Verify every link and anchor.** Nothing checks them for you: `make doc-links` +skips external URLs by design, and every link in a notes file is absolute. Anchors +are the usual failure, and the built site is the authoritative oracle โ€” read the +ids MkDocs actually emitted rather than re-deriving a slug by hand, which gets +punctuation, backticks, and parenthesised clauses wrong: + +```bash +make docs-build +grep -oE 'id="[^"]*"' site/operations/upgrade/index.html | sed 's/id="//;s/"//' +``` + +Better, resolve every link in the file against `site/` in one pass. Whatever you +write, **include a URL you know is broken** and confirm it is reported: a checker +that silently resolves nothing looks exactly like a clean file. `v1.3.0`'s notes +were verified this way โ€” 20 site URLs resolved, plus one deliberately bogus +anchor that the same run flagged. + +**Do not hard-wrap.** GitHub renders a release body with comment-flavour GFM, +where a single newline becomes `
`. Keep every paragraph, blockquote, and list +continuation on one line. `v1.3.0`'s hard-wrapped draft rendered 46 of them. + +Check this against the **renderer**, not the source. `gh release view --json body` +returns the raw Markdown, which never contains `
` however badly it is wrapped, +so grepping that is a check that cannot fail. Render it the way GitHub will: + +```bash +gh api -X POST /markdown -f mode=gfm -f "text=$(cat docs/releases/vX.Y.Z.md)" \ + | grep -c '
' +``` + +`mode=gfm` is the comment flavour; `mode=markdown` is not, and reports 0 on a +hard-wrapped file. The same render confirms the rest of the GitHub-only markup +survived โ€” expect one `markdown-alert-*` class per `> [!โ€ฆ]` block, one `
` +per fold, and no literal `[!` anywhere. + +**In-page anchors do not work in a release body.** Release-body headings carry no +`id`, so `[Upgrading](#upgrading)` is a dead link. Refer to a section by name in +bold instead. Verify on a published release rather than trusting this โ€” the page +does emit ids, but only on GitHub's own chrome: + +```bash +curl -sS https://github.com///releases/tag/ \ + | grep -oE ']*>' | grep -c 'id=' +``` + +**So a table of contents is not available, and should not be faked.** An unlinked +list of section names is dead weight that costs collapsed height against the +truncation limit while navigating nothing. The folds already serve that role: a +collapsed `
` is a labelled one-line entry, so a body with ten folds +reads as an outline whether or not the reader expands any of them. Navigation +comes from section order and the danger banner, not from a ToC. (The in-repo copy +under `docs/releases/` does get GitHub's auto-generated file outline for free, +which is a second reason not to hand-roll one.) + +**Watch the chart-version form.** Images are tagged `vX.Y.Z`, charts `X.Y.Z`. +A copy-pasteable `helm` command with a `v` in it fails. + +**Run the `deslop` skill over the draft before publishing.** Release notes are the +most-read prose the project ships. + +##### Before publishing: the mechanical checks + +Every rule above that can be checked by a machine, in the order they are cheapest +to run. None of these is a substitute for reading the notes โ€” but each one caught +a defect in `v1.3.0` that several careful readings had not. + +| Check | How | What it catches | +|---|---|---| +| Fold counts | recount every `` against its bullets, counting the noun the label names | a count that drifted during curation | +| Enumerations | reconcile each surface fold against the tag-to-tag diff **both ways** | a name listed that no longer exists, or shipped and never listed | +| Citations resolve | `gh pr view --json title` for each `#NNNN` | a planning commit cited as the implementation | +| Citations ship | `git show --stat --name-only ` against product paths | a non-product commit listed as a feature | +| Anchors | resolve every site URL against a built `site/`, **with one planted bad anchor** | a heading that moved, and a checker that silently resolves nothing | +| Rendering | `gh api -X POST /markdown -f mode=gfm` | hard-wrapped `
`s, alerts that did not render, literal `[!` | +| Published body | re-fetch and `diff` against the file | an edit made on the Release and not in the repo | + +Two habits make the difference. **Plant a known failure in anything that reports +"all clear"** โ€” a checker with a broken query and a clean file produce identical +output. And **report the negative when it is true**: "16 metrics added, none +removed" and "13 spec fields, nothing removed" are what an operator is actually +scanning for, and neither is worth stating unless it was measured. + ### 6. Chart version & metadata The `chart-publish` job sets the published chart's `version` and `appVersion` to diff --git a/docs/releases/README.md b/docs/releases/README.md new file mode 100644 index 000000000..f4f970527 --- /dev/null +++ b/docs/releases/README.md @@ -0,0 +1,53 @@ +# Release notes + +One file per stable tag, `vX.Y.Z.md`, holding that release's **GitHub Release +body verbatim** โ€” no front matter, no heading above the first line. + +**No title heading**, specifically. The Releases page renders the tag name as the +page's own `

`, so a `# vX.Y.Z` in the body is a duplicate title one line under +the real one. The file therefore opens on its first real sentence, and its +identity comes from the filename โ€” which is also why stripping a heading at +publish time is not the answer: the published body has to stay byte-identical to +this file, or the drift these files exist to prevent comes back. + +The file is the authoring source; the published body is a copy of it: + +```bash +gh release edit vX.Y.Z --notes-file docs/releases/vX.Y.Z.md +``` + +Curating a release body is a `publish.yml` step with its own method and its own +traps (what to promote, what the caveats script actually reports, why hard-wrapping +breaks the rendering). All of it lives in +[operations/release.md ยง Writing the curated notes](../operations/release.md#writing-the-curated-notes). + +| Release | Notes | +|---|---| +| v1.3.0 | [v1.3.0.md](v1.3.0.md) | + +Releases before `v1.3.0` predate this convention; their bodies live only on the +Releases page. Retrieve one with `gh release view vX.Y.Z --json body --jq .body`. + +## Not published to the docs site + +These files are written for github.com's **comment-flavour GFM** renderer, which +the MkDocs site is not. Two differences matter: + +- GFM alerts (`> [!WARNING]`, `> [!CAUTION]`, `> [!NOTE]`) render on github.com and + publish as literal text under MkDocs. +- A single newline inside a paragraph becomes `
` in a release body, so these + files are deliberately **not hard-wrapped**. Do not reflow them. Verify against + the renderer (`gh api -X POST /markdown -f mode=gfm โ€ฆ`), never against the raw + Markdown โ€” the source has no `
` in it either way. + +`docs/releases/` is therefore excluded from every site version, `dev` included. +The exclusion is spelled out in four places that must agree โ€” `mkdocs.yml`, two +`env:` blocks and one `export` in `.github/workflows/pages.yml`, and +`scripts/docs/docs-preview.sh`. Because the directory is never a site page, a doc +that links here must use the absolute `github.com` URL; a relative link fails +`mkdocs build --strict`. + +Their own links point into the **versioned docs site** +(`https://actions-gateway.com/X.Y.Z/โ€ฆ`, no leading `v`), so an operator reading +the notes for a release gets that release's instructions. `make doc-links` skips +external URLs by design, so those links are not checked โ€” verify them by hand. diff --git a/docs/releases/v1.3.0.md b/docs/releases/v1.3.0.md new file mode 100644 index 000000000..748d11a8e --- /dev/null +++ b/docs/releases/v1.3.0.md @@ -0,0 +1,455 @@ +Zero-idle self-hosted GitHub Actions runners for multi-tenant Kubernetes. + +> [!WARNING] +> **Two required upgrade steps, and one rollback that is not symmetric.** Read **Upgrading** below before deploying. Rolling back past the PriorityClass allowlist migration re-arms a cluster-wide apiserver outage, and upgrading correctly does not protect you from it. + +## Highlights + +**[Measured worker right-sizing](https://actions-gateway.com/1.3.0/operations/worker-rightsizing/).** The AGC sizes workers from what jobs actually use, not what the template asks for. Two opt-in profiles: `NodeShare` divides a node's allocatable capacity, `Throughput` derives from observed usage. Validated on GKE: a worker took a derived `1500m` where its templates asked for 2 and 3 CPU. + +**[A capacity gate that stops claiming jobs the cluster cannot place](https://actions-gateway.com/1.3.0/operations/troubleshooting/#runnerset-reports-workercapacitydeclined-the-gateway-stopped-claiming-jobs).** A `RunnerSet` refuses work it has nowhere to run instead of acquiring a job and stranding it. Whether the cluster can grow is now a gateway-level fact, not something each tenant asserts. Opt-in via `spec.capacityGate.mode`, default `Off`. + +**[Automatic re-run for interrupted workers](https://actions-gateway.com/1.3.0/operations/troubleshooting/#which-disruptions-auto-re-run-a-job-and-which-never-do).** Evicted, preempted, drained, and hand-deleted workers get their job re-run instead of vanishing. The eviction counter's new `cause` label tells the four apart. + +**[GitHub Enterprise Server gateways address their own appliance](https://actions-gateway.com/1.3.0/operations/troubleshooting/#a-ghes-tenants-traffic-never-reaches-the-appliance).** They never did: the GMC set no `GITHUB_API_BASE_URL`, so token exchange fell back to `api.github.com` and a GHES tenant failed before acquiring any job. A private CA in front of the appliance is now trusted via `spec.githubCABundleRef`. + +> [!NOTE] +> **Untested against a real GHES appliance.** The flow and the plumbing have test coverage. No GHES host has ever run it. + +## Upgrading + +Two steps, both guarded: skip either and the upgrade stops with a message naming the fix. Full procedure in [the upgrade guide](https://actions-gateway.com/1.3.0/operations/upgrade/), which also documents rollback per component: [GMC](https://actions-gateway.com/1.3.0/operations/upgrade/#gmc-rollback), [AGC](https://actions-gateway.com/1.3.0/operations/upgrade/#agc-rollback), [proxy](https://actions-gateway.com/1.3.0/operations/upgrade/#proxy-rollback), and [worker image](https://actions-gateway.com/1.3.0/operations/upgrade/#worker-image-rollback). + +**1. Apply the chart's CRDs.** Part of every upgrade of this chart, not a one-off: Helm installs the chart-root `crds/` directory on a fresh install only. + +```bash +helm show crds oci://ghcr.io/actions-gateway/charts/actions-gateway \ + --version 1.3.0 | kubectl apply -f - +``` + +Charts carry no leading `v`. Images are tagged `v1.3.0`, charts `1.3.0`. + +**2. If you set `priorityClassAllowlist.configMapName`, move those names into `allowedPriorityClasses`.** The allowlist is now a cluster-scoped `PriorityClassAllowlist` CR. It **fails closed**, so names you do not carry across start being rejected. + +> [!CAUTION] +> **Rolling back past step 2 re-arms a cluster-wide outage, and upgrading correctly does not protect you from it.** The allowlist moved off a `ConfigMap` because a ValidatingAdmissionPolicy `paramKind` on a core type stays broken for the life of a kube-apiserver process once its bindings go empty, which `helm uninstall` does. Read [Rolling back past this change](https://actions-gateway.com/1.3.0/operations/upgrade/#rolling-back-past-this-change-re-arms-the-outage-it-fixes) before reverting. + +**Three changes that need no action, but will be visible.** Each links its own section of the upgrade guide. + +- **[Worker quota accounting now counts native sidecars, `RuntimeClass` overhead, and storage](https://actions-gateway.com/1.3.0/operations/upgrade/#worker-quota-accounting-now-counts-native-sidecars-runtimeclass-overhead-and-storage)** โ€” a tight `ResourceQuota` may admit fewer workers than it did (#883, #887). New this release: [Sizing the platform-owned `ResourceQuota`](https://actions-gateway.com/1.3.0/operations/resourcequota-sizing/) turns runner shapes and concurrency ceilings into the numbers, so the first install is a calculation rather than a guess. +- **[An `EgressProxy` pool drops the `app: actions-gateway-proxy` label](https://actions-gateway.com/1.3.0/operations/upgrade/#non-breaking-an-egressproxy-pools-pods-drop-the-app-actions-gateway-proxy-label-its-pool-is-recreated-once)** โ€” the pool is recreated once on upgrade (#1117). +- **[`v2alpha1` now warns on every apply](https://actions-gateway.com/1.3.0/operations/upgrade/#non-breaking-v2alpha1-is-deprecated-and-the-apiserver-now-warns)** โ€” deprecated, not removed; see **Deprecations** below (#828). + +## Deprecations + +**Nothing is removed in this release.** Both notices below point at `v2.0.0`. + +**`v2alpha1` is deprecated**, and as of this release the apiserver warns on every apply. It stays served until `v2.0.0`, which removes it. Moving to `v2beta1` is a re-apply rather than a migration tool: [Before you upgrade past v2.0.0](https://actions-gateway.com/1.3.0/operations/v1alpha1-deprecation/#before-you-upgrade-past-v200). + +**`v1alpha1` and Classic are removed at `v2.0.0` too.** Tenants still on `v1alpha1` migrate with `gag-migrate`: [Migrating a tenant to v2](https://actions-gateway.com/1.3.0/operations/migration-v1-to-v2/#why-upgrade-to-v2). Unlike `v2alpha1`, `v1alpha1` emits **no** apiserver warning today, so nothing will remind you it is going away. + +## Everything since v1.2.0 + +Over 450 commits. These 63 are the ones that ship in the product; build, CI, test, and docs work is left out. + +
+New features (22) + +**[Worker right-sizing](https://actions-gateway.com/1.3.0/operations/worker-rightsizing/)** + +- Per-`RunnerSet` worker usage sampling (#741) +- Measured sizing recommendations in `RunnerSet` status (#744) +- Opt-in sizing profiles applied at pod-build time (#746) +- Report a `Throughput` profile cancelled by an injected CPU limit (#956) + +**[Capacity-aware job intake](https://actions-gateway.com/1.3.0/operations/troubleshooting/#runnerset-reports-workercapacitydeclined-the-gateway-stopped-claiming-jobs)** + +- Opt-in capacity gate: refuse jobs the cluster cannot place (#902) +- Gate intake on the node autoscaler's own declination (#913) +- Refuse to claim jobs when namespace quota has no headroom (#793) +- Pre-claim quota rung ported to the scale-set tier (#868) + +**[Worker recovery](https://actions-gateway.com/1.3.0/operations/troubleshooting/#which-disruptions-auto-re-run-a-job-and-which-never-do)** + +- Recover preempted workers off `PreemptionByScheduler` (#972) +- Recover drained workers on the deletion mark (#1011) +- Eviction recovery ported to the scale-set tier (#851) +- Bound a worker orphaned while the AGC is down with a lifetime cap (#914) + +**[GitHub Enterprise Server](https://actions-gateway.com/1.3.0/operations/troubleshooting/#a-ghes-tenants-traffic-never-reaches-the-appliance)** (untested against a real appliance) + +- Trust a private CA in front of a GHES appliance (`spec.githubCABundleRef`) (#1137) + +**[Autoscaling and observability](https://actions-gateway.com/1.3.0/operations/observability-metrics/)** + +- Bring-your-own proxy autoscaler via `EgressProxy` `managedAutoscaling` (#758) +- Managed VPA opt-in for the GMC and per-gateway AGC (#761) +- Export `AGCAutoscalingUnavailable` as a Prometheus gauge (#763) +- Export `GitHubEgressIncomplete` as a gauge for fleet alerting (#1038) +- Count scale-set poll errors in the shared counter (#878) +- Alias `runner_set` onto `worker_pods_reaped_total` so scale-set series join (#1020) + +**[API](https://actions-gateway.com/1.3.0/operations/v1alpha1-deprecation/)** + +- Mark `v2alpha1` deprecated so the apiserver warns (#828) +- Rename `sizingRecommendation.windowStart` to `windowStartTime` (breaking, pre-GA) (#943) +- Exempt deletion-only updates from PriorityClass allowlist re-validation (#1017) + +
+ +
+Fixes (41) + +**Scale-set listener** + +- Deregister scale-set runner records on reap, and sweep unclaimed ones (#1052) +- Re-offer a scale-set job whose runner name will not register (#1048) +- Measure and fix the AGC restart queue replay (#1138) +- Flush the listener's pending message deletes on shutdown (#1148) +- Stop stranding a worker pod on a reclaimed job Secret (#1119) +- Reclaim the per-job JIT-config Secret (#727) +- Latch the capacity gate past the reap with a one-probe slot (#1000) +- One autoscaler loop's two verdicts must not gate intake (#949) +- Give up on assignments GitHub no longer holds (#1108) +- Defer ceiling-blocked jobs instead of spinning on redelivery (#1105) + +**Shutdown and lifecycle** + +- Delete broker sessions reliably on `SIGTERM` (#739) +- Forward `SIGTERM` to `Runner.Worker` so cancelled jobs report (#747) +- Arm the worker termination relay before starting the child (#872) +- Drain in-flight `CONNECT` tunnels on proxy shutdown (#748) +- Linger for endpoint removal inside the drain budget (#755) +- Give a Running scale-set worker a reap deadline (#824) +- Delete the worker pod of a job the listener abandons (#1032) +- Reap tenant worker pods when a v2 gateway is deleted (#1140) + +**Quota accounting** + +- Count native sidecars and `RuntimeClass` overhead in the worker footprint (#883) +- Count the storage keys in the worker footprint (#887) + +**Naming and identity** + +- Keep a truncated worker pod name valid and unique (#921) +- Bound the derived `RunnerGroup` name via one shared helper (#922) +- Keep v1 and v2 agent pools apart during coexistence (#915) +- Key a v2 proxy pool solely on its own identity label (#1117) +- A gateway-scoped AGC no longer reconciles v1 `RunnerGroups` (#1036) + +**GitHub Enterprise Server** (untested against a real appliance) + +- Make GHES gateways address their own appliance (#1034) +- Address `GITHUB_API_BASE_URL` for the re-run call (#973) +- Retry an evicted job's re-run until GitHub accepts it (#1010) + +**Admission and API** + +- Move the PriorityClass VAP `paramKind` off a core type (#958) +- Reject a `nodeShare.allocatable` declaring neither cpu nor memory (#946) +- Ratchet the v2beta1 single-`runnerLabel` rule onto the field (#847) +- Generate the missing `ClusterCapacity` deepcopy (#928) +- Dedupe apiserver deprecation warnings to one log line per process (#1021) + +**Tenant migration (`gag-migrate`)** + +- Dual-read the privileged grant across both label domains (#911) +- Retry a transiently unreachable webhook in `gag-migrate --apply` (#904) +- Make a Docker-in-Docker tenant actually migratable (#856) + +**GMC and proxy** + +- Let a rollout restart of a managed Deployment take effect (#1104) +- Derive the AGC `NO_PROXY` apiserver exemption from the cluster (#912) +- Own every namespaced v1 child so a stripped finalizer cannot leak them (#848) +- Fail AGC startup on an unreadable proxy CA cert (#1133) +- Read classic-tier run identity from the `github` context (#967) + +
+ +## API and metric surface + +**One new CRD: [`PriorityClassAllowlist`](https://actions-gateway.com/1.3.0/operations/upgrade/#priorityclass-allowlist-configmap-to-cr)**, cluster-scoped, the only kind added since v1.2.0. It is also the only CRD the chart ships in its chart-root `crds/` directory, which Helm installs on a fresh install and **skips on every upgrade**. That is why applying CRDs is step 1 of Upgrading above, and why the upgrade hard-fails without it. + +**Thirteen spec fields and two status fields are added, and nothing is removed** โ€” verified against the CRD schemas at both tags, not assembled by hand. The rest are new fields, metrics, and settings this tag publishes for the first time. There is no generated API reference yet, so the CRDs themselves are the authority: `kubectl explain`, or the signed `actions-gateway-crds-v2.yaml` attached below. + +**Sixteen metrics are added and none removed**, so existing queries keep working. The shipped Grafana dashboards were rebuilt around them and the `PrometheusRule` gains an `ActionsGatewayGitHubEgressIncomplete` alert โ€” both in [`deploy/monitoring/`](https://github.com/actions-gateway/github-actions-gateway/tree/v1.3.0/deploy/monitoring), described in [observability-dashboards](https://actions-gateway.com/1.3.0/operations/observability-dashboards/). + +
+New spec fields (13) + +`RunnerSet` + +- `spec.sizing.profile`: `Static`, `Binpack`, `Throughput`, or `NodeShare` (default `Static`) +- `spec.sizing.nodeShare.allocatable` / `.workersPerNode`: the NodeShare envelope +- `spec.sizing.minRequests` / `.maxRequests` / `.limitHeadroomPercent`: bounds on what a profile may derive +- `spec.capacityGate.mode`: `Off` or `Observe` (default `Off`) +- `spec.maxWorkerLifetime`: cap on a worker pod's lifetime + +`ActionsGateway` + +- `spec.clusterCapacity.nodeAutoscaling`: `Present` or `Absent` (default `Present`) +- `spec.agcAutoscaling.mode`: managed VPA for the AGC, `Off`, `Initial`, or `Recreate` +- `spec.githubCABundleRef.name`: ConfigMap holding a private CA for a GHES appliance + +`EgressProxy` + +- `spec.managedAutoscaling`: bring-your-own proxy autoscaler opt-in (default `true`) + +[`PriorityClassAllowlist`](https://actions-gateway.com/1.3.0/operations/security-operations/#priority-classes-the-allowed-priority-classes-allowlist) (new cluster-scoped kind) + +- `spec.allowedPriorityClasses`: the platform allowlist, replacing the watched ConfigMap + +
+ +
+New status fields (2) + +`RunnerSet.status` + +- `sizingProfileState`: `Active` or `AwaitingSamples` +- `sizingRecommendation[]`: per-container `observedPeak`, `observedP95`, `requests`, `limits`, `sampleCount`, `windowStartTime` + +
+ +
+New configuration: 4 chart values, 3 tunables, 1 renamed flag + +Nothing here is required โ€” every item defaults to the previous behaviour. Listed because none of it is discoverable from the CRDs. + +**Chart values** โ€” the GMC's own managed VPA, off by default + +- `vpa.enabled` (default `false`) +- `vpa.updateMode` (default `"Off"`) +- `vpa.minAllowed` / `vpa.maxAllowed` (default `{}`) + +**Environment tunables** โ€” documented with their defaults in [troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/) and [observability-metrics](https://actions-gateway.com/1.3.0/operations/observability-metrics/) + +- `PROXY_SHUTDOWN_DRAIN_TIMEOUT` (default `45s`) โ€” the proxy's whole shutdown budget, covering endpoint linger then in-flight `CONNECT` drain +- `PROXY_SHUTDOWN_LINGER` (default `10s`, `-1s` to skip) โ€” how long to wait for endpoint removal before draining +- `AGC_QUOTA_ADMISSION` (default on) โ€” set `false` to turn the namespace-quota rung off AGC-wide + +**Renamed flag** โ€” only affects an install that passes GMC arguments directly; the chart handles it + +- `--priority-class-allowlist-configmap` โ†’ `--priority-class-allowlist-name` (#958) + +
+ +
+New metrics (16) + +All prefixed `actions_gateway_`, listed here without it. Their remedies and alerting guidance are in [observability-metrics](https://actions-gateway.com/1.3.0/operations/observability-metrics/). + +**Worker usage sampling** + +- `worker_usage_cpu_peak_cores` +- `worker_usage_memory_peak_bytes` +- `worker_usage_job_cpu_peak_cores` +- `worker_usage_job_memory_peak_bytes` +- `worker_usage_jobs_sampled_total` +- `worker_usage_jobs_unsampled_total` +- `worker_usage_poll_errors_total` + +**Scale-set capacity and intake** + +- `scaleset_advertised_capacity` +- `scaleset_capacity_withheld` +- `scaleset_jobs_deferred` +- `scaleset_jobs_abandoned_total` + +**Worker recovery** + +- `eviction_rerun_failures_total` +- `eviction_recovery_identity_unknown_total` + +**Gateway health** + +- `agc_autoscaling_unavailable` +- `github_egress_incomplete` +- `broker_session_leaks_total` + +
+ +
+New Kubernetes Event reasons (8) + +A surface separate from the condition reasons below โ€” these land in `kubectl get events`, and six have their own section in [troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/). None was removed. + +- `WorkerPodLifetimeExceeded` โ€” the lifetime cap killed a worker +- `WorkerPodOrphanedRunning` โ€” a Running worker was reaped +- `WorkerPodCompletedPending` โ€” a worker was reaped while Pending after its job finished +- `WorkerPodsReapedOnGatewayTeardown` โ€” workers cleared as a gateway was deleted +- `WorkerDrainTimeout` โ€” a drain did not finish in budget +- `WaitingForWorkerDrain` โ€” deletion is blocked pending drain +- `AssignmentAbandoned` โ€” the listener gave up an assignment +- `WorkerCapacityDeclined` โ€” also a condition reason, listed below + +
+ +
+New condition reasons (28) + +Most are documented with their remedies in [observability-metrics](https://actions-gateway.com/1.3.0/operations/observability-metrics/) and [troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/). Five are not yet: `SizingWithinRange`, `SizingDriftDetected`, `InsufficientSamples`, `JobsProvisioning`, and `GitHubEgressAllowed`. Until the operator docs catch up, their godoc in `api/apiconditions/conditions.go` is the reference. + +**Sizing** + +- `SizingProfileActive` +- `SizingProfileOverridden` +- `SizingWithinRange` +- `SizingDrift` +- `SizingDriftDetected` +- `InsufficientSamples` +- `CPULimitInjected` +- `NoCPULimitInjected` + +**Capacity and job intake** + +- `CapacityAvailable` +- `WorkerCapacityDeclined` +- `WorkerCeilingReached` +- `ScaleUpDeclined` +- `GateModeUnsupported` +- `AwaitingProbe` +- `AwaitingWorkerPods` +- `JobsProvisioning` +- `JobProvisionStalled` +- `RunnerNameConflict` + +**AGC autoscaling** + +- `AGCAutoscalingActive` +- `AGCAutoscalingDisabled` +- `AGCAutoscalingUnavailable` +- `VPACRDNotInstalled` + +**GHES, egress, and CA trust** + +- `GitHubEgressAllowed` +- `GitHubEgressIncomplete` +- `ApplianceRangesRequired` +- `CABundleNotFound` +- `CABundleInvalid` + +**Lifecycle** + +- `GatewayTerminating` + +
+ +## Validation + +**This is the first release whose tag is gated by a passing end-to-end run on real infrastructure, rather than advised by one.** The gate predates `v1.2.0`. Getting a verdict out of it took five candidates: rc.1 through rc.3 aborted before finishing, rc.4 passed but never caught a live worker pod, and `v1.3.0-rc.5` passed with the pod-level check rc.4 missed. rc.5 needed its own verdict regardless โ€” it carries product changes past rc.4, so rc.4's pass does not transfer. + +
+73/73 e2e specs on Kata microVM workers, on live GKE โ€” the four legs, and what none of them assert + +Run against the signed release artifacts rather than a build ([run](https://github.com/actions-gateway/github-actions-gateway/actions/runs/30786972228)): + +- The full e2e matrix on gateway-provisioned runners, **73/73 specs**, on Kata microVM-isolated workers +- Both sizing profiles actuating on real worker pods, with the derived value confirmed at the pod: `1500m` where the templates asked for 2 and 3 CPU +- The signed v2 CRD artifact downloaded, signature-verified, applied, and all five CRDs registered +- Teardown back to zero nodes + +That is what the gate itself asserts. It does not assert the capacity gate, the namespace-quota rung, or anything GHES โ€” the run targets github.com, and it builds an autoscaling cluster with headroom rather than a constrained one. Those rest on the unit, envtest, and e2e suites instead โ€” several of which grew this cycle to reach a live cluster, listed under **Testing and internal quality** below. + +
+ +**Every PR now upgrades from the last released chart, not just HEAD to HEAD** (#1008). It immediately caught what became step 1 above: #958 shipped a CRD into chart-root `crds/`, breaking every v1.2.0 upgrade while CI stayed green. + +This tag adds no product code beyond that candidate. + +**Full changelog:** [`v1.2.0...v1.3.0`](https://github.com/actions-gateway/github-actions-gateway/compare/v1.2.0...v1.3.0) + +## Project and tooling + +**Documentation is now versioned.** Since this release cycle the site publishes a build per stable tag, so these notes link `1.3.0` pages rather than whatever `main` says today, and an operator on an older release reads that release's docs. Older versions were seeded, so `1.0.0` through `1.3.0` are all live. + +**Three operator guides are new**, and one grew by a third. [Worker right-sizing](https://actions-gateway.com/1.3.0/operations/worker-rightsizing/) and [Sizing the platform-owned `ResourceQuota`](https://actions-gateway.com/1.3.0/operations/resourcequota-sizing/) are linked from their features above. [Node shutdown budgets across platforms](https://actions-gateway.com/1.3.0/operations/node-shutdown-budgets/) is reference material with no feature of its own: `terminationGracePeriodSeconds` is a request, not a guarantee, and it collects what each platform actually grants โ€” the proxy asks for 60s and gets 15 on a GKE Spot node. [Troubleshooting](https://actions-gateway.com/1.3.0/operations/troubleshooting/) gained **36 sections**, most of them the remedy for a condition or event reason listed above. + +The rest is contributor-facing, listed because a release is also a claim about how the project is built. + +
+Release, docs-site, CI, and tooling (25) + +**Documentation site** + +- Versioned docs via mike โ€” stable default, dev opt-in (#768) +- Publish docs from stable release tags (#766) +- Derive the site announce bar from the git tags (#775) +- Gate the release on the docs-site announce bar naming the tag (#801) +- Publish the backlog on the dev docs site, filterable (#1059) +- Gate site-side link resolution, fix 13 site-only 404s (#1062) + +**Release process** + +- Release delta report + scope-ledger convention (#1058) +- Surface operator caveats in the pre-flight, don't rely on memory (#992) +- A status file and a sentinel for the release gate (#1161) +- Relay e2e spec progress into the release-validation gate (#1159) +- Make the release gate able to fail on a dead sizing profile (#867) + +**Test and CI speed** + +- Show live spec progress and render the JUnit report (#1152) +- Report unit-run progress from the `go test -json` stream (#1162) +- Compile the e2e images' dependency tree once, and cache it across runs (#870) +- Overlap the e2e runner disk cleanup with job setup (#871) +- Cut the suite's serial tail (#874) +- Measure coverage in one workspace-wide invocation (#952) +- Run workspace unit tests as one multi-module invocation (#722) +- Change-scope local golangci-lint to the modules a diff affects (#724) + +**Repository tooling** + +- Allocate Q-IDs from a ref claim, not a counter line (#823) +- Merge driver resolving STATUS.md Queue rows by ID (#879) +- Repo-wide leftover conflict-marker gate (#732) +- Auto-rebase stranded Dependabot Go PRs (#827) +- Parse workflow YAML with a real parser, in a new devtools/ module (#1114) +- Add nolintlint + a ratcheted funlen gate (#771) + +
+ +
+Testing and internal quality (13) + +Coverage that reaches real infrastructure rather than a fake, and the consolidation work behind it. 52 test and 15 refactor commits landed in this window; these are the ones that changed what the suites can observe. + +**Tests that drive a live cluster** + +- Assert the capacity gate's autoscaler matcher against a live cluster-autoscaler (#929) +- Extend the live-autoscaler drift gate to Karpenter (#1024) +- Gate day-2 `helm upgrade` over a live release (#935) +- Validate the v1โ†’v2 migration live on GKE (#909) +- Prove a quota-blocked job redelivers to a sibling gateway (#1028) +- Scale-set disruption recovery under the chart's real RBAC, and the ordering fix it caught (#1018) +- Measure the node-drain path on both tiers (#895) +- Execute the cosign download path and `verify-release` itself (#1145) + +**Consolidation** + +- Code-restraint sweep: trim narrative comments, dedup helpers, drop dead weight (#1013) +- Merge the twin worker-quota evaluators into one shared core (#1022) +- Decompose the GMC `main()` and AGC `run()` god functions (#765) +- Unify the broker/scale-set error taxonomy in `githubapp/httpx` (#794) +- Share the version-neutral v2 API code, gate the identical remainder (#760) + +
+ +## Security + +**No security advisory accompanies this release**, and no fix here is embargoed or backported. What it carries is dependency maintenance: a security bump of gRPC to v1.82.1 and, in the build-tools module, `x/net` to v0.56.0 (#749), plus six routine dependency-group updates. + +Three fixes below harden handling of credentials and trust material rather than patching a reported vulnerability: an AGC that now fails startup on an unreadable proxy CA cert (#1133), and two that stop a per-job JIT-config `Secret` being stranded or reused (#727, #1119). + +Every pull request โ€” with no path filter, deliberately โ€” runs `govulncheck` across the workspace and a Trivy scan of the images, and `main` re-runs both on every push. + +## Verifying this release + +We sign every image and both charts. + +```bash +make verify-release VERSION=v1.3.0 +``` + +The signed v2 CRD manifest and `SHA256SUMS` ship as release assets with detached cosign bundles, and the publish workflow attests build provenance. diff --git a/mkdocs.yml b/mkdocs.yml index 3dfca76b7..9af7d478a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -19,8 +19,14 @@ copyright: Licensed under the Apache License 2.0. # a backlog in one would be a snapshot stale from tag day. Only the dev path in # .github/workflows/pages.yml sets it, and it keeps /README.md excluded on every # version: MkDocs drops it anyway as a conflict with the index.md landing page. +# +# /releases/ is excluded on EVERY version including dev, the one path here that is +# not repo-internal-vs-operator: those files are GitHub Release bodies, authored +# for github.com's renderer. Their GFM alerts (`> [!WARNING]`) have no MkDocs +# equivalent and would publish as literal text, and their links already point at +# the versioned site. The dev override in pages.yml carries this entry too. # See website.md ยง Publication scope. -exclude_docs: !ENV [MKDOCS_EXCLUDE_DOCS, "/README.md\n/STATUS.md\n/plan/\n/development/\n/assets/README.md\n/assets/fonts/README.md\n"] +exclude_docs: !ENV [MKDOCS_EXCLUDE_DOCS, "/README.md\n/STATUS.md\n/plan/\n/development/\n/releases/\n/assets/README.md\n/assets/fonts/README.md\n"] # Published but deliberately absent from the nav, which stays the operator-facing # table of contents on every version: the repo-internal tree the `dev` version diff --git a/scripts/docs/docs-preview.sh b/scripts/docs/docs-preview.sh index 0f1a77d88..af2d7cf3b 100755 --- a/scripts/docs/docs-preview.sh +++ b/scripts/docs/docs-preview.sh @@ -77,7 +77,9 @@ main() { # gate: the release scope excludes docs/plan/ and docs/development/, so a # break in those pages only ever surfaces in the dev scope. "${venv_dir}/bin/mkdocs" build --strict - MKDOCS_EXCLUDE_DOCS=/README.md \ + # Must match the dev override in pages.yml: /releases/ holds GitHub Release + # bodies, excluded from every version including dev (mkdocs.yml says why). + MKDOCS_EXCLUDE_DOCS=$'/README.md\n/releases/\n' \ "${venv_dir}/bin/mkdocs" build --strict --site-dir site-dev }