Add trusted-publisher + secrets pre-flight health check for the release pipeline by schickling-assistant · Pull Request #1291 · livestorejs/livestore

schickling-assistant · 2026-06-03T07:21:23Z

Problem

The release publish job touches five external surfaces (npm, Netlify, Cloudflare, Mixedbread, GitHub OIDC) and roughly half a dozen tools on `PATH`. Today the first time we discover any of them have rotted is during an actual release — see #1284 / #1285 for the `PATH` regression class.

Solution

Weekly pre-flight workflow (Mon 08:00 Europe/Berlin) that exercises every external token, OIDC binding, and tooling path the publish job depends on — without ever publishing. The workflow also runs on any PR that touches `.github/workflows/release.yml*` so the next regression of that class trips this check, not the actual release.

Probes:

`gh secret list` — required (the release job's non-npm surfaces): `NETLIFY_AUTH_TOKEN`, `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, `MXBAI_API_KEY`, `MXBAI_VECTOR_STORE_ID`. `NPM_TOKEN` is intentionally not probed — npm publishing uses tokenless OIDC trusted publishing (`release.yml` rejects `NPM_TOKEN`/`NODE_AUTH_TOKEN`).
`gh api repos/.../actions/oidc/customization/sub` — npm trusted-publisher OIDC readiness (GitHub OIDC endpoint reachable; 200 or 404 both OK).
`npm view @livestore/livestore versions --json` — npm registry read.
HEAD `https://api.netlify.com/api/v1/sites/livestore-docs.netlify.app\` with `NETLIFY_AUTH_TOKEN` — Netlify read.
HEAD `https://api.cloudflare.com/client/v4/accounts/\${CLOUDFLARE_ACCOUNT_ID}\` with `CLOUDFLARE_API_TOKEN` — Cloudflare read.
HEAD `https://api.mixedbread.com/v1/vector_stores/\${MXBAI_VECTOR_STORE_ID}\` with `MXBAI_API_KEY` — Mixedbread read.
`which pnpm node bun gh jq` — PATH sanity.

Failures aggregate into one structured report; the workflow opens (or warms) a `bug`-labelled `release pre-flight regressed` issue. On all-green the report is uploaded as the `release-preflight-state` artifact (30d retention).

The script is reusable: pass `--json` to consume the structured report from `validate-publish-substance.yml` (in flight).

Trade-offs / follow-ups

All API probes use the existing release-job tokens. Each token is already read-capable on its respective endpoint (Netlify lists sites, Cloudflare describes accounts, Mixedbread reads vector stores), so no new write-scoped credentials are introduced — least-privilege is preserved.
The OIDC trusted-publisher claim is currently only validated as endpoint-reachable. Validating the actual `aud` / `sub` against the npm-side config requires the workflow to mint an `id-token`; the workflow already declares `id-token: write` so a follow-up commit can implement the full handshake without touching the workflow permissions.

Validation

`bun scripts/src/commands/health/release-preflight.ts --dry-run` runs locally: 4 ok / 0 fail / 3 skip — PATH sanity, required non-npm secrets present, npm registry read, and npm trusted-publisher OIDC readiness all green; Netlify/Cloudflare/Mixedbread skipped because no env tokens locally (the workflow exports them from secrets).
`devenv tasks run lint:check` and oxlint with the `overeng` plugin both pass.
`devenv tasks run genie:run --mode before --no-tui` regenerates the workflow cleanly.

Files

`.github/workflows/health-release-preflight.yml.genie.ts` — workflow source.
`scripts/src/commands/health/release-preflight.ts` — probes + structured report + issue management. Exposes `runPreflight()` for reuse.
`CHANGELOG.md` — entry under `### Internal Changes > Development Tooling`.

🤖 Generated with Claude Code

Posted on behalf of @schickling

field	value
`agent_name`	🎈 cl1-swell
`agent_session_id`	5a1e2097-a640-400a-a2b8-be55050a58b4
`agent_tool`	Claude Code
`agent_tool_version`	2.1.202
`agent_runtime`	Claude Code 2.1.202
`agent_model`	claude-opus-4-8
`runtime_profile`	/nix/store/393xxjd29yx4b41rsjaglp3r47q9k7s5-coding-agent-runtime-profile/share/coding-agents/profile.json
`skills_manifest`	/nix/store/xhnp6r3s5n46kpcq182d56ib1dia1179-agent-skills-corpus/share/agent-skills/manifest.json
`worktree`	agent-ad02f75ddd0f605bc/finalize-1291
`machine`	dev3
`tooling_profile`	dotfiles@ab9375c

Weekly workflow (and on every release-workflow PR) exercises every external token, OIDC binding, and DNS path the publish job depends on without ever publishing: - npm / Netlify / Cloudflare / Mixedbread API read probes, - presence of `NPM_TOKEN`, `NETLIFY_AUTH_TOKEN`, `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, `MXBAI_API_KEY`, `MXBAI_VECTOR_STORE_ID`, - PATH sanity for `pnpm`/`node`/`bun`/`gh`/`jq`, - GitHub OIDC endpoint reachability. Failures aggregate into one structured report. On any failure the workflow opens (or updates) a `bug`-labelled issue listing every probe that regressed. On all-green the report is uploaded as an artifact so later runs can see when the pipeline was last verified end-to-end. The script also accepts `--json` so other release-validation workflows can consume the structured report directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> agent-session-id: c1934b02-14a7-4f20-a3e4-9e7b987c77ab agent-tool: Claude Code agent-tool-version: 2.1.145 agent-model: claude-opus-4-7 tooling-profile: dotfiles@unknown-dirty

The release publish path uses npm trusted publishing via OIDC (`id-token: write`) and explicitly rejects token auth — `release.yml` fails any publish job where `NPM_TOKEN`/`NODE_AUTH_TOKEN` is set. So the pre-flight requiring an `NPM_TOKEN` secret probed a credential the pipeline forbids. - Drop `NPM_TOKEN` from `REQUIRED_SECRETS`; the probe now checks only the release job's non-npm surfaces (Netlify / Cloudflare / Mixedbread). - Reframe the OIDC endpoint check as the npm trusted-publisher readiness probe (`npm trusted-publisher OIDC readiness`): same reachability mechanism, now the explicit stand-in for npm publish readiness. No workflow-permission or genie-output change: the generated workflow already declared `id-token: write` and never passed `NPM_TOKEN`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-27T09:17:15Z

PR preview

Subject	Status	Report	Details	Updated
LiveStore docs preview	success	Docs preview deployed (livestore-docs-dev)	PR aliases: pr-1291-8aa4ee7, pr-1291	2026-07-27 09:10 UTC
web-linearlite	success	web-linearlite deployed (preview)	Worker: example-web-linearlite-preview.livestore.workers.dev	2026-07-27 08:53 UTC
web-todomvc	success	web-todomvc deployed (preview)	Worker: example-web-todomvc-preview.livestore.workers.dev	2026-07-27 08:53 UTC
web-todomvc-script	success	web-todomvc-script deployed (preview)	Worker: example-web-todomvc-script-preview.livestore.workers.dev	2026-07-27 08:53 UTC
web-todomvc-sync-cf	success	web-todomvc-sync-cf deployed (preview)	Worker: example-web-todomvc-sync-cf-preview.livestore.workers.dev	2026-07-27 08:53 UTC

Report history

PR 1291 · 2026-07-27 09:16 UTC

Subject	Status	Report	Details	Updated
LiveStore docs preview	success	Docs preview deployed (livestore-docs-dev)	PR aliases: pr-1291-8aa4ee7, pr-1291	2026-07-27 09:10 UTC
web-linearlite	success	web-linearlite deployed (preview)	Worker: example-web-linearlite-preview.livestore.workers.dev	2026-07-27 08:53 UTC
web-todomvc	success	web-todomvc deployed (preview)	Worker: example-web-todomvc-preview.livestore.workers.dev	2026-07-27 08:53 UTC
web-todomvc-script	success	web-todomvc-script deployed (preview)	Worker: example-web-todomvc-script-preview.livestore.workers.dev	2026-07-27 08:53 UTC
web-todomvc-sync-cf	success	web-todomvc-sync-cf deployed (preview)	Worker: example-web-todomvc-sync-cf-preview.livestore.workers.dev	2026-07-27 08:53 UTC

schickling-assistant · 2026-07-27T10:07:30Z

Closing in favour of #1505, which keeps this PR's genuinely useful half — credential liveness — and drops the parts that could not work as intended.

What #1505 keeps

Tokens expire on wall-clock time, so a PR-triggered check can never prove they are still valid. That is a real gap and a scheduled probe is the right answer. #1505 probes Netlify, Cloudflare and Mixedbread weekly, using endpoints that report token status where available.

Why this branch is superseded rather than landed

Both cited motivating regressions would have passed it green.
- Run mxbai search sync via devenv task (pnpm not on PATH) #1285 was pnpm: command not found, because pnpm exists only inside the devenv shell. This workflow runs pnpm/action-setup@v4 and then checks which pnpm — it installs the very tool whose absence was the bug.
- Use MXBAI_VECTOR_STORE_ID (not _PROD suffix) in publish-release #1284 was release.yml referencing MXBAI_VECTOR_STORE_ID_PROD when the secret is MXBAI_VECTOR_STORE_ID. REQUIRED_SECRETS hardcodes the correct name, which does exist, and never cross-references what release.yml actually reads.
The root cause is environment divergence: release.yml's publish path runs through Nix + pinned devenv, while this workflow runs on the stock runner image. Any environment probe here describes a machine the release job never runs on — and that is inherent to maintaining a second workflow that approximates the first, not a fixable probe list. Declare release credential surfaces once and probe them weekly #1505 therefore probes only credentials, where there is no environment to reproduce, and derives the secret names from shared generator data so the two cannot diverge.
on: pull_request combined with ref: main. On a PR event this checks out main, so a PR changing release.yml is validated against main's code and the PR's own changes are never examined — the opposite of the stated intent.
gh secret list proves a name exists, not that the credential works. An expired NETLIFY_AUTH_TOKEN still appears in the list. This is the one probe class that justifies the whole workflow, and it was the weakest. Cloudflare's /client/v4/user/tokens/verify reports actual token status; Declare release credential surfaces once and probe them weekly #1505 uses it as the primary probe rather than a fallback.
The OIDC probe was near-tautological — both 200 and 404 counted as healthy, so it could essentially only fail if GitHub's API were down. It proved nothing about npm's trusted-publisher binding.
id-token: write was declared but unused, for a future handshake. Granting a powerful permission ahead of the work that needs it inverts least privilege.

Minor: HTTP status was parsed by substring-matching a header dump (firstLine.includes('200')), and probe stderr was interpolated unredacted into a public issue body.

One thing this PR got right that I initially misread: the reasoning for not probing NPM_TOKEN is correct against today's main — publishing uses OIDC trusted publishing and release.yml asserts no npm token is present. #1505 carries that conclusion over.

Posted on behalf of @schickling

field	value
`agent_name`	cl1-lagoon
`agent_session_id`	4f0de924-6d8e-4779-9a7d-c30c8fa3d2a4
`agent_tool`	Claude Code
`agent_tool_version`	2.1.220
`agent_runtime`	Claude Code 2.1.220
`agent_model`	claude-opus-5
`runtime_profile`	/nix/store/57hz15fbcg9c3hasm0ny3axd5pg33rh3-coding-agent-runtime-profile/share/coding-agents/profile.json
`skills_manifest`	/nix/store/d9hvhhg1k9n3j20dgcch7sz3sg98l377-agent-skills-corpus/share/agent-skills/manifest.json
`worktree`	schickling
`machine`	dev3
`tooling_profile`	dotfiles@b785b9f

schickling-assistant and others added 2 commits July 27, 2026 10:07

schickling-assistant force-pushed the schickling-assistant/2026-06-03-health-release-preflight branch from 95124d5 to cd37349 Compare July 27, 2026 08:40

schickling-assistant mentioned this pull request Jul 27, 2026

Declare release credential surfaces once and probe them weekly #1505

Open

schickling-assistant closed this Jul 27, 2026

Navigation Menu

Uh oh!

Uh oh!

Add trusted-publisher + secrets pre-flight health check for the release pipeline - #1291

Add trusted-publisher + secrets pre-flight health check for the release pipeline#1291
schickling-assistant wants to merge 2 commits into
mainfrom
schickling-assistant/2026-06-03-health-release-preflight

schickling-assistant commented Jun 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 27, 2026

PR 1291 · 2026-07-27 09:16 UTC

Uh oh!

schickling-assistant commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

schickling-assistant commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Trade-offs / follow-ups

Validation

Files

Uh oh!

github-actions Bot commented Jul 27, 2026

PR preview

PR 1291 · 2026-07-27 09:16 UTC

Uh oh!

schickling-assistant commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

schickling-assistant commented Jun 3, 2026 •

edited

Loading