Skip to content

fix(release): gate the publish on CI instead of re-running it, and widen the wait#356

Merged
anconina merged 1 commit into
mainfrom
fix/release-pipeline-gate-and-race
Jul 26, 2026
Merged

fix(release): gate the publish on CI instead of re-running it, and widen the wait#356
anconina merged 1 commit into
mainfrom
fix/release-pipeline-gate-and-race

Conversation

@anconina

Copy link
Copy Markdown
Contributor

Description

Two structural defects made every release a slow coin flip. v1.0.56 hit both.

1. The publish path re-ran CI's entire gate suite, serially. npm-publish.yml called pnpm validate:full before publishing — the same suite ci.yml had already run for that exact commit, sharded across ~7 parallel jobs when it landed on main. Measured on v1.0.56: that one step took 48m57s of a 51-minute publish. It is precisely the pipeline ci.yml's own header describes escaping — "was a single ~40-min serial job".

Replaced with the invariant that actually matters: poll ci.yml's run for the tagged commit and refuse to publish unless it concluded success. This is strictly stronger than before, because validate:full never checked whether CI had passed at all — it just re-ran a subset by hand. Then only publish-specific work remains: build:clean (prepack needs a dist, and a release must not inherit stale output) and smoke:tarball (packs and inspects the real artifact — the check that catches bundling regressions in what users install).

2. The workflow waiting on that publish could not outlast it. dockerhub-release.yml's github-release job polled on a hard-coded 120 × 15s = 30-minute budget. A ~50-minute publish cannot fit, so it timed out 2m38s early and failed:

Waiting for npm publish workflow (in_progress, attempt 120/120)
Timed out waiting for npm-publish.yml for v1.0.56 at ef86b7a0d...

v1.0.55 cleared the identical wait with 18 seconds to spare — the gate was passing by accident. The window also floats: the job starts only after the image builds, so faster builds make it expire earlier. Budget is now stated once as ATTEMPTS + INTERVAL_SECONDS, raised to 60 minutes, and the timeout message reports the budget it exhausted so "slow" is distinguishable from "hung".

3. Also: the publish job had no timeout-minutes, inheriting GitHub's 6-hour default — one earlier release run sat 2h26m before failing. Now bounded at 90.

Net effect: the publish drops from ~51 minutes to roughly the CI wait plus ~5 minutes of build-and-pack, and the Docker Hub race disappears.

Related Issue

N/A: release-pipeline CI configuration. No runtime behavior, security boundary, or public API changes — the defects live entirely in .github/workflows/ and were observed live on the v1.0.56 release.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Refactor

Checklist

  • Targeted checks for the changed area pass
  • Full repository validation passes (pnpm validate) — ran via the pre-push hook
  • New or changed behavior has a test that demonstrated the RED state before the production change
  • Documentation updated (if applicable) — CLAUDE.md release section
  • No secrets or credentials committed
  • Security implications considered — the new gate only adds a precondition (CI must be green) and needs actions: read to read workflow runs
  • The change is focused on one concern

RED Test Proof

test/architecture/release-pipeline-gate.test.ts fails on the pre-patch workflows for all three properties. Captured by restoring HEAD's workflow files and re-running:

 × does not re-run CI's gate suite on the publish path 3ms
 × refuses to publish a commit whose CI did not pass 1ms
 × budgets the publish wait longer than a publish can take 1ms

⎯⎯⎯⎯⎯⎯⎯ Failed Tests 3 ⎯⎯⎯⎯⎯⎯⎯
  "runs `pnpm validate:full` — duplicates ci.yml serially and stretches the
   publish past the budget dockerhub-release.yml waits on"
  "does not look up ci.yml's run for the tagged commit — nothing proves the
   commit was validated"
  "missing `actions: read` permission, so the CI lookup cannot read workflow runs"
  "no job timeout-minutes — a wedged step burns the 6-hour GitHub default
   (one run sat 2h26m)"
  "wait budget is not stated as ATTEMPTS + INTERVAL_SECONDS — a hard-coded loop
   bound drifts from the message that prints it"

 Tests  3 failed (3)

All 3 pass on this branch.

Evidence and Residual Risk

  • Contribution path: not applicable — release tooling. The defects were observed on the live v1.0.56 release, not inferred.
  • Evidence level: deterministic test plus a controlled harness. Beyond the architecture guard, both new shell blocks were extracted from the YAML and exercised against a stubbed gh: CI-success → exit 0; CI-failure → prints Refusing to publish a commit whose CI did not pass and exit 1; exhausted budget → bounded loop, timeout message, exit 1. Both files parse as YAML and both blocks pass bash -n. Budget arithmetic verified: publish waits 45 min for CI, Docker Hub waits 60 min for the publish.
  • Tested profile: macOS local (pnpm validate green via pre-push); workflow logic exercised against a stubbed GitHub CLI rather than a live tag.
  • Residual risk: the only true end-to-end proof is the next tag push — no PR exercises a tag-triggered workflow, which is exactly why both defects survived. Specific things to watch on the next release: (a) ci.yml must actually produce a run for the tagged commit, otherwise the new gate waits its full 45 minutes and fails — if a tag is ever cut on a commit that never landed on main via a push, this blocks rather than silently publishing, which is the intended direction but is a behavior change; (b) the publish now depends on the GitHub Actions API being readable, a new failure mode that did not exist before; (c) the 60-minute Docker Hub budget is sized for a publish that no longer waits ~49 minutes — if the CI wait ever dominates, revisit both numbers together. Unrelated and still open from the v1.0.56 clean-room install: comis init accepting an under-length gateway token into a systemd restart loop, comis doctor's PID-file false negative under systemd, and init --quick relocating the documented log path.

🤖 Generated with Claude Code

…den the wait

Two structural defects turned every release into a slow coin flip. v1.0.56 hit
both.

`npm-publish.yml` ran `pnpm validate:full` before publishing. That re-executed
the entire gate suite SERIALLY in one job — the same suite `ci.yml` had already
run for that exact commit, sharded across ~7 parallel jobs when it landed on
main. The step took 48m57s of a 51-minute publish and proved nothing new. It is
the pipeline ci.yml's own header describes escaping: "was a single ~40-min
serial job".

Replace it with the invariant that actually matters: poll `ci.yml`'s run for the
tagged commit and refuse to publish unless it concluded success. That is
strictly stronger than before, because `validate:full` never checked whether CI
had passed at all — it just re-ran a subset by hand. Then do only the work
specific to publishing: `build:clean` (prepack needs a dist, and a release must
not inherit stale output) and `smoke:tarball` (packs and inspects the real
artifact, the check that catches bundling regressions in what users install).

`dockerhub-release.yml`'s `github-release` job waits for that publish on a
fixed 120x15s = 30-minute budget. A ~50-minute publish cannot fit, so it timed
out 2m38s early and failed the workflow. v1.0.55 had cleared the identical wait
with 18 SECONDS to spare — the gate was passing by accident. The window also
floats: the job starts only after the image builds, so faster builds make it
expire earlier. State the budget once as ATTEMPTS + INTERVAL_SECONDS, raise it
to 60 minutes, and have the timeout message report the budget it exhausted so
"slow" is distinguishable from "hung".

Also bound the publish job with timeout-minutes. It had none, so it inherited
GitHub's 6-hour default; one earlier release run sat for 2h26m before failing.

Guarded by test/architecture/release-pipeline-gate.test.ts, which fails on the
pre-patch workflows for all three properties. Workflow files are not built,
linted, or executed by `pnpm validate`, and the race only manifests on a tag
push that no PR exercises — so nothing else could have caught either defect.

Fixes the stale CLAUDE.md claim that no workflow creates the GitHub release:
dockerhub-release.yml's github-release job does.
@anconina
anconina merged commit 516a444 into main Jul 26, 2026
14 checks passed
@anconina
anconina deleted the fix/release-pipeline-gate-and-race branch July 26, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant