Skip to content

Release confidence gate - #42

Merged
KTSCode merged 4 commits into
mainfrom
release_confidence_gate
Aug 14, 2026
Merged

Release confidence gate#42
KTSCode merged 4 commits into
mainfrom
release_confidence_gate

Conversation

@KTSCode

@KTSCode KTSCode commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Release confidence gate

Goal: after mix tag --minor, the release works — no tag rollbacks, no re-releases. v0.20.0 failed twice after the tag was pushed:

  1. CI build failed on the tag — Burrito 1.6.0 hard-checks Zig 0.16.0; release.yml was pinned to 0.15.2. Caught now by a workflow_dispatch dry-run of release.yml inside the mix tag gate.
  2. The re-released binary was broken at runtime — Burrito 1.6.0 (feat: Zig 0.16.0 compatibility burrito-elixir/burrito#225) split the launcher argv, so Elixir consumed --daemon as a script path and halted with No file named --daemon. Caught now by bin/checks/release_smoke.sh, which runs in full mix quality (pre-push) and against the CI artifact at tag time.

Changes

  • bin/checks/release_smoke.sh — builds (or accepts) a packaged binary; asserts argv passthrough (--version) and daemon boot (EPMD registration + sev status); teardown in a trap on every exit path; skips (exit 2) when the developer's daemon is running or the local toolchain can't build the release.
  • .quality.exsrelease_smoke custom writer stage; quick profile that skips it (plus dialyzer/coverage).
  • Hooks inverted — pre-commit drops to mix quality --profile quick; new pre-push runs full mix quality. Partly reverts 56232b7: dialyzer still blocks the push, just not every commit. .claude/settings.json commit gate matches, and now uses $CLAUDE_PROJECT_DIR instead of a hardcoded path.
  • Burrito downgraded to 1.5.0 (what v0.19.0 shipped with) + CI Zig re-pinned to 0.15.2. The two pins must move together — Burrito's Zig check is exact equality. Upstream fix for the argv bug is Stop Elixir's CLI consuming app arguments and halting the VM burrito-elixir/burrito#230 (unmerged); switch back when it lands.
  • mix tag gatecheck_release_preconditions! now dispatches release.yml, watches the run, downloads the artifact, and smokes the exact bytes CI built. Any failure (including missing/unauthenticated gh, or a skipped smoke) refuses to tag. Dispatch runs upload a workflow artifact and can never publish a release.

Red validation (smoke script vs the shipped v0.20.0 binary)

$ ./bin/checks/release_smoke.sh ~/bin/sev
release_smoke: smoking /Users/kylesanclemente/bin/sev
release_smoke: booting daemon with SEVERANCE_SHUTDOWN_TIME=17:57
{"summary":"release daemon failed to boot: never registered with EPMD","stats":{"finding_count":1},"findings":[{"file":"mix.exs","message":"sev --daemon output: No file named --daemon"}]}
exit=1

No stray process after the run; a registered severance EPMD node makes the script skip (exit 2) instead of racing the real daemon.

Green validation (CI artifact on Burrito 1.5.0)

Dispatched run 31752527839 on this branch, downloaded the burrito_out artifact, smoked sev_macos_arm64:

$ ./bin/checks/release_smoke.sh .../ci_artifact/sev_macos_arm64
release_smoke: smoking .../ci_artifact/sev_macos_arm64
release_smoke: booting daemon with SEVERANCE_SHUTDOWN_TIME=18:08
{"summary":"release binary ok: argv passthrough, daemon boot, EPMD registration, status","stats":{"finding_count":0},"findings":[]}
exit=0

No stray process, no lingering EPMD registration after the run.

Known local ceiling

MIX_ENV=prod mix release sev still fails on this machine (burrito-elixir/burrito#221: Zig 0.15.2 libc defs vs macOS 26 SDK — a build-host problem; ubuntu cross-compiles are unaffected). Locally the smoke stage reports a visible skip:

○ Release smoke: skipped (local toolchain cannot build the release (see burrito/Zig vs macOS SDK); CI builds are unaffected)

It goes green locally when burrito #230 lands.

The v0.20.0 binary shipped broken because nothing executes what CI
builds: mix test cannot reach the Zig launcher and release.yml never
runs its artifacts. bin/checks/release_smoke.sh builds (or accepts) a
packaged binary, asserts argv passthrough and daemon boot via EPMD,
and tears down on every exit path.

The script runs as an ExQuality writer stage in full mix quality,
now wired to a new pre-push hook. Commits drop to the quick profile,
which skips the smoke stage, dialyzer, and coverage. On a toolchain
that cannot build the release (Zig 0.15 vs macOS 26 SDK) the stage
skips visibly instead of failing.
Burrito 1.6.0 (PR #225) split the launcher's argv so Elixir's CLI
consumes --daemon as a script path and halts with "No file named
--daemon" — the v0.20.0 daemon crash-loop. 1.5.0 keeps the fused
argv BEAM never parses; v0.19.0 shipped on it and ran fine.

Pin CI Zig back to 0.15.2 to match: Burrito's Zig check is exact
equality, and the mismatched pair is what broke the v0.20.0 tag
build. Local mix release stays broken on macOS 26 (burrito #221,
a build-host SDK problem); ubuntu cross-compiles are unaffected.
The v0.20.0 tag was pushed twice against an unproven release: CI
failed on the tag once, and the rebuilt binary was broken at runtime.
Nothing local exercises the ubuntu build config, so prove the release
before the tag exists.

check_release_preconditions! now dispatches release.yml via
workflow_dispatch, watches the run, downloads the burrito_out
artifact, and runs bin/checks/release_smoke.sh against the exact
bytes CI built. Any failure — including a missing or unauthenticated
gh — refuses to tag. Dispatch runs upload a workflow artifact and are
barred from publishing a release; only refs/tags/ pushes hit
action-gh-release.
@KTSCode

KTSCode commented Aug 13, 2026

Copy link
Copy Markdown
Owner Author

0.20.0 rehearsal (verification step 10)

Throwaway branch with Zig re-pinned to 0.16.0 (mismatched with Burrito 1.5.0), dispatched via the same workflow_dispatch path the mix tag gate uses.

Run 31752758337: failed as expected, at MIX_ENV=prod mix release sev:

> Your Zig version does not match the one Burrito requires! We need `0.15.2`, you have: `0.16.0`
    (burrito 1.5.0) lib/burrito.ex:59: Burrito.check_zig_version/0

This is the exact v0.20.0 tag-build failure mode. The mix tag gate (gh run watch --exit-status) would refuse to tag on this result. Throwaway branch and CI run's remote branch ref deleted after capture.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 80e08f07dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread mix.exs Outdated
Comment thread mix.exs Outdated
Comment thread README.md
Codex flagged that release.yml's push:tags trigger rebuilt on
ubuntu-latest and published binaries mix tag never smoked (Burrito
macOS binaries can't run the smoke script in that job). Drop the
tag trigger; mix tag now publishes via gh release create using the
exact artifact it already downloaded and smoked.

Also filter gh run list by --commit/--event so dispatch polling
can't grab a concurrent run's result, and add the missing changelog
entry for the release confidence gate.
@KTSCode
KTSCode merged commit afc9b22 into main Aug 14, 2026
2 checks passed
@KTSCode
KTSCode deleted the release_confidence_gate branch August 14, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant