ci(conda): publish release archives to the prefix.dev channel - #616
Conversation
dcmcand
left a comment
dcmcand
left a comment
There was a problem hiding this comment.
Verified the load-bearing claims rather than taking them on trust: the cosign check reproduces exactly with the pinned v3.1.1 (control Verified OK, one-byte-tampered manifest rejected), the rendered digests match v0.14.0's real checksums.txt for all six assets, and I built linux-64 and cross-built win-64 on Linux - the test ran and reported the release version and commit on the first, skipped with the documented cross-compilation message on the second, and the package contains Library/bin/nic.exe. Verifying the signature before reading any digest is the right ordering and the comment explaining why earns its place.
Two blockers, both small: the build number needs to be a substituted slot, and one sentence in packaging.md is false for win-arm64. Four suggestions inline, none gating.
On the open question - provider tools do not belong in the package. conda-forge has never published opentofu for win-arm64, across every version including the current 1.12.6, so an unconditional run dependency trades a working install for a missing one on a platform you otherwise cover. NIC is also provider-agnostic at the binary level and local/existing never touch OpenTofu. The workspace layer already handles this better than a package dependency could: #617 derives the AWS starter's constraint from pkg/tofu.MinVersion/MaxVersionExclusive, which is a tighter pin and scoped to the provider that needs it. The gap that leaves is Azure, and that belongs with whatever adds the Azure starter.
| # of dying on the first already-published filename. It cannot detect | ||
| # different bytes under the same filename, which is why the build is | ||
| # reproducible from the signed release rather than from local state. | ||
| rattler-build upload prefix -c nebari-dev/nebari --skip-existing "$pkg" |
There was a problem hiding this comment.
--skip-existing here plus the hardcoded number: 0 in recipe.yaml.tmpl:18 means a republish after a recipe fix uploads nothing and exits green.
The filename is name-version-<varianthash>_<buildnumber>.conda, and the variant hash covers the variant configuration, not the recipe body. I built the same source three ways:
| recipe | filename |
|---|---|
| as merged | nebari-infrastructure-core-0.14.0-hb0f4dca_0.conda |
extra line in build.script |
nebari-infrastructure-core-0.14.0-hb0f4dca_0.conda |
requirements.run: [opentofu >=1.11.3,<2.0.0] added |
nebari-infrastructure-core-0.14.0-hb0f4dca_0.conda |
So fixing a recipe bug and re-dispatching for a published tag hits six filenames the channel already has, skips all six by design, and reports success. The channel keeps serving the broken package.
That third row is this PR's own open question: deciding to add a provider-tool dependency would require republishing existing versions, and the republish would no-op. docs/operations/packaging.md:122 also sends the operator down exactly this path as troubleshooting step 4.
The comment above is the specific thing to correct - the package bytes are a function of the release archive and the recipe, and the recipe is mutable on main, so reproducibility from the signed release covers only half of it.
Fix: make the build number a substituted slot (__BUILD_NUMBER__, default 0) so a recipe change bumps it, or take --force on a deliberate republish path.
Unaffected: a tag's first publish, and resuming a genuinely partial upload. Both work as advertised.
| [https://github.com/nebari-dev/nebari-infrastructure-core/issues/552](https://github.com/nebari-dev/nebari-infrastructure-core/issues/552)), | ||
| and conda-forge ships `opentofu` for all platforms NIC supports, so a pixi | ||
| workspace that pins both never has to phone home on first run. | ||
| set `NIC_TOFU_PATH` in an activation script. conda-forge ships `opentofu` for all |
There was a problem hiding this comment.
conda-forge does not ship opentofu for win-arm64. Its published subdirs are linux-64, linux-aarch64, linux-ppc64le, osx-64, osx-arm64, win-64 - across every version, including the current 1.12.6.
This PR ships a win-arm64 conda package (build-packages.sh:55, and nebari-infrastructure-core_0.14.0_windows_arm64.zip is in the release), so a packager following this guidance there gets an unsolvable workspace. The PR description already states the fact: "would also impose it on local users and make win-arm64 unsolvable, since conda-forge ships no opentofu there."
The reading that rescues the sentence is "platforms NIC supports" meaning the four in the starter templates rather than the six NIC publishes, but your own text does not use it that way. A clause naming win-arm64 as the exception fixes it.
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: ./packaging/conda/build-packages.sh "$TAG" dist/conda | ||
|
|
||
| - name: Upload to prefix.dev |
There was a problem hiding this comment.
Worth trying --generate-attestation (and --store-github-attestation) on the upload - rattler-build 0.74.0's upload prefix has both, and this job already declares id-token: write.
The provenance chain is airtight up to the channel boundary and stops there: you verify the release signature going in, but someone pulling nebari-infrastructure-core from prefix.dev has nothing to check. Given the argument for this whole approach is that the bytes a conda user runs are the bytes you signed, letting them verify that is the natural close.
I could not test these against your channel without the trusted-publisher setup, so this is worth trying on the first dispatch rather than adding blind.
| # The tag must name a real release. On the dispatch path no release | ||
| # event vouches for it, so this is also where stable-only is enforced | ||
| # - a `release: published` filter alone would let a dispatched rc through. | ||
| if ! prerelease="$(gh release view "$TAG" --repo "$GITHUB_REPOSITORY" --json isPrerelease -q .isPrerelease 2>/dev/null)"; then |
There was a problem hiding this comment.
2>/dev/null turns any gh failure - rate limit, token scope, outage - into no such release: $TAG.
The PR description notes GitHub's /releases endpoint returning empty for this repo for about a day. That is the exact scenario this line misdiagnoses, at the top of the workflow whose documented recovery is to re-run it. Capturing stderr and echoing it with the error costs two lines.
| # cutting a new release. | ||
| on: | ||
| release: | ||
| types: [published] |
There was a problem hiding this comment.
types: [released] would stop release candidates before the approval prompt rather than after it.
.goreleaser.yml sets draft: false and prerelease: auto, so RC tags land as prereleases. With published, an RC starts a run, waits on the release environment approval, and only then fails the tag check at line 77. GitHub's docs point at published as the type to use when you want both stable and prerelease, so released is the stable-only one.
The API check at lines 73-80 still has to stay - it is the only thing covering the dispatch path.
| resumes rather than failing on the first one. This is also how a release | ||
| published before the workflow existed gets backfilled. | ||
|
|
||
| This channel is a bridge. The intended home is prefix.dev's shared |
There was a problem hiding this comment.
The sentence this PR correctly removed from line 82 still stands in two other places:
docs/adr/0016-opentofu-runtime-version-policy.md:19- "NIC is distributed via the prefix.devgithub-releaseschannel", word for word.starters/templates/pixi.toml.tmpl:8- "nic resolves from the prefix.dev github-releases channel; see the distribution issue for how that channel is populated." This one ships inside every generated starter.
Not reopening the scoping call on the template's channels list - waiting until the channel is confirmed serving is sound, and #620 may make github-releases correct for real. But that comment is wrong today independent of the decision, and it is the line a starter user reads while wondering why pixi install will not resolve. Worth a tracking issue either way, since #620 is about moving off this channel rather than onto it.
nic is not installable from any conda channel today, so the starter workspaces cannot resolve the dependency they declare. The intended home is prefix.dev's shared github-releases channel, which needs no recipe, but onboarding onto it is an upstream pull request we do not control and it has been open without movement. Publish to our own channel in the meantime. Nothing is compiled here: the recipe unpacks the archive GoReleaser already published and copies the binary into the prefix, taking the sha256 from that release's cosign-signed checksums.txt and failing when a checksum is absent, so a conda user runs the bytes the release signed. One Linux runner emits all six subdirs. Only the smoke test is platform-bound, and --test native runs it for the build platform and skips it elsewhere. Windows needs the zip archive and Library/bin. Kept out of release.yml so it can backfill versions published before it existed and re-run a failed upload without cutting a release. Uploads via OIDC trusted publishing, so there is no long-lived token.
The pixi guidance claimed NIC was distributed via the shared github-releases channel. It is not, and was not when the sentence was written: onboarding there is an unmerged upstream change. Replace the claim with the channel we actually publish to, how packages get there, and what to check when a release does not show up. Records the stable-only posture and why a prerelease tag cannot be packaged unchanged, and keeps the github-releases channel documented as the intended destination so the bridge has a stated end.
Review found four defects in the first cut. The tag reached two run: blocks as a GitHub expression, which is substituted into the script text before bash parses it, so a quote in the value executed arbitrary commands in a job holding the publishing identity. It now travels through env and is quoted, and it is validated for shape before anything consumes it, because it still reaches gh, sed, asset names and generated YAML. A dispatch could name any ref, so the scripts this job runs came from whatever branch was dispatched while the approver saw only a tag string. Dispatches must now come from main, and the checkout is pinned to main: the scripts and the release being repackaged are independent, and only the latter comes from the tag. The docs claimed only stable releases are published while the trigger fired for prereleases. The tag is now checked against the release API and a prerelease is refused, which also covers the dispatch path a trigger filter would have missed. The provenance claim was not backed by anything: checksums.txt was fetched and trusted, though the release signs it. It is now cosign-verified against the release workflow's identity before a digest is read, so the claim describes what the code does. Also: a build that emitted nothing reported a green publish, uploads could not resume after a partial failure, and the job had no timeout or concurrency group.
da11afc to
4ffd75c
Compare
Review found that a corrected recipe could never reach the channel. The filename is name-version-<varianthash>_<number>, and the variant hash covers the variant configuration rather than the recipe body: adding a requirements block leaves the name identical, verified by building the same source twice. Uploads skip filenames the channel already has, so republishing a fixed recipe for a published version was skipped and reported green, leaving the broken package served. The build number is now a substituted slot with a dispatch input, and the comment claiming reproducibility from the signed release is corrected - the bytes depend on the recipe too, and the recipe is mutable. The release trigger moves from published to released, so a release candidate stops before the environment approval instead of after it. The API check stays: it is the only thing covering the dispatch path. gh's stderr is no longer discarded. A rate limit, a token-scope problem or an outage was being reported as "no such release" in the step whose documented recovery is to re-run this workflow. Also records why provider tools stay in the starter workspaces rather than the package: conda-forge publishes no opentofu for win-arm64, which NIC does ship a package for, so an unconditional run dependency would trade a working install for an unsolvable one.
4ffd75c to
cfe505b
Compare
|
Hey @dcmcand, thanks for actually building the three-way filename test. All six are addressed in Build number. You're right, and I reproduced it before changing anything: adding a
Attestation. Agreed on the gap, and I've left it out for now for the reason you gave: neither of us can test The two stale sentences. Split out as #622. On provider tools in the package: convinced, and I've recorded it in Still unverified is the Re-requesting review since the branch has moved twice. |
tylerpotts
left a comment
tylerpotts
left a comment
There was a problem hiding this comment.
Built this end to end locally before reading the review threads: build-packages.sh v0.14.0 with rattler-build 0.74.0 and cosign 3.x emits all six subdirs, Verified OK on the control, one-byte-tampered checksums.txt rejected with invalid signature, and a pixi add from the local channel runs nic version reporting 0.14.0 / c1aa182b, so the ldflags survive. The nebari-dev/nebari spelling you flagged as unverified is fine on both counts: prefix.dev documents <namespace>/<channel> as the canonical reference for a non-primary channel and https://prefix.dev/<namespace>/<channel> as the pixi URL, and @nebari-dev/nebari already exists (empty).
One blocker, on the trigger, inline below. Two small non-gating notes:
docs/operations/packaging.md:110andpackaging/conda/README.md:31say the workflow "runs on every published release"; it deliberately runs onreleased, so "every stable release" is the true sentence (the next paragraph inpackaging.mdalready says so).docs/operations/verifying-releases.md"Maintainer prerequisites" is where the one-timereleaseenvironment setup lives; registering the prefix.dev trusted publisher (repo +publish-conda.yml+ environment) belongs on that list, sincepackaging.mdcorrectly notes nothing in-repo fails when it is missing.
| # would start a run and sit on the environment approval before failing the | ||
| # tag check. The API check below still has to stay - it is the only thing | ||
| # covering the dispatch path. | ||
| types: [released] |
There was a problem hiding this comment.
With the way this repo actually cuts releases, this event fires about twenty minutes before the assets the build needs exist.
Releases here are created as published releases by a human, which pushes the tag and fires released immediately; release.yml then starts on the tag push and GoReleaser attaches checksums.txt and checksums.txt.sigstore.json at the end. From the API and run history:
v0.14.0 published_at=2026-08-25T13:48:10Z release.yml created=13:48:13Z finished=14:09:20Z
v0.13.0 published_at=2026-08-18T16:09:32Z release.yml created=16:09:34Z finished=16:32:06Z
GoReleaser's own fresh-release path would be fine (v2.18.0, the version the v0.14.0 run resolved, always creates a draft and undrafts after all uploads in internal/pipe/release/release.go doPublish). But createOrUpdateRelease in internal/client/github.go takes the other branch when the release already exists: it preserves the existing draft state and just updates it, so the release stays published through the whole upload window.
On that path this workflow starts, passes the isPrerelease == false check, and gh release download finds nothing. Worth knowing that gh release download --pattern A --pattern B exits 0 on a partial match (tested), so the failure surfaces at cosign verify-blob on a missing bundle rather than at the download. It does fail loudly and the dispatch recovery works, but the automatic path breaks for every release cut this way, and the release approval gate only hides it if the approver is slow.
Two small fixes, either works:
- Trigger on
workflow_run: workflows: [Release], types: [completed], guard ongithub.event.workflow_run.conclusion == 'success', take the tag fromgithub.event.workflow_run.head_branch. The API check stays for the dispatch path. - Keep this trigger and add a bounded wait in the tag step: poll
gh release view "$TAG" --json assets -q '.assets[].name'forchecksums.txt.sigstore.jsonfor a few minutes before proceeding.
Whichever lands should keep working if someone cuts a release by plain git push --tags, where GoReleaser's draft-then-publish ordering makes the current trigger correct.
| fi | ||
|
|
||
| recipe="${workdir}/recipe-${subdir}.yaml" | ||
| sed -e "s|__VERSION__|${VERSION}|g" \ |
There was a problem hiding this comment.
Suggestion, not blocking: starters.yml asserts no __[A-Z_]*__ survives rendering before shipping; a one-liner here (grep -q '__[A-Z_]*__' "$recipe" && { echo "unsubstituted token in $recipe" >&2; exit 1; }) would give a new template slot without a matching sed -e a clear failure instead of a bad URL or a literal __BIN_DIR__ directory.
The release event fires about twenty minutes before the assets this build reads exist. Releases here are usually created as published releases by hand, which pushes the tag and publishes at once, and Release then starts on the tag push and attaches checksums.txt and its signature at the end: v0.14.0 published at 13:48:10 and Release finished at 14:09:20; v0.13.0 published at 16:09:32 and finished at 16:32:06. On that path the old trigger passed the prerelease check and then found no assets to download, surfacing as a cosign failure on a missing bundle rather than as a missing download, since gh release download exits 0 on a partial pattern match. Keying off a successful Release run is correct on that path and on git push --tags, where GoReleaser creates a draft and undrafts only after every upload. The tag now comes from the triggering run, and the release API check stays: with no release-event type to filter on, it is the only thing keeping release candidates off the channel. Also adds the unsubstituted-token guard starters.yml already applies to rendered starters, so a new template slot without a matching substitution fails loudly instead of shipping a literal __TOKEN__ as a URL or a directory name. It ignores comments, because the template's own header explains the convention in prose. Docs: the workflow no longer runs on every published release, and registering the prefix.dev trusted publisher joins the one-time maintainer prerequisites, since nothing in the repository fails when it is missing.
|
Hey @tylerpotts, the timing data settles it. I checked the API myself and it matches exactly: v0.14.0 published at 13:48:10 with Release finishing 14:09:20, v0.13.0 published 16:09:32 finishing 16:32:06. The release exists as published about three seconds before the workflow that uploads its assets even starts, so the trigger was firing into an empty release on the path this repo actually uses. Went with Worth flagging what that shifts: with no release-event type to filter on, the API check is now the only thing keeping release candidates off the channel, on both paths. It was already covering the dispatch path, so it is the same check doing more work rather than a new one, but it is load-bearing in a way it was not before. Good catch on Both non-gating notes taken. Also took the token guard suggestion. It caught a false positive on its first run, which was worth having: the template's own header explains the Thanks for building it end to end before reading the threads, and for confirming the channel spelling. That was the last thing on the PR I could not check myself. |
tylerpotts
left a comment
tylerpotts
left a comment
There was a problem hiding this comment.
Re-verified at beee1b7: clean full build via pixi exec (rattler-build 0.74.0, cosign 3.x) still gives Verified OK, six packages, native test passing, and the new token guard is silent on the real template. Mutation on the guard: appending probe: __PROBE__ to the template exits 1 right after the cosign step and names the line; the header comment's __PLACEHOLDER__ prose does not trip it. shellcheck and check-action-pins.sh clean.
On the trigger fix: workflows: ["Release"] matches release.yml's name, and workflow_run.head_branch is the tag on real tag-triggered Release runs (32855542098 event=push head_branch=v0.14.0), so the tag plumbing holds. The one thing nobody can test before merge is workflow_run itself, since GitHub only fires it once the file is on the default branch; the first release after merge is the first real run, same caveat as the upload.
Both doc notes and the guard suggestion are addressed. One non-gating suggestion inline: rc runs will now sit on the approval prompt before failing the API check, which is what released was chosen to avoid, and a one-clause pre-filter in the if removes that noise without changing where stable-only is enforced.
Approving.
Co-authored-by: Tyler <49161327+tylerpotts@users.noreply.github.com>
tylerpotts
left a comment
tylerpotts
left a comment
There was a problem hiding this comment.
Thanks for all the work on this! The final suggestion was implemented so I'm green-lighting this. Feel free to merge when CI is passing
3 participants
Closes
Closes #579. Moving to prefix.dev's shared
github-releaseschannel and deleting this bridge is #620. Part of #552.What this is
pixi installs conda packages. There is no other artifact type it consumes, so "install
nicwith pixi" and "a conda package fornicexists" are the same requirement, and this produces one.It does not compile anything.
rattler-buildunpacks the archive GoReleaser already published and copies the binary into the prefix - three lines ofbuild.script, no Go toolchain, no source tarball. The recipe is a repackaging manifest, not a build.That is also what octoconda does for the shared channel: it generates a recipe with the same
source: url+sha256and the samebinary_relocation: false/prefix_detection.ignorepair, then runs a generic extract-and-relocate script. The difference here is the digest. octoconda reads GitHub's API asset digest; this reads the sha256 from the release'schecksums.txtafter cosign-verifying that file against the release workflow's identity. Given the point is that the bytes a conda user runs are the bytes we signed, that is the half worth owning.Why now
The starter workspaces merged in #594 already declare the dependency:
Nothing carries it. Onboarding onto
github-releasesis a one-line entry in octoconda'sconfig.toml, open upstream and not ours to merge. So this publishes tonebari-dev/nebari, a channel we control, and says plainly inpackaging/conda/README.mdwhat deleting it looks like.Shape
packaging/conda/recipe.yaml.tmplpackaging/conda/build-packages.sh.github/workflows/publish-conda.ymlOne Linux runner emits all six subdirs. Copying files is not platform-bound; only the
nic versionsmoke test is, and--test nativeruns it for the build platform and skips it elsewhere with a logged reason.Separate from
release.ymlon purpose. The recipe fetches assets from an already-published release by URL, so it has to run after that release exists and be able to run for one cut long ago.workflow_dispatchfrommainbackfills versions published before this workflow existed, and re-runs an upload that failed without cutting a release. That last part earns its keep: GitHub's/releaseslist endpoint returned empty for this repo for about a day earlier this month while every other endpoint kept working.No long-lived token. Upload goes through prefix.dev trusted publishing over OIDC, bound to the
releaseenvironment.Stable releases only, enforced. The tag is checked against the release API and a prerelease refused, which covers the dispatch path a trigger filter would miss.
Two things that bite
conda forbids
-in a version string. It separates name, version and build string in a package filename, so av0.14.0-rc.1tag builds an artifact whose own version cannot be resolved, surfacing asError: Test failed / failed to setup test environment. Normalised to0.14.0rc1.Windows differs twice:
.ziprather than.tar.gz, and binaries underLibrary/binrather thanbin.How to Test
The commit and build date are the release's, not
dev, so the ldflags survive repackaging. A one-byte-tamperedchecksums.txtfails verification before any digest is read.actionlint,shellcheckandscripts/check-action-pins.shall pass.Scope and open items
Provider tools stay in the starter workspaces rather than this package. conda-forge publishes no
opentofuforwin-arm64, which NIC does ship a package for, andlocal/existingnever invoke OpenTofu; #617 already scopes the constraint to the AWS starter and derives it frompkg/tofu. Recorded indocs/operations/packaging.md.Repointing
starters/templates/pixi.toml.tmplat this channel is not here. It has to wait until the channel actually serves, which needs this merged, the trusted publisher registered, and a first publish.--generate-attestationon the upload is deferred rather than declined: it would close the provenance chain past the channel boundary, but it cannot be tested until the trusted publisher is live, so it belongs on the first real dispatch.Still unverified: the
-c nebari-dev/nebarichannel spelling. The first dispatch is the first real test; if it is wrong the fix is one line.