Skip to content

ci: the jac-kit action checks its download against a manifest sealed into the kit, and fetches again on a miss - #8942

Open
marsninja wants to merge 1 commit into
jaseci-labs:mainfrom
marsninja:ci/kit-manifest-check
Open

ci: the jac-kit action checks its download against a manifest sealed into the kit, and fetches again on a miss#8942
marsninja wants to merge 1 commit into
jaseci-labs:mainfrom
marsninja:ci/kit-manifest-check

Conversation

@marsninja

Copy link
Copy Markdown
Collaborator

What failed on main

The CI run for 8081ec8 (run 33816389635) failed in test-jac-pack-smoke and test-scale (microservices, data, server). All four died in the jac-kit action's warm-up step with:

jac (launcher): no runtime payload trailer (this is not a bundled jac binary)

The kit itself was whole: the same artifact (ID 9917157116) ran in every other lane of that run, and build-kit had already smoked the binary with its 110 MiB payload. In the four failing jobs actions/download-artifact@v4 took about 70s instead of about 7s, never printed its "SHA256 digest of downloaded artifact" / "Artifact download completed successfully" lines, and still ended with outcome=success. The blob stream stalled mid-extraction, the step exited 0, and the jobs ran a jac binary cut short of its payload trailer.

(The Release run that failed at the same commit was a re-dispatch of an already-tagged v0.37.3 and passed on the next dispatch; nothing to fix there.)

The fix

  • build-kit (ci.yml) and build-kit-macos (_macos-native.yml) seal a manifest into the kit: zig-out/kit.sha256 hashes every kit file, and zig-out/kit.sha256.sha256 hashes the manifest, so a manifest cut short by the same truncation cannot vouch for the files after it.
  • The jac-kit action checks its download against the manifest right after download-artifact, strictly (a short, missing, or malformed line fails). On a miss it warns, fetches the artifact once more, and checks again, so a stalled download costs a retry instead of the job.
  • .github/actions/jac-kit/manifest.sh holds both sides (write / check). It uses sha256sum where present and shasum -a 256 otherwise, so the macOS lane's kit gets the same treatment.

Verified

  • manifest.sh exercised locally against a fake kit tree: a clean check passes; a truncated binary, a missing file, a manifest cut at a line boundary, a missing manifest, and a missing manifest-of-manifest each fail with exit 1; writing with one hash tool and checking with the other agrees.
  • The three YAML files parse; shellcheck is clean on the script; actionlint reports nothing new in the changed ranges.

…into the kit, and fetches again on a miss

Main's CI run for 8081ec8 lost test-jac-pack-smoke and three
test-scale lanes to a jac binary with no payload trailer. The kit was
whole (the same artifact ran in every other lane, and build-kit had
smoked it), but actions/download-artifact@v4 took 70s where the other
jobs took 7s, never printed its download-completed lines, and still
exited 0: a stalled blob stream, a jac binary cut short of its
trailer, and a step that reported success.

build-kit and build-kit-macos now seal a manifest into the kit
(zig-out/kit.sha256, with kit.sha256.sha256 vouching for the manifest
itself, so a manifest cut short by the same truncation cannot pass).
The jac-kit action checks its download against it strictly, and on a
miss fetches the artifact once more and checks again. manifest.sh
carries both sides and falls back from sha256sum to shasum -a 256 for
the macOS lane.
@marsninja

Copy link
Copy Markdown
Collaborator Author

Attempt 1 lost only test-runtime, to tests/language/test_console.jac::all edge case patterns preserved ("Pattern was corrupted or stripped: #2000"). The kit steps in that lane had passed ("kit verified: 862 files"), and the identical failure hit main's own run 33775038316 earlier today at fe7c62b and cleared on the next push with no change to that test, so it is a pre-existing intermittent. Reran the failed lane; attempt 2 is green.

@christianwilkins christianwilkins left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 3989a6d30d1de42a67547e4208666c554cd94d0e.

No blocking finding in the complete patch. Both kit producers include the manifest and its digest, and the consumer retries once before a mandatory second check. I also read Jason's note about the first-attempt runtime failure and checked the current results rather than treating that attempt as the final CI state.

Validation: eight local manifest cases behaved as expected (intact kit, whitespace in filenames, truncated binary, missing file, truncated manifest, missing manifest/digest, malformed line with a valid manifest digest, and restored-download success; the intact and whitespace checks share one case). Cross-checking a sha256sum manifest with shasum, and the reverse, passed; shasum also rejected a corrupted payload. bash -n passed. Current-head CI is green (25 successful checks, 3 skipped); GitHub reports no conflicts. No source changes made during review.

Approved and ready to merge. No further author action identified on this head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants