✨ Ship standalone wk-* binaries on each release - #18
Merged
Conversation
Adds a PyInstaller --onedir bundle with all seven wk-* CLIs sharing one
embedded Python + dependency tree, attached as a versioned zip per arch
on every published release.
Builds on every PR / push / release / workflow_dispatch (so PRs catch
breakage early); the publish step only fires on release: published.
- packaging/pyinstaller/{warpkit.spec,build_bundle.py,launchers/} drive
the build. Launchers each call multiprocessing.freeze_support() up
front; PyInstaller's user runtime_hooks run before its built-in ones,
so calling freeze_support from a custom hook hits the unpatched
no-op and any wk-* using a process pool dies with BrokenProcessPool.
- Linux builds run inside quay.io/pypa/manylinux2014_{x86_64,aarch64}
to keep the glibc floor identical to the wheels (2.17).
- macOS arm64 binaries are ad-hoc signed; bundle README documents the
Gatekeeper xattr workaround.
- Smoke test runs --version on all 7 binaries plus a real wk-unwrap-phase
end-to-end against tests/data/test_data/ — exercises numpy, scipy,
skimage, the C++ extension, and the multiprocessing worker pool.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a PyInstaller-based packaging path to ship standalone wk-* CLI binaries (one shared --onedir bundle per platform) and integrates build + smoke-test + release-asset upload into CI.
Changes:
- Add PyInstaller multi-entry spec plus per-command launcher scripts to produce a shared
_internal/runtime bundle. - Add a
build_bundle.pydriver that builds, post-processes (strip/sign), writes an in-bundle README, and zips with version/target naming. - Extend the GitHub Actions workflow to build and smoke-test bundles on PR/push and upload zipped assets on release publish.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
packaging/pyinstaller/warpkit.spec |
Defines a multi-binary PyInstaller build that merges shared dependencies into a single onedir bundle. |
packaging/pyinstaller/launchers/wk-*.py |
Adds frozen-entry launchers that call freeze_support() before importing warpkit CLIs. |
packaging/pyinstaller/build_bundle.py |
Implements the build pipeline: run PyInstaller, strip/sign, generate README, and create a versioned zip. |
packaging/pyinstaller/bundle_README.md |
Provides end-user install/PATH and macOS Gatekeeper instructions embedded into each bundle. |
.github/workflows/build.yml |
Adds CI jobs to build/smoke-test bundles per arch and attach zips to GitHub Releases. |
.gitignore |
Ensures the committed PyInstaller spec is not ignored. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #18 +/- ##
=======================================
Coverage 94.44% 94.44%
=======================================
Files 15 15
Lines 1081 1081
=======================================
Hits 1021 1021
Misses 60 60 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
actions/checkout@v6 ships with Node 24, which requires glibc 2.27+; manylinux2014 is based on CentOS 7 with glibc 2.17, so Node fails to load before any of our build steps run. cibuildwheel 3.x already defaults to manylinux_2_28 for cp39+, so the binaries' glibc floor (2.28) now matches the wheels' rather than being more conservative than them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two unrelated failures from the first CI run: 1. Linux: manylinux's /opt/python/cp311-cp311 is built without --enable-shared, so PyInstaller cannot find libpython3.11.so. Switch to uv-managed Python (built --enable-shared, matching the macOS job). 2. macOS: warpkit_cpp.cpython-*-darwin.so was missing from the bundle when warpkit is wheel-installed (works locally only because the editable install routes the import through the repo build dir so PyInstaller's import-graph picks it up; a wheel install does not). Add a hook that explicitly collects the .so. Note default collect_dynamic_libs search_patterns is `lib*.so` which excludes our extension since it has no `lib` prefix; passing `*.so` explicitly fixes it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Running build_bundle.py creates packaging/pyinstaller/build/warpkit/ (PyInstaller's workdir). Without an explicit exclude, setuptools' packages.find walks into it, sees a `warpkit` directory, and treats it as the warpkit package — failing on `packaging/pyinstaller/build/warpkit/localpycs` does not exist during subsequent uv sync runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
collect_dynamic_libs was returning empty even with explicit *.so search patterns when warpkit is wheel-installed in CI. Switch to importlib's import resolution which finds the .so regardless of install layout, and add diagnostic prints so the next failure is debuggable from the build log. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local repro showed the wheel-install path was tripping over the source tree's `warpkit/` being on sys.path (CWD=repo root) but lacking the .so next to it: PyInstaller's import-graph analyzer found the source tree first and never consulted the venv's wheel-installed copy. Switch the binaries jobs to `uv sync --group dev` (editable install via cmake-build-extension), which drops the .so right into the source tree's `warpkit/` directory — exactly where the analyzer is already looking. Side benefits: - No more `needs: [wheels-build]`, so binaries build in parallel with wheels instead of waiting for the slowest matrix entry. - One less artifact dance (download wheel → install). - The source-tree build is the same path local devs use, so any future CI failure is reproducible with `uv sync` + the build script. The hook-warpkit.py keeps a simple importlib-based binary registration as defense-in-depth (and so it works for both editable and wheel installs); the diagnostic prints are gone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
setuptools-scm calls git during uv sync to compute the version, but the manylinux container runs as a different UID than the actions runner — git's "dubious ownership" check trips and the build fails. actions/checkout sets safe.directory for the runner user, not for processes inside the container. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Running `strip --strip-unneeded` on every `.so*` in the bundle breaks ELF segment alignment in NumPy's bundled libscipy_openblas64_*.so — NumPy then fails to import at runtime with `ELF load command address/offset not properly aligned`. Drop the strip step entirely; the size cost (~30–50MB unzipped, less zipped) is acceptable for a release artifact and removes a class of bugs where strip mangles SIMD-aligned scientific libraries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Gatekeeper flags every .dylib and .so under _internal/, not just the top-level wk-* binaries. Drop the right-click alternative (it would need to be done per-binary AND the libs still wouldn't be cleared) and use `xattr -r -d` on the whole bundle dir. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add a Standalone binaries section pointing at the per-arch zips attached to each GitHub release, so PyPI/README readers can find them without Python. - Fix the wheel matrix line: linux aarch64 wheels are also published. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- adhoc_sign_macos: raise on codesign failures instead of silently shipping a bundle Gatekeeper would reject. Collect every failure with stderr and surface them all in one RuntimeError. - write_readme: pass encoding="utf-8" explicitly to read_text / write_text so bundle generation is deterministic across platforms with non-UTF-8 default locales. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--onedirbundle (one shared_internal/for all 7wk-*CLIs) attached as a versioned zip per arch onrelease: publishedworkflow_dispatch; onlybinaries-publishis gated on releaseWhat's in
packaging/pyinstaller/warpkit.spec— multi-binary spec: 7Analysis→MERGE→ 7EXE→ 1COLLECTlaunchers/wk-*.py— entrypoint per CLI. Each callsmultiprocessing.freeze_support()before the warpkit import. Why this matters: PyInstaller runs userruntime_hooksbefore its built-in ones, so callingfreeze_support()from a custom hook hits the unpatched no-op — any wk-* using a process pool then dies withBrokenProcessPool. Putting the call in the launcher (the actual main script) ensurespyi_rth_multiprocessinghas already patchedfreeze_supportby the time we call ithooks/hook-warpkit.py— explicitly registerswarpkit_cpp.cpython-*.soviaimportlib.util.find_spec, since PyInstaller's defaultcollect_dynamic_libsglob (lib*.so) doesn't match our extension and the import-graph analyzer can be foiled by the source tree shadowing the venv copybuild_bundle.py— runs PyInstaller, ad-hoc signs on macOS, writes the bundle README, zips with versioned filenamebundle_README.md— install/PATH instructions + recursivexattr -d com.apple.quarantinefor macOS Gatekeeper (covers all.dylib/.sofiles in_internal/, not just the top-level binaries)Workflow changes
New jobs in
.github/workflows/build.yml:binaries-build-linux— matrix over x86_64 / aarch64, runs insidequay.io/pypa/manylinux_2_28_${arch}for ABI compat. Builds the C extension viauv sync --group dev(editable install, drops.sointo the source tree where PyInstaller's analyzer is already looking)binaries-build-macos—macos-latestrunner, arm64 onlybinaries-publish—if: release && action == published, attaches zips viagh release uploadSmoke test in each build job:
--versionon all 7 binaries plus a realwk-unwrap-phaseend-to-end against committedtests/data/test_data/— exercises numpy, scipy, skimage, the pybind11 C++ extension, and the multiprocessing worker pool.Also updates the main README with a new "Standalone binaries" section and corrects the wheel matrix line (linux aarch64 was missing).
Notable gotchas hit + fixed during this PR
manylinux2014choice brokeactions/checkout@v6(Node 24 needs glibc 2.27+). Bumped tomanylinux_2_28./opt/python/cp311-cp311is built without--enable-shared; PyInstaller can't findlibpython3.11.so. Switched the install step to uv-managed Python.setuptools.findwas includingpackaging/pyinstaller/build/warpkit/(PyInstaller's workdir) as a phantomwarpkitpackage after a localbuild_bundle.pyrun; added\"packaging*\"toexclude.git config --global --add safe.directoryset byactions/checkoutdoesn't reach the manylinux container (different UID). Added an explicit step.strip --strip-unneededon every.soin the bundle was corrupting NumPy's bundled OpenBLAS (ELF load command address/offset not properly aligned). Dropped the strip step.warpkit/(which has no.so) shadowed the venv's wheel-installed copy on sys.path. Switched to building from an editable install so the.solives in the source tree where the analyzer expects it — also drops theneeds: [wheels-build]dependency.Test plan
--versionwk-unwrap-phaseend-to-end against test data completes in ~18s with parallel workers🤖 Generated with Claude Code