Clean rattler-build migration: CPU packages + standalone CUDA EP plugin, no staging - #198
Draft
hmaarrfk wants to merge 13 commits into
Draft
Clean rattler-build migration: CPU packages + standalone CUDA EP plugin, no staging#198hmaarrfk wants to merge 13 commits into
hmaarrfk wants to merge 13 commits into
Conversation
<details><summary>Claude's draft</summary> Clean-slate rattler-build migration per conda-forge#197 (comment): no staging outputs anywhere, restarted from main and combining the proven parts of main's conda-build recipe with the EP-plugin work from conda-forge#196/conda-forge#197. Three independent outputs: - onnxruntime-ep-cuda (CUDA variants only): the CUDA execution provider built as a standalone plugin (onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON). Python-free, one build per platform x CUDA version, loads into any onnxruntime core >= 1.24.4 via the EP plugin API. Nushell script and license list carried from the validated conda-forge#197 work. - onnxruntime{-novec}-cpp (CPU variants): standalone build.py-driven build, no Python bindings, no unit tests; ships the nested upstream header layout, exactly like the conda-build packages did. Build python pinned to 3.13.* so the python variant key does not fan this output out. - onnxruntime{-novec} (CPU variants): per-python build.py-driven build with wheel + the full upstream test run — main's build.sh/bld.bat with the CUDA branches removed. With CUDA gone these are comparatively cheap CPU builds; linux/win build all pythons in one CI job, osx splits per python. CUDA branches, install-cpp scripts and meta.yaml are gone; scripts are main's proven bash/bat adapted, not the megabuild's cmake-direct nushell flow, so main's per-platform license lists apply verbatim (the plugin output carries its own list matching its leaner vendored set). EXPLORATORY skip (remove before merge): novec variants, to save CI while iterating. TODO in-recipe: restore -cpp run_exports once the v1 linter handles templated pin_subpackage names. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
…026.08.09.00.32.18 Other tools: - conda-build 26.7.0 - rattler-build 0.72.2 - rattler-build-conda-compat 1.4.19
Contributor
|
Hi! This is the friendly automated conda-forge-linting service. I just wanted to let you know that I linted all conda-recipes in your PR ( I do have some suggestions for making it better though... For recipe/recipe.yaml:
Documentation on acceptable licenses can be found conda-forge.org > Docs > Maintainer Documentation > Contributing packages > SPDX Identifiers and Expressions. This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/31651507124. Examine the logs at this URL for more detail. |
<details><summary>Claude's draft</summary> The plugin links only cudart/cublas/cublasLt (+ the driver); cuFFT is loaded lazily at runtime and cusparse/curand/nvrtc are unused — their -dev packages are host deps purely for headers. CI flagged overdepending warnings for exactly these four. Ignore their run exports so the plugin package's runtime dependencies match reality (cuFFT-backed FFT contrib ops fail cleanly with NOT_IMPLEMENTED when libcufft is absent, per upstream's optional-dependency design). Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
Contributor
Author
|
Status note on the CUDA jobs: they build, test, and package successfully (~53 min for the aarch64 cross build) and fail only at conda-forge's output validation, because Registration requested in conda-forge/feedstock-outputs#117. Until that merges, expect all six CUDA jobs to go red at the validation step despite green builds. CI also emitted overdepending warnings for cuda-nvrtc/libcufft/libcusparse/libcurand — the libraries the plugin deliberately does not link (cuFFT is lazy-loaded; the others are header-only build deps). A commit trimming those run exports via |
<details><summary>Claude's draft</summary> Following the approach of conda-forge/h5py-feedstock#168: - linux_aarch64: drop the linux_64 cross-compile mapping and build natively on GitHub's hosted ubuntu-24.04-arm runners (4 cores; no namespace arm-linux profile exists yet). Native builds run the full test suite, which the cross builds could not. - osx_arm64: compilation was already native; move the jobs from Azure to GitHub Actions on namespace-profile-12cpu-on-osx-arm64 (the profile pytorch-cpu-feedstock uses) for considerably faster macs. Side effect: conda-smithy now inlines all python versions into a single osx job like the other platforms, and the Azure pipeline files disappear. The whole matrix is now 10 GitHub Actions jobs. Watch the aarch64 CPU job duration: 5 python builds + tests on 4 cores may approach the timeout; if so, options are trimming the test run or restoring cross-compilation. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
…026.08.09.00.32.18 Other tools: - conda-build 26.7.0 - rattler-build 0.72.2 - rattler-build-conda-compat 1.4.19
<details><summary>Claude's draft</summary>
Native-runner results from the previous round:
- Both native aarch64 CUDA jobs starved GitHub's free 4-core/16GB
ubuntu-24.04-arm runners to death ("runner lost communication") after
~4.5h — CUDA compiles need more memory than the hosted arm runners have,
and there is no namespace arm-linux profile yet. Restore the proven
linux_64 cross-compilation (53min CUDA plugin builds).
- linux_64 CPU was killed at exactly 8h00m (SIGTERM/exit 143) — the
namespace runner session cap — because main's scripts hard-code
--parallel=8, wasting half of the 16-core runner. Windows never passed
--parallel (build.py defaults to all cores) and finished comfortably.
Switch linux/osx to --parallel=${CPU_COUNT}, which should bring the
5-python job to roughly 5h.
osx stays on the (native) namespace-profile-12cpu-on-osx-arm64 runners.
Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume ${SESSION_ID}
```
</details>
…026.08.09.00.32.18 Other tools: - conda-build 26.7.0 - rattler-build 0.72.2 - rattler-build-conda-compat 1.4.19
<details><summary>Claude's draft</summary> Round-3 findings: - linux_64 CPU hit the namespace runner's 8h session cap again (exit 143 at 8h00m) even at --parallel=$CPU_COUNT. The job runs 1 cpp + 5 python builds, each with the full upstream test suite (gtests + python tests). - The osx namespace runner dies at ~2h45m, so the consolidated 6-build osx job cannot fit there at all. - Per-python CI job splitting is not achievable in the split-output design: conda-smithy distributes jobs over the intersection of variant keys used by ALL outputs, and the cpp/plugin outputs must not carry the python key. - The Windows CPU failure this round was a transient github.com connection timeout during a FetchContent download; the push retries it. Changes: - build-python.sh: run build.py --test only for python 3.13 (the gtest suite is identical across pythons). Unit tests are still built for every variant so the vendored dependency set (and license list) stays identical; every python still runs the recipe-level import and entry-point tests. Windows keeps the full per-python test runs — it fits comfortably there. Should cut the linux job to ~6h. - EXPLORATORY (marked): osx builds only python 3.13 while iterating — the osx flow is byte-identical to main's proven conda-build flow, and one end-to-end python validates it within the mac runner's session cap. Restoring full osx coverage needs either a longer-lived mac runner or accepting multiple pushes of the same job. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
<details><summary>Claude's draft</summary> linux_64 CPU hit the namespace runner's 8h session cap a second time, even with the full test suite gated to one python. The decisive comparison: Windows builds all five pythons WITH full per-python tests in ~2.5-3h — and bld.bat never passed --enable_lto. GCC LTO is the dominant linux cost. Precedent: the rattler megabuild (conda-forge#196) also built with onnxruntime_ENABLE_LTO=OFF and its full test suites were green; only the conda-build recipe on main used LTO on unix. Flagged in the PR as a deliberate trade-off for reviewers. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
… tests <details><summary>Claude's draft</summary> Root cause of three consecutive linux_64 CPU deaths at exactly the 8h runner cap: every one carried the annotation "Error reading output: stream did not contain valid UTF-8". The RegexFullMatch.NonUtf8Pattern gtest deliberately prints invalid UTF-8 to stderr; rattler-build's output reader dies on it, the test process then blocks on a full pipe, and the job hangs until the session cap kills it. The megabuild recipe (conda-forge#196) carried GTEST_FILTER=-RegexFullMatch.NonUtf8Pattern for exactly this reason ("breaks rattler build output due to invalid utf8 output on stderr") and this recipe failed to inherit it. Supporting evidence that builds were never the bottleneck: aarch64 (cross, no tests) does cpp + 5 pythons in 24 minutes; Windows (different console pipeline, full tests) passes in hours; only native-gtest unix jobs died. The round-2 osx death mid-gtests was likely the same hang, not a runner session cap. With the true cause fixed, revert the tests-once-per-job mitigation: every python variant runs the full upstream test suite again. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
<details><summary>Claude's draft</summary> With the invalid-UTF-8 gtest hang fixed, the CPU jobs are cheap (linux_64: 40 min for cpp + 5 pythons with full tests; aarch64 cross: 26 min; osx single-python: 12 min), so the exploratory reductions are no longer needed: - novec variants are built again on every CPU platform - osx builds the full python 3.10-3.14 matrix again (estimated ~40 min) This is the intended merge state of the matrix: 14 jobs. LTO remains off (as in the megabuild conda-forge#196); whether to re-enable --enable_lto at ~2-3x CI cost for parity with the current conda-build binaries is left as an open review question. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
…026.08.10.11.23.07 Other tools: - conda-build 26.7.0 - rattler-build 0.72.2 - rattler-build-conda-compat 1.4.19
Contributor
Author
|
Final validation complete — the full 14-job merge-candidate matrix has now run with no exploratory reductions:
The six CUDA checks remain red only at output validation pending conda-forge/feedstock-outputs#117. Every python version runs the full upstream test suite on every native platform; novec variants are back; nothing is skipped. Open review items are in the PR description (LTO, auto-registration UX, cudnn dep, run_exports TODO). |
Contributor
|
The author of pybind11 is also the author of nanobind, which has this to say about pybind11<O:
I.e., We should at least take a look at the effect of turning off LTO here. |
<details><summary>Claude's draft</summary> Per review: pybind11 depends on LTO to produce reasonably-sized bindings (https://github.com/wjakob/nanobind/blob/master/docs/why.rst), and main's shipped binaries are LTO-built. LTO was only dropped here while chasing what turned out to be the invalid-UTF-8 gtest hang; with that fixed the ~3x unix build-time cost fits the runner caps easily (linux_64 estimated ~2h for cpp + 5 pythons with full tests, vs the 8h session cap). Windows stays without LTO, matching main's bld.bat which never enabled it. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
Contributor
Author
|
Trying now... claudeGood point on pybind11's LTO dependence — |
Contributor
Author
|
LTO'd round complete — everything green. Timing effect of
So GCC LTO costs ~2–3× on linux but fits the 8h runner cap with lots of headroom; Apple clang's LTO is effectively free on osx. The one red CPU check (win novec) was a transient github.com download failure during FetchContent (second such flake on the win runners — CMake has no download retry); it has been rerun. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clean-slate take per @cbourjau's comment on #197: a rattler-build migration with no staging outputs, starting fresh from
main, combining the proven parts of the conda-build recipe with the EP-plugin work from #196/#197.Structure — three independent outputs
onnxruntime-ep-cudaonnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON), Python-free, loads into any core ≥ 1.24.4 via the EP plugin APIonnxruntime{-novec}-cpponnxruntime{-novec}build.sh/bld.batminus the CUDA branchesGPU usage until deeper integration lands:
(C++:
Env::RegisterExecutionProviderLibrary.)CI status: everything builds; timings are excellent
The six CUDA checks stay red only at conda-forge's output validation until conda-forge/feedstock-outputs#117 (registering
onnxruntime-ep-cuda) is merged — the builds themselves are green.The one debugging saga worth knowing about: the linux CPU job hung to the runner's 8h session cap three times. Root cause was the
RegexFullMatch.NonUtf8Patterngtest deliberately printing invalid UTF-8, which kills rattler-build's output reader and stalls the test process on a full pipe. The megabuild recipe carriedGTEST_FILTER=-RegexFullMatch.NonUtf8Patternfor exactly this reason; this recipe now does too. (Upstream issue candidate for rattler-build: non-UTF-8 script output should not deadlock the build.)Deliberate choices for review
bld.bat.namespace-profile-12cpu-on-osx-arm64via GitHub Actions (12 min single-python job); Azure and its per-python job split are gone.cudnnkept as a hard run dep of the plugin (most users want Conv); upstream treats it as lazily-optional. Relax torun_constrained?cuda-plugin-ep=1build info, so registration is explicit for now. Options: patchonnxruntime/__init__.pyto probe$CONDA_PREFIX/lib, or ship upstream'sonnxruntime_ep_cudahelper module.-cpprun_exportstemporarily dropped (TODO in recipe): the conda-smithy v1 linter cannot handle templatedpin_subpackagenames.Supersedes #197. CC @cbourjau