Migrate to rattler-build with staged outputs (refresh of #171 for 1.28.0) - #196
Migrate to rattler-build with staged outputs (refresh of #171 for 1.28.0)#196hmaarrfk wants to merge 53 commits into
Conversation
(cherry picked from commit bbb7162)
(cherry picked from commit a603d7a)
(cherry picked from commit 013a296)
(cherry picked from commit b9e50c1)
(cherry picked from commit b4ddaab)
(cherry picked from commit 8178088)
(cherry picked from commit b621bf2)
(cherry picked from commit 0c0eb26)
[cherry picked from commit 256790e; recipe changes only, rerender output dropped]
(cherry picked from commit 53d6eb8)
[cherry picked from commit d3457f3; recipe changes only, rerender output dropped]
(cherry picked from commit 0f79f21)
(cherry picked from commit 2332847)
(cherry picked from commit 9724cd6)
(cherry picked from commit de3e786)
(cherry picked from commit 8fa9d24)
(cherry picked from commit 68ded51)
(cherry picked from commit 8a294e1)
(cherry picked from commit 9417bea)
(cherry picked from commit f1211de)
(cherry picked from commit 739f3e5)
[cherry picked from commit 666a846; recipe changes only, rerender output dropped]
(cherry picked from commit 85eb668)
(cherry picked from commit af69e8d)
(cherry picked from commit 542e19a)
(cherry picked from commit 53c4f88)
(cherry picked from commit 0dfeb68)
(cherry picked from commit e041e7a)
(cherry picked from commit 9dcca6c)
(cherry picked from commit ae48510)
(cherry picked from commit dd1d48d)
(cherry picked from commit eb1156e)
(cherry picked from commit 72e183d)
(cherry picked from commit 9407828)
(cherry picked from commit 8f432b4)
(cherry picked from commit 66edfee)
<details><summary>Claude's draft</summary> Adapt the cherry-picked rattler-build migration (conda-forge#171 by @cbourjau) to the current 1.28.0 feedstock state: - Bump version to 1.28.0 with the matching sha256 and add patches 0006/0007 (onnx-less test guards introduced for 1.28.0). - Re-enable the -novec variants that were skipped while debugging conda-forge#171; this required templating the staging output name in the `inherit:` keys. - Port CoreML execution provider support for osx-arm64 (conda-forge#182): onnxruntime_USE_COREML=ON, nlohmann_json host dep, provider test, BSD-3-Clause license and vendored license files. - Port Jetson Thor support (conda-forge#190): add 110-real to the CUDA 13.0 arch list. - Port nvcc_threads=2 (Windows CUDA 13.0 OOM mitigation). - Pin cmake <4 (vendored protobuf 3.21 does not build with CMake 4). - Match main's 1.28.0 runtime deps: drop sympy, add __cuda run dep on the python output. - Add onnxruntime_test --help entry-point test. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
…026.08.08.17.43.08 Other tools: - conda-build 26.7.0 - rattler-build 0.72.2 - rattler-build-conda-compat 1.4.19
<details><summary>Claude's draft</summary> The top-level recipe name is a placeholder (onnxruntime-dummy), so the conda-smithy linter needs an explicit extra.feedstock-name: onnxruntime. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
|
Hi! This is the friendly automated conda-forge-linting service. I just wanted to let you know that I linted all conda-recipes in your PR ( I do have some suggestions for making it better though... For recipe/recipe.yaml:
Documentation on acceptable licenses can be found conda-forge.org > Docs > Maintainer Documentation > Contributing packages > SPDX Identifiers and Expressions. This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/31292524604. Examine the logs at this URL for more detail. |
<details><summary>Claude's draft</summary> All CI jobs failed at cmake configure with "CMAKE_CXX_STANDARD must be at least 20. It is 17." — 1.28.0 enforces C++20 via onnxruntime_language_standard_versions.cmake. The C++17 choice in conda-forge#171 was a workaround for C++20 module-scanning issues at 1.24.4; upstream 1.28.0 now sets CMAKE_CXX_SCAN_FOR_MODULES=OFF itself (cmake/CMakeLists.txt:21), so C++20 is safe. This also matches main's build.py invocation which passed CMAKE_CXX_STANDARD=20. run_cpp_test.bat stays at /std:c++17: consuming the public headers only needs C++17 and main is green with that combination. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
<details><summary>Claude's draft</summary>
Both Windows CUDA jobs failed at configure with "Could not find compiler
set in environment variable CC: cl.exe" while the Windows CPU job (which
skips the CUDA branch) proceeded fine.
Cause: on Windows, nushell exposes the search path as a *list* named
`Path`. The CUDA branch did
$env.PATH = $"($build_lib_prefix)/bin;($env.PATH)"
which interpolates that list into a string ("[C:\..., C:\...]") under the
separate key `PATH`, shadowing the real `Path` in child processes — so
cmake could no longer find cl.exe. Prepend to the `Path` list instead.
Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
<details><summary>Claude's draft</summary> The Windows CUDA 12.9 job now compiles (PATH fix worked) but fails on every .cu file targeting sm_100a/sm_120a: CCCL's clusterlaunchcontrol.h inline asm uses long2, and long is 32-bit under MSVC, producing "asm operand type size(4) does not match constraint 'l'". main already carries the answer in bld.bat: SM 100+ is broken with CUDA 12.9 on Windows (fixed in 13.0), and SM 110 (Thor) is Linux-only. My 1.28.0 adaptation had wrongly applied the Linux arch lists to all platforms. Split the arch lists per platform to match main exactly: - win 12.9: 70..90-real (no 100/120) - win 13.0: 75..100-real,120 (no 110) - linux unchanged Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
<details><summary>Claude's draft</summary> The osx_arm64 jobs failed compiling onnxruntime_autoep_test: the vendored googletest's gtest-printers.h formats std::chrono::time_point via std::format, whose floating-point path needs std::to_chars — gated by libc++ availability to macOS 13.4+, while MACOSX_DEPLOYMENT_TARGET is 11.0. main never hits this because its meta.yaml keeps gmock in host, so onnxruntime's FetchContent FIND_PACKAGE_ARGS resolves conda-forge's gtest/gmock instead of vendoring one. Restore that dep, scoped to osx — Linux and Windows are green with the vendored copy, and the megabuild prefers a minimal host env to keep the shared cmake cache stable. Added to both the staging and python stages to keep their dependency sets identical (cache-sharing requirement). Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
<details><summary>Claude's draft</summary>
The osx jobs now compile, pass all 10 ctest suites, and install — but fail
at packaging with rattler-build's "No license files were copied". That
error fires when ANY itemized license path matches nothing (the detailed
missing-path list goes to a log level the CI does not surface), and the
set of vendored _deps differs per platform (CoreML adds coremltools/fp16/
psimd on osx; deps provided from conda-forge, e.g. nlohmann_json/gtest on
osx, may not be vendored at all).
Replace the itemized _deps license list with globs:
build-ci/Release/_deps/*/LICENSE*
build-ci/Release/_deps/*/COPYING*
This ships the license of every dep actually vendored into the build on
each platform — including several that are compiled in but were previously
not shipped (mp11, date, dlpack, kleidiai, cutlass on cuda) — and cannot
break when the vendored set changes.
Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
<details><summary>Claude's draft</summary> Replace the license globs with an explicit list, conditioned on exactly the variants that vendor each dependency. The per-variant sets were read from the packaging logs of the passing glob-based CI run: - base (all variants): LICENSE + abseil, date, dlpack, eigen (COPYING.MPL2 only, EIGEN_MPL2_ONLY=ON), flatbuffers, gsl, onnx, protobuf, pytorch_cpuinfo, re2, safeint - nlohmann_json: not osx (conda-forge host dep there, not vendored) - googletest: only native non-CUDA builds (tests off ⇒ never fetched), and not osx (conda-forge gtest/gmock there) - kleidiai: ARM targets (linux-aarch64, osx-arm64) - microsoft_wil: win - coremltools/fp16/psimd: osx (CoreML EP) Verified with rattler-build --render-only that each variant's rendered list matches what its CI packaging log actually copied. Not itemized: cutlass / cudnn_frontend on CUDA builds (parity with main, which never shipped them); can be added once a CUDA packaging log confirms the paths. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
|
Thanks for pushing this forward! Are you aware that upstream introduced an execution provider plugin system which should decouple the expensive cuda builds from the Python version, AFAIU. |
|
I think I remember you mentioning that. But I did not look into it too deeply. What would it change in terms of organization here? |
|
Also. Full disclosure. I am trying to help here and threw my AI at your improvements since it is better than me at these problems. I didn’t have time to explicitly write that at the top level but I have my AI often disclose itself in every commit. I have not yet reviewed these changes. |
It would virtually remove the need for staged outputs, I think. We would compile one Python-independent cuda package per platform, and N fairly cheap Python-dependent frontends that depend on the former. |
<details><summary>Claude's draft</summary> linux_64 CPU hit the namespace runner's 8h session cap a second time, even with the full test suite gated to one python. The decisive comparison: Windows builds all five pythons WITH full per-python tests in ~2.5-3h — and bld.bat never passed --enable_lto. GCC LTO is the dominant linux cost. Precedent: the rattler megabuild (conda-forge#196) also built with onnxruntime_ENABLE_LTO=OFF and its full test suites were green; only the conda-build recipe on main used LTO on unix. Flagged in the PR as a deliberate trade-off for reviewers. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
… tests <details><summary>Claude's draft</summary> Root cause of three consecutive linux_64 CPU deaths at exactly the 8h runner cap: every one carried the annotation "Error reading output: stream did not contain valid UTF-8". The RegexFullMatch.NonUtf8Pattern gtest deliberately prints invalid UTF-8 to stderr; rattler-build's output reader dies on it, the test process then blocks on a full pipe, and the job hangs until the session cap kills it. The megabuild recipe (conda-forge#196) carried GTEST_FILTER=-RegexFullMatch.NonUtf8Pattern for exactly this reason ("breaks rattler build output due to invalid utf8 output on stderr") and this recipe failed to inherit it. Supporting evidence that builds were never the bottleneck: aarch64 (cross, no tests) does cpp + 5 pythons in 24 minutes; Windows (different console pipeline, full tests) passes in hours; only native-gtest unix jobs died. The round-2 osx death mid-gtests was likely the same hang, not a runner session cap. With the true cause fixed, revert the tests-once-per-job mitigation: every python variant runs the full upstream test suite again. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
<details><summary>Claude's draft</summary> With the invalid-UTF-8 gtest hang fixed, the CPU jobs are cheap (linux_64: 40 min for cpp + 5 pythons with full tests; aarch64 cross: 26 min; osx single-python: 12 min), so the exploratory reductions are no longer needed: - novec variants are built again on every CPU platform - osx builds the full python 3.10-3.14 matrix again (estimated ~40 min) This is the intended merge state of the matrix: 14 jobs. LTO remains off (as in the megabuild conda-forge#196); whether to re-enable --enable_lto at ~2-3x CI cost for parity with the current conda-build binaries is left as an open review question. Resume this Claude session: ``` cd /home/mark/git/feedstock/onnxruntime-feedstock claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc ``` </details>
Refresh of #171 by @cbourjau against the current 1.28.0 feedstock. All credit for the rattler-build "megabuild" design goes to Christian — this branch cherry-picks his commits (with original authorship) and rebases the effort onto today's
main.Status: CI is fully green — all 14 build jobs pass (linux_64/aarch64 × CPU/novec/CUDA 12.9/13.0, win × CPU/novec/CUDA 12.9/13.0, osx_arm64 × regular/novec).
What was done
Cherry-picked from #171 (44 commits): the full rattler-build migration — staged
onnxruntime-buildoutput whose build cache is shared by the-cppand per-python outputs, nushell build scripts, and all of Christian's CUDA/cross-compilation/Windows fixes.Dropped from #171:
main: python 3.14 migration file, namespace runner labels, patch0005, theosx_arm64: defaultprovider change, and CI workflow hand-patches that rerendering regenerates.Adaptation for 1.28.0 + ports from main:
0006/0007(onnx-less test guards new in 1.28.0)onnxruntime_USE_COREML=ON,nlohmann_jsonhost dep, provider assert test, license additions110-realin the Linux CUDA 13.0 arch listnvcc_threads=2(-Donnxruntime_NVCC_THREADS=2, Windows CUDA OOM mitigation)-novecvariants Migrate to rattler-build with staged outputs ("Megabuild") #171 had skipped while debugging; requiredinherit: onnxruntime${{ suffix }}-buildcmake <4pin, droppedsympyrun dep,__cudarun dep on the python output,extra.feedstock-nameFixes found while driving CI green (details in the commit messages):
CMAKE_CXX_STANDARD >= 20; the C++17 module-scanning workaround from Migrate to rattler-build with staged outputs ("Megabuild") #171 is obsolete (upstream now setsCMAKE_CXX_SCAN_FOR_MODULES=OFFitself).$env.PATH = $"...;($env.PATH)"mangles thePathlist on Windows and madecl.exeunfindable in CUDA builds; prepend to$env.Pathinstead.bld.bat): SM 100+ triggers broken CCCLclusterlaunchcontrol.hinline asm on Windows CUDA 12.9 (longis 32-bit under MSVC); SM 110 is Linux-only.chrono::time_pointviastd::format, needing floatto_chars(macOS 13.4+) while the deployment target is 11.0 — same reason main keepsgmockin host._depsset differs per variant (e.g. nlohmann_json/googletest are not vendored on osx, kleidiai is ARM-only, microsoft_wil is win-only, googletest only exists when unit tests are built). Each entry is now conditioned on exactly the variants that vendor it, validated against CI packaging logs.Notes for review
include/onnxruntime/*.h(wasinclude/onnxruntime/core/session/...). Downstream consumers ofonnxruntime-cppwill see the new layout.Supersedes #171 (kept as reference; happy to hand this back to @cbourjau's branch instead if preferred).