Skip to content

Migrate to rattler-build with staged outputs (refresh of #171 for 1.28.0) - #196

Draft
hmaarrfk wants to merge 53 commits into
conda-forge:mainfrom
hmaarrfk:rattler-build-171
Draft

Migrate to rattler-build with staged outputs (refresh of #171 for 1.28.0)#196
hmaarrfk wants to merge 53 commits into
conda-forge:mainfrom
hmaarrfk:rattler-build-171

Conversation

@hmaarrfk

@hmaarrfk hmaarrfk commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Refresh of #171 by @cbourjau against the current 1.28.0 feedstock. All credit for the rattler-build "megabuild" design goes to Christian — this branch cherry-picks his commits (with original authorship) and rebases the effort onto today's main.

Status: CI is fully green — all 14 build jobs pass (linux_64/aarch64 × CPU/novec/CUDA 12.9/13.0, win × CPU/novec/CUDA 12.9/13.0, osx_arm64 × regular/novec).

What was done

Cherry-picked from #171 (44 commits): the full rattler-build migration — staged onnxruntime-build output whose build cache is shared by the -cpp and per-python outputs, nushell build scripts, and all of Christian's CUDA/cross-compilation/Windows fixes.

Dropped from #171:

  • The three rerender commits (recipe edits bundled inside them were preserved) — rerendered ourselves with conda-smithy 2026.8.9.
  • Commits already absorbed into main: python 3.14 migration file, namespace runner labels, patch 0005, the osx_arm64: default provider change, and CI workflow hand-patches that rerendering regenerates.

Adaptation for 1.28.0 + ports from main:

Fixes found while driving CI green (details in the commit messages):

  1. C++20: 1.28.0 enforces CMAKE_CXX_STANDARD >= 20; the C++17 module-scanning workaround from Migrate to rattler-build with staged outputs ("Megabuild") #171 is obsolete (upstream now sets CMAKE_CXX_SCAN_FOR_MODULES=OFF itself).
  2. Windows nushell PATH: $env.PATH = $"...;($env.PATH)" mangles the Path list on Windows and made cl.exe unfindable in CUDA builds; prepend to $env.Path instead.
  3. Per-platform CUDA arch lists (mirrors main's bld.bat): SM 100+ triggers broken CCCL clusterlaunchcontrol.h inline asm on Windows CUDA 12.9 (long is 32-bit under MSVC); SM 110 is Linux-only.
  4. conda-forge gtest/gmock on osx: the vendored googletest's printers format chrono::time_point via std::format, needing float to_chars (macOS 13.4+) while the deployment target is 11.0 — same reason main keeps gmock in host.
  5. Explicit per-variant license lists: the vendored _deps set differs per variant (e.g. nlohmann_json/googletest are not vendored on osx, kleidiai is ARM-only, microsoft_wil is win-only, googletest only exists when unit tests are built). Each entry is now conditioned on exactly the variants that vendor it, validated against CI packaging logs.

Notes for review

  • Header layout change: the cmake-direct install flattens headers to include/onnxruntime/*.h (was include/onnxruntime/core/session/...). Downstream consumers of onnxruntime-cpp will see the new layout.
  • cutlass/cudnn_frontend licenses are not shipped on CUDA builds (parity with main, which never shipped them); can be itemized later.
  • The megabuild works as intended: one CI job per platform/CUDA/suffix builds cpp + all of py310–py314 from a single shared cmake cache.

Supersedes #171 (kept as reference; happy to hand this back to @cbourjau's branch instead if preferred).

cbourjau added 30 commits August 8, 2026 20:41
(cherry picked from commit bbb7162)
(cherry picked from commit 1729bf1)
(cherry picked from commit a603d7a)
(cherry picked from commit 013a296)
(cherry picked from commit b9e50c1)
(cherry picked from commit b4ddaab)
(cherry picked from commit 0ceeae9)
(cherry picked from commit 8178088)
(cherry picked from commit 0c0eb26)
[cherry picked from commit 256790e; recipe changes only, rerender output dropped]
[cherry picked from commit d3457f3; recipe changes only, rerender output dropped]
(cherry picked from commit 0f79f21)
(cherry picked from commit 2332847)
(cherry picked from commit 9724cd6)
(cherry picked from commit de3e786)
(cherry picked from commit 68ded51)
(cherry picked from commit 9417bea)
[cherry picked from commit 666a846; recipe changes only, rerender output dropped]
(cherry picked from commit 85eb668)
(cherry picked from commit af69e8d)
(cherry picked from commit 542e19a)
cbourjau and others added 11 commits August 8, 2026 20:42
(cherry picked from commit 9dcca6c)
(cherry picked from commit ae48510)
(cherry picked from commit dd1d48d)
(cherry picked from commit 72e183d)
(cherry picked from commit 9407828)
(cherry picked from commit 8f432b4)
(cherry picked from commit 66edfee)
<details><summary>Claude's draft</summary>

Adapt the cherry-picked rattler-build migration (conda-forge#171 by @cbourjau) to the
current 1.28.0 feedstock state:

- Bump version to 1.28.0 with the matching sha256 and add patches 0006/0007
  (onnx-less test guards introduced for 1.28.0).
- Re-enable the -novec variants that were skipped while debugging conda-forge#171; this
  required templating the staging output name in the `inherit:` keys.
- Port CoreML execution provider support for osx-arm64 (conda-forge#182):
  onnxruntime_USE_COREML=ON, nlohmann_json host dep, provider test,
  BSD-3-Clause license and vendored license files.
- Port Jetson Thor support (conda-forge#190): add 110-real to the CUDA 13.0 arch list.
- Port nvcc_threads=2 (Windows CUDA 13.0 OOM mitigation).
- Pin cmake <4 (vendored protobuf 3.21 does not build with CMake 4).
- Match main's 1.28.0 runtime deps: drop sympy, add __cuda run dep on the
  python output.
- Add onnxruntime_test --help entry-point test.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
…026.08.08.17.43.08

Other tools:
- conda-build 26.7.0
- rattler-build 0.72.2
- rattler-build-conda-compat 1.4.19
<details><summary>Claude's draft</summary>

The top-level recipe name is a placeholder (onnxruntime-dummy), so the
conda-smithy linter needs an explicit extra.feedstock-name: onnxruntime.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
@conda-forge-admin

conda-forge-admin commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/recipe.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/recipe.yaml:

  • ℹ️ output 0 output overrides versions pinned in the feedstock:
    ['- In section host: python 3.13.*']
    Requirement spec should not list version specifiers to respect conda-forge-pinning. If you need to force another version, please override the pin via conda_build_config.yaml.
  • ℹ️ License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found conda-forge.org > Docs > Maintainer Documentation > Contributing packages > SPDX Identifiers and Expressions.

This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/31292524604. Examine the logs at this URL for more detail.

<details><summary>Claude's draft</summary>

All CI jobs failed at cmake configure with "CMAKE_CXX_STANDARD must be at
least 20. It is 17." — 1.28.0 enforces C++20 via
onnxruntime_language_standard_versions.cmake.

The C++17 choice in conda-forge#171 was a workaround for C++20 module-scanning issues
at 1.24.4; upstream 1.28.0 now sets CMAKE_CXX_SCAN_FOR_MODULES=OFF itself
(cmake/CMakeLists.txt:21), so C++20 is safe. This also matches main's
build.py invocation which passed CMAKE_CXX_STANDARD=20.

run_cpp_test.bat stays at /std:c++17: consuming the public headers only
needs C++17 and main is green with that combination.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
<details><summary>Claude's draft</summary>

Both Windows CUDA jobs failed at configure with "Could not find compiler
set in environment variable CC: cl.exe" while the Windows CPU job (which
skips the CUDA branch) proceeded fine.

Cause: on Windows, nushell exposes the search path as a *list* named
`Path`. The CUDA branch did

    $env.PATH = $"($build_lib_prefix)/bin;($env.PATH)"

which interpolates that list into a string ("[C:\..., C:\...]") under the
separate key `PATH`, shadowing the real `Path` in child processes — so
cmake could no longer find cl.exe. Prepend to the `Path` list instead.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
<details><summary>Claude's draft</summary>

The Windows CUDA 12.9 job now compiles (PATH fix worked) but fails on every
.cu file targeting sm_100a/sm_120a: CCCL's clusterlaunchcontrol.h inline asm
uses long2, and long is 32-bit under MSVC, producing "asm operand type
size(4) does not match constraint 'l'".

main already carries the answer in bld.bat: SM 100+ is broken with CUDA 12.9
on Windows (fixed in 13.0), and SM 110 (Thor) is Linux-only. My 1.28.0
adaptation had wrongly applied the Linux arch lists to all platforms. Split
the arch lists per platform to match main exactly:

- win 12.9: 70..90-real (no 100/120)
- win 13.0: 75..100-real,120 (no 110)
- linux unchanged

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
<details><summary>Claude's draft</summary>

The osx_arm64 jobs failed compiling onnxruntime_autoep_test: the vendored
googletest's gtest-printers.h formats std::chrono::time_point via
std::format, whose floating-point path needs std::to_chars — gated by
libc++ availability to macOS 13.4+, while MACOSX_DEPLOYMENT_TARGET is 11.0.

main never hits this because its meta.yaml keeps gmock in host, so
onnxruntime's FetchContent FIND_PACKAGE_ARGS resolves conda-forge's
gtest/gmock instead of vendoring one. Restore that dep, scoped to osx —
Linux and Windows are green with the vendored copy, and the megabuild
prefers a minimal host env to keep the shared cmake cache stable.

Added to both the staging and python stages to keep their dependency
sets identical (cache-sharing requirement).

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
<details><summary>Claude's draft</summary>

The osx jobs now compile, pass all 10 ctest suites, and install — but fail
at packaging with rattler-build's "No license files were copied". That
error fires when ANY itemized license path matches nothing (the detailed
missing-path list goes to a log level the CI does not surface), and the
set of vendored _deps differs per platform (CoreML adds coremltools/fp16/
psimd on osx; deps provided from conda-forge, e.g. nlohmann_json/gtest on
osx, may not be vendored at all).

Replace the itemized _deps license list with globs:

    build-ci/Release/_deps/*/LICENSE*
    build-ci/Release/_deps/*/COPYING*

This ships the license of every dep actually vendored into the build on
each platform — including several that are compiled in but were previously
not shipped (mp11, date, dlpack, kleidiai, cutlass on cuda) — and cannot
break when the vendored set changes.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
<details><summary>Claude's draft</summary>

Replace the license globs with an explicit list, conditioned on exactly the
variants that vendor each dependency. The per-variant sets were read from
the packaging logs of the passing glob-based CI run:

- base (all variants): LICENSE + abseil, date, dlpack, eigen (COPYING.MPL2
  only, EIGEN_MPL2_ONLY=ON), flatbuffers, gsl, onnx, protobuf,
  pytorch_cpuinfo, re2, safeint
- nlohmann_json: not osx (conda-forge host dep there, not vendored)
- googletest: only native non-CUDA builds (tests off ⇒ never fetched), and
  not osx (conda-forge gtest/gmock there)
- kleidiai: ARM targets (linux-aarch64, osx-arm64)
- microsoft_wil: win
- coremltools/fp16/psimd: osx (CoreML EP)

Verified with rattler-build --render-only that each variant's rendered list
matches what its CI packaging log actually copied. Not itemized: cutlass /
cudnn_frontend on CUDA builds (parity with main, which never shipped them);
can be added once a CUDA packaging log confirms the paths.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
@cbourjau

cbourjau commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for pushing this forward! Are you aware that upstream introduced an execution provider plugin system which should decouple the expensive cuda builds from the Python version, AFAIU.

@hmaarrfk

hmaarrfk commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

I think I remember you mentioning that.

But I did not look into it too deeply.

What would it change in terms of organization here?

@hmaarrfk

hmaarrfk commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Also. Full disclosure. I am trying to help here and threw my AI at your improvements since it is better than me at these problems.

I didn’t have time to explicitly write that at the top level but I have my AI often disclose itself in every commit.

I have not yet reviewed these changes.

@cbourjau

cbourjau commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

I think I remember you mentioning that.

But I did not look into it too deeply.

What would it change in terms of organization here?

It would virtually remove the need for staged outputs, I think. We would compile one Python-independent cuda package per platform, and N fairly cheap Python-dependent frontends that depend on the former.

hmaarrfk added a commit to hmaarrfk/onnxruntime-feedstock that referenced this pull request Aug 10, 2026
<details><summary>Claude's draft</summary>

linux_64 CPU hit the namespace runner's 8h session cap a second time, even
with the full test suite gated to one python. The decisive comparison:
Windows builds all five pythons WITH full per-python tests in ~2.5-3h — and
bld.bat never passed --enable_lto. GCC LTO is the dominant linux cost.

Precedent: the rattler megabuild (conda-forge#196) also built with
onnxruntime_ENABLE_LTO=OFF and its full test suites were green; only the
conda-build recipe on main used LTO on unix. Flagged in the PR as a
deliberate trade-off for reviewers.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
hmaarrfk added a commit to hmaarrfk/onnxruntime-feedstock that referenced this pull request Aug 11, 2026
… tests

<details><summary>Claude's draft</summary>

Root cause of three consecutive linux_64 CPU deaths at exactly the 8h
runner cap: every one carried the annotation "Error reading output: stream
did not contain valid UTF-8". The RegexFullMatch.NonUtf8Pattern gtest
deliberately prints invalid UTF-8 to stderr; rattler-build's output reader
dies on it, the test process then blocks on a full pipe, and the job hangs
until the session cap kills it. The megabuild recipe (conda-forge#196) carried
GTEST_FILTER=-RegexFullMatch.NonUtf8Pattern for exactly this reason
("breaks rattler build output due to invalid utf8 output on stderr") and
this recipe failed to inherit it.

Supporting evidence that builds were never the bottleneck: aarch64 (cross,
no tests) does cpp + 5 pythons in 24 minutes; Windows (different console
pipeline, full tests) passes in hours; only native-gtest unix jobs died.
The round-2 osx death mid-gtests was likely the same hang, not a runner
session cap.

With the true cause fixed, revert the tests-once-per-job mitigation: every
python variant runs the full upstream test suite again.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
hmaarrfk added a commit to hmaarrfk/onnxruntime-feedstock that referenced this pull request Aug 11, 2026
<details><summary>Claude's draft</summary>

With the invalid-UTF-8 gtest hang fixed, the CPU jobs are cheap
(linux_64: 40 min for cpp + 5 pythons with full tests; aarch64 cross:
26 min; osx single-python: 12 min), so the exploratory reductions are no
longer needed:

- novec variants are built again on every CPU platform
- osx builds the full python 3.10-3.14 matrix again (estimated ~40 min)

This is the intended merge state of the matrix: 14 jobs. LTO remains off
(as in the megabuild conda-forge#196); whether to re-enable --enable_lto at ~2-3x CI
cost for parity with the current conda-build binaries is left as an open
review question.

Resume this Claude session:
```
cd /home/mark/git/feedstock/onnxruntime-feedstock
claude --resume bceefe2e-0d08-4ad8-9b4c-3518466f1ccc
```
</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants