Skip to content

Pass -gpu= at link time so the Thrust fatbin keeps every architecture - #17

Merged
hfwen0502 merged 1 commit into
mainfrom
fix-thrust-fatbin-arch-at-link
Sep 3, 2026
Merged

Pass -gpu= at link time so the Thrust fatbin keeps every architecture#17
hfwen0502 merged 1 commit into
mainfrom
fix-thrust-fatbin-arch-at-link

Conversation

@hfwen0502

@hfwen0502 hfwen0502 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

The Thrust extension passes -gpu=<arch> when compiling but not when linking.

That is not a cosmetic omission. The device-link step is where the final SASS is generated — nvc++ keeps device IR in the object file and codegens at link. So without -gpu= there, the architecture requested at compile time is discarded entirely and the artifact gets whatever the toolchain's own default target is. No warning, exit 0.

Measured on NVHPC 26.1 (aarch64, a cc100 host):

compile flag link flag resulting .so
(none) (none) sm_100
-gpu=cc90 (none)today's behaviour sm_100 ← asked cc90, got sm_100
-gpu=cc90 -gpu=cc90 — this PR sm_90

Note the middle row: it is not that a comma-separated list gets truncated, it is that any explicit request is overridden. A single-arch cc90 build is equally affected whenever the build host's default differs — which is why this surfaced as a Thrust-only failure on H100 from a build that had asked for the right architecture. Nothing masks it: these builds embed no PTX, so there is no JIT fallback.

_core_gpu_omp_offload was never affected — its extension already passes -gpu= in extra_link_args. That asymmetry is exactly why the symptom looked backend-specific rather than like a build-flag bug.

Fix

extra_link_args=extra_link_args + ['-mp', '-cuda', f'-gpu={gpu_arch}', '-lcudart'],

One line, matching what the OMP-offload extension already does.

Effect on main

main resolves the arch as _resolve_gpu_arch(default='cc90'), and the README documents that default. With this PR that default becomes effective rather than nominal: an unset SBD_GPU_ARCH now genuinely produces an sm_90 binary, as documented. Today it produces whatever the build host defaults to, which may not be sm_90 at all.

So this PR makes main's behaviour match main's README. It deliberately changes nothing else — no README edits, no change to the default.

Verification on a real build of this package

Multi-arch, on GB200:

build _core_gpu_thrust SASS size
before sm_100 10 M
after sm_80 sm_90 sm_100 16 M

Still numerically correct — run_sbd_diag.py, h2o-1em3, 4 ranks (2×2):

--device gpu      Energy = -76.23586672306634
--device gpu-omp  Energy = -76.23586672306634

matching the published reference for that determinant set.

Notes

  • The kept architecture comes from the toolchain, not from probing hardware: building on a GPU-less host (a container build stage) does not change it.
  • -gpu=ccall-major also works on the Thrust path, yielding sm_80 sm_90 sm_100 sm_110 sm_120 — a reasonable choice for distributed images instead of enumerating.

Follow-ups (deliberately not in this PR)

  1. Assert after build_ext that the architectures in the linked extension match SBD_GPU_ARCH (cuobjdump --list-elf), so a silent arch drop fails the build instead of surfacing on someone else's GPU months later.
  2. Making SBD_GPU_ARCH optional — unset meaning "target the build host", set meaning "honor exactly, including several generations" — is implemented on fix-conda-build-and-mpi-detection and will arrive on main with that branch at the next PyPI release.

The device-link step is where the final SASS is generated, so an arch passed
only at compile time is discarded and the extension silently gets nvc++'s own
default target instead of what was requested.

Measured on NVHPC 26.1 (aarch64), compile with -gpu=cc90:
  .so linked without -gpu=  ->  sm_100   (the host default, not cc90)
  .so linked with    -gpu=  ->  sm_90

_core_gpu_omp_offload already passes -gpu= at link, which is why only the Thrust
backend was affected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hfwen0502
hfwen0502 force-pushed the fix-thrust-fatbin-arch-at-link branch from 1b85c87 to ae463c8 Compare September 3, 2026 20:30
@hfwen0502
hfwen0502 merged commit 6c436c9 into main Sep 3, 2026
6 checks passed
@hfwen0502
hfwen0502 deleted the fix-thrust-fatbin-arch-at-link branch September 3, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant