You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] CuTe DSL: cute.compile stalls for minutes when no GPU is visible #3681
With no GPU visible and CUTE_DSL_ARCH unset, cute.compile stalls for minutes with no output. It then compiles for the default arch. On 4.8.0 a trivial kernel took 773 s; on 4.7.0 it took 325 s.
Arch detection recurses through its own error message:
env_manager.arch calls detect_gpu_arch, which calls cuInit.
cuInit fails with CUDA_ERROR_NO_DEVICE, so checkCudaErrors raises DSLCudaRuntimeError.
That error's constructor calls _get_friendly_cuda_error_message, which reads env_manager.arch again. _arch is still unset, so this is step 1 again.
Each round pays a failing cuInit, about 2 s here, until Python's recursion limit. Then the except in detect_gpu_arch unwinds it. A faulthandler dump shows these frames repeating:
$ CUDA_VISIBLE_DEVICES= python repro.pycutlass-dsl 4.8.0: compiled in 773 s
$ CUDA_VISIBLE_DEVICES= CUTE_DSL_ARCH=sm_100a python repro.pycutlass-dsl 4.8.0: compiled in 0 s
Expected behavior
With no device, compile quickly for the default arch, or raise an error that points at CUTE_DSL_ARCH. Building an error message shouldn't re-enter device detection.
Environment details (please complete the following information):
Environment location: Bare-metal, run with CUDA_VISIBLE_DEVICES= to hide the GPU
nvidia-cutlass-dsl 4.8.0 and 4.7.0 (cu13), cuda-python 13.4.1
Activity