[BUG] sm120 GEMM hangs when K = 0 (grouped and non-grouped) after can_implement(), initialize() and run() return success #3733

Describe the bug

On sm120, a CUTLASS 3.x GEMM with K = 0 passes can_implement(), and initialize() and run() return kSuccess in the tested release builds. The records show zero-K kernels not completing within 10, 15 or 60 seconds.

• Grouped GEMM, where K = 0 is documented as supported. With the sm120 ptr-array kernel, the recorded K = {128, 0, 256, 0} and {0, 0, 0, 0} groups do not complete within the recorded timeouts, while {128, 128, 128, 128} and {256, 512} pass. The mainloop source indicates that one K = 0 group is sufficient. CHANGELOG 4.3.0 says (CHANGELOG.md L715):

  Enable GEMM_K = 0 in grouped gemm.

  The same case was fixed for Hopper in [BUG] Grouped GEMM with GroupScaling Fails with TMA Error on Hopper When a Problem has K=0 #2547 and for sm100 grouped GEMM in Support for GEMM-K=0 for Blackwell Grouped GEMMs #2746.

• Non-grouped GEMM, a validation gap. The docs don't say whether K = 0 is supported, and a debug build stops at assert(gmem_prob_shape[0] >= 1) in make_tma_copy_desc (copy_traits_sm90_tma.hpp L957). A release build (-DNDEBUG, as the examples use) prints Error: Failed to initialize the TMA descriptor 1 to stderr, initialize() still returns kSuccess, and the launched kernel does not complete before the recorded timeout. With M = 0 or N = 0, the same kernel's run() returns Error Internal instead. Upstream examples 79a (NVFP4) and 79c (MXFP8 x MXFP6) time out at --k=0. Our own tests time out the same way with FP8 (cooperative and pingpong) and NVFP4 cooperative kernels at 128x128, and with FP8 at 300x392 with L = 2 and with beta = 0. They use real (non-null) A/B buffers.

A caller whose K can be 0 at run time (for example a weight-gradient GEMM for an MoE expert that received no tokens) is an illustrative scenario. This makes reachability plausible, not demonstrated for a deployed SM120 framework, product, or protected workload.

The submitted computation does not complete within the recorded timeout. Software waiting for it can lose availability; recovery and impact on other GPU users were not assessed. No confidentiality or integrity failure was demonstrated.

Cause (our reading):

1. The producer issues TMA loads only inside for ( ; k_tile_count > 0; --k_tile_count) (sm120_mma_tma.hpp L400, sm120_blockscaled_mma_tma.hpp L665), so at K = 0 it loads nothing.
2. The consumer mma() clears the accumulators and then calls pipeline.consumer_wait(smem_pipe_read) unconditionally (sm120_mma_tma.hpp L526, sm120_blockscaled_mma_tma.hpp L848). No TMA transaction ever completes that barrier. The array mainloops have the same first wait (sm120_mma_array_tma.hpp L516, sm120_blockscaled_mma_array_tma.hpp L934).
3. The collectives' can_implement() checks only TMA alignment (sm120_mma_tma.hpp L278, sm120_blockscaled_mma_tma.hpp L399). When the descriptor encode fails, copy_traits_sm90_tma.hpp L1066-L1080 prints the error and calls assert(false), which NDEBUG compiles out, so initialize() still reports success.

Other architectures, read, not run (we have no sm90 or sm100 GPU):

• Hopper's ptr-array mainloop guards the first wait with if (k_tile_count > 0) (sm90_mma_array_tma_gmma_ss_warpspecialized.hpp L533). Hopper's non-grouped mainloop has only a debug assert(k_tile_count >= 1) before an unconditional consumer_wait (sm90_mma_tma_gmma_ss_warpspecialized.hpp L481-L487).
• The sm100 mainloops skip the first wait when k_tile_count <= 0 (sm100_mma_warpspecialized.hpp L670, sm100_blockscaled_mma_warpspecialized.hpp L991).

What a one-line guard did. We added if (k_tile_count <= 0) { return; } right after clear(accum) in mma() and rebuilt on the RTX 5090:

• In sm120_mma_tma.hpp and sm120_blockscaled_mma_tma.hpp: the non-grouped K = 0 cases we re-ran completed with D = beta*C, bit-exact against a double-precision host reference (FP8 cooperative at 128x128 and at 300x392 with L = 2, FP8 pingpong, NVFP4 cooperative). Example 79a's own check printed Disposition: Passed at --k=0. The K = 128 controls still passed.
• In sm120_mma_array_tma.hpp: the grouped K = {128, 0, 256, 0} and {0, 0, 0, 0} runs completed with 0 mismatches in every group.

That ties the hang to the unguarded first wait, but it isn't a proposed complete fix. initialize() still prints the TMA descriptor error for the zero-extent operands. We didn't patch or run the block-scaled array mainloop, didn't check the blockwise-scaling or sparse sm120 collectives, and didn't run CUTLASS's unit tests with the guard. For non-grouped GEMM, rejecting K = 0 in can_implement() would be the other option.

Steps/Code to reproduce bug

Upstream example 79a at the pinned commit, with no added code:

nvcc -std=c++17 -O3 --expt-relaxed-constexpr -DCUTLASS_ENABLE_TENSOR_CORE_MMA=1 -DNDEBUG \
  -gencode=arch=compute_120a,code=sm_120a -Iinclude -Itools/util/include -Iexamples/common \
  examples/79_blackwell_geforce_gemm/79a_blackwell_geforce_nvfp4_bf16_gemm.cu -o 79a
timeout 60 ./79a --m=128 --n=128 --k=0 --beta=1 --iterations=0; echo rc=$?   # rc=124, killed by timeout
./79a --m=128 --n=128 --k=128 --beta=1 --iterations=0                        # Disposition: Passed

At --k=0, stderr shows four descriptor dumps like this one, then nothing until timeout kills the process:

globalDim      (0,128,1,1,1)
...
Error: Failed to initialize the TMA descriptor 1

Grouped case: our 146-line test vt_grouped_k0.cu uses the kernel from the first test in test/unit/gemm/device/sm120_tensorop_gemm/sm120_gemm_f8_f8_f32_tensor_op_group_gemm.cu (e4m3 x e4m3 -> f32, 128x128x128 tile, KernelPtrArrayTmaWarpSpecializedPingpongSm120<2>), with the arguments set up as in example 57. Each group is 128 x 128 x K_i, alpha = 1.25, beta = -0.5. No TMA descriptor error is printed in this case.

$ ./vt_grouped_k0 128,0,256,0
can_implement: Success
initialize: Success (workspace 108800)
run (launch): Success
RESULT: TIMEOUT kernel not complete after 15.0 s
$ ./vt_grouped_k0 128,128,128,128
...
RESULT: PASS

We can attach vt_grouped_k0.cu and the two guard diffs. They can be used under CC0 as tests. The 101-line standalone non-grouped FP8 reproducer (k0_repro.cu) polls the stream and prints TIMEOUT instead of hanging. k0_repro.cu includes the separate Stream-K crash mode.

Expected behavior

• Grouped: a K = 0 group writes D = beta*C and the kernel completes.
• Non-grouped: either can_implement() rejects K = 0, or the kernel completes with D = beta*C. It shouldn't hang after every call has reported success.

Environment details (please complete the following information):

• Environment location: Cloud (RunPod), Docker. Reproduced on two different hosts.
• GPU: NVIDIA GeForce RTX 5090 (cc 12.0, 170 SMs), drivers 595.91.07 and 580.173.02.
• nvcc 13.0.88, GCC 13.3.0, Ubuntu 24.04.5. Built with -gencode=arch=compute_120a,code=sm_120a, -O2/-O3 and -DNDEBUG unless stated otherwise.
• CUTLASS 0b55a2f691d69981583568fd9eb69687b1f0de8a (4.8.0; main on 2026-10-11 UTC).

Additional context

We searched NVIDIA/cutlass issues and PRs (open and closed) through the GitHub search API for K=0, "K = 0", "k == 0", zero K, GEMM_K = 0, K=0 hang, sm120 hang, sm120 K=0, sm120 deadlock, sm120 grouped K=0, geforce hang, RTX 5090 hang, consumer_wait hang, "Failed to initialize the TMA descriptor", k_tile_count, skip_wait, can_implement zero and similar terms, and found no existing report. Related but different: #2547 (Hopper grouped GEMM hang on a K = 0 group, fixed in 4.2), #2674 and #2746 (sm100 grouped K = 0), #1593 (2.x multistage mainloops at k = 0), and #3542 (other can_implement gaps, in the 2.x wrappers).

Found by ChipLab's AI verification agents. Reproduced independently on a second RTX 5090 host.

_{ChipLab ref: CLAB-2026-072}