tests: gate cooperative-matrix tests on device feature support - #12734
tests: gate cooperative-matrix tests on device feature support#12734jvepsalainen-nv wants to merge 1 commit into
Conversation
Tests in tests/neural/ and tests/cooperative-matrix/ that emit cooperative matrix SPIR-V ran their Vulkan variants unconditionally, without declaring -render-feature cooperative-matrix. On a device that reports no cooperative matrix support the shader is still submitted, and the driver faults while executing it: on Mesa lavapipe this kills the slang-test process with SIGSEGV or SIGABRT after the subtest has already reported success, so the remainder of the directory never runs. Add -render-feature cooperative-matrix to the Vulkan execution lines of every test whose emitted SPIR-V contains cooperative matrix instructions, matching the gating convention already used elsewhere in tests/cooperative-matrix/. Devices that report the feature are unaffected and still run these tests.
📝 WalkthroughWalkthroughChangesVulkan test directives across the cooperative-matrix and neural suites now pass Cooperative matrix Vulkan tests
Neural Vulkan tests
Suggested reviewers: Merge Risk: ⚪ Minimal · up to The change makes unsupported cooperative-matrix Vulkan tests skip instead of crashing the test process. The remaining issue is limited to line-length formatting in ten test directives and has no runtime or product impact, so no actionable merge-blocking risk remains after normal checks and review. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (61 skipped: 61 unsupported.) ✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: b8934676-50ed-48d6-9279-289e929a9181
📒 Files selected for processing (61)
tests/cooperative-matrix/add.slangtests/cooperative-matrix/array.slangtests/cooperative-matrix/bfloat16-arith.slangtests/cooperative-matrix/bfloat16-comparison.slangtests/cooperative-matrix/bfloat16.slangtests/cooperative-matrix/coherent-load-store-pointer.slangtests/cooperative-matrix/comparison.slangtests/cooperative-matrix/conversion-1.slangtests/cooperative-matrix/conversion.slangtests/cooperative-matrix/copyFrom.slangtests/cooperative-matrix/div.slangtests/cooperative-matrix/fill.slangtests/cooperative-matrix/fp8.slangtests/cooperative-matrix/inout.slangtests/cooperative-matrix/length.slangtests/cooperative-matrix/load-store-arbitrary-array-vec.slangtests/cooperative-matrix/load-store-arbitrary-array.slangtests/cooperative-matrix/load-store-groupshared.slangtests/cooperative-matrix/load-store-pointer.slangtests/cooperative-matrix/load-store-rwbyteaddressbuffer.slangtests/cooperative-matrix/load-store-rwstructuredbuffer.slangtests/cooperative-matrix/load-store.slangtests/cooperative-matrix/mat-mul-add.slangtests/cooperative-matrix/mod.slangtests/cooperative-matrix/mod1.slangtests/cooperative-matrix/mul.slangtests/cooperative-matrix/out.slangtests/cooperative-matrix/parameter.slangtests/cooperative-matrix/return.slangtests/cooperative-matrix/scalar-mul.slangtests/cooperative-matrix/struct.slangtests/cooperative-matrix/sub.slangtests/cooperative-matrix/subscript-in-func.slangtests/cooperative-matrix/subscript.slangtests/cooperative-matrix/unary_neg.slangtests/neural/basic-coopmat-vector-test.slangtests/neural/basic-coopmat-vector-tiled-layout-test.slangtests/neural/bias-sum-reduce.slangtests/neural/fflayer-wavetangled-vector-test.slangtests/neural/fflayer-wavetangled-vector-tiled-test.slangtests/neural/mma-helper-test-multi-warps-arbitrary-size.slangtests/neural/mma-helper-test-multi-warps.slangtests/neural/mma-helper-test-single-warp-arbitrary-size.slangtests/neural/mma-helper-test-single-warp.slangtests/neural/mma-helper-test-transpose-multi-warps-arbitrary-size.slangtests/neural/mma-helper-test-transpose-multi-warps.slangtests/neural/mma-helper-test-transpose-single-warp-arbitrary-size.slangtests/neural/mma-helper-test-transpose-single-warp.slangtests/neural/mma-tiled-backward-test.slangtests/neural/mma-tiled-layout-test-multi-warps-arbitrary-size.slangtests/neural/mma-tiled-layout-test-multi-warps.slangtests/neural/mma-tiled-layout-test-single-warp-arbitrary-size.slangtests/neural/mma-tiled-layout-test-single-warp.slangtests/neural/mma-tiled-layout-test-transpose-multi-warps-arbitrary-size.slangtests/neural/mma-tiled-layout-test-transpose-multi-warps.slangtests/neural/mma-tiled-layout-test-transpose-single-warp-arbitrary-size.slangtests/neural/mma-tiled-layout-test-transpose-single-warp.slangtests/neural/outerproduct-accumulate-test-arbitrary-size.slangtests/neural/outerproduct-accumulate-test.slangtests/neural/outerproduct-accumulate-tiled-test-arbitrary-size.slangtests/neural/outerproduct-accumulate-tiled-test.slang
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.
| //TEST(compute):COMPARE_COMPUTE(filecheck-buffer=CHECK):-vk -emit-spirv-directly -render-feature cooperative-matrix | ||
| //TEST(compute):COMPARE_COMPUTE(filecheck-buffer=CHECK):-vk -emit-spirv-directly -Xslang -DBAB -render-feature cooperative-matrix |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Keep the added Vulkan directives within 100 columns.
The same added option makes the changed //TEST directives exceed the Slang line-length limit.
tests/cooperative-matrix/mod1.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/mul.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/out.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/parameter.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/return.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/scalar-mul.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/struct.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/sub.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/subscript-in-func.slang#L1-L2: wrap the two Vulkan directives.tests/cooperative-matrix/subscript.slang#L1-L2: wrap the two Vulkan directives.
As per coding guidelines, follow a 100-column line limit in Slang files.
📍 Affects 10 files
tests/cooperative-matrix/mod1.slang#L1-L2(this comment)tests/cooperative-matrix/mul.slang#L1-L2tests/cooperative-matrix/out.slang#L1-L2tests/cooperative-matrix/parameter.slang#L1-L2tests/cooperative-matrix/return.slang#L1-L2tests/cooperative-matrix/scalar-mul.slang#L1-L2tests/cooperative-matrix/struct.slang#L1-L2tests/cooperative-matrix/sub.slang#L1-L2tests/cooperative-matrix/subscript-in-func.slang#L1-L2tests/cooperative-matrix/subscript.slang#L1-L2
Source: Coding guidelines
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: 🟡 Has issues — 2 gap(s)
Test-only PR that appends -render-feature cooperative-matrix to the Vulkan (-vk) execution lines of tests/cooperative-matrix/*.slang and tests/neural/*.slang, so devices lacking VK_KHR_cooperative_matrix skip these tests (reported ignored) instead of segfaulting the whole slang-test process. The gating mechanism is correct: the flag was added only to -vk lines (CUDA/Metal/compile-only -target spirv lines untouched), every -vk line within each changed file was gated, and on capable devices the tests still run (a supported required-feature does not skip). The gaps below are about completeness of the sweep, not the mechanism.
Changes Overview
Cooperative-matrix test gating (tests/cooperative-matrix/*.slang — 35 files)
- What changed: each
-vkCOMPARE_COMPUTEline gained-render-feature cooperative-matrix. Two files (bfloat16-arith.slang,fp8.slang) now carry both-render-features <x>and-render-feature cooperative-matrixon one line.
Neural test gating (tests/neural/*.slang — 25 files)
- What changed:
-vklines of coopmat-backed tests (*-coopmat-*,*-wavetangled-*,mma-*,outerproduct-accumulate-*) gained-render-feature cooperative-matrix; CUDA/Metal lines left as-is.
Findings (2 total)
| Severity | Location | Finding |
|---|---|---|
| 🟡 Gap | tests/neural/activation-coopmat-vector-test.slang:6 |
Coopmat WaveTangledVector Vulkan test left un-gated and absent from the PR; still segfaults on non-coopmat devices (commented on the gated sibling fflayer-wavetangled-vector-test.slang:6) |
| 🟡 Gap | tests/cooperative-matrix/bfloat16-arith.slang:1 |
Mixed -render-features/-render-feature spellings on one line (also fp8.slang:1) — functional but inconsistent |
reviewed: 7473c7c · diff sha256 d52f6044afd8
| // LinearLayout — single-warp: | ||
| // TEST(compute, vulkan):COMPARE_COMPUTE_EX(filecheck-buffer=BUFFER):-vk -compute -shaderobj -xslang -experimental-feature -output-using-type -xslang -DTEST_POINTER=0 -xslang -DWARP_COUNT=1 -xslang -DTEST_TENSORVIEW=0 -emit-spirv-directly | ||
| // TEST(compute, vulkan):COMPARE_COMPUTE_EX(filecheck-buffer=BUFFER):-vk -compute -shaderobj -xslang -experimental-feature -output-using-type -xslang -DTEST_POINTER=1 -xslang -DWARP_COUNT=1 -xslang -DTEST_TENSORVIEW=0 -emit-spirv-directly | ||
| // TEST(compute, vulkan):COMPARE_COMPUTE_EX(filecheck-buffer=BUFFER):-vk -compute -shaderobj -xslang -experimental-feature -output-using-type -xslang -DTEST_POINTER=0 -xslang -DWARP_COUNT=1 -xslang -DTEST_TENSORVIEW=0 -emit-spirv-directly -render-feature cooperative-matrix |
There was a problem hiding this comment.
🟡 Gap: A sibling cooperative-matrix Vulkan test was missed by the sweep
This file's -vk WaveTangledVector lines are correctly gated here, but its close sibling tests/neural/activation-coopmat-vector-test.slang was left un-gated and is not in this PR. That file's own header documents that it uses the same coopmat-backed type, yet its -vk line still has no -render-feature cooperative-matrix:
// Test activation functions with CooperativeMatrix accelerated WaveTangledVector type.
// ...
// Only run on CUDA and Vulkan (SPIR-V) which support cooperative matrix operations.
//TEST(compute, vulkan):COMPARE_COMPUTE_EX(filecheck-buffer=BUFFER):-vk -compute -shaderobj -xslang -experimental-feature -output-using-type -emit-spirv-directly
By the PR's own membership criterion (gate every test whose emitted SPIR-V contains cooperative-matrix instructions), this WaveTangledVector-on-Vulkan test qualifies. Leaving it un-gated means it still submits a coopmat shader to a device lacking VK_KHR_cooperative_matrix and segfaults the slang-test process — the exact failure mode this PR exists to prevent, taking down every subsequently-scheduled test.
Suggested fix: Append -render-feature cooperative-matrix to activation-coopmat-vector-test.slang:6 (the -vk line only), matching this file.
While here, it's worth re-running the SPIR-V-membership check over the other untouched neural -vk tests in the same MMA family — the tests/neural/tiled-mma-load-test-*.slang set (testLoadShA/testLoadShB) — and gating any whose SPIR-V contains OpTypeCooperativeMatrixKHR/OpCooperativeMatrix*.
| @@ -1,4 +1,4 @@ | |||
| //TEST(compute):COMPARE_COMPUTE(filecheck-buffer=CHECK):-vk -emit-spirv-directly -output-using-type -capability spvBFloat16KHR -render-features bfloat16 | |||
| //TEST(compute):COMPARE_COMPUTE(filecheck-buffer=CHECK):-vk -emit-spirv-directly -output-using-type -capability spvBFloat16KHR -render-features bfloat16 -render-feature cooperative-matrix | |||
There was a problem hiding this comment.
🟡 Gap (consistency): mixed -render-features / -render-feature on one line
This line now carries both the plural -render-features bfloat16 and the singular -render-feature cooperative-matrix. tools/render-test/options.cpp:154 treats the two spellings as aliases and both split on commas, so this is functionally correct — but mixing them on one directive reads as accidental. tests/cooperative-matrix/fp8.slang:1 has the same pattern (-render-features fp8 -render-feature cooperative-matrix).
Suggestion: collapse into a single plural form — -render-features bfloat16,cooperative-matrix (and -render-features fp8,cooperative-matrix for fp8.slang) — so each line declares its features once.
2 participants
1. Motivation
Running the neural test directory on a machine whose Vulkan device does not implement
cooperative matrices does not skip those tests — it kills the test process.
The subtest reports passed, and the process then dies. Reproduced 20/20 on Mesa lavapipe
(Mesa 25.2.8 / LLVM 20.1.2), Ubuntu 24.04, Release, clang-18.
This is not one stray test. Running the whole directory:
Because the crash takes down the
slang-testprocess itself, everything scheduled after it islost. Sweeping every test in the directory individually, 19 of 60 exit non-zero (SIGSEGV 139 or
SIGABRT 134).
tests/cooperative-matrix/behaves the same way.The device is not at fault, and neither is the driver. Querying it directly:
Lavapipe correctly reports that it does not support cooperative matrices. Slang submits a
cooperative-matrix shader to it anyway; the driver cannot lower the intrinsics (it prints
Unsupported intrinsic: @cmat_muladd, a string that lives inlibvulkan_lvp.so), emits broken JITcode, and a worker thread later jumps into it. The faulting PC sits in an anonymous
r-xpmappingwith no backing file and frame #1 is
0x0— a jump into freed JIT memory.The tests never declared that they need the feature. That is the bug.
2. Proposed solution
Declare the requirement, using the mechanism the repository already has for exactly this.
-render-feature <name>makes render-test checkIDevice::hasFeature()and returnSLANG_E_NOT_AVAILABLEwhen it is missing, which slang-test reports as ignored. It is alreadyused 301 times across the suite — including for
cooperative-vector(118 uses) and, intests/cooperative-matrix/itself, for several cooperative-matrix sub-features. Thecooperative-matrix tests are simply the ones that omitted it.
This PR adds
-render-feature cooperative-matrixto the Vulkan execution lines of every test whoseemitted SPIR-V actually contains cooperative-matrix instructions.
Membership was determined by compiling each candidate to SPIR-V with the flags from its own vk
TESTline and checking forOpTypeCooperativeMatrixKHR/OpCooperativeMatrix*, rather than bygrepping the source or by taking the list of tests that happened to crash. That distinction matters:
only 6 files in
tests/neural/mention a cooperative-matrix identifier at all (the rest reach itthrough
import slang.neural), and the crashing set (19) is smaller than the set that actuallyuses the feature (26). Tests like
mma-tiled-backward-test.slanguse cooperative matrices and didnot crash in a given run — gating only the crashers would have left a latent failure behind.
3. Change summary
tests/neural/*.slang-render-feature cooperative-matrixto vk execution linestests/cooperative-matrix/*.slangTESTlinesOnly lines that (a) are execution tests (
COMPARE_COMPUTE*/COMPARE_RENDER*/EXECUTABLE),(b) target
-vk, and (c) did not already declare a cooperative-matrix feature were touched.-cuda,-dx12and-mtllines are deliberately left alone — see the process report.4. Concepts and vocabulary
-render-feature <name>— a render-test option (accepted also as-render-features, taking acomma-separated list) that names an
rhi::Featurethe test requires. Unmet ⇒ render-test returnsSLANG_E_NOT_AVAILABLE⇒ slang-test marks the test ignored, not failed.that crashes the process is worse than either, because it also destroys the results of every test
that would have run after it.
executable memory, which is why a mis-lowered shader shows up as a PC in an unnamed
r-xpregion.5. Process report
Why the gate is the right layer, and not a driver work-around or an expected-failure entry.
The input shape here is a Vulkan device that reports
cooperativeMatrix = 0. That shape iscorrect and principled — it is a device honestly reporting a capability it lacks, and it is the
shape any non-cooperative-matrix device presents. The defect is on the consumer side: the test
declared no requirement, so the harness had nothing to check and submitted the shader regardless.
Fixing this in an expected-failure list would be wrong twice over: it would encode "this test
crashes" as acceptable, and expected-failure lists mark test outcomes, whereas a SIGSEGV takes
out the process and the rest of the run with it. Fixing it in the driver is not available to us and
would not be right anyway.
Why not gate at the module or compiler level. One could imagine
slang.neuraldeclaring acapability so compilation fails early. That is a separate design question and would not remove the
need for this change: compilation succeeding says nothing about whether the device this test is
about to run on implements the extension. The runtime feature check is the only thing that can
answer that, and
-render-featureis how this repository asks it.Why the SPIR-V-derived list rather than the crash list. Described in §2. The crash set is
timing-dependent —
basic-coopmat-vector-test.slangpasses at the default lavapipe thread count andsegfaults at
LP_NUM_THREADS=28. A list derived from observed crashes would therefore be unstablebetween runs and machines. Emitted SPIR-V is a property of the test, not of the run.
Why
-vklines only. The failure is demonstrated on Vulkan, andrhi::Feature::CooperativeMatrixis reported by the Vulkan backend from
cooperativeMatrix1Features.cooperativeMatrix. Whether theCUDA and D3D12 backends report the same feature for the paths those
TESTlines exercise was notverified here, and adding the gate to a line whose backend does not report the feature would
silently disable a test that currently works — the exact failure mode PR 2 is about. Restricting to
-vkkeeps this change to what is evidenced.No coverage is lost. The gate is a runtime check: on a device that reports
cooperative-matrix,hasFeature()returns true and the test runs exactly as before. Only devicesthat already could not execute these shaders are affected, and there the outcome changes from
"crash the run" to "ignored".
6. Verification
All on Mesa lavapipe 25.2.8, Ubuntu 24.04 / glibc 2.39, Release, clang-18.
basic-coopmat-vector-tiled-layout-test.slangslang-test tests/neural/slang-test tests/cooperative-matrix/claim that these tests still run there rests on the gate being a
hasFeature()check that returnstrue on such a device. CI on NVIDIA hardware is the check for that, and is the one thing a reviewer
should confirm before merging.
Formatting:
./extras/formatting.shcould not run here (clang-format, prettier, shfmt notinstalled). The change is confined to
//TESTcomment lines in.slangfiles, which none of thoseformatters rewrite.