Skip to content

test: xfail regression for #1051 (bwds crash on runtime-start diff loop) + docs workaround - #1053

Draft
nv-slang-bot[bot] wants to merge 1 commit into
mainfrom
dev/slangpy-fixer/1051
Draft

test: xfail regression for #1051 (bwds crash on runtime-start diff loop) + docs workaround#1053
nv-slang-bot[bot] wants to merge 1 commit into
mainfrom
dev/slangpy-fixer/1051

Conversation

@nv-slang-bot

@nv-slang-bot nv-slang-bot Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Regression coverage + a documented workaround for #1051, where backward dispatch (.bwds()) of a [Differentiable] function crashes when the function body contains a loop with a runtime (non-constant) induction start, e.g. for (int dx = -radius; dx <= radius; ++dx) where radius is a no_diff runtime param.

This PR does not fix the crash — the root cause is upstream in the Slang compiler's reverse-mode autodiff (tracked as shader-slang/slang#12070), and SlangPy cannot fix it: it hands the user function verbatim to bwd_diff and does no loop/induction rewriting. Confirmed against the generated kernel (SLANGPY_PRINT_GENERATED_SHADERS=1): the bwds kernel prepends [Differentiable] to a _trampoline, calls the user function unchanged, then dispatches bwd_diff(_trampoline).

The trigger is the loop start being non-constant, not negative. A constant negative start such as for (int dx = -2; dx <= 2; ++dx) differentiates fine; the reporter's -radius was incidentally both runtime and negative. Reproduced on CUDA and Vulkan (SIGSEGV, exit 139); the constant-start controls produce correct gradients.

Uses Refs (not Fixes) intentionally: the issue must stay open to track the upstream fix. It will be closed by the upstream Slang change, not by merging this.

Refs #1051
Refs shader-slang/slang#12070

Changes

  • slangpy/tests/slangpy_tests/test_diff_loop_runtime_start.py (new):
    • test_diff_loop_runtime_start runs the crashing runtime_start bwds in an isolated subprocess (a SIGSEGV is a hard native crash that would otherwise take down the pytest worker / disrupt pytest-xdist). It brackets the .bwds() dispatch with begin/end markers so a crash is attributed to the backward dispatch specifically (not setup/readback/teardown). Marked xfail(raises=_KnownCrash, strict=True):
      • today's native crash inside the dispatch → XFAIL;
      • a wrong-gradient result (compared element-wise against an independent numpy oracle) → hard FAIL, not masked;
      • a clean child exit once the upstream fix lands → strict XPASS → hard FAIL, forcing removal of the marker + subprocess scaffolding.
    • test_diff_loop_constant_start runs two in-process controls that must always pass: const_neg_start (constant -2 → shows negative-ness is not the trigger) and zero_start (the documented workaround). Together they isolate the trigger to runtime-ness.
  • docs/src/autodiff/autodiff.rst: a note documenting the constant-start workaround, cross-linking [Bug] Backward dispatch segfaults when a [Differentiable] function has a loop with a negative runtime start value #1051 and slang#12070.

Verification

  • pytest slangpy/tests/slangpy_tests/test_diff_loop_runtime_start.py4 passed, 2 xfailed on CUDA + Vulkan, exit 0; identical under pytest-xdist -n2.
  • Parent-side returncode→outcome mapping verified: crash-in-dispatch→XFAIL, exit0→XPASS/fail, wrong-grad→fail, no-device→skip, crash-outside-dispatch/other→fail.
  • black clean, pure ASCII (satisfies check-ascii-source).

5-bullet status

🤖 Generated by an automated SlangPy coworker — may be inaccurate. A human maintainer should verify.

@nv-slang-bot
nv-slang-bot Bot requested a review from a team as a code owner July 12, 2026 17:03
@nv-slang-bot
nv-slang-bot Bot requested review from bmillsNV and removed request for a team July 12, 2026 17:03
@nv-slang-bot
nv-slang-bot Bot marked this pull request as draft July 12, 2026 17:17
nv-slang-bot Bot added a commit that referenced this pull request Jul 13, 2026
The runtime_start test deliberately SIGSEGVs a GPU subprocess. On CI unit-test
lanes SlangPy's Crashpad handler is active (crashpad flag), and it intercepts
the intentional crash to capture a minidump -- which wedges the Python test step
for hours (observed 2h+ on linux/windows, 6h on macos on PR #1053) instead of
exiting cleanly. Locally crashpad is disabled (is_supported() == False) so the
crash was clean and the hang did not reproduce.

Skip the crashing subprocess case when spy.crashpad.is_supported(). The
constant-start controls (const_neg_start, zero_start) still run on every lane
and carry the regression value; the crashing tripwire runs on non-Crashpad
builds and local dev, where a clean SIGSEGV -> XFAIL, and XPASS once the upstream
fix (shader-slang/slang#12072) lands.

Refs #1051
…wds crash)

Backward dispatch (.bwds()) of a [Differentiable] function crashes when the body
contains a loop whose start value is a runtime (non-constant) induction start,
e.g. `for (int dx = -radius; dx <= radius; ++dx)` with radius a runtime no_diff
param. The trigger is the start being non-constant, not negative: a constant
negative start (-2) differentiates fine. Root cause is upstream in Slang
reverse-mode autodiff (shader-slang/slang#12070, fixed by shader-slang/slang#12072),
not SlangPy, which hands the function body verbatim to bwd_diff.

Add two constant-start control tests (const_neg_start, zero_start) that
differentiate correctly, compared element-wise against an independent numpy
oracle on a non-square shape. They pin the #1051 trigger to runtime-ness and
guard the documented zero-based workaround. The crashing runtime-start form is
not exercised as a live test (it is a hard native crash; documented in the test
docstring and docs/src/autodiff/autodiff.rst instead). Document the workaround
in the autodiff docs.

Refs #1051
Refs shader-slang/slang#12070
Refs shader-slang/slang#12072
@nv-slang-bot
nv-slang-bot Bot force-pushed the dev/slangpy-fixer/1051 branch from bc4e5ba to 0fb3344 Compare July 13, 2026 20:48
@nv-slang-bot

nv-slang-bot commented Jul 14, 2026

Copy link
Copy Markdown

Status: this PR is BLOCKED on a CI/test-harness issue that is not a defect in this PR's code — do not merge; held pending a maintainer/operator decision.

🤖 Generated by an automated SlangPy coworker — may be inaccurate. A human maintainer should verify.

@jhelferty-nv jhelferty-nv self-assigned this Jul 14, 2026
@jhelferty-nv
jhelferty-nv requested review from ccummingsNV, jhelferty-nv and saipraveenb25 and removed request for bmillsNV July 14, 2026 15:36
@jhelferty-nv

Copy link
Copy Markdown
Contributor

Assigning over to @saipraveenb25 since he owns shader-slang/slang#12070 where this will ultimately need to be fixed

@nv-slang-bot

nv-slang-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Update — the upstream root cause is fixed; this PR's remaining gate is the bundled-Slang pin, not a release.

Correcting my 2026-07-14 note above, which is now stale in three ways:

  • Upstream fix landed. shader-slang/slang#12299 ("Preserve runtime induction values in reverse differentiation", @saipraveenb25) merged to master 2026-08-03T21:28:38Z as 546ad18f. It supersedes the earlier draft slang#12072. slang#12070 and slangpy#1051 are both closed.
  • The gate for re-adding the runtime_start case is the SGL_SLANG_VERSION bump — not "the next wheel". SlangPy bundles Slang as a release tarball pinned at external/CMakeLists.txt:85, currently 2026.12 (last bumped 2026-06-30, Update to slang 2026.12 #1037). 546ad18f is 25 commits ahead of the newest Slang tag v2026.14.1, so it is in no released tarball yet. A wheel cut today would still bundle Slang 2026.12 and the test would fail for the original reason. Sequence: merge ✅ → Slang tag containing 546ad18f ⏳ → slangpy pin bump ⏳.
  • Earlier verification is available without waiting for either. ci-latest-slang.yml builds slangpy main against Slang master; the two repository_dispatch runs it fired for slang#12299 (30836651948, 30848628629) both passed, GPU Unit Tests (Python) included. A workflow_dispatch with slang_branch=master would confirm the fix against slangpy before any tag exists.

On the CI blocker (#1070), one caveat worth flagging before assuming this PR is unblocked. #1070 was closed as "the race condition in profiling", but two things don't fit, and I'd rather raise them than quietly rely on them:

Across 07-29 → 08-03 there were 46 ci runs on 14 branches with zero wedges (Unit Tests (Python) median 1.70 min, max 7.00 min), so the lane looks healthy — but the signature only ever fired on dev/slangpy-fixer/1051, which hasn't run CI since 07-13, and its test file never reached main. That reads as unexercised, not proven fixed. Re-running this PR's CI is the measurement; I'm not assuming either outcome. Note the structural gap is still open on main: Unit Tests (Python) has no timeout-minutes, and tools/ci.py still uses --maxprocesses=4 (#1024 closed unmerged), so a recurrence would again burn 6h with no signal.

Suggested next step, @saipraveenb25 / @jhelferty-nv — since this PR is assigned to you, flagging rather than pushing: the docs half (the autodiff.rst workaround note) is worth landing on its own merits, though its "Until the upstream fix lands" wording should become past-tense now that #12299 is in. I can (a) rebase onto main and re-run CI to re-measure the hang, (b) update the docs wording, and/or (c) add runtime_start as a plain passing test once the pin moves to a Slang tag containing 546ad18f. Happy to do any of these on your word — no changes pushed in the meantime.

🤖 Generated by an automated SlangPy coworker — may be inaccurate. A human maintainer should verify.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Backward dispatch segfaults when a [Differentiable] function has a loop with a negative runtime start value

4 participants