CI: GPU "Unit Tests (Python)" step wedges to the 6h timeout with a specific test file (surfaced in #1053)

[nv-slang-bot]

Summary

The Unit Tests (Python) step of the ci workflow hangs to the 6-hour job timeout on the GPU-backed runners when a specific ~140-line test file is present, even though that file's tests pass locally in seconds and contain no obvious pathological construct. Surfaced while adding regression coverage for #1051 (see PR #1053); filing separately because it is an independent CI/test-harness problem, not specific to that change's correctness.

Observed behaviour (PR #1053, run 29283873237, HEAD 0fb3344, rebased 0-behind main)

Lane	Unit Tests (Python) outcome
macOS aarch64 (Release + Debug)	cancelled at 360 min (6h job timeout)
linux x86_64 gcc, windows x86_64 msvc	killed ~128 min mid-step
linux aarch64 (no GPU test load)	pass, ~4–5 min

The whole GPU Unit Tests (Python) step wedges — not a single failing assertion — so no useful failure output is produced; the job just runs to timeout.

What has been ruled out

• Not the deliberately-crashing test. [Bug] Backward dispatch segfaults when a [Differentiable] function has a loop with a negative runtime start value #1051's original regression test intentionally SIGSEGV'd inside a subprocess; it was dropped entirely. The hang persists with only two ordinary constant-start .bwds() control tests + a docstring remaining.
• Not a stale base. Rebasing dev/slangpy-fixer/1051 onto current main (0 commits behind) did not change the behaviour.
• Not a general CI outage / not fleet-wide. ~11 other contemporaneous open PRs pass the identical 12-job matrix (same tools/ci.py --maxprocesses=4) in ~12–17 min with zero hang.
• Not obviously CUDA OOM. The macOS-aarch64 lane (Metal, no CUDA) is among those that hang, so a CUDA-specific out-of-memory cascade does not explain it on its own.

Why it needs runner access to diagnose

With the above ruled out, the remaining question — why a small test file that passes locally wedges the entire GPU Unit Tests (Python) step for hours — is not reproducible without access to the GPU CI runners (process state, driver/Crashpad interaction, pytest-xdist worker behaviour under load). It is a latent hazard: the next PR that adds or perturbs GPU-exercising Python tests may hit the same 6h wedge with no clear signal.

References

• Blocked coverage PR: test: xfail regression for #1051 (bwds crash on runtime-start diff loop) + docs workaround #1053 (held draft; regression test for [Bug] Backward dispatch segfaults when a [Differentiable] function has a loop with a negative runtime start value #1051 + docs workaround)
• Underlying functional bug (fixed upstream, unrelated to this harness issue): [Bug] Backward dispatch segfaults when a [Differentiable] function has a loop with a negative runtime start value #1051 → [autodiff] bwd_diff of a [Differentiable] loop with a runtime (non-constant) induction start crashes (SPIR-V ICE / CUDA SIGSEGV) slang#12070 → Fix reverse-mode autodiff for runtime induction start (#12070) slang#12072

_{🤖 Generated by an automated SlangPy coworker — may be inaccurate. A human maintainer should verify.}

[jhelferty-nv]

@nv-slang-bot Have more failures like this occurred since? Can you triage and see if we have any updated/additional information to help narrow this down?

[nv-slang-bot]

@jhelferty-nv — triaged. Short version: no new occurrences of this signature since filing, but that's because the trigger is parked, not because anything upstream resolved. Details below.

1. Has this recurred? — No new hits of this signature

Read-only sweep of the last ~60 ci runs across all branches, 2026-07-14 → today:

• No Unit Tests (Python) GPU-lane wedge has recurred. The only run matching CI: GPU "Unit Tests (Python)" step wedges to the 6h timeout with a specific test file (surfaced in #1053) #1070's exact signature is still the original — run 29283873237 on dev/slangpy-fixer/1051 (macOS lanes cancelled at 6h; linux-gcc / windows-msvc killed ~2h9m mid-step).
• That branch (PR test: xfail regression for #1051 (bwds crash on runtime-start diff loop) + docs workaround #1053) has not re-run CI since 2026-07-13 — it was parked as a draft ~1 min after this issue was split out. So the absence of recurrence is "the trigger hasn't been re-exercised," not "the hazard is gone." Any PR that adds/perturbs GPU-exercising Python tests can still hit it.
• ⚠️ One unrelated 6h-timeout did occur in-window — run 29460256843 (dev/slangpy-fixer/1067, 07-16), a single macOS-Debug lane. But it wedged on Unit Tests (C++), a different step (Unit Tests (Python) was never reached). It shares only the coarse "job rode to the 6h cap" symptom, not this signature — flagging it as a possibly-distinct latent hazard worth its own look, not a recurrence of CI: GPU "Unit Tests (Python)" step wedges to the 6h timeout with a specific test file (surfaced in #1053) #1070.

2. Updated / additional info

• The upstream fix has not landed. shader-slang/slang#12070 is still open, and its proposed fix shader-slang/slang#12072 is still an open draft PR (not merged). So the trigger's root cause is unchanged; #12070 is now maintainer-owned (@saipraveenb25).
• Nothing has rolled into SlangPy's Slang. SlangPy still pins SGL_SLANG_VERSION 2026.12 (external/CMakeLists.txt:85), last bumped 2026-06-30 (Update to slang 2026.12 #1037), no bump since. Since #12072 isn't even merged, the fix definitively isn't in the pinned Slang.
• PR test: xfail regression for #1051 (bwds crash on runtime-start diff loop) + docs workaround #1053 is still draft/parked (controls-only test file + docs; last touched 2026-07-17), so its controls-only file can't be confirmed to still reproduce without re-running CI on a GPU runner — and per the original filing this isn't reproducible without runner access.

The narrowing point — this is a harness gap, independent of the upstream fix

The reason a single hung GPU dispatch burns 6h with zero signal is that there is no sub-6h guardrail anywhere in the Python test path (verified on live main today):

• .github/workflows/ci.yml:218-220 — the Unit Tests (Python) step (and its build job) set no timeout-minutes, so a hang rides to GitHub Actions' 360-min job default.
• tools/ci.py:151-157 runs pytest slangpy/tests -vra -n auto --maxprocesses=4; pytest-timeout is not a dependency and pyproject.toml sets no --timeout. A hung xdist worker never returns and reports nothing.

Concrete suggestion, independent of whether/when slang#12072 lands:

1. Step-level timeout-minutes (~30–45 min) on Unit Tests (Python) — the reliable, backend-agnostic backstop (works on the Metal lane too, where a thread-based pytest timeout can't interrupt a native GPU call). Healthy runtime is ~12–17 min, so there's ample margin.
2. pytest-timeout with a per-test --timeout — for attribution, so the next occurrence fails fast against a named test in one run instead of six hours of silence.

That converts the wedge from a 6h silent burn into a fast, attributable failure and makes the offending case bisectable. Happy to open that as a small CI-hardening PR if you'd like — it's orthogonal to the autodiff fix and would de-risk the whole GPU Python-test lane, not just this one file.

_{🤖 Generated by an automated SlangPy coworker — may be inaccurate. A human maintainer should verify.}

[jhelferty-nv]

From talking with Jay, it sounds like this is the race condition in profiling. We have a fix in the review awaiting @skallweitNV's review. Closing for now.

[nv-slang-bot]

Follow-up on the close — not asking to reopen, just recording two verified facts that may not reconcile with the profiling attribution, in case they matter for where the fix lands. If Jay's runner-side diagnosis already covers these, ignore me.

1. The profiler postdates 5 of the 6 observed wedges. src/sgl/utils/profiler.cpp and slangpy/tests/slangpy_tests/test_profiler.py were both introduced by the same commit, dac9e0e3 ("Profiler" #1063, merged 2026-07-13 17:02Z). Checking each wedged sha for those two paths via contents/...?ref=<sha>:

wedged run	sha	run started	profiler.cpp	test_profiler.py
29201285694	2b2379d2	07-12 17:03Z	absent	absent
29207192226	6c656dc9	07-12 20:09Z	absent	absent
29219333336	a7294115	07-13 02:21Z	absent	absent
29235474956	6c64cee0	07-13 08:26Z	absent	absent
29258433795	bc4e5ba2	07-13 14:33Z	absent	absent
29283873237	0fb33449	07-13 20:48Z	present	present

So a profiler race can account for at most the last of the six. The five earlier ones carry the same signature on a tree with no profiler in it: on the macos aarch64 clang Release+Debug lanes, Unit Tests (C++) finishes green in 37–68 s and then Unit Tests (Python) runs 174–352 min and is cancelled.

2. Nothing that could affect the Python lane has merged. #1073 — the fix referenced as "in the review" — is still open and mergeable_state: blocked. What did merge, #1076, touches only tests/sgl/device/test_profiler.cpp, adding * doctest::skip() to two C++ TEST_CASEs (still skipped on main at :332 and :598). Both live in the Unit Tests (C++) step, which was never the step that wedged.

On the "has it recurred" question I answered earlier: still no — but I'd read that as unexercised rather than resolved. The signature only ever fired on dev/slangpy-fixer/1051, and that branch has run no CI since 2026-07-13 (#1053 is still draft and behind main; its test file never landed). Across 08-01 → 08-04, 12 ci runs on 4 branches, Unit Tests (Python) maxed at 7.0 min.

The part that outlives whichever trigger it was: main still has no sub-6h backstop. There is no timeout-minutes anywhere in ci.yml (the step is at .github/workflows/ci.yml:218-220), no pytest-timeout in requirements-dev.txt, and tools/ci.py:156,164 still runs -n auto --maxprocesses=4. No open PR changes any of those. So if a GPU Python test wedges again — profiler or otherwise — it burns another 6 h of runner time and produces no attributable output. A timeout-minutes on that step is a two-line change, independent of root cause; happy to open it if you want it.

_{🤖 Generated by an automated SlangPy coworker — may be inaccurate. A human maintainer should verify.}