Add two-stage pytest retry to SlangPy CI test jobs

[nv-slang-bot]

Summary (5-bullet)

• Status: Draft — two-stage pytest retry added to SlangPy CI test jobs. The GPU-free unit tests pass when run in isolation against tools/ci.py (pytest slangpy/tests/test_ci_retry.py, author's machine — not yet run on a built checkout / in CI); peer review dispatched to internal reviewer (verdict outstanding at open time).
• Link: Fixes Add similar retry logic than in Slang SlangPy tests #829.
• Verdict: Ready for CI + maintainer review; held as draft pending human vetting.
• Next-action: CI run on this PR; reviewer confirmation on a built checkout; maintainer promotes to ready if satisfied.
• Blocker: None (self-contained CI-tooling change).

Motivation

SlangPy's parallel CI test jobs run pytest -n auto (xdist). A test can go red from GPU device contention between xdist workers rather than a real defect, turning an entire job red with no second chance. Slang's old test-slangpy job had a retry for exactly this (removed in shader-slang/slang#9900 when SlangPy CI was decoupled); the review comment that removed it called the missing SlangPy-side retry a "pre-existing gap … add the retries later in separate change." This is that change.

Proposed solution

Mirror the old Slang pattern: on a parallel run failure, re-run only the failed tests, once, sequentially (-n 0 --lf). A single-process rerun escapes flakiness caused by contention between xdist workers; a deterministic failure that reproduces sequentially still fails on the rerun and keeps the job red.

The retry is added once, in tools/ci.py, so it covers every parallel CI call site that routes through unit_test_python / test_examples (ci.yml, ci-gcp.yml, and the composite action); the non-parallel sanitizer calls in sanitizers.yml route through the same code but take the single-run branch (see the --parallel gate below). No new dependency — --lf is core pytest and pytest-xdist is already a dev requirement.

Gated on --parallel. The sequential rerun only earns its keep against parallel-only contention. The sanitizer jobs (sanitizers.yml) run without --parallel; auto-retrying an intermittent ASan/UBSan finding would mask it, so the non-parallel path runs exactly once.

Change summary

File	Change
tools/ci.py	New run_pytest_with_retry(test_path, env, parallel); unit_test_python and test_examples route through it. Parallel: first attempt -n auto --maxprocesses=4 --cache-clear, on failure rerun -n 0 --lf --lfnf=all. Non-parallel: single run.
slangpy/tests/test_ci_retry.py	New GPU-free tests (5 control-flow via a run_command stub, 4 real-pytest integration) covering the retry flow and the false-green and re-execution safeguards.

Concepts and vocabulary

• --lf / last-failed cache: pytest records failed node ids in <rootdir>/.pytest_cache; --lf reselects them on the next run. This cache lives at the pytest rootdir and is independent of --basetemp (which relocates the temporary-directory fixtures — for example tmp_path and tmpdir — not the cache).
• --lfnf=all (last-failed-no-failures): when the last-failed set is empty, --lfnf=all (the pytest default) reruns the whole suite rather than selecting nothing.
• Exit 5: pytest's EXIT_NOTESTSCOLLECTED; run_command treats any nonzero exit (including 5) as a failure.

Process report

Why gate on --parallel (input-shape check). The rerun's purpose is to distinguish contention between xdist workers from real failures. That specific contention exists only under -n auto; the non-parallel sanitizer runs don't spawn competing workers, and an intermittent sanitizer finding must not be retried away. So the non-parallel input is handled by not retrying — a deliberate, not incidental, choice.

False-green and re-execution safeguards (cf. shader-slang/slang#11911). A retry that re-runs zero tests but reports success would convert a real failure into a green. The base guarantee is exit-code propagation; two flags address separate failure modes, each covered by a test that fails if the flag is removed:

1. --cache-clear on the first attempt (false-green guard). pytest merges into the existing last-failed mapping, so a stale entry for a test not collected this run can survive. Clearing up front guarantees the rerun's --lf set contains only this run's failures — without it, a rerun could select a now-passing stale set and go green. (test_integration_stale_cache_cannot_mask_failure fails without it.)
2. --lfnf=all on the rerun (re-execution guard). If the first attempt dies before writing the cache (crash/OOM at session start), the rerun's last-failed set is empty. Note the run is already kept red in that case by exit-code propagation — an empty --lf selection exits 5 (EXIT_NOTESTSCOLLECTED), which run_command treats as a failure, even under an inherited PYTEST_ADDOPTS=--lfnf=none. --lfnf=all makes the rerun re-run the whole suite so the real failure is actually re-executed and re-observed rather than merely inferred from an empty-collection exit code. test_integration_crash_rerun_reexecutes_failing_test asserts (via a marker file) that the failing test truly re-ran.

Exit-code propagation (the base guarantee). run_command raises RuntimeError on any nonzero exit (including exit 5 for an empty collection), and the rerun's exception is never swallowed, so a still-failing rerun keeps the job red.

Scoped out (stated so a reviewer needn't ask): the raw inline python -m pytest slang/tests/integration/slangpy/ at .github/actions/build-and-test-with-slang/action.yml:181 (not routed through ci.py, Windows-only, non-parallel — lowest contention risk) and benchmark_python (its current CI invocations already continue after per-device failures).

_{🤖 Generated by an automated SlangPy coworker — may be inaccurate. A human maintainer should verify.}

[jhelferty-nv]

Automated notice (PR board sync) — do not reply to this comment.

Auto-assigned @jkiviluoto-nv as shepherd for this Bot PR.

FYI for maintainers: committer signal on the changed files is highest for skallweitNV among collaborators other than the assignee. They were not auto-requested; a human may optionally add them as a reviewer.

[jhelferty-nv]

Automated notice (PR board sync) — do not reply to this comment.

Auto-assigned @jkiviluoto-nv as shepherd for this Bot PR.

FYI for maintainers: committer signal on the changed files is highest for skallweitNV among collaborators other than the assignee. They were not auto-requested; a human may optionally add them as a reviewer.