Fix #12637: lower portable HLSL ReportHit to OptiX/CUDA

[nv-slang-bot]

Motivation

The portable, standard HLSL intersection-shader hit report fails to compile for CUDA/OptiX:

struct Attributes { uint id; float weight; };
[shader("intersection")]
void main() {
    Attributes attributes = { 7u, 0.5f };
    ReportHit(1.0f, 1u, attributes);   // error[E36107] on -target cuda / -target ptx
}

The core-module ReportHit<A> (source/slang/hlsl.meta.slang) was declared
[require(glsl_hlsl_spirv, raytracing_intersection)] and its __target_switch had only
hlsl / glsl / spirv cases — CUDA was excluded, so the capability check rejected it. Prior
work (#4948 / PR #4949) added the target-specific ReportHitOptix, which takes attributes as an
already-flattened variadic pack of integer scalars; portable shaders passing an aggregate
attributes (the same code that compiles for D3D/Vulkan) still could not target CUDA. Closes #12637.

Proposed solution

Give the standard ReportHit<A> a cuda case that flattens the aggregate attributes
field-wise into OptiX attribute registers and emits a single optixReportIntersection call.

This is the principled contract because the read side already defines it. When a hit shader
reads its attributes parameter back, emitOptiXAttributeFetch
(slang-ir-legalize-varying-params.cpp) recurses the aggregate and consumes one attribute
register per scalar leaf (struct fields in order, then array/vector/matrix elements in index
order), rendering optixGetAttribute_N() for each — float leaves via __int_as_float(...) to
preserve their bit pattern. The write side is made the exact inverse of that, so a value reported by
ReportHit round-trips through the fixed-function registers. A whole-struct byte-packed reinterpret
(e.g. bit_cast<uint[N]>) would in general not round-trip: whenever padding or sub-dword fields
make the byte layout's register boundaries differ from the reader's per-leaf boundaries, the two
disagree.

Two OptiX/CUDA facts shape the mechanism:

1. OptiX has no per-attribute-register setter. Unlike ray payloads (optixSetPayload_N), hit
   attributes are written only through the single call optixReportIntersection(float, unsigned int [, unsigned int a0 .. a7]) (fixed 0–8 register overloads, external/optix-dev/.../optix_device.h).
   So the write side gathers all leaves into one call.
2. CUDA lowers IRBitCast as a numeric C cast, not a bit cast (slang-emit-c-like.cpp). So a
   float leaf cannot rely on an IR BitCast to reach the register bit-for-bit; the emitter writes
   __float_as_uint(x) explicitly, so the register holds the same 32-bit pattern the reader
   recovers with __int_as_float(optixGetAttribute_N()).

Change summary

File	Change
source/slang/hlsl.meta.slang	Add cuda to ReportHit<A>'s [require] set and a case cuda: forwarding to a new hidden marker __reportOptiXIntersection.
source/slang/slang-ir-insts.lua	Register the new IR op reportOptiXIntersection.
source/slang/slang-ir-insts-stable-names.lua	Assign the op a stable serialization id.
source/slang/slang-ir-legalize-varying-params.cpp / .h	Add flattenOptiXHitAttributes (write-side mirror of emitOptiXAttributeFetch, restricting leaves to types that survive a single 32-bit register), legalizeOptiXReportIntersections (rewrites the marker to carry flattened leaves, enforcing the shared 8-register cap), and a legalizeOptiXReportIntersectionsForCUDA entry point.
source/slang/slang-emit.cpp	Run the flatten pass for CUDA/CUDAHeader after post-inline DCE, before empty-type legalization.
source/slang/slang-emit-cuda.cpp	Emit case kIROp_ReportOptiXIntersection: as optixReportIntersection(tHit, hitKind, a0..aN), wrapping float leaves in __float_as_uint.
source/slang/slang-diagnostics.lua	Add user-facing diagnostics E55216 (unsupported attribute type) and E55217 (>8 registers).
tests/cuda/report-hit-portable.slang, tests/cuda/report-hit-portable-diagnostic.slang	New regression + diagnostic tests.

Concepts and vocabulary

• Hit attribute registers — OptiX exposes up to 8 32-bit intersection attribute registers
  (a0..a7), written once via optixReportIntersection and read individually via
  optixGetAttribute_N. Distinct from ray payload registers, which do have per-register
  setters (optixSetPayload_N) — the two are easy to conflate, but only payloads are per-register
  writable.
• emitOptiXAttributeFetch — the existing read-side recursion that reconstructs a hit-attribute
  aggregate from the registers, one register per scalar leaf. This PR's write side mirrors it.
• Marker IR op — reportOptiXIntersection carries the un-flattened aggregate from the core
  module through inlining to the CUDA flatten pass, keeping the emitter trivial (no aggregate
  walking at emit time), matching how the read side splits work between the pass and the emitter.

Process report

hlsl.meta.slang — capability + cuda case. The root cause of E36107 is purely that CUDA was
absent from ReportHit<A>'s capability set and switch. Adding cuda to [require(...)] (reusing
the existing cuda_glsl_hlsl_spirv alias ReportHitOptix already uses) and a
case cuda: return __reportOptiXIntersection(tHit, hitKind, attributes); is the minimal front-end
change. __reportOptiXIntersection<A> is a hidden __intrinsic_op declaration (modeled on the
adjacent __reportIntersection and generic __intrinsic_op helpers like select), so the
aggregate is preserved as a single IR operand rather than decomposed at the language layer.

Why an IR op + pass rather than emit-time flattening. The aggregate must be broken into scalar
leaves to fill the fixed a0..a7 registers. Doing that in the emitter would require the backend to
synthesize field/element access and re-derive the aggregate's shape — the "consumer-side patching /
semantic-to-syntax reconstruction" the methodology warns against. The read side already puts this in
the IR pass (emitOptiXAttributeFetch) and hands the emitter trivially-renderable per-register ops;
the write side mirrors that split. flattenOptiXHitAttributes is a near-structural copy of the
reader's recursion, collecting extracted leaf values instead of fetching registers, guaranteeing
identical leaf order so the two paths round-trip.

Pass placement. ReportHit is [ForceInline], so its reportOptiXIntersection marker appears
at each call site after performForceInlining (the pass scans every function). The flatten pass runs after the
post-inline DCE (so the dead specialized core-module ReportHit body is gone and each marker is
diagnosed only once) and before the generic empty-type / resource legalization passes, which do
not understand the marker's aggregate operand. Placing it after inlining but before those passes is
what makes an empty attribute struct (0 leaves) reach the valid zero-attribute overload
optixReportIntersection(tHit, hitKind) instead of aborting in empty-type legalization.

Input-shape check. The pass handles kIROp_ReportOptiXIntersection whose operand 2 is the
aggregate the core module passed — a canonical, intentionally-produced shape, not a malformed one
being repaired. flattenOptiXHitAttributes accepts only leaves that survive a single 32-bit
register: float (bit-reinterpreted), and integer/bool/char types of 32 bits or fewer (which
sign/zero-extend on write and truncate back on read, symmetrically with the reader). It rejects
double and 64-bit scalars (they exceed one register) and half (the reader has no bit-preserving
half read — it would assign the register's integer value to the half numerically), as well as
pointer-typed attributes (a scalar register cannot round-trip a pointer, and dereferencing would
change which value is reported) — with the user-facing E55216 rather than a silent miscompile.
Aggregates exceeding 8 registers get E55217 (the shared
kMaxHitAttributeRegisters cap, now used by both the read and write paths).

Emitter. case kIROp_ReportOptiXIntersection renders one optixReportIntersection(...) call;
float leaves get __float_as_uint(...), the exact inverse of the reader's __int_as_float, so a
float attribute reads back bit-identical. Non-float leaves pass through as the same unsigned int
register value the reader returns and assigns to the leaf type.

Testing. tests/cuda/report-hit-portable.slang checks the emitted flattened call for a struct
with signed/narrow-integer/float/vector fields, an empty-attribute entry point, and a PTX/NVRTC
compile against the real OptiX header. tests/cuda/report-hit-portable-diagnostic.slang covers the
E55216 (double, pointer) and E55217 (>8 registers) rejection paths. Locally I verified via direct
slangc emission plus a PTX/NVRTC compile (the FileCheck-based slang-test harness needs
slang-llvm, unavailable in my environment, so CI runs the harness); the pre-existing
report-hit.slang, optix-hit-attributes.slang, and optix-get-attributes-mixed.slang remain as
regression coverage and emitted unchanged in my checks.

[jhelferty-nv]

Automated notice (PR board sync) — do not reply to this comment.

Auto-assigned @jkwak-work as shepherd for this Bot PR.

FYI for maintainers: committer signal on the changed files is higher for szihs than for the auto-requested reviewer (jkwak-work). They were not auto-requested; a human may optionally add them as a reviewer.

[jhelferty-nv]

Automated notice (PR board sync) — do not reply to this comment.

Auto-assigned @jkwak-work as shepherd for this Bot PR.

FYI for maintainers: committer signal on the changed files is higher for szihs than for the auto-requested reviewer (jkwak-work). They were not auto-requested; a human may optionally add them as a reviewer.

[jkwak-work]

I will need to review it more carefully.
But I don't think we need an explicit legalization pass for this.
I think that adding a new inst and emit the cuda for the new inst should be enough.

@nv-slang-bot please explain to me why the legalization is needed if it is needed.

[jkwak-work]

@nv-slang-bot can you review my previous comment? I am waiting for your input.