[ling0322]

This PR moves the audio operators out of waifu::audio and into flint:

• conv_transpose1d, snake, stft and istft were already declared but nothing ran them; they are now real flint operators.
• pad1d (zero/reflect/replicate) and clamp are new operators.
• The CPU, CUDA and Vulkan run conv_transpose1d, snake, pad1d and clamp as kernels.
• Metal, and stft/istft on every device, run a composition in the Operators base class (flint/functional.cc).

The compositions that stood in for these in waifu::audio are removed (audio.rs: 981 → 473 lines), and every caller uses the graph ops.

Why these

Four of them were already plumbed through the C API, Op and Graph, but no backend ran them, so waifu::audio built each out of 6–25 graph nodes. The snake alone was six passes over the tensor, and BigVGAN runs it in all 109 of its activations. pad1d had 13 call sites, mirroring by a matmul against an anti-diagonal. clamp replaces at_least (three copies), at_most and clamp_unit.

What moved

• flint:
  • New Operators::pad1d/clamp, with FL_PAD_*, fl_pad1d and fl_clamp in the C API.
  • conv_transpose1d is one GEMM through the device's own matmul (F::transposedConvolutionColumns), then a col2im kernel that adds the bias.
  • Kernels are in flint/cpu/signal.cc, flint/cuda/signal.cu, and flint/vulkan/signal.cc with flint/vulkan/shaders/{snake,clamp,pad1d,overlap_add1d}.comp, in f32 and f16.
• waifu: new Op::Pad1d, Op::Clamp, Graph::pad1d, Graph::clamp, PadMode and audio::halves.
• Removed from waifu::audio: pad1d, Padding, reversal, conv_transpose1d, snake, stft, istft, stft_basis, istft_basis and window_envelope.
• Signatures: functions that only carried dtype/device for those helpers drop them, as in audio: remove conv1d and depthwise_conv1d; callers use Graph::conv1d #122.
• Kept: Whisper's "eight below the loudest" floor stays at_least, because its bound is a graph value and clamp takes numbers.

Numerics

• Half precision. The fused kernels round once where the compositions rounded at every step, so they are nearer float32 per element. On CUDA, snake's mean error is 5.6e-4 against 6.8e-4 for the composition; on Vulkan, conv_transpose1d's is 8.0e-3 against 1.0e-2.
• Vulkan's snake. It reduces its angle into [-π, π] before sin, because GLSL bounds sin's error only there.
• istft divides by the window's overlap in float32 whatever the spectrum's type.
• clamp keeps a NaN on every device, as torch.clamp does. Inside the interval it returns the input exactly on every path, including the composition.

BigVGAN's GPU test

The old test compared the card's half-precision output with BigVGAN on the worst of 24 samples, under 4e-3. That statistic is noise:

• Over 20 six-frame mels it ranges from 2.5e-3 to 7.6e-3 for the same code.
• The old compositions exceed 4e-3 on 13 of the 20.

It now compares the card in half with the processor in float32 over 200 frames: mean error ≤ 2.5e-3, max ≤ 1.5e-2. It runs as it_runs_on_a_card (CUDA) and it_runs_on_vulkan. Both measure a mean of about 1.5e-3.

Review fixes (f80a4f5, c962b67)

• istft in half made NaNs at the ends when not centred. The 1e-11 floor and w[1]² ≈ 9e-11 rounded to zero. Fixed by dividing in float32, and tested with a half, non-centred 1024-point round trip on CUDA and Vulkan, which fails at sample 0 without the fix.
• clamp turned a NaN into a bound on CUDA (fmaxf), on Vulkan (GLSL min/max) and in the composition (relu). The kernels now pass a NaN through; the composition adds exp(-|x|) * 0.
• The clamp composition erased small values inside the interval. lo + relu(x - lo) rounded x - lo to the input type, so 1e-4 in half (and 1e-8 in float32) came back as 0. It now uses the max and min reductions of each element beside its bound, which return one of their inputs unchanged. The tests require exact equality with the kernel, and they fail with the old composition.

Tests

• ./build/unittest: all 115193 assertions pass.
  • The CPU kernels and the compositions are checked against the definitions.
  • CUDA and Vulkan are checked against the CPU in f32 and f16.
  • The compositions are also run on a real Vulkan device.
• cargo test: 393 passed, 0 failed. The CUDA- and Vulkan-gated audio_cuda and bigvgan tests pass too.
• Not run:
  • Metal: not buildable on the Linux box this was written on. It runs the same composition that was tested on Vulkan.
  • The model tests:

cargo test --release --manifest-path waifu/Cargo.toml --no-fail-fast --test cosyvoice3 -- --ignored --test-threads=1
cargo test --release --manifest-path waifu/Cargo.toml --no-fail-fast --test indextts --test indextts_gpt -- --ignored --test-threads=1

Not done

New activations (gelu-tanh, mish, leaky_relu, elu) and a graph rotary op are left for later. They are duplicated in waifu, but each needs Metal shader work.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WkfaWVmr9m5uxoJ4grPuLm

[ling0322]

Reviewed commit 585ef95. Two correctness issues are noted inline. Validation: code inspection and minimal numerical reproductions of float16 underflow and fminf/fmaxf NaN behavior; the full project test suite was not run.

[ling0322]

[P2] Keep the ISTFT envelope floor representable in float16

The envelope is floored at 1e-11f and then cast to the spectrum dtype by hostConstant, but 1e-11 rounds to zero in float16. For a Hann window with centered=false, the first envelope sample therefore remains zero and the subsequent division produces 0/0 (NaN); with nFft=1024, nearby small squared-window values also underflow. Please normalize in float32 or use a dtype-appropriate positive floor, and add a float16, non-centered ISTFT test. The new CUDA STFT/ISTFT test currently only exercises float32.

[ling0322]

Fixed in f80a4f5. composedIstft now divides by the overlap in float32 whatever the spectrum type, and casts only the kept samples back, so the floor and the small end values (w[1]^2 ≈ 9e-11 for a 1024-point Hann) never pass through half. Added a half, non-centred 1024-point stft→istft round trip on CUDA (flint/cuda/signal_test.cc) and on Vulkan. It checks every sample is finite and the covered region matches to 2e-2. With the old normalization it fails at sample 0.

[ling0322]

[P2] Preserve NaN inputs in the CUDA clamp kernel

fmaxf(NaN, lo) returns lo, so this expression turns a NaN input into a finite bound (for example, clamp(NaN, -0.5, 0.25) returns -0.5). The CPU implementation using std::min/std::max preserves the input NaN, so the operator behaves differently across backends and CUDA silently hides upstream numerical failures. Please explicitly propagate NaN inputs and cover this case in the CPU/CUDA comparison.

[ling0322]

Fixed in f80a4f5, and not only on CUDA. The Vulkan shader (GLSL leaves min/max of a NaN undefined) and the base-class composition (relu makes 0 of a NaN) dropped it too. The CUDA, Vulkan and CPU kernels now test isnan and pass a NaN through. The composition adds exp(-|x|) * 0, which is 0 for every number and for ±inf, and NaN only for a NaN. Tests cover NaN and ±inf for kernel and composition on the CPU, CUDA and Vulkan.

[ling0322]

Thanks for the review. Both findings were real, and both are fixed in f80a4f5. Details are in the inline threads.

1. ISTFT in float16 (thread): composedIstft now divides by the window overlap in float32, whatever the spectrum type, and only casts the kept samples back.
   • New test: a half-precision, non-centred, 1024-point stft→istft round trip on CUDA and on Vulkan. It checks that every sample is finite.
   • With the old normalization, the test fails at sample 0.
2. NaN through clamp (thread): the same bug was in the Vulkan shader and in the base-class composition, not just CUDA, so all four paths now keep a NaN.
   • The CPU, CUDA and Vulkan kernels check isnan explicitly.
   • The composition adds exp(-|x|) * 0, which is NaN only for a NaN input, so ±inf still go to their bounds.
   • Tests cover NaN and ±inf on the CPU, CUDA and Vulkan.

The same push also adds Vulkan kernels for conv_transpose1d, snake, pad1d and clamp (6b82e1e), so Metal is now the only backend on the compositions.

./build/unittest (114375 assertions) and cargo test (393 passed, 0 failed) are green. The model tests and Metal were not run; the description says why and gives the commands.

[ling0322]

Re-reviewed f80a4f5, including the Vulkan kernels and fixes to both previous findings. The previous ISTFT denominator-underflow and CUDA clamp NaN issues are addressed in the implementation and regression tests. Built with Metal enabled on macOS and ran the signal suite: 6 cases, 2519 assertions passed. The process could not access a Metal device, so GPU execution was not validated locally. One additional issue in the shared clamp composition is reproduced below by running that same composition on the CPU in float16.

[ling0322]

[P2] Avoid cancellation when clamping half-precision values already inside the interval

The arithmetic composition stores each subtraction/addition in the input dtype. For a float16 input containing 0.0001, composedClamp(..., -0.5f, 0.25f, ...) returns 0 rather than the original representable value (~0.000100017): x - lo rounds to exactly 0.5, losing x before the bound is added back. The upper-only case composedClamp(..., -inf, 100.0f, ...) also returns 0. I reproduced both by executing this exact composition with CPU operators in float16; the CPU clamp kernel retains the input. Metal inherits this composition, so values inside the interval can be erased depending on the backend. Please avoid the subtract-and-add cancellation (for example with a comparison/select kernel) and test small in-range half values; the current comparisons predominantly exercise float32 or allow 1e-3 error, which hides this case.

[ling0322]

Fixed in c962b67. This was real, and it isn't limited to half precision: in float32, 1e-8 against a lower bound of -0.5 was lost the same way.

composedClamp no longer clamps by arithmetic. It puts each element beside its bound in an (n, 2) tensor and takes the max and then the min reduction of the last dimension, operators every backend (Metal included) already has. A reduction returns one of its two inputs unchanged, so:

• a value inside the interval comes out exactly itself;
• a value outside it comes out exactly the bound, including ±inf;
• the exp(-|x|) * 0 term still puts back a NaN if a device's reduction drops one.

The tests now require exact equality between the composition and the kernel:

• CPU, float32: 1e-8 and -1e30.
• CUDA and Vulkan, float16: your 1e-4, a half subnormal, values next to each bound, and ±1e4, for both [-0.5, 0.25] and (-inf, 100].

With the previous composition all three fail: 0 against 1e-08 on the CPU, and 0 against 0.000100017 on CUDA and Vulkan.