[klimovich008]

What this fixes

Single-precision FMUL/FNMUL/FMLA results that the FPX1 fast path can't prove fell back to the soft-float exact body every time. These are tiny or infinite results, or a subnormal input under FZ. Code that runs with those cumulative FPSR flags already set paid that cost on each such op.

Why merge

The rescue keeps the hardware result only when it can raise no new FPSR bit. Everything else still goes to the exact body. The review found no correctness problem.

Change

• a64_fpx_sticky32 runs in the outlined wrappers a64_fpx_mul32_fma32 and a64_fpx_fma32_fma32, inside the gate.
• It uses RECOMP_FPX_FMA64; without a hardware FMA only FMUL is rescued.
• Cheap rejects run before any FP op.
• Dead fz_mul32 code is removed.
• The FPX1 contract comment now points to the wrappers.

Tested

tests/test_fpx_sticky.py builds at x86-64-v3 and x86-64-v2. Each build runs:

• guest FMUL/FNMUL/FMADD/FMLA S words through the lifter's pipeline (96 runs);
• 16M random checks;
• 150M sweep checks.

There were 0 mismatches against the exact body. Rescues are counted inside the wrappers, and a rescue under a closed gate fails the test. Sabotages of the emitter route, the wrapper route, the FIZ/AH/NEP mask and the UFC/IDC checks each fail it.

Not covered

• No Unicorn, insn or leaf test run.
• No full re-lift.
• No game re-measure of this revision.
• No aarch64, wasm or GCC build.
• The shadow build doesn't exercise the rescue.

🤖 Generated with Claude Code

[hckrman101]

No correctness problem found. I checked a64_fpx_sticky32 against the exact body (EmitFPMulAddValue in suyu/arm64_to_c.h) and Arm's FPMulAdd/FPRound:

• RECOMP_FPX_OPEN already requires RN, no DN, no AH/FIZ/NEP, no trap enables and IXC set, so other rounding modes and DN can't reach it, and IXC never needs adding.
• The FZ input flush only happens with IDC already set.
• Tininess is detected before rounding: results <= the smallest normal are treated as possibly tiny, and exactly the smallest normal falls back.
• The binary64 cancellation check compares the exact product with the negated addend (no binary64 sum), so it's exact.
• An overflow needs OFC.
• The FNMUL/FMLS/FNMLA negation is outside the call and symmetric under RN.
• The real FMA is only used under __FMA__ with GCC/clang.

Nice work, and a nicely adversarial test. A few points, mostly about what the test can and can't catch, which I'd like answered first:

Test coverage (inline)

1. The "rescue decided cases" counter doesn't observe the generated wrappers.
2. Only the x86-64-v3 build is tested, so the A64_FPX_FMA32 == 0 path never runs.
3. gen_state never sets FPCR bits 0-2, and the sweep only uses two flag states.

Design
4. The rescue lives only in the generated outlined wrappers, not behind the runtime's recomp_fpx_* / recomp_fpx_shadow layer. The shadow build emits the inline form (EmitFPNativeSuffix), which FPX_GUARD doesn't rewrite, so the shadow build (which validates the fast paths against the exact body in-game) never exercises this path. Also, the FPX1 contract comment in arm64_to_c.h ("the result is finite and larger in magnitude than the smallest normal, so it is neither NaN, infinite nor tiny...") no longer matches what lifted code keeps. It needs at least a note there pointing to the wrappers, or the rescue could move down into the runtime header so the shadow build covers it.

Smaller things (inline)
5. A64_FPX_FMA32 duplicates RECOMP_FPX_FMA64.
6. Some dead mul32 code is left over.
7. A minor perf note on the fall-through path.

[hckrman101]

This counts by calling a64_fpx_sticky32 directly, so it doesn't show that the generated wrappers ever reach it. Say the (fast, exact) in (("mul32","fma32"),("fma32","fma32")) branch in fpx_header (a64_lift.py ~724-725) stopped matching, or passed the wrong args. The wrappers would then always return recomp_fpe_fma32, which matches the reference, so there'd be no mismatch, and these counters would still be nonzero. So the "can't pass by never running" guarantee doesn't really hold.

Could you count from inside the wrapper path instead? For example, a probe macro (empty by default) in the sticky branch of the generated wrapper, which this driver defines to bump rescued[mul].

[hckrman101]

This only builds at -march=x86-64-v3, so the A64_FPX_FMA32 == 0 side is never compiled or checked: the SSE-multiply rescue for FMUL and the else return 0; for FMLA. That's what x86-64-v2 builds get from host_march() (used by nx2/build.py), e.g. the Ivy Bridge from issue 363, so it's a real configuration. And on a non-AVX2 machine the whole test skips.

Could you add a second build at -march=x86-64-v2? It can run without the AVX2 check.

[hckrman101]

0x07ff9f00 never sets FPCR bits 0-2 (FIZ/AH/NEP), and the sweep only uses FPSR 0x9f or 0x10. So:

• The AH exclusion the rescue relies on is never exercised. With AH=1, Arm detects tininess after rounding and doesn't set IDC on the input flush. If RECOMP_FPX_FPCR_MASK lost bits 0-2, this test would still pass.
• Partial flag states (e.g. UFC set and IDC clear, with FZ, a subnormal operand and a tiny result) aren't swept.

Could you add bits 0-2 to the "anything else" mask and sweep a few more FPSR combinations?

[hckrman101]

This duplicates RECOMP_FPX_FMA64's condition (suyu/arm64_to_c.h ~6568), which is already in scope here. Could you define A64_FPX_FMA32 from it (or just use it), so the "correctly rounded hardware FMA" test can't drift?

[hckrman101]

Cleanup: FMUL's exact body is always fma (EmitFPOutlined maps mul to fma), so mul32 only ever pairs with fma32, and that pair now takes the branch above. That leaves unreachable code:

• "mul32" in this elif
• a64_fpx_fz_mul32 (~626 and ~631)
• a64_fpx_fz_result's mul param and its (mul && m == 0x00800000u) branch (~617)

Could you remove them and update the FPX_FZ_HEADER comment?

[hckrman101]

Minor perf: every slow-path call now pays for the hardware op and the checks before falling back. The old FMUL rescue exited cheaply on A64_FPX_FZ_OPEN. NaN operands, and the cases where the needed IDC/OFC/UFC bit is clear, could be rejected before computing. Also, RECOMP_FPX_OPEN(c) is tested twice in the generated wrapper (once for the fast path and again here).

[klimovich008]

Thanks. All seven points are addressed in 1ff4abe and 0a2b38e:

1. The wrappers now call A64_FPX_STICKY_PROBE() (empty by default) when the sticky rescue keeps a result, and the test counts those calls. A new lifted-route case runs hand-encoded FMUL, FNMUL, FMADD and FMLA S guest words through a64emit.translate and a64_lift.op_text/hot_text, the lifter's own pipeline. It checks the whole Ctx against the same words with FPX off, plus the probe count, on 5 must-rescue and 7 must-fall-back inputs. With the emitter no longer outlining FMUL (EmitFPMulAddValue), that case fails while the by-name runs pass. With fpx_header's tuple made never to match, every run fails.
2. There's a second build at -march=x86-64-v2 (-DEXPECT_FMA=0, #error if RECOMP_FPX_FMA64 disagrees). It needs only SSSE3, SSE4.2 and POPCNT, not AVX2. It requires FMUL rescues > 0 and FMLA rescues == 0: 16.5M / 0 in the sweep.
3. gen_state now sets FPCR bits 0-2. The sweep adds 16 states: partial IDC/UFC/OFC, IXC clear, and FIZ/AH/NEP. The exact body models none of FIZ/AH/NEP, so a rescue under a closed gate fails the test. Dropping bits 0-2 from the mask fails it (190k random and 6.1M sweep at v3).
4. The FPX1 comment in FpxH now points to the wrappers' rescues and says the shadow build doesn't reach them. Nothing moved into the runtime.
5. A64_FPX_FMA32 is gone; the helper uses RECOMP_FPX_FMA64.
6. Removed a64_fpx_fz_mul32, fz_result's mul and its branch, "mul32" in the elif, and the unused A64_FPX_FZ_OPEN. Comment updated.
7. One OPEN test per wrapper. Before any FP op, the rescue rejects NaN operands, FMLA without an FMA, and FMUL with OFC and UFC clear and nothing flushed. Rescue counts are unchanged, and the fallback microbenchmark is flat within noise.

Results: 2 × (16M random + 150M sweep + 96 lifted) checks, 0 mismatches.

[hckrman101]

Thanks, went through 1ff4abe and 0a2b38e. All seven points are covered:

• The probe sits in the wrappers' sticky branch, and the lifted case runs real guest words through op_text/hot_text, so the count now comes from the code the build actually emits.
• The v2 build plus the #error against RECOMP_FPX_FMA64 covers the no-FMA side, with FMLA required to stay at 0 there.
• With the test's own copy of the gate mask, a rescue under FIZ/AH/NEP now fails the test even when the value matches. That's the part a value comparison couldn't catch.

I also checked the new early rejects. NaN, IDC clear with a flushed operand, and FMUL with OFC/UFC both clear only drop cases the old code already returned 0 for. Taking RECOMP_FPX_OPEN out of the helper is fine, since both wrappers now call it inside the gate, and the add/sub condition matches the old A64_FPX_FZ_OPEN.

LGTM.