Repository navigation
Avoid scaled-index table loads on the double path under -fPIC (fixes ~2x slowdown on AMD Zen4) - #21
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under
-fPIC/-fPIE, thedoublepath (xjb64) is ~2x slower on AMD Zen4than it should be, while the
floatpath and all other x86-64 microarchitecturesare unaffected. This PR folds two critical-path table loads into base+disp form,
recovering 1.55×–1.85× on Zen4 with no change to output and no change to
non-PIC builds.
Root cause
With
-fPIC, static data uses RIP-relative addressing, so thedouble_tablebase is pinned into a register and the hot
pow10_double/exp_result_doublelookups compile to scaled-index loads:
The pow10 load sits on the double path's critical dependency chain (its result
feeds the 128-bit multiply). On Zen4, a scaled-index load with a large
displacement misses the fast AGU path and adds load-use latency, stalling the
chain every call. Intel's AGU has no such penalty, which is why only Zen4
regresses — and why it reproduces on clang, gcc and icpx alike (it's the
addressing mode, not a compiler artifact). The float path is smaller and not
register-starved, so it's unaffected.
Fix
Fold the index into the pointer (moving address math off the critical load-use
chain) and use an empty asm barrier to stop the compiler from re-merging it
back into a scaled-index load:
Same treatment for exp_result_double[e10 + 324].
Gated on PIC so static / LTO / -no-pie builds are byte-for-byte
unchanged (there the loads are already base+abs-disp, no index penalty).
znver4 can't be used because the affected builds are generic x86-64, where
that macro isn't defined.
Results — AMD Ryzen 7 7840H (Zen4), double, median ns/call
┌──────────────┬────────┬───────┬─────────┐
│ build │ before │ after │ speedup │
├──────────────┼────────┼───────┼─────────┤
│ clang SSE2 │ 16.68 │ 10.77 │ 1.55× │
├──────────────┼────────┼───────┼─────────┤
│ clang SSE4.1 │ 15.36 │ 8.88 │ 1.73× │
├──────────────┼────────┼───────┼─────────┤
│ gcc SSE2 │ 17.33 │ 10.63 │ 1.63× │
├──────────────┼────────┼───────┼─────────┤
│ gcc SSE4.1 │ 16.72 │ 9.08 │ 1.84× │
└──────────────┴────────┴───────┴─────────┘
perf stat (clang/SSE4.1): cycles 11.19G → 8.00G (−28%) at flat instruction
count and identical L1 load/miss counts, IPC 2.80 → 3.96 — i.e. pure load-latency
stall relief, exactly matching the diagnosis.
Cost
On Intel (i7-13700K) the extra leas cost ~4% under PIC (no scaled-index penalty
to recover). Non-PIC Intel builds are unchanged. The Zen4 win far outweighs this,
and xjb64 remains faster than the alternatives on Intel either way.
Correctness
Output is bit-identical to the current code over 30M random doubles plus edge
cases (0, -0, ±inf, nan, subnormals, 5e-324, 1e15/1e16 boundaries).
The asm barrier is a pure codegen barrier — semantically a no-op.
🤖 Generated with Claude Code