Skip to content

Add complete Eigen 5 benchmark coverage - #125

Merged
pauldreik merged 5 commits into
mainfrom
eigen-full
Aug 5, 2026
Merged

Add complete Eigen 5 benchmark coverage#125
pauldreik merged 5 commits into
mainfrom
eigen-full

Conversation

@pauldreik

@pauldreik pauldreik commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • update the Eigen benchmark from 3.4.0 to 5.0.1 and refresh its checked-in performance report
  • complete the cmath, complex, and list functionality required by Eigen
  • add eigen-full for the complete upstream official test target, with memory-aware compilation and nightly coverage

Performance

The refreshed nine-repetition GCC 14 benchmark reports:

  • Debug compilation: 25.08 s with libstdc++ versus 13.75 s with psychicstd, a 1.82x speedup
  • Release compilation: 25.13 s with libstdc++ versus 13.86 s with psychicstd, a 1.81x speedup

The always-inline cmath wrappers reduce Debug call overhead and emitted code without a measurable regression in the broader compile-time suite. A large compiler ifdef alternative was benchmarked and rejected because it provided no project-level gain while substantially duplicating the header.

Validation

  • Eigen 5.0.1 benchmark, Debug and Release, nine repetitions per system/psychicstd side
  • complete Eigen official test target with psychicstd
  • GCC debug: 337 of 337 tests passed
  • strict compatibility: 337 of 337 tests passed
  • Python lint, Markdown formatting, YAML formatting, and diff checks

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Compile-time performance diff

Compiler: g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.

Repetitions: 5 per configuration.

psychicstd compile time, main vs this PR (same runner). 🟢 faster · 🔴 slower · 🟡 within noise. A change is colored only when its bootstrap 95% CI excludes 0 and it clears ±5% / 3ms. vs system is this PR's speedup over system libstdc++, for context -- a regression is less alarming if psychicstd is still far ahead. Median system-time drift (noise proxy): 1.9%.

No significant changes (🟡 3 within noise).

🟡 3 benchmark(s) within noise
benchmark main PR Δ (95% CI) vs system
all-headers 208.8ms 210.6ms 🟡 +0.9% [-0.3, +2.0] 6.7x
iostream 76.2ms 78.0ms 🟡 +2.4% [+0.2, +3.8] 3.4x
wordcounter 177.5ms 175.5ms 🟡 -1.1% [-2.9, +3.3] 3.2x

Reproduce this on your machine: scripts/compare_compile_time_to_ref.py --reps 5 g++ origin/main

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

rdfind build-time diff (configure / compile / run tests)

Compiler: c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.

Repetitions: 5 per configuration.

rdfind build time with psychicstd, main vs this PR (same runner). 🟢 faster · 🔴 slower · 🟡 within noise. A change is colored only when its bootstrap 95% CI excludes 0 and it clears ±5% / 3ms. vs system is this PR's speedup over system libstdc++, for context -- a regression is less alarming if psychicstd is still far ahead. Median system-time drift (noise proxy): 0.2%.

No significant changes (🟡 3 within noise).

🟡 3 benchmark(s) within noise
benchmark main PR Δ (95% CI) vs system
compile 475.6ms 477.3ms 🟡 +0.4% [-8.7, +0.9] 4.1x
configure 1237.8ms 1237.7ms 🟡 -0.0% [-0.3, +0.2] 1.0x
run tests 1599.8ms 1598.4ms 🟡 -0.1% [-1.2, +0.8] 1.0x

Reproduce this on your machine: scripts/compare_realworld_to_ref.py rdfind --build-type debug --reps 5

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

eigen build-time diff (compile / run tests)

Compiler: c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.

Repetitions: 2 per configuration.

eigen build time with psychicstd, main vs this PR (same runner). 🟢 faster · 🔴 slower · 🟡 within noise. A change is colored only when its bootstrap 95% CI excludes 0 and it clears ±5% / 3ms. vs system is this PR's speedup over system libstdc++, for context -- a regression is less alarming if psychicstd is still far ahead. Median system-time drift (noise proxy): 1.3%.

No significant changes (🟡 2 within noise).

🟡 2 benchmark(s) within noise
benchmark main PR Δ (95% CI) vs system
compile 20623.8ms 20556.6ms 🟡 -0.3% [-0.8, +0.2] 1.7x
run tests 216.7ms 215.5ms 🟡 -0.5% [-0.8, -0.3] 1.5x

Reproduce this on your machine: scripts/compare_realworld_to_ref.py eigen --build-type debug --reps 2

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

flatbuffers build-time diff (configure / compile / run tests)

Compiler: c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.

Repetitions: 1 per configuration.

flatbuffers build time with psychicstd, main vs this PR (same runner). 🟢 faster · 🔴 slower · 🟡 within noise. A change is colored only when its bootstrap 95% CI excludes 0 and it clears ±5% / 3ms. vs system is this PR's speedup over system libstdc++, for context -- a regression is less alarming if psychicstd is still far ahead. Median system-time drift (noise proxy): 19.9%.

No significant changes (🟡 3 within noise).

🟡 3 benchmark(s) within noise
benchmark main PR Δ (95% CI) vs system
compile 25848.7ms 25599.8ms 🟡 -1.0% 2.2x
configure 487.3ms 481.3ms 🟡 -1.2% 1.0x
run tests 87.9ms 88.7ms 🟡 +0.9% 1.1x

Reproduce this on your machine: scripts/compare_realworld_to_ref.py flatbuffers --build-type debug --reps 1

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

fmt build-time diff (compile / run tests)

Compiler: c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.

Repetitions: 2 per configuration.

fmt build time with psychicstd, main vs this PR (same runner). 🟢 faster · 🔴 slower · 🟡 within noise. A change is colored only when its bootstrap 95% CI excludes 0 and it clears ±5% / 3ms. vs system is this PR's speedup over system libstdc++, for context -- a regression is less alarming if psychicstd is still far ahead. Median system-time drift (noise proxy): 26.5%.

No significant changes (🟡 2 within noise).

🟡 2 benchmark(s) within noise
benchmark main PR Δ (95% CI) vs system
compile 30297.2ms 30228.7ms 🟡 -0.2% [-0.7, +0.2] 1.6x
run tests 230.3ms 230.4ms 🟡 +0.0% [-3.7, +3.8] 1.7x

Reproduce this on your machine: scripts/compare_realworld_to_ref.py fmt --build-type debug --reps 2

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

simdutf build-time diff (compile / run tests)

Compiler: c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.

Repetitions: 1 per configuration.

simdutf build time with psychicstd, main vs this PR (same runner). 🟢 faster · 🔴 slower · 🟡 within noise. A change is colored only when its bootstrap 95% CI excludes 0 and it clears ±5% / 3ms. vs system is this PR's speedup over system libstdc++, for context -- a regression is less alarming if psychicstd is still far ahead. Median system-time drift (noise proxy): 1.8%.

No significant changes (🟡 2 within noise).

🟡 2 benchmark(s) within noise
benchmark main PR Δ (95% CI) vs system
compile 16844.4ms 16852.7ms 🟡 +0.0% 2.1x
run tests 51347.4ms 50956.5ms 🟡 -0.8% 1.0x

Reproduce this on your machine: scripts/compare_realworld_to_ref.py simdutf --build-type debug --reps 1

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

catch2 build-time diff (compile / run tests)

Compiler: c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.

Repetitions: 3 per configuration.

catch2 build time with psychicstd, main vs this PR (same runner). 🟢 faster · 🔴 slower · 🟡 within noise. A change is colored only when its bootstrap 95% CI excludes 0 and it clears ±5% / 3ms. vs system is this PR's speedup over system libstdc++, for context -- a regression is less alarming if psychicstd is still far ahead. Median system-time drift (noise proxy): 1.4%.

No significant changes (🟡 2 within noise).

🟡 2 benchmark(s) within noise
benchmark main PR Δ (95% CI) vs system
compile 16661.7ms 16667.2ms 🟡 +0.0% [-2.6, +0.4] 3.6x
run tests 3866.5ms 3897.2ms 🟡 +0.8% [-1.9, +2.8] 1.2x

Reproduce this on your machine: scripts/compare_realworld_to_ref.py catch2 --build-type debug --reps 3

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Compliance diff

🔴 1 header(s) regressed, 🟢 1 header(s) improved.

header before after Δ
memory 15/40 14/40 🔴 -1
complex 17/40 21/40 🟢 +4
70 header(s) unchanged
header passes/sampled
algorithm 8/40
any 9/18
array 27/40
atomic 17/40
bit 2/15
bitset 9/40
cassert 3/9
cctype 2/5
cerrno 5/29
cfloat 5/29
chrono 1/40
ciso646 5/29
climits 15/40
clocale 5/29
cmath 1/19
compare 5/14
concepts 2/35
condition_variable 13/26
csetjmp 3/9
csignal 5/29
cstddef 13/21
cstdint 3/5
cstdio 5/29
cstdlib 3/9
cstring 2/5
ctime 5/29
cwchar 5/29
deque 29/40
exception 20/21
filesystem 3/40
forward_list 9/40
fstream 14/40
functional 17/40
initializer_list 0/0
iomanip 24/40
ios 17/40
iostream 0/2
istream 23/40
iterator 5/40
limits 36/37
list 6/40
locale 9/40
map 13/40
mutex 15/40
new 19/40
numeric 4/40
optional 11/40
ostream 33/40
random 4/40
ranges 0/40
ratio 13/13
regex 3/40
set 1/40
sstream 7/40
stack 16/36
stdexcept 9/9
streambuf 30/37
string 2/40
string_view 15/40
system_error 30/40
thread 6/40
tuple 10/40
type_traits 22/40
typeinfo 2/5
unordered_map 15/40
unordered_set 12/40
utility 8/40
valarray 1/40
vector 29/40
version 0/40

Reproduce this on your machine: LLVM_ROOT=$PWD/llvm-project cp base_cache.json .compliance_cache.json && tools/compliance.py --recheck

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Sanitizer diff (ASan/UBSan)

🟢 1 header(s) improved.

header before after Δ
complex 11/25 14/25 🟢 +3
71 header(s) unchanged
header passes/sampled
algorithm 5/25
any 6/18
array 17/25
atomic 13/25
bit 2/15
bitset 3/25
cassert 3/9
cctype 2/5
cerrno 4/25
cfloat 4/25
chrono 0/25
ciso646 4/25
climits 11/25
clocale 4/25
cmath 1/19
compare 5/14
concepts 2/25
condition_variable 13/25
csetjmp 3/9
csignal 4/25
cstddef 13/21
cstdint 3/5
cstdio 4/25
cstdlib 3/9
cstring 2/5
ctime 4/25
cwchar 4/25
deque 17/25
exception 20/21
filesystem 2/25
forward_list 6/25
fstream 8/25
functional 10/25
initializer_list 0/0
iomanip 14/25
ios 12/25
iostream 0/2
istream 10/25
iterator 4/25
limits 24/25
list 4/25
locale 6/25
map 7/25
memory 8/25
mutex 13/25
new 6/25
numeric 3/25
optional 8/25
ostream 19/25
random 3/25
ranges 0/25
ratio 13/13
regex 1/25
set 0/25
sstream 5/25
stack 9/25
stdexcept 9/9
streambuf 18/25
string 1/25
string_view 9/25
system_error 20/25
thread 5/25
tuple 7/25
type_traits 15/25
typeinfo 2/5
unordered_map 10/25
unordered_set 7/25
utility 6/25
valarray 0/25
vector 18/25
version 0/25

Reproduce this on your machine: LLVM_ROOT=$PWD/llvm-project cp sanitize_base_cache.json .compliance_cache.sanitize.json && tools/compliance.py --sanitize --recheck

@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

cppcheck build-time diff (compile)

Compiler: c++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0.

Repetitions: 2 per configuration.

cppcheck build time with psychicstd, main vs this PR (same runner). 🟢 faster · 🔴 slower · 🟡 within noise. A change is colored only when its bootstrap 95% CI excludes 0 and it clears ±5% / 3ms. vs system is this PR's speedup over system libstdc++, for context -- a regression is less alarming if psychicstd is still far ahead. Median system-time drift (noise proxy): 0.4%.

No significant changes (🟡 1 within noise).

🟡 1 benchmark(s) within noise
benchmark main PR Δ (95% CI) vs system
compile 48559.4ms 49104.5ms 🟡 +1.1% [-1.4, +3.6] 2.0x

Reproduce this on your machine: scripts/compare_realworld_to_ref.py cppcheck --build-type debug --reps 2

@pauldreik
pauldreik force-pushed the eigen-full branch 6 times, most recently from 51b5f88 to 58042c7 Compare August 4, 2026 20:35
@pauldreik
update Eigen to 5.0.1
f6af29c 
Use the latest Eigen release while preserving the existing focused test set.
Verify the official archive checksum and keep the benchmark recipe otherwise
unchanged.
@pauldreik
complete cmath arithmetic overloads
169cdd6 
Provide correctly typed unary and binary overloads for floating-point and
integral arguments while retaining the C names from <math.h>.

Match GCC's builtin-backed constant evaluation without requiring unsupported
constexpr math from Clang.
@pauldreik
expand complex math support
3bc3aea 
Add the standard operations needed by Eigen and use compiler complex builtins
for overflow-safe magnitude, square root, and exponential edge cases. Keep
stream insertion lightweight while consuming Eigen field widths correctly.
@pauldreik
implement list resize
a51ef2c 
Support shrinking and both forms of list growth while preserving existing
nodes and iterators. Eigen exercises these operations through its STL tests.
@pauldreik
add complete Eigen benchmark
11ed7e4 
Build Eigen's complete official test target as a separate real-world
project.

Cap compilation at two jobs because the heaviest translation units consume
several GiB. Allow tests to use the requested parallelism.
@pauldreik
pauldreik merged commit 9886559 into main Aug 5, 2026
54 checks passed
@pauldreik
pauldreik deleted the eigen-full branch August 5, 2026 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pauldreik