Bladebit runs 7% slower on Fedora-35 and Ubuntu-21.10 compared to RHEL-8.4

[bladeuserpi]

1) Problem
Bladebit runs 7% slower on Fedora-35 and Ubuntu-21.10 compared to RHEL-8.4

2) Test description
I installed these 3 distributions on the SAME machine and run 5 plots. 
And I also used the same bladebit executable on all three (mounted from 2nd drive).
These are results:

2a) RHEL-8.4, Kernel 4.18.0-305.25.1.el8_4.x86_64
124044.12443.out:Finished plotting in 350.49 seconds (5.84 minutes).
124044.12443.out:Finished plotting in 344.27 seconds (5.74 minutes).
124044.12443.out:Finished plotting in 343.97 seconds (5.73 minutes).
124044.12443.out:Finished plotting in 345.64 seconds (5.76 minutes).
124044.12443.out:Finished plotting in 360.38 seconds (6.01 minutes).

2b) Fedora-35-server, Kernel 5.14.10-300.fc35.x86_64
2385.29416.out:Finished plotting in 384.14 seconds (6.40 minutes).
2385.29416.out:Finished plotting in 370.80 seconds (6.18 minutes).
2385.29416.out:Finished plotting in 370.82 seconds (6.18 minutes).
2385.29416.out:Finished plotting in 370.98 seconds (6.18 minutes).
2385.29416.out:Finished plotting in 385.65 seconds (6.43 minutes).

2c) Ubuntu-21.10-server, Kernel 5.13.0-21-generic
3603.14006.out:Finished plotting in 382.11 seconds (6.37 minutes).
3603.14006.out:Finished plotting in 367.04 seconds (6.12 minutes).
3603.14006.out:Finished plotting in 366.88 seconds (6.11 minutes).
3603.14006.out:Finished plotting in 367.07 seconds (6.12 minutes).
3603.14006.out:Finished plotting in 382.60 seconds (6.38 minutes).

3) Discussion of results:
- Fedora-35 and Ubuntu-21.10 have comparable performance, about 367-370 seconds
  (Looking into details I would say U21.20 is 3 seconds faster than F35, but that is not topic here).
- RHEL-8.4 runs in ~344 seconds, that is ~25 seconds faster (=7%) 

A performance loss of 7% is HUGE and I did not expect this as bladebit
should not put stress on the kernel.

That might explain conflicting benchmark results when comparing
same/different hardware/CPU configs. Until now it was assumed
the Linux distribution has only minor influence. But this proves
choosing a Linux distribution has a major influence on bladebit performance


4) Going further

I would be interested if this is also seen/confirmed by other bladebit users ?

What could be causing this performance regression?

[harold-b]

This has indeed been like this from the beginning. Bladebit was originally mainly developed on Amazon Linux 2, which is based on RHEL, and consistently got better times than Ubuntu when testing on the same machine.

I have not tested CentOS, so I would be curious what times are obtained there, and if it is comparable to RHEL times.

As for the reasoning, I'd like to know myself, but it could be a number of things, starting with each distro including different versions of GCC. Though I suspect that might not be it. Have you tried tests with clang to see what times you get?

I'd love to hear what @mocksoul has to say as his more in-depth Linux expertise.

[bladeuserpi]

I run only my one "golden" self-compiled executable (AVX) on all three distributions, so GCC/Clang 
are out of scope.
But it uses the distribution-specific libraries:
ldd bladebit
	linux-vdso.so.1 (0x00007fff64fa9000)
	libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f4028b10000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f402877b000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f40283f9000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f40281e1000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4027fc1000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f4027bfc000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4028d1c000)

[harold-b]

Yes but the distros usually have different versions of GCC, which will yield different outputs, as will Clang.

[bladeuserpi]

Looking into the details where the performance is lost:

1a) RHEL-8.4

   119	Finished F1 generation in 1.33 seconds.
   121	Finished F1 sort in 7.24 seconds.
   124	  Finished pairing L/R groups in 8.5250 seconds. Created 4294941671 pairs.
   127	  Finished computing Fx in 8.1830 seconds.
   143	  Finished sorting in 13.09 seconds.
   144	Finished forward propagating table 2 in 30.13 seconds.
   147	  Finished pairing L/R groups in 8.4670 seconds. Created 4294837444 pairs.
   150	  Finished computing Fx in 8.6370 seconds.
   152	  Finished sorting in 13.58 seconds.
   153	Finished forward propagating table 3 in 31.00 seconds.
   156	  Finished pairing L/R groups in 8.4410 seconds. Created 4294667958 pairs.
   159	  Finished computing Fx in 8.8220 seconds.
   161	  Finished sorting in 13.47 seconds.
   162	Finished forward propagating table 4 in 31.04 seconds.
   165	  Finished pairing L/R groups in 8.4570 seconds. Created 4294329627 pairs.
   168	  Finished computing Fx in 8.8040 seconds.
   170	  Finished sorting in 13.42 seconds.
   171	Finished forward propagating table 5 in 30.99 seconds.
   174	  Finished pairing L/R groups in 8.4600 seconds. Created 4293654823 pairs.
   177	  Finished computing Fx in 8.4870 seconds.
   179	  Finished sorting in 12.95 seconds.
   180	Finished forward propagating table 6 in 30.20 seconds.
   183	  Finished pairing L/R groups in 8.4330 seconds. Created 4292276761 pairs.
   186	  Finished computing Fx in 8.0270 seconds.
   187	Finished forward propagating table 7 in 16.77 seconds.
   188	Finished Phase 1 in 178.70 seconds.
   191	  Finished prunning table 6 in 0.34 seconds.
   193	  Finished prunning table 5 in 7.55 seconds.
   195	  Finished prunning table 4 in 7.20 seconds.
   197	  Finished prunning table 3 in 7.07 seconds.
   199	  Finished prunning table 2 in 7.02 seconds.
   200	Finished Phase 2 in 29.47 seconds.
   203	  Finished compressing tables 1 and 2 in 20.72 seconds
   206	  Finished compressing tables 2 and 3 in 21.52 seconds
   209	  Finished compressing tables 3 and 4 in 21.36 seconds
   212	  Finished compressing tables 4 and 5 in 21.69 seconds
   215	  Finished compressing tables 5 and 6 in 22.77 seconds
   218	  Finished compressing tables 6 and 7 in 27.23 seconds
   220	Finished Phase 3 in 135.30 seconds.
   223	  Finished writing P7 in 0.45 seconds.
   225	  Finished writing C1 table in 0.00 seconds.
   227	  Finished writing C2 table in 0.00 seconds.
   229	  Finished writing C3 table in 0.34 seconds.
   230	Finished Phase 4 in 0.80 seconds.
   231	Finished plotting in 344.27 seconds (5.74 minutes).

1b) Ubuntu
  119	Finished F1 generation in 1.33 seconds.
   121	Finished F1 sort in 6.88 seconds.
   124	  Finished pairing L/R groups in 8.5860 seconds. Created 4294946841 pairs.
   127	  Finished computing Fx in 8.2450 seconds.
   143	  Finished sorting in 15.91 seconds.
   144	Finished forward propagating table 2 in 33.09 seconds.
   147	  Finished pairing L/R groups in 8.3990 seconds. Created 4294925770 pairs.
   150	  Finished computing Fx in 8.5450 seconds.
   152	  Finished sorting in 17.28 seconds.
   153	Finished forward propagating table 3 in 34.52 seconds.
   156	  Finished pairing L/R groups in 8.3750 seconds. Created 4294794102 pairs.
   159	  Finished computing Fx in 8.9210 seconds.
   161	  Finished sorting in 17.23 seconds.
   162	Finished forward propagating table 4 in 34.82 seconds.
   165	  Finished pairing L/R groups in 8.3740 seconds. Created 4294632504 pairs.
   168	  Finished computing Fx in 8.9600 seconds.
   170	  Finished sorting in 17.22 seconds.
   171	Finished forward propagating table 5 in 34.85 seconds.
   174	  Finished pairing L/R groups in 8.3700 seconds. Created 4294313130 pairs.
   177	  Finished computing Fx in 8.6860 seconds.
   179	  Finished sorting in 15.92 seconds.
   180	Finished forward propagating table 6 in 33.27 seconds.
   183	  Finished pairing L/R groups in 8.4050 seconds. Created 4293664765 pairs.
   186	  Finished computing Fx in 8.2180 seconds.
   187	Finished forward propagating table 7 in 16.92 seconds.
   188	Finished Phase 1 in 195.68 seconds.
   191	  Finished prunning table 6 in 0.33 seconds.
   193	  Finished prunning table 5 in 7.63 seconds.
   195	  Finished prunning table 4 in 7.28 seconds.
   197	  Finished prunning table 3 in 7.14 seconds.
   199	  Finished prunning table 2 in 7.09 seconds.
   200	Finished Phase 2 in 29.72 seconds.
   203	  Finished compressing tables 1 and 2 in 21.88 seconds
   206	  Finished compressing tables 2 and 3 in 22.42 seconds
   209	  Finished compressing tables 3 and 4 in 22.62 seconds
   212	  Finished compressing tables 4 and 5 in 23.04 seconds
   215	  Finished compressing tables 5 and 6 in 24.16 seconds
   218	  Finished compressing tables 6 and 7 in 26.69 seconds
   220	Finished Phase 3 in 140.82 seconds.
   223	  Finished writing P7 in 0.47 seconds.
   225	  Finished writing C1 table in 0.00 seconds.
   227	  Finished writing C2 table in 0.00 seconds.
   229	  Finished writing C3 table in 0.35 seconds.
   230	Finished Phase 4 in 0.82 seconds.
   231	Finished plotting in 367.04 seconds (6.12 minutes).

2) Discussion of results
- Each sort loses 3-4 seconds, causing phase1 to lose 15seconds
- compressing tables loses ~1 seconds each

So the main loss is in "sorting".

[bladeuserpi]

Yes but the distros usually have different versions of GCC, which will yield different outputs, as will Clang.

I compiled only once (some weeks ago with RHEL-8.x)
md5sum bladebit
9da6366fbd61e7902509076add3d4ddc bladebit

The SAME executables runs fine on all 3 distributions (of course it uses the system libraries as shown with ldd which are different)

[harold-b]

Oh. Your RHEL-compiled exe was ABI compatible with Ubuntu?

[bladeuserpi]

Oh. Your RHEL-compiled exe was ABI compatible with Ubuntu?

Yes!