About the computation efficiency

[haibianshifeng]

I use the same tutorials as following：https://github.com/OpenFOAM/OpenFOAM-dev/tree/master/tutorials/multiphase/interFoam/laminar/capillaryRise。

I run it on the same compute, one is installed with ubuntu 14.04 and OF 5, another one is installed with windows 7 and blueCFD 2017.

However, the time of running on ubuntu is 20 minutes and the time of running on windows is 45 minutes.

So, the question is that why the computation efficiency is so slow on windows. Do you have trip to improve the computation efficiency.

[wyldckat]

Unfortunately we haven't had the time needed to conduct timing experiments on Windows and Linux, but past experiments led us to up to ~50% more time to run a case: https://sourceforge.net/p/bluecfd-sc/wiki/Runtimes202_211/

Now taking more than twice as long to run, seems to be due to missing process binding between the process and the logical CPU on that machine. In other words, if the process is allowed to jump between cores, this means that time is spent switching between cores will lead to various issues:

1. RAM access latency increases, because the process will spend more time re-getting the data from RAM, instead of cache.

2. The power management mechanisms will go up and down on power levels, depending on the core that it's using at a particular moment; for example, an idle core at 1.1GHz will have to speed up to 2.5 or 3.7 GHz when it receives the process to work on, but a moment later the process skips to another core that may be idle.

3. On machines with multiple sockets (usually on motherboards with Intel Xeon processors), if the process jumps to another socket, then it needs to retrieve the data from RAM through the other processor, which will further increase the time it takes to run.

Another thing that could be reducing performance is the anti-virus, which may be up and running and scanning the computer, while the simulation is running.

If possible, please run the tutorial case once again and use the Task Manager to force the processor affinity for the interFoam application to only use a specific core. Furthermore, if possible, disable the CPU power reduction policy and make it always use the maximum throughput, to see if it improves performance.

Please do tell us of your findings on this!

Beyond these, there are at least a few known/expected bottlenecks for our ports on Windows:

1. The random number generator algorithm on Linux is a lot more efficient than the one we use on Windows. This shouldn't impact all that much, since this should only be used on some algorithms for probability related modeling (usually Lagrangian modeling).

2. File read-write efficiency on Windows isn't known to be better than Linux's.

3. Math libraries on Windows that are used by MinGW-w64 may have less performance, due to various reasons. I don't know how well they work nowadays, but some 10 years ago, MinGW was using the outdated math libraries that were in an Windows' old runtime library (msvcrt.dll if I remember correctly).

[wyldckat]

I was curious and I've run some tests, although I don't have an Ubuntu installation directly installed on my workstation at the office. However, here are some timings for that tutorial case, with OpenFOAM 5 Deb on Windows 10 WSL with Ubuntu 16.04 versus blueCFD-Core 2017-2.

The CPU the workstation has got is an i5-6600 @ 3.3GHz, running at ~3.75GHz when using a single core. This is the machine blueCFD-Core 2017-2 was built with.

I only left if running for 0.05s on each run, namely that it only spends some time reading and writing to disk and because I couldn't spend much time on this.

• blueCFD-Core 2017-2:

  • With balanced mode and no affinity: 183s
  • With performance mode and no affinity: 183s
  • With performance mode and with affinity: 183s
  • Using time ./Allrun, the differences were only around 0.5s apart from each other.
  • Compiler used: GCC 7.2.0

• OpenFOAM 5 Deb package on Ubuntu 16.04 via Windows 10 WSL:

  • With balanced mode and no affinity: 164s
  • With performance mode and no affinity: 162s
  • Compiler used: GCC 5.4.0

• OpenFOAM 5.x, on CentOS 6.9 inside a VM with VirtualBox, with the same Windows 10 host...

  • With balanced mode and no affinity: 181s
  • With performance mode and no affinity: 180s
  • Compiler used: GCC 4.8.5

All builds and OS's are in x86_64/x64 modes, 32-bit labels and Double Precision.

So it's at least roughly 12% slower using blueCFD-Core 2017, when comparing these modes, where using VirtualBox doesn't give much of an advantage in performance.

Edits:

1. Added information for the CentOS 6.9 build.

[wyldckat]

@haibianshifeng If possible, can you tells us the CPUs you are testing blueCFD-Core with?

[haibianshifeng]

The system for testing is Windows 7-64 bits. The software for testing is BlueCFD 2017-2. The CPU is Intel(R) Core(TM) i7-5960X @3.00GHz. *edit: Removed sensitive information.*

[wyldckat]

I've done additional tests with my machine at home, which has an AMD A10-7850K, 32GB of RAM at 2133MHz and a Samsung SSD 850 Evo 250GB.

The tests performed were with:

• OpenFOAM 5 Deb on Windows 10 WSL with Ubuntu 16.04
• OpenFOAM 5 Deb on Windows 10 WSL with Ubuntu 18.04
• blueCFD-Core 2017-2 on Windows 10
• OpenFOAM 5 Deb on Ubuntu 16.04
• OpenFOAM 5.x custom build on Ubuntu 16.04, using the native GCC

Note: Edited this comment with new information as I ran each test, but I've now finished it (over a period of 4h30).

Again, I only left it running for 0.05s on each run, namely that it only spends some time reading and writing to disk and because I couldn't spend much time on this.

• blueCFD-Core 2017-2:

  • With balanced mode and no affinity: 323s (at around 3.9GHz)
  • With performance mode and no affinity: 325s (at around 3.9GHz)
    • By setting the CPU to 100%, it looks like it didn't allow for the single-core turbo mode to kick in...
  • With balanced mode and with affinity: 331s (at around 3.9GHz)
    • Looks like binding to Core 0 wasn't much of a good idea...
  • Compiler used: GCC 7.2.0

• OpenFOAM 5 Deb package on Ubuntu 16.04 via Windows 10 WSL:

  • With balanced mode and no affinity: 298s (at around 3.9GHz)
  • With performance mode and no affinity: 301s (at around 3.9GHz)
    • By setting the CPU to 100%, it looks like it didn't allow for the single-core turbo mode to kick in...
  • Compiler used: GCC 5.4.0

• OpenFOAM 5 Deb package on Ubuntu 18.04 via Windows 10 WSL:

  • With balanced mode and no affinity: 296s (at around 3.9GHz)
  • Compiler used: GCC 7.3.0

• OpenFOAM 5 Deb package on Ubuntu 16.04:

  • With on-demand mode and no affinity: 299s (at around 3.7GHz)
  • With performance mode and no affinity: 298s (at around 3.7GHz)
  • With on-demand mode and with affinity: 299s (at around 3.7GHz)
  • Compiler used: GCC 5.4.0

• OpenFOAM 5.x on Ubuntu 16.04:

  • With balanced mode and no affinity: 299s (at around 3.7GHz)
  • Compiler used: GCC 5.4.0

A few additional notes about the timings above:

1. 3.9GHz on Windows vs 3.7GHz on Ubuntu, has to do with how the kernel is getting information from the CPU timings; in other words, the actual speeds are pretty much the same, it's just that a different multiplier is being read from the CPU, hence the discrepancy. Allegedly I could install a different kernel build or module to get the correct timings, but I haven't bothered.

2. As mentioned before, these timings do not account well for disk performance, given that the majority of the time spent was in solving the case; furthermore, the case is small enough that it is read from cache on RAM and written to cache on RAM.

Conclusion:

• Even when using a different CPU architecture, the performance was only around 8% slower when using blueCFD-Core 2017-2 (323/299 ~= 1.08).

• The custom build of OpenFOAM 5.x did not use any special CPU native options, therefore no additional performance was revealed.

[wyldckat]

Following on the previous comment, I've then left the case running for its full completion to see what timings to expect:

• OpenFOAM 5.x on Ubuntu 16.04:

  • Solver time in ascii mode: ExecutionTime = 2846.28 s ClockTime = 2850 s
  • Total time in ascii mode:

    real	47m29.581s
    user	46m26.708s
    sys	1m0.020s
    

  • Solver time in binary mode: ExecutionTime = 2783.81 s ClockTime = 2786 s
  • Total time in binary mode:

    real	46m29.412s
    user	45m29.680s
    sys	0m54.576s
    

• OpenFOAM 5.x on Ubuntu 16.04, with different CPU controller:

  • Solver time in ascii mode: ExecutionTime = 2724.12 s ClockTime = 2726 s
  • Total time in ascii mode:

    real	45m28.041s
    user	44m24.960s
    sys	0m59.604s
    

  • Solver time in binary mode: ExecutionTime = 2727.36 s ClockTime = 2730 s
  • Total time in binary mode:

    real	45m30.279s
    user	44m30.204s
    sys	0m57.444s
    

• blueCFD-Core 2017-2:

  • Solver time in ascii mode: ExecutionTime = 5303.58 s ClockTime = 5303 s
  • Total time in ascii mode:

    real    88m25.207s
    user    0m0.015s
    sys     0m0.060s
    

  • Solver time in binary mode: ExecutionTime = 3114.86 s ClockTime = 3115 s
  • Total time in binary mode:

    real    51m56.400s
    user    0m0.015s
    sys     0m0.123s
    

• OpenFOAM 5 Deb package on Ubuntu 16.04 via Windows 10 WSL:

  • Solver time in ascii mode: ExecutionTime = 2824.73 s ClockTime = 2829 s
  • Total time in ascii mode:

    real    47m10.582s
    user    43m46.281s
    sys     3m19.016s  
    

  • Solver time in binary mode: ExecutionTime = 2840.5 s ClockTime = 2842 s
  • Total time in binary mode:

    real    47m24.309s
    user    44m9.078s
    sys     3m12.031s
    

Thoughts and conclusions:

• It is not clear why running in ascii on Ubuntu via WSL was a bit faster than in binary, although it is possible that since the ascii mode was run right after the binary mode, it may have used caching a bit better.

  • Actually, after comparing the log files side by side, there is a clear indication that there is something introducing randomness into the simulation, therefore the results aren't exactly the same, hence taking different times to run.

• If the time application is accurate when running under WSL, then it is possible that the CPU on Linux is really not properly taking advantage of the maximum performance capabilities, namely not using the turbo mode to its full extent; furthermore, that it takes 3 times more to write to disk while under WSL?

  • Both of these hypothesis requires further investigation (CPU limitation on Ubuntu and disk writing performance).

• Writing in ascii with blueCFD-Core is painfully slow. This will have to be dealt with on its own task, although it isn't exactly a high priority. The primary suspect would be that flushing to disk is always used when writing in ascii mode, which would account properly for the massive performance bottleneck.

• Even when running in binary mode, it is roughly 12% slower to use blueCFD-Core 2017-2 than running OpenFOAM natively on Linux... Although these values could change depending on further investigation, namely:

  • If it's flushing to disk at the end of each buffer, this could explain the change from 8% to 12% performance loss, when compared to running directly on Ubuntu.

  • If it's also associated to CPU turbo mode not being properly used on Ubuntu, the difference would increase considerably.

[wyldckat]

@haibianshifeng I don't know if you've been periodically looking into this report page, but I've been editing the past couple of comments of mine with more details on performance issues. In essence, if you run the case with writeFormat set to binary, you will get the most performance from your machines. However, ascii mode on Windows seems to be having serious performance issues, which I'm going to isolate on this comment.

---

Using the code provided on this page: OF23_IOPerformance_Analysis_2, section Sixth approach, it's easier to try and isolate how each double value export functions work.

On Ubuntu, used ondemand CPU power scheduler and bound to the core 0, to try and get the most out of the performance.

• On Ubuntu 16.04, using the following command:

  (
  time taskset -c 0 ./ascii
  time taskset -c 0 ./ascii_sprintf_f
  time taskset -c 0 ./ascii_sprintf_g
  time taskset -c 0 ./binary
  )
  

  Got the following times, although this has a margin of error that can take longer up to 0.5s for the slower runs:

  real	0m2.519s
  user	0m2.436s
  sys	0m0.080s
  
  real	0m2.793s
  user	0m2.460s
  sys	0m0.076s
  
  real	0m1.953s
  user	0m1.884s
  sys	0m0.064s
  
  real	0m0.443s
  user	0m0.100s
  sys	0m0.048s
  

• On Windows 10, with blueCFD-Core 2017-2, using the following command:

   ( time ./ascii; time ./ascii_sprintf_f; time ./ascii_sprintf_g; time ./binary; )
  

  There seems to be a margin of error of 0.2s, for the slower ones:

  real    0m8.287s
  user    0m0.000s
  sys     0m0.000s
  
  real    0m6.433s
  user    0m0.000s
  sys     0m0.015s
  
  real    0m5.487s
  user    0m0.000s
  sys     0m0.015s
  
  real    0m0.890s
  user    0m0.000s
  sys     0m0.015s
  

  There is clearly a big problem with the conversion from double to ascii...

• On Ubuntu 16.04, under WSL, using the same command as on the real machine:

  real    0m2.568s
  user    0m2.453s
  sys     0m0.109s
  
  real    0m2.646s
  user    0m2.438s
  sys     0m0.188s
  
  real    0m2.003s
  user    0m1.828s
  sys     0m0.156s
  
  real    0m0.240s
  user    0m0.094s
  sys     0m0.141s
  

  There seemed to be a smaller error range, but the main disk here is an SSD and the real Ubuntu machine is using an HDD.

OK, the problem is very real when using ASCII on Windows, via GCC, being roughly 2.5x slower this way on all conversion methods.

---

When commenting out the file opening/write/close code on the ascii_sprintf_f code, it was possible to get the following timing on Windows 10:

real    0m4.901s
user    0m0.000s
sys     0m0.015s

On Ubuntu 16.04 via WSL:

real    0m2.253s
user    0m2.234s
sys     0m0.016s

Well, this is interesting.... it's a somewhat old and known problem: https://stackoverflow.com/questions/17236352/mingw-w64-slow-sprintf-in-cstdio
With this information, using stdio.h reduced the timing without writing down to:

real    0m3.478s
user    0m0.015s
sys     0m0.000s

And ascii_sprintf_g.c with stdio.h included first, did also speed up things a bit more:

real    0m4.825s
user    0m0.000s
sys     0m0.015s

Although it didn't reduce all that much, when compared to 0m5.487s... and the 2s on Ubuntu...

Fortunately someone did a better job in looking into this sometime ago: https://codeforces.com/blog/entry/47180

[wyldckat]

I kept forgetting to compare the log files when running in binary and in ascii on Windows and have finally (sort-of) pinpointed the problem: there is something affecting performance when running in ascii mode. The following contiguous time steps on ascii mode:

Courant Number mean: 0.0047381 max: 0.200875
Interface Courant Number mean: 0.000169456 max: 0.200875
deltaT = 1.4019e-005
Time = 0.299117

PIMPLE: iteration 1
MULES: Solving for alpha.water
Phase-1 volume fraction = 0.472119  Min(alpha.water) = -6.75066e-019  Max(alpha.water) = 1.00017
MULES: Solving for alpha.water
Phase-1 volume fraction = 0.472123  Min(alpha.water) = -7.31421e-019  Max(alpha.water) = 1.00017
DICPCG:  Solving for p_rgh, Initial residual = 0.00450714, Final residual = 0.000167714, No Iterations 3
time step continuity errors : sum local = 3.76151e-007, global = 4.99405e-010, cumulative = 1.78934e-007
DICPCG:  Solving for p_rgh, Initial residual = 0.000331231, Final residual = 1.56572e-005, No Iterations 16
time step continuity errors : sum local = 3.52233e-008, global = 1.10126e-010, cumulative = 1.79044e-007
DICPCG:  Solving for p_rgh, Initial residual = 5.15231e-005, Final residual = 9.98829e-008, No Iterations 162
time step continuity errors : sum local = 2.24621e-010, global = 5.94179e-013, cumulative = 1.79045e-007
ExecutionTime = 1809.34 s  ClockTime = 1809 s

Courant Number mean: 0.00465817 max: 0.19705
Interface Courant Number mean: 0.000165975 max: 0.19705
deltaT = 1.4019e-005
Time = 0.299131

PIMPLE: iteration 1
MULES: Solving for alpha.water
Phase-1 volume fraction = 0.472127  Min(alpha.water) = -1.40441e-018  Max(alpha.water) = 1.00017
MULES: Solving for alpha.water
Phase-1 volume fraction = 0.472131  Min(alpha.water) = -3.51098e-019  Max(alpha.water) = 1.00017
DICPCG:  Solving for p_rgh, Initial residual = 0.00465599, Final residual = 0.00022328, No Iterations 2
time step continuity errors : sum local = 5.02166e-007, global = 4.82269e-010, cumulative = 1.79527e-007
DICPCG:  Solving for p_rgh, Initial residual = 0.00028515, Final residual = 1.30929e-005, No Iterations 16
time step continuity errors : sum local = 2.95377e-008, global = 1.21727e-010, cumulative = 1.79649e-007
DICPCG:  Solving for p_rgh, Initial residual = 4.36855e-005, Final residual = 8.7545e-008, No Iterations 166
time step continuity errors : sum local = 1.97555e-010, global = -1.13206e-012, cumulative = 1.79648e-007
ExecutionTime = 4095.69 s  ClockTime = 4095 s

But there is no clear reason as to why this would have stalled for such a long time... so either there is an infinite loop somewhere or the Anti-virus kicked in and stalled the run...

If we remove this massive difference in time, the simulation would have taken 3017s, which is similar to the run in binary mode.

It's not the first time I see something like this and what comes to mind was an infinite loop that occurred several years ago in the Lagrangian code, where a loop for randomness calculations was not able to exiting the loop, because the value was always zero, due to the randomness range going from 0 to 65535... but I'm not familiar with the transport models in this case and will require additional studying of the problem.

[haibianshifeng]

Many thanks for your help and information.

I had test my model by using different writeFormat binary and ascii. I found that the time of writeFormat binary is much faster the time of writeFormat ascii. The conclusion is agree with your result.

However, I found the problem about the contiguous time steps. I do not know the reason. I guess the problem is the convergence of the pressure. I do not how to solve it.