Reconstructing large meshes doesn't finish, stops with incomplete point file

[kastnerp]

When I am trying to reconstruct a large mesh that was meshed with snappyHexMesh, I am facing problems with reconstructParMesh -constant.
The job stops at Writing merged mesh to "C:/Users/..." with the point file being 2'184'368'264 bytes large but doesn't abort --- it just stops to continue.
The same case fully reconstructs with OpenFOAM in a Windows Docker environment without any problems.
Are there any caps on how large individual files can be due to the way BlueCFD is ported?

Best,

Patrick

[wyldckat]

There shouldn't be such a limitation with blueCFD-Core...

I need to know a few details:

1. How many cells does the mesh have after snappyHexMesh is finished? You can check this by running:

foamJob -s -p checkMesh

2. How big are these files inside each parallel folder? You can check this by running:

ls -l processor*/constant/points

3. Are you writing in ASCII or binary format? Namely the writeFormat setting defined in the file system/controlDict.

[kastnerp]

Dear Bruno,

Thank you for taking the time to look into this!

How many cells does the mesh have after snappyHexMesh is finished? You can check this by running:

cells: 64508056

How big are these files inside each parallel folder? You can check this by running:

$ ls -l processor*/constant/polyMesh/points
-rw-r--r-- 1 xxx Domain Users 356709811 Apr 22 05:51 processor0/constant/polyMesh/points
-rw-r--r-- 1 xxx Domain Users 368350699 Apr 22 05:51 processor1/constant/polyMesh/points
-rw-r--r-- 1 xxx Domain Users 368438779 Apr 22 05:51 processor2/constant/polyMesh/points
-rw-r--r-- 1 xxx Domain Users 365307739 Apr 22 05:51 processor3/constant/polyMesh/points
-rw-r--r-- 1 xxx Domain Users 362528059 Apr 22 05:51 processor4/constant/polyMesh/points
-rw-r--r-- 1 xxx Domain Users 367162315 Apr 22 05:51 processor5/constant/polyMesh/points

I remember I had the same problem occur for cases of similar size that were decomposed for even more processors (~20) but didn't troubleshoot the issue at the time.

Are you writing in ASCII or binary format? Namely the writeFormat setting defined in the file system/controlDict.

I am writing everything in binary format as of now.

Best,

Patrick

[wyldckat]

Hi Patrick,

I haven't had time to look into this and there are a few more questions I forgot to ask sooner:

1. Which exact version of blueCFD-Core are you using?

   I ask this because we haven't had this issue with the latest versions and I only expect that this would potentially be an issue with our older ports of OpenFOAM 1.7 or 2.0.

2. How many points and how many faces does the full mesh have?

3. Does this error occur in both binary and ASCII formats?

Best regards,
Bruno

[kastnerp]

Hi Bruno,

Which exact version of blueCFD-Core are you using?

I am using blueCFD-Core 2017-2.

How many points and how many faces does the full mesh have?

    points:           91187111
    faces:            218842709
    internal faces:   198998583
    cells:            64508056
    faces per cell:   6.4773505
    boundary patches: 409

Does this error occur in both binary and ASCII formats?

(I am checking this right now and will edit later when I have the results.)

Best,

Patrick

[wyldckat]

Hi Patrick,

I've tested the following case in a machine with 32GB of RAM. Attached is the file motorBike.meshTest.tar.gz, in which you can unpack and run the script Allrun as usual, e.g.:

tar -xf motorBike.meshTest.tar.gz
cd motorBike.meshTest
./Allrun

This case is a modified motorBike tutorial case, which only runs blockMesh, decomposePar and reconstructParMesh, in an attempt to recreate a similar situation to yours. The file constant/polyMesh/faces should have 2.3GB of disk space and not fail in between.

Now this does lead me to the following question: How much RAM does you machine have? Does it have at least 32GB or 64GB?

Best regards,
Bruno

[kastnerp]

Hi Bruno,

your case works just fine, thank you for preparing it. My initial problem still persists however.
I tried to reproduce my problem by adapting a tutorial case just like you did but couldn't reproduce it yet. I'll follow up with this in a couple of days...

Now this does lead me to the following question: How much RAM does you machine have? Does it have at least 32GB or 64GB?

It's a university computer with 128 GB RAM so I don't think this was the issue.

Best,

Patrick

[wyldckat]

OK, then when you get a chance, test with the following command:

foamJob -s -p checkMesh -allTopology -allGeometry

this is for checking the mesh in parallel, while the mesh is decomposed. I suspect that there is a flaw in the mesh that is resulting in this crash.