Cube Example
Aurora Directory: /home/bylaska/PWDFT/QA/Cube
-rw-r--r-- 1 bylaska users 52246 Apr 15 21:36 C.psp
-rw-r--r-- 1 bylaska users 7781 Apr 15 21:32 cube1.nw
-rw-r--r-- 1 bylaska users 136599 Apr 15 21:38 cube1.out
-rw-r--r-- 1 bylaska users 24586666 Apr 15 21:36 C.vpp
-rw-r--r-- 1 bylaska users 7763 Apr 16 21:37 distort.nw
-rw-r--r-- 1 bylaska users 259494 Apr 16 23:58 distort.out12
-rw-r--r-- 1 bylaska users 260152 Apr 16 23:28 distort.out24
-rw-r--r-- 1 bylaska users 258092 Apr 17 02:03 distort.out36
-rw-r--r-- 1 bylaska users 257648 Apr 16 23:07 distort.out48
-rw-r--r-- 1 bylaska users 262023 Apr 17 01:46 distort.out60
-rw-r--r-- 1 bylaska users 258010 Apr 17 00:14 distort.out72
-rw-r--r-- 1 bylaska users 170194 Apr 17 01:15 distort.out84
-rw-r--r-- 1 bylaska users 259154 Apr 17 00:32 distort.out96
-rw-r--r-- 1 bylaska users 39212 Apr 15 21:36 H.psp
-rw-r--r-- 1 bylaska users 9834810 Apr 15 21:36 H.vpp
-rw-r--r-- 1 bylaska users 52208 Apr 15 21:36 N.psp
-rw-r--r-- 1 bylaska users 24586666 Apr 15 21:36 N.vpp
-rw-r--r-- 1 bylaska users 52248 Apr 15 21:36 O.psp
-rw-r--r-- 1 bylaska users 24586666 Apr 15 21:36 O.vpp
drwxr-xr-x 2 bylaska users 12288 Apr 17 02:03 perm
-rw-r--r-- 1 bylaska users 61050 Apr 16 01:02 pwdft.json
-rw-r--r-- 1 bylaska users 1504678024 Apr 16 01:16 pwdft.movecs
-rw-r--r-- 1 bylaska users 91325 Apr 15 21:36 Zn.psp
-rw-r--r-- 1 bylaska users 63924778 Apr 15 21:36 Zn.vpp
This input file performs a plane-wave DFT calculation using the pwdft code (NWChemEx), followed by generation of a total electron valence density cube file for visualization.
- Code: NWChemEx (PWDFT code)
- Task: Energy gradient + cube file generation
- Goal: Validate valence density output and enable isosurface visualization
cutoff: wfnc=30 au, density=60 au mpirun -np 1 ../../build_cuda2/pwdft distort.nw
| Column | Meaning |
|---|---|
| machine | Name of the machine (Aurora) |
| nodes | Number of compute nodes |
| ngpus | Total GPUs (12 per node → 1 GPU/socket) |
| cputime | Total CPU-side runtime (seconds per step) |
| non-local | Non-local pseudopotential ops |
| ffm | tall skinny x tall skinny |
| fmf | tall skinny x Mat |
| fft | Time spent in FFTs |
| diagonalize | Time spent in diagonalization / eigensolver |
Overview
- FFT scaling saturates at ~6–8 nodes
- Other operations (nonlocal ops, "ffm", "fmf") remain small and manageable
- Suggests FFT and diagonalization are the dominant scaling bottlenecks
- Entering the latency-dominated regime beyond 6 nodes
- At 6 nodes (72 GPUs), total time is ~6.5x faster than 1 node — very healthy
- Diagonalization is still <10ms — you could consider overlap or GPU-native slvers here if it grows
- FFT time doesn’t decrease further beyond 6 nodes — communication cost becomes dominant
| machine | nodes | ngpus | cputime | non-local | ffm | fmf | fft | diagonalize |
|---|---|---|---|---|---|---|---|---|
| Aurora | 1 | 12 | 2.850e+00 | 7.563e-02 | 2.511e-02 | 4.039e-02 | 1.264e+00 | 5.293e-03 |
| Aurora | 2 | 24 | 1.906e+00 | 4.293e-02 | 1.713e-02 | 2.209e-02 | 5.783e-01 | 4.617e-03 |
| Aurora | 3 | 36 | 2.139e+00 | 4.607e-02 | 2.238e-02 | 2.539e-02 | 6.352e-01 | 6.734e-03 |
| Aurora | 4 | 48 | 1.965e+00 | 3.587e-02 | 1.788e-02 | 1.870e-02 | 4.631e-01 | 5.932e-03 |
| Aurora | 5 | 60 | 4.512e-01 | 4.256e-02 | 2.286e-02 | 2.160e-02 | 6.076e-01 | 8.305e-03 |
| Aurora | 6 | 72 | 4.330e-01 | 6.061e-02 | 1.605e-02 | 1.502e-02 | 4.460e-01 | 5.916e-03 |
| Aurora | 7 | 84 | 3.164e-01 | 1.735e-02 | 4.685e-03 | 5.957e-03 | 1.422e-01 | 1.805e-03 |
| Aurora | 8 | 96 | 3.700e-01 | 4.986e-02 | 1.871e-02 | 1.481e-02 | 4.893e-01 | 8.064e-03 |
