MHDTurbulence

This repository contains a 3D MHD solver (MPI + OpenACC/OpenMP target/OpenMP). The current default test problem is the Kelvin–Helmholtz instability.

---

How to copy the source code

After you login the server, copy the programs.

git clone git@github.com:cfcanaoj/MHDTurbulence MHDTurbulence
cd MHDTurbulence

---

How to run

Compile

To run the code, first compile source on a GPU server. The OpenACC version (src_f90_acc_device) is the default and recommended setup.

cd src_f90_acc_device
make

The executable Simulation.x is created in ../exe directory.
If you prefer another implementation, use the appropriate directory and codes listed below.

directory	Language	GPU/CPU	Parallelization
src_f90_acc_device	Fortran	GPU	MPI OpenACC
src_f90_omp_device	Fortran	GPU	MPI OpenMP Target
src_f90_omp_host	Fortran	CPU	MPI OpenMP
src_cpp_omp_device	C++	GPU	MPI OpenMP
src_cpp_kokkos_device	C++	GPU	MPI Kokkos
src_cpp_kokkos_host	C++	CPU	MPI Kokkos

For stable simulations, please use the OpenACC (src_f90_acc_device) or CPU (src_f90_omp_host) versions.

Execution

Copy the batch script and submit the job. The script for Slurm is prepared.

cd ../exe
cp ../misc/sj_g00.sh .
sbatch sj_g00.sh

Batch scripts depend on your parallelization scheme and environment.

script	GPU/CPU	directory
sj_g00.sh	GPU	src_f90_acc_device, src_f90_omp_device, src_cpp_omp_device, src_cpp_kokkos_device
sj_xd.sh	CPU	src_f90_omp_host, src_cpp_kokkos_host

---

Data Output Specification

The code supports multiple output modes depending on your purpose: benchmarking, quick inspection, visualization, or detailed analysis.

0. Performance Measurement Mode (No Intermediate Output)

For performance benchmarking, the code provides an option in config.f90 to suppress intermediate outputs. In this case the initial and final snapshots are only damped.

logical,parameter:: benchmarkmode = .true. !! If true, only initial and final outputs are damped.

When enabled:

• Intermediate snapshots are not written
• I/O overhead is minimized
• Only initial and final data are output

This mode is recommended for:

• Strong/weak scaling tests
• GPU/CPU performance comparisons
• Pure solver benchmarking

Performance Information

During execution, the following performance information is printed to the standard output:

sim time [s]: 1.145357e+03
time/count/cell : 2.262434e-09

• sim time [s] : total wall-clock time of the main loop of simulation
• time/count/cell : wall-clock time per cell per time step These values are useful for benchmarking and performance comparison.

Benchmark Results

The typical performance in representative environments is shown below.

Code	Grid size x Physical time	Wall time [s]	time/cell/step [s]	Environment
src_f90_acc_device	150^3 x 15	421	1.14e-9	CfCA GPU server, A100 4 GPU
src_f90_omp_device	150^3 x 15	1299	3.52e-9	CfCA GPU server, A100 4 GPU
src_cpp_omp_device	150^3 x 15	1151	3.18e-9	CfCA GPU server, A100 4 GPU
src_f90_omp_host	156^3 x 15	3293	7.5e-9	CfCA XD2000, Xeon Max 1 node
src_f90_omp_host	156^3 x 15	3239	7.5e-9	Miyabi-C, Xeon Max 1 node

• Grid size: number of cells in each direction
• Physicsl time (t): physical end time of the simulation
• Wall time: total elapsed wall-clock time
• time/cell/step: wall-clock time per cell per time step

Quick check

RealTimeAnalysis evaluates bulk diagnostics during the run and writes them to t-prof.csv. The file t-prof.csv contains four columns:

1. time
2. mix
3. sqrt(<v_y^2>)
4. A exp(\Gamma t)

The third columm is an average velocity that should be compared with the forth column in an appropriate time $1<t<6$. The fourth column is a reference exponential growth curve, $A\exp(\Gamma t)$ , with the hard-coded parameters. The parameters are used as a practical comparison metric for KH growth in this setup.