Parallelizing wxvx

[timothyas]

As noted in #49, I ran into some trouble while scaling up the stats computation via wxvx. I'm not really sure if some of the trouble stemmed from using multiple tasks/processes within wxvx, but I decided to try something else to parallelize the operations and I thought I'd share them here in case its useful to others.

I ended up using GNU Parallel basically because that's what Perlmutter recommends since it enables one big slurm job rather than overloading the scheduler. My process was as follows:

Create a slurm script that pulls in the observations and then creates an input file to be processed with GNU parallel.

wxvx -c obs.wxvx.yaml -t obs -n 16 > log.wxvx.obs 2>&1

# Write a yaml for each cycle and process separately
echo "Starting grids and stats"
export cycles_dir="cycles-wxvx"
find ${cycles_dir}  -maxdepth 1 -type f -name "*.yaml" > input.txt
parallel -j 128 ./parallel_wxvx.sh {} :::: input.txt

The cycles directory has 1 separate yaml for each cycle processed by wxvx, created with a separate python script. Then, the parallel_wxvx.sh script is what does the main tasks:

yamlfile=$1 # Use a local variable for the input path

# Extract the date string (e.g., 20230201T060000) from the filename
base_filename=$(basename "$yamlfile")
file_date_str=${base_filename%%.*}

echo "-> [PID: $$] Running wxvx for ${file_date_str}"

# Run the commands, redirecting output.
# The 'cycles_dir' variable is available within the function.
wxvx -c "$yamlfile" -t grids > "${cycles_dir}/log.wxvx.grids.${file_date_str}" 2>&1
wxvx -c "$yamlfile" -t stats > "${cycles_dir}/log.wxvx.stats.${file_date_str}" 2>&1

echo "-> [PID: $$] Finished ${file_date_str}"

This ended up working pretty well for me. Probably not the cleanest approach, but I just thought I'd share!

[maddenp-cu]

@timothyas Thanks for sharing this recipe! I'm sure it will be useful to others, even after we get to the bottom of whatever was causing the concurrency issues you observed. I'll be testing it out myself to make sure I understand the technique.