Creates separate output directories for workflow run in pipeline associated with runner name

[singhd789]

• creates output dir based off runner number (to associate with runner if there are multiple running)
• (to-do: ensures there's no race condition for 2 runners accessing same output directory)
• could potentially solve permission denied errors we see every once in a while with the ppp-setup folder in the cloud?

Ciheim brought in this branch in his fork, created 2 different runners, and started them around the same time:

• they both ran at the same time (no waiting around)
• created different ppp-setup output directories with cylc-run dir
  • can see this with the uploaded archive files:
  • runner 1: https://github.com/Ciheim/fre-workflows/actions/runs/19113157075/job/54615250677#step:7:4
  • runner2: https://github.com/Ciheim/fre-workflows/actions/runs/19113159063/job/54615257983#step:7:4

BUT - in those pipelines, the workflow took 3 HOURS AND 33 MINUTES, no bueno
Currently testing running 2 at the same time but accessing different history file locations

Here is successful output from a pipeline run:
https://github.com/NOAA-GFDL/fre-workflows/actions/runs/20470921836/job/58825830669

[singhd789]

Note: Could still be relevant but looking into free github runners at the moment

[singhd789]

Update (7/16): not entirely sure if it's needed or desired. It could be desired if more than 1 person wants to test/use the runner at a time. I'll push to 2027 until it potentially becomes wanted

[ilaflott]

Update (7/16): not entirely sure if it's needed or desired. It could be desired if more than 1 person wants to test/use the runner at a time. I'll push to 2027 until it potentially becomes wanted

peeking at this i think it's got some clear improvements. why is it in draft? is the conflict tractable? do the numerous commits reflect fragility, or something that was frustratingly opaque?

[singhd789]

Update (7/16): not entirely sure if it's needed or desired. It could be desired if more than 1 person wants to test/use the runner at a time. I'll push to 2027 until it potentially becomes wanted

peeking at this i think it's got some clear improvements. why is it in draft? is the conflict tractable? do the numerous commits reflect fragility, or something that was frustratingly opaque?

The state of the runners and the cloud resource were in flux for a bit.

1. During the times of moving from one cloud resource to another (google to AWS), we had to reconfigure and recreate runners
2. Pertaining to cloud allocation, another runner was also just using more. I don't think our workflow uses too much buuuuut this was in draft due to the time the resource was having allocation problems
3. At the time when this was being developed, not too many people were using it much, so 1 runner sufficed.

I agree, there are probably some improvements in here and having at least 2 runners could still be good, we might just need to re-assess where we are. @Ciheim, do you agree?