Error in polarisation pipeline when stokes I cube and stokes QU cubes are launched at the same time.

[ErikOsinga]

I've found that WSClean can crash (maybe this is filesystem dependent) when multiple tasks are running against the same MS.

Note that -no-update-model was enabled in my tests (no_update_model_required: true ).

The fix might be to switch around the loop in the polarisation_pipeline.py from looping over all MSes and then doing every polarisation per MS to instead looping over all polarisations, and then doing one polarisation per MS, waiting for the results of one polarisation to finish completely, before moving on to the next polarisation.

[tjgalvin]

I am a little surprised by this one. In the past we have been able to get many wsclean instances runners running on when working channel / scan wise on an MS in.

What was the error thrown?It would be good to understand this behavior before we unknowingly end up relying on quirks of a filesystem

@AlecThomson - thoughts ?

[ErikOsinga]

e.g.

[AlecThomson]

The above screenshot seems to show Stokes V being run. I wonder if there's a name clash between Stokes I and V (i.e. single pol) that's causing the issue?

[tjgalvin]

I don't think it is that. In the -name attribbute there is a .v indicator, which would brack the name clash.

My bet is that the -temp-dir is not unique between jobs. When a -temp-dir is specified, a hold_then_move context manager is invoked when wsclean is executed. When the context manager exits, all files in that directory are moved to a final location. If more than one process is using this context manager which ever one finishes first takes everything.

Can you see what happens if you turn off the usage of temp_dir in your imaging strategy file.

[ErikOsinga]

Just noting that it seems like it's a WSClean restarting issue

[ErikOsinga]

I'm running it on a slightly different cluster now (trillium instead of niagara) and disabled the temp_dir, and I'm getting a new weird error that I think might also be related to multiple processes trying to access the same MS file?

+ >>> Error opening meta file for ms /scratch/eosinga/doradogroup/production/697826981770314/SB697826981770314.NGC1566_band1.beam28.round4.ms, dataDescId 0

These are the full logs for the dask worker that crashed... Not sure what's going on here 😕

dask-worker-165264.log

[tjgalvin]

Are these with wscleans no_model_uprate option set to true in the imaging strategy? I can try to reproduce tomorrow. In the mean time can you share the strategy yaml and the cli you are using?

…

On Wed, Oct 1, 2025, 10:03 PM Erik Osinga ***@***.***> wrote: *ErikOsinga* left a comment (flint-crew/flint#332) <#332 (comment)> I'm running it on a slightly different cluster now (trillium instead of niagara) and disabled the temp_dir, and I'm getting a new weird error that I think might also be related to multiple processes trying to access the same MS file? + >>> Error opening meta file for ms /scratch/eosinga/doradogroup/production/697826981770314/SB697826981770314.NGC1566_band1.beam28.round4.ms, dataDescId 0 These are the full logs for the dask worker that crashed... Not sure what's going on here 😕 dask-worker-165264.log <https://github.com/user-attachments/files/22642161/dask-worker-165264.log> — Reply to this email directly, view it on GitHub <#332 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACQOAJRIJQER3VJ5WAX3G5T3VPNJJAVCNFSM6AAAAACFUTWZ3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNJWGUYDINRZGA> . You are receiving this because you commented.Message ID: ***@***.***>

[ErikOsinga]

Yup, I've got no_update_model_required set to true, here's my config files and how I called it

go_flint_channel_imaging.sh

config_channel_imaging.yaml

trillium.yaml

[ErikOsinga]

Note that I'm now trying to run it with

temp_dir: /scratch/eosinga/doradogroup/wsclean_tmpdir/$FLINT_UUID/

and the error did not pop up yet... Hoping this will indeed resolve it 🤞

[ErikOsinga]

No failed WSClean tasks with that temp_dir parameter.. At least for one run.

[tjgalvin]

Excellent! I am sure that an alternative would be to rework the initial flow to ensure a single process is accessing each MS at a time. But given how you were already using the temp_dir option to something that I thought was unique to each job / node, my curiosity is screaming to dig deeper.