Using --rename-ms causes gaincal_applycal_ms to fail if slurm job cancelled

[ddobie]

I noticed this a while ago but didn't have time to dig into it until now. If you are using --rename-ms then gaincal_applycal_ms can fail irrecoverably if the slurm job it's running inside is cancelled (e.g. due to timeout).

My understanding is that if a slurm job times out while running a task, the task is simply re-run from scratch on a different job. However, the use of rename-ms causes files to be moved and renamed, meaning that the input files no longer exist and hence the task fails almost immediately upon retry.

Short version of the relevant logs below, but I can supply complete ones if that's useful

First attempt times out with:

slurmstepd: error: *** JOB 6619916 ON john8 CANCELLED AT 2025-11-06T00:08:55 DUE TO TIME LIMIT ***

Second attempt starts 40 minutes later with:

Measurement set to be self-calibrated: ms=MS(path=PosixPath('/fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms'), column='CORRECTED_DATA', beam=None, spw=None, field=None, model_column='MODEL_DATA')

Updating gaincal options with: {'solint': '480s', 'calmode': 'ap', 'round': 0, 'minsnr': 0.0, 'uvrange': '>400m', 'selectdata': True, 'gaintype': 'G', 'nspw': 1}

Detected a previous round of self-calibration. 

Output MS name will be /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round3.ms.

Renaming /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms to /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round3.ms.

/fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round3.ms already exists. Removing it. 

ms.path=PosixPath('/fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms') ms.path.exists()=False

out_ms_path=PosixPath('/fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round3.ms') out_ms_path.exists()=True

Removing folder /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round3.ms

Encountered exception during execution:
Traceback (most recent call last):
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/prefect/engine.py", line 2169, in orchestrate_task_run
    result = await call.aresult()
             ^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/flint/prefect/common/imaging.py", line 304, in task_gaincal_applycal_ms
    return gaincal_applycal_ms(
           ^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/flint/selfcal/casa.py", line 276, in gaincal_applycal_ms
    cal_ms = copy_and_clean_ms_casagain(ms=ms, round=round, rename_ms=rename_ms)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/flint/selfcal/casa.py", line 91, in copy_and_clean_ms_casagain
    ms = rename_ms_and_columns_for_selfcal(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/flint/ms.py", line 929, in rename_ms_and_columns_for_selfcal
    raise FileNotFoundError(
FileNotFoundError: /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms does not exists or is not a directory (hence measurement set).

Finished in state Failed('Task run encountered an exception FileNotFoundError: /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms does not exists or is not a directory (hence measurement set).')

Third attempt starts 6 hours later with:

Measurement set to be self-calibrated: ms=MS(path=PosixPath('/fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms'), column='CORRECTED_DATA', beam=None, spw=None, field=None, model_column='MODEL_DATA')

Updating gaincal options with: {'solint': '480s', 'calmode': 'ap', 'round': 0, 'minsnr': 0.0, 'uvrange': '>400m', 'selectdata': True, 'gaintype': 'G', 'nspw': 1}

Detected a previous round of self-calibration. 

Output MS name will be /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round3.ms.

Renaming /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms to /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round3.ms.

Encountered exception during execution:
Traceback (most recent call last):
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/prefect/engine.py", line 2169, in orchestrate_task_run
    result = await call.aresult()
             ^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/flint/prefect/common/imaging.py", line 304, in task_gaincal_applycal_ms
    return gaincal_applycal_ms(
           ^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/flint/selfcal/casa.py", line 276, in gaincal_applycal_ms
    cal_ms = copy_and_clean_ms_casagain(ms=ms, round=round, rename_ms=rename_ms)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/flint/selfcal/casa.py", line 91, in copy_and_clean_ms_casagain
    ms = rename_ms_and_columns_for_selfcal(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fred/oz330/ddobie/FLINT/flint_venv/lib/python3.11/site-packages/flint/ms.py", line 929, in rename_ms_and_columns_for_selfcal
    raise FileNotFoundError(
FileNotFoundError: /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms does not exists or is not a directory (hence measurement set).

Finished in state Failed('Task run encountered an exception FileNotFoundError: /fred/oz330/ddobie/FLINT/62657/SB62657.ELAIS-S1.beam23.round2.ms does not exists or is not a directory (hence measurement set).')

The solution is probably just to copy the ms as usual and then only wipe the previous round at the very end of the task

[tjgalvin]

Thanks for raising this one. It really seems like a case of bad timing. These types of side effect functions are painful.

Your understanding is correct as I understand the situation - the task would be picked up by another worker. The temporary copy of the MS for operation and then moved into place would likely be the way to go. This comes at the cost of an extra copy operation, but for the moment that is not something I care too much about.

The only thing to do is to clearly indicate the the copy being operated on is 'in transit'. Should the same thing happen (worker killed, task handed over to another worker) and that 'in transit' file exists, it can be safely deleted with no consequence.

[tjgalvin]

I may ponder what I think should be done.

There is also another potential issue similar to the one you highlighted. Should archive_input_ms=True and rename_ms=False and the worker were to be killed after the zip finishes than the next work starting the task will also throw a fit on a missing MS.

I'll try to rearrange a little coming day or two when I have some spare time - if there is an easy reorder that saves us I am for it. Aside from an extra copy like what I described above the only other alternative that comes to mind is to rename at the end after all self-calibration has been applied. But then the issue is around a half self-cal round potentially being baked into the data.

I would prefer a loud error in this situation than a quiet 'it works but data has been double self-caled' type error. The solution may be to keep as is for the moment.

[tjgalvin]

Putting this here as a reminder, but the following context manager does have a copy option, which would perhaps be the thing to use or extend to ensure that the function remain idempotent.

def critical_ms_interaction(
    input_ms: Path, copy: bool = False, suffix: str = ".critical"
):