It occurs to me now that we are starting to have multiple pipelines that potentially run against the same calibrated measurement set and produce similar imaging / linmos products we should consider what to do with the overlapping outputs.
Here I am speaking to multiple pipelines producing, for example, stokes I images. The outputs of the polarisation imaging could overwrite the outputs of the self-calibration imaging, for example.
As far as I see there are a couple of options:
- each pipeline has its own folder outputs are placed in
- we extend the flint naming format to include some other characteristic
On point 1:
The self-calibration and imaging pipeline transforms the input measurement sets (either raw data or MSs from CASDA) and preprocesses them so they are compatible with wsclean and have a different naming scheme. They are placed in a folder whose name is the SBID (folder is created), and any outputs are placed next to them. This has been a natural consequence of appending components to the path of some data file, and building up from there.
We could consider a situation where each pipeline writes output products to a sub-directory within the SBID folder. I think the implementation is somewhat straight-forward. We would have to added something to the imaging stage to appropriately accept a new root output folder. The imager task should not be responsible for creating this, but may be necessary to pass something through to indicate an up-dated output path.
Alternatively, something be designed so the sub-directory has a symlink back the MS, then it would mostly be business as usual. My principal concern would be how symlinks are treated when we call into a container and specify a set of paths to bind to. I am not sure what the typical behavior is in such a situation (though I am pretty sure singularity would not follow a symlink if it points to an item on a different logical file system and it is not part of the bindpath). This could be overcome easily with an environment change (SINGULARITY_BINDPATH).
On point 2:
We could consider adding some other property to the processed component name format to indicate a specific project, something hardcoded per pipeline mode, or perhaps drawn from the operation used to tap into the strategy file.
This would be something that does not modify the name of the MS, which for the moment don't think needs to be updated (though in time this is going to change for high-spectral resolution down to an averaged MS). Rather it would again be something that would have to be passed through to appropriate tasks that create the output products that could conflict.
The regular expression used to break apart the processed naming scheme into fields would have to be updated to pick up this new field, and I am not sure if we should use some type of marker to indicate where it starts / ends.
I am at the moment slightly drawn to option 1, but would like to think this through and get some other thoughts from @AlecThomson and others
It occurs to me now that we are starting to have multiple pipelines that potentially run against the same calibrated measurement set and produce similar imaging / linmos products we should consider what to do with the overlapping outputs.
Here I am speaking to multiple pipelines producing, for example, stokes I images. The outputs of the polarisation imaging could overwrite the outputs of the self-calibration imaging, for example.
As far as I see there are a couple of options:
On point 1:
The self-calibration and imaging pipeline transforms the input measurement sets (either raw data or MSs from CASDA) and preprocesses them so they are compatible with wsclean and have a different naming scheme. They are placed in a folder whose name is the SBID (folder is created), and any outputs are placed next to them. This has been a natural consequence of appending components to the path of some data file, and building up from there.
We could consider a situation where each pipeline writes output products to a sub-directory within the SBID folder. I think the implementation is somewhat straight-forward. We would have to added something to the imaging stage to appropriately accept a new root output folder. The imager task should not be responsible for creating this, but may be necessary to pass something through to indicate an up-dated output path.
Alternatively, something be designed so the sub-directory has a symlink back the MS, then it would mostly be business as usual. My principal concern would be how symlinks are treated when we call into a container and specify a set of paths to bind to. I am not sure what the typical behavior is in such a situation (though I am pretty sure singularity would not follow a symlink if it points to an item on a different logical file system and it is not part of the bindpath). This could be overcome easily with an environment change (
SINGULARITY_BINDPATH).On point 2:
We could consider adding some other property to the processed component name format to indicate a specific project, something hardcoded per pipeline mode, or perhaps drawn from the operation used to tap into the strategy file.
This would be something that does not modify the name of the MS, which for the moment don't think needs to be updated (though in time this is going to change for high-spectral resolution down to an averaged MS). Rather it would again be something that would have to be passed through to appropriate tasks that create the output products that could conflict.
The regular expression used to break apart the processed naming scheme into fields would have to be updated to pick up this new field, and I am not sure if we should use some type of marker to indicate where it starts / ends.
I am at the moment slightly drawn to option 1, but would like to think this through and get some other thoughts from @AlecThomson and others