I recently had to look over the self-calibration function we have that uses CASA's gaincal and apply solutions, and think it should be refactored to reduce complexity.
Initially this function would copy a MS, renaming it to indicate a new round of self-cal, and then delete+rename columns inline with whats expected for CASA's gaincal+applysolutions.
Later I added functionality to compress old measurement sets (optionally), compress calibration tables (optionally), and process a MS by simply renaming it and further modifying columns. Supporting this logic has introduced new and exciting keyword parameters into the function that can make the flow difficult to flow, particularly if we are examining what happens when a task is restarted.
I would like to reimplement this so that:
- The MS is only ever renamed, never copied
- Old DATA columns are either renamed (emulating the MS copying) or deleted outright. This column renaming would include the round-1 value to avoid duplication.
- Calibration tables are always compressed when finished with
I hope that setting the function up this way will let us ensure fewer side effects, and when we move to a newer prefect version can right a simpler roll back function should the task itself fail/be killed.
I recently had to look over the self-calibration function we have that uses CASA's gaincal and apply solutions, and think it should be refactored to reduce complexity.
Initially this function would copy a MS, renaming it to indicate a new round of self-cal, and then delete+rename columns inline with whats expected for CASA's gaincal+applysolutions.
Later I added functionality to compress old measurement sets (optionally), compress calibration tables (optionally), and process a MS by simply renaming it and further modifying columns. Supporting this logic has introduced new and exciting keyword parameters into the function that can make the flow difficult to flow, particularly if we are examining what happens when a task is restarted.
I would like to reimplement this so that:
I hope that setting the function up this way will let us ensure fewer side effects, and when we move to a newer prefect version can right a simpler roll back function should the task itself fail/be killed.