copy_and_preprocess_casda_askap_ms resource usage is inefficient

[ddobie]

This step is relatively simple and only uses a single core (I think?) but it gets submitted to the same dask cluster as every other task. This results in a huge number of cores sitting idle for a reasonable length of time (7/8 cores for each of 36 workers idle for ~15 minutes in my case running an EMU observation on ozstar).

Processing them in serial wouldn't make sense because they can be parallelised on a per-beam basis, but is there a way to make better use of the resources available? Either spinning up a small dask cluster for this specific task, or grouping the tasks and running multiple on each worker?

[tjgalvin]

Thanks for flagging this.

The immediate thing I could do without too much trouble is to eliminate the initial copy command, and use the existing mstransform operation that is currently the very last step of this stage. The long and short is that the mstransform was added to rewok around some bug introduced through a casacore version change. Currently rewrites the near final ms into the final ms, and deletes the input ms. It would save however much time is normally spent copying the data.

Otherwise, there might be some scope to retune how htis stage is executed with an eye towards a smaller resource allocation (like your suggestion). I could try to mock something up, but to-date my 'keep it simple, stupid' philosophy has kept me from trying to overly tune the tasks like this. There could be some benefit to it, but my main concern is that the extra leg work to optimise something that is ~10 minutes of a ~4 hour work flow may not balance out against the extra config+sub-flow construction.

For the moment I am happy to do the first part, but the second I may need to sit on for a little, mock up etc.

Thanks for the feedback!

[ddobie]

Partially related to this - was just talking to someone about GPUs, which got me thinking about GPUs in the context of FLINT.

If you wanted to use GPU nodes efficiently you would need to have a separate dask cluster for GPU tasks. If (and it's a big if - similar to your comment above, not sure if the tradeoff is worth it) you added that it would be trivial to also add some sort of small_compute cluster for simple serialised tasks.

[tjgalvin]

Yes - getting some GPU acceleration involved here would be a prime pathway to accelerate it. We can surely find some dask-gpu magic, or write our own pycuda type solution (I have no experience with either). Both solutions could be added to fixms, and would indeed require a separate dask cluster file that could be tapped into.

The larger suggestion you made around having a small dask cluster configuration that could be used for smaller/less intensive parts of the pipeline - that is indeed where my mind was going. I want to try to mock up a neat way of spinning up/down the flow/sub-flow for this. The last thing I want to do is to write both @flow wrappers around many separate @task decorated functions. It means more code than I would like to have to manage (e.g. change it once means changing it twice if not done carefully).