Move to Prefect Version 3

[tjgalvin]

There is been a new major release of prefect that I have been staying away from. It does involve some changes to the API which means changes on our end. It also requires a new server to be set up.

The latter is easy enough to do. The former will require some testing.

https://docs.prefect.io/v3/resources/upgrade-to-prefect-3

[tjgalvin]

I have started to mock this up in a separate prefect3 branch.

I think the biggest change is needing to resolve the futures ourself before some flow finishes, e.g. calling .result() or explicitly waiting for a list of futures to resolve. That is to say the prefect engine doesn't auto-resolve the futures like it did on prefect2.

I could imaging that the useable of a markdown artefact may also need to have an explicit _sync=True argument.

I am not sure whether this is something I can get working for prefect v2 magically.

[tjgalvin]

Ok - been playing around with this move. There are two conflicting version issues.

1 - PrefectHQ/prefect#18998

This one raises an error when .map() results go into a .submit() task, rending dask unable to resolve the futures under certain contexts. The ultimate fix is to upgrade dask / distributed to a newer >2024.12.0

2 - ratt-ru/dask-ms#350

dask-ms is used by crystalball to predict model visibilities. The issue here is that dask-ms is using some internal components of dask.core that have been lost when dask went through a larger internal refactor, stabalising around 2025.2.0 (from some discussions in ray-project/ray#52589)

So, either an alternative to the crystalball is used (we do have addmodel available), an update to dask-ms, or we remain on prefect2.

@AlecThomson - ideas and or thoughts?

[tjgalvin]

A new version of dask-ms has fixed this conflict above. I think that this enables moving over to prefect version 3 in the near future.