Create an intake-ESM datastore for the ct11 collection

[rbeucher]

I have started working on an intake-ESM datastore for the ct11 collection here

The COSIMA community and the Ocean team will likely start using some of the datasets there so it would be good to finalise this work ASAP.
Priority should be on the ESMValTool Data Pool stored in /g/data/ct11/access-nri/replicas/ESMVALTOOL
It would be good if you could look into that @charles-turner-1 or @marc-white

[charles-turner-1]

Just started looking into this properly. It looks like there are 7250 netcdf files in here:

[ct1163@gadi-login-05 esmvaltool]$ pwd
/g/data/ct11/access-nri/replicas/esmvaltool
[ct1163@gadi-login-05 esmvaltool]$ tree . | grep ".*.nc" | wc -l
7250

Am I correctly understanding that we want to do roughly what the notebook you linked above is doing, but for the total set of files in that directory in order to generate an intake ESM datastore for the whole lot?

I'm having a slightly tricky time inferring the subtleties of the intent of that notebook (eg. "OBS6" vs "obs4MIPS", "CowtanWay", and the numgauge stuff) but if my paragraph above is roughly what we're after I should be able to get that done pretty straightforwardly.

[rbeucher]

We need an intake-ESM catalog for the whole lot yes. Something that can be added to the ct11 NCI collections like the other projects we have already added to the access-nri-intake-catalog.

OBS6, obs4MIP are projects (similar to CMIP6, CMIP5).

For the workflow I was describing, I hope we can use something like the 'to_dataset' function you were mentioning. That being said, we need to align this with the work you have been doing with Bouwe.

Happy to have a chat next week if you can.

[charles-turner-1]

Gotcha, I think I can work with that.

I still need to spend a little more time playing with ESMValTool before I'll be confident with what exactly we're trying to do with the intake integration, but I'm confident enough that I know how to implement adding a something like a to_dataset method to intake-esm so we're at least halfway there.

[rbeucher]

This is related: ACCESS-NRI/access-nri-intake-catalog#250