era5_rechunking: add script to rechunk and concatenate ERA5 forcing files

[ezhilsabareesh8]

Adds era5_rechunking/ with a Python script and PBS script to rechunk ERA5 single-level monthly files from the rt52 archive into yearly files optimised for ACCESS-OM3 / DATM.

Related issue ACCESS-NRI/access-om3-configs#1381

What it does:

• Reads directly from /g/data/rt52/era5/single-levels/reanalysis/{stream}/{year}/
• Rechunks from [93, 91, 180] (ERA5 default) → [1, 721, 1440] (one full-resolution timestep per chunk)
• Concatenates 12 monthly files into one yearly file per stream
• Preserves raw int16 values bit-for-bit (no unpack/repack)
• Covers all 13 ERA5 streams, 1940–2026

Test output is here /g/data/tm70/ek4684/era5_rechunk_test/

[anton-seaice]

This appears somewhat complicated

• are using xarray (with Dask) or nco/ncks not good options ?
• do we need this level of configuration in input options ?

[anton-seaice]

I was thinking something like this:

import xarray as xr

from dask.distributed import Client

client = Client()

client.dashboard_link

ds = xr.open_mfdataset(
    '/g/data/rt52/era5/single-levels/reanalysis/2t/1940/*.nc',
    chunks={'latitude':-1,'longitude':-1}
)

ds_new = ds.t2m.chunk({'time':1})

ds_new.to_netcdf('2t_1940_era5.nc')

is that too slow ?

[ezhilsabareesh8]

is that too slow ?

Changed to xarray in this commit 4b0efbf, it is taking 3 minutes per variable now