SegFault w/ chunked data loading

[bnubald]

Attempting to read netCDF using chunked sizes (also coincidental with compressed dataarrays) fails with Segmentation Fault.

[2025-09-03 11:23:43,848][root][processor.py:276][INFO] - Got 1 files for train:ua500
[2025-09-03 11:23:43,848][root][processor.py:276][INFO] - Got 1 files for train:ua700
[2025-09-03 11:23:43,848][root][processor.py:276][INFO] - Got 1 files for train:zg500
[2025-09-03 11:23:43,848][root][processor.py:276][INFO] - Got 1 files for train:zg700
[2025-09-03 11:23:43,848][root][processor.py:276][INFO] - Got 1 files for train:tas
[2025-09-03 11:23:43,848][root][processor.py:276][INFO] - Got 1 files for train:tos
[2025-09-03 11:23:43,848][root][processor.py:276][INFO] - Got 1 files for train:sic
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for val:ua500
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for val:ua700
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for val:zg500
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for val:zg700
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for val:tas
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for val:tos
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for val:sic
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for test:ua500
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for test:ua700
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for test:zg500
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for test:zg700
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for test:tas
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for test:tos
[2025-09-03 11:23:43,849][root][processor.py:276][INFO] - Got 1 files for test:sic
[2025-09-03 11:23:43,849][root][processor.py:393][INFO] - Opening 1 files for ua500
[2025-09-03 11:23:43,854][root][processor.py:460][INFO] - Normalising ua500
Segmentation fault (core dumped)

Culprit:

preprocess-toolbox/preprocess_toolbox/processor.py

Lines 397 to 400 in 8bcdebe

	ds = xr.open_mfdataset(
	source_files,
	parallel=self._parallel,
	lock=False)

Fix:

Pass engine="h5netcdf" as argument to this call which is a more reliable (esp. with Dask and multi-file access) and faster engine to use.

We should consider using it throughout in lieu of the default engine=netcdf4.