fix: nc_def_var_chunking_ints should dispatch to format-aware handler - #3399
fix: nc_def_var_chunking_ints should dispatch to format-aware handler#3399tomdurrant wants to merge 2 commits into
Conversation
nc_def_var_chunking_ints() is the Fortran-compatible wrapper (accepts int* chunksizes instead of size_t*) called by nf90_def_var_chunking() in netcdf-fortran. It had two broken paths for NCZarr files: 1. NCZarr-only builds (no HDF5): the stub in libsrc4/nc4cache.c returned NC_NOERR without doing anything, so chunk sizes were never applied. The variable kept default chunk sizes (full dimension), producing a single chunk file with full-variable data. 2. HDF5-enabled builds: the implementation in libhdf5/hdf5var.c called nc_def_var_extra() which set var->chunksizes (metadata correct) but did NOT update NCZarr-specific cache state (zvar->chunkproduct, zvar->chunksize, zvar->cache->valid). The write path used stale cache values, writing full-variable-sized chunk data files. The fix: make nc_def_var_chunking_ints() convert int* to size_t* and delegate to nc_def_var_chunking(), which dispatches to the correct format backend (NCZ_def_var_chunking for NCZarr, NC4_def_var_chunking for HDF5). Both correctly update all internal state. Fixes Unidata/netcdf-fortran#487. Reported-by: tdurrant - Replace stub in libsrc4/nc4cache.c with dispatch-aware implementation - ifdef-out the HDF5-only version in libhdf5/hdf5var.c - Add regression test nczarr_test/tst_chunking_ints.c
|
Can you explain in plain language what issue you are addressing? This appears to be AI generated, and will be closed without a human in the loop. |
|
Hi @WardF , |
|
The tests appear to be failing. |
The regression test for nc_def_var_chunking_ints failed CI for two reasons: 1. Missing declaration: nc_def_var_chunking_ints is a Fortran-compat wrapper declared in netcdf_f.h, which the test did not include. Every autotools/cmake job (Ubuntu, Docker, Cygwin, MINGW, UCRT64, macOS) failed to compile tst_chunking_ints.c with 'implicit declaration of function'. Add #include <netcdf_f.h>. 2. Raw stat() on Windows: MSVC compiled (lenient implicit decl) but the runtime size check read garbage (1782925324 bytes for a 64KB chunk) because plain struct stat/stat() is not portable on Windows. Use the codebase-standard NCstat()/struct _stat64 pattern from ncpathmgr.h (same as libnczarr/zmap_file.c), which also performs the required NCpathcvt path conversion.
|
CI fix for the failing tests in this PR (pushed as The failing checks traced to two problems in the new regression test
Verified locally: |
Problem
nc_def_var_chunking_ints()is the Fortran-compatible wrapper (acceptsint*chunksizes) called bynf90_def_var_chunking()in netcdf-fortran. It had two broken paths for NCZarr files:NCZarr-only builds (no HDF5): the stub in
libsrc4/nc4cache.creturnedNC_NOERRwithout doing anything, so chunk sizes were never applied. The variable kept default chunk sizes (full dimension), producing a single chunk file with full-variable data.HDF5-enabled builds: the implementation in
libhdf5/hdf5var.ccallednc_def_var_extra()which setvar->chunksizes(metadata correct) but did NOT update NCZarr-specific cache state (zvar->chunkproduct,zvar->chunksize,zvar->cache->valid). The write path used stale cache values, writing full-variable-sized chunk data files.Both cases produced correct
.zarraymetadata but wrong chunk data files — unreadable byzarr-pythonandxarray.See: Unidata/netcdf-fortran#487
Fix
Make
nc_def_var_chunking_ints()convertint*tosize_t*and delegate tonc_def_var_chunking(), which dispatches to the correct format backend (NCZ_def_var_chunkingfor NCZarr,NC4_def_var_chunkingfor HDF5). Both correctly update all internal state.Changes
libsrc4/nc4cache.cwith dispatch-aware implementationlibhdf5/hdf5var.cnczarr_test/tst_chunking_ints.cVerification
zarr-pythonandxarray