feat: C-library interop testbench + 6 write-compatibility fixes - #59
Open
rhaist wants to merge 1 commit into
Open
feat: C-library interop testbench + 6 write-compatibility fixes#59rhaist wants to merge 1 commit into
rhaist wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
rhaist
force-pushed
the
feat/c-interop-testbench
branch
5 times, most recently
from
July 17, 2026 08:59
8d44447 to
5580730
Compare
Add an interop test suite that validates Go-written files against the
official HDF5 C tools (h5dump/h5diff/h5repack), plus a CI job that
installs hdf5-tools on Ubuntu and runs it. Tests self-skip when the
tools are absent.
- TestCInterop_WriteMatrix: 15 cases across the write feature matrix
(all numeric types, multidim, chunked+gzip, shuffle+fletcher32,
fixed/vlen strings, ragged vlen, groups+attributes, hard/soft links,
enum, array types, superblock v0), each verified via h5dump output
including data values.
- TestCInterop_RepackRoundTrip: Go write -> h5repack (C rewrite) ->
h5diff -> Go reads both files back and compares every value.
The suite immediately caught six write-side format bugs, all fixed:
1. Deflate filter wrote gzip (RFC 1952) streams instead of zlib
(RFC 1950): every compressed dataset was unreadable by the C
library, h5py, and our own Open() path. Reader keeps a gzip-magic
sniff fallback so files written by earlier releases stay readable.
2. Filter pipeline message stamped version 2 with a version-1 body;
h5dump parsed filter ID 0. Now emits spec-correct v1 (padded
null-terminated names, pad after odd CD counts).
3. Enum datatype message interleaved name/value pairs; the spec stores
all names then all values ("0 length enum name" in h5dump). Also
corrected the version stamp (v3 -> v1, matching the padded layout).
4. RefCount message (hard links) lacked its version byte and the
grown header write left the superblock EOA 8 bytes short.
5. Soft links were written as standalone object headers holding a
Link message; the C library cannot classify such an object. Now
written per spec as cache-type-2 symbol table entries with the
target path in the parent's local heap. SNOD entries now serialize
scratch-pad data (cached STAB addresses, soft-link offsets).
6. Object headers rewritten in place (attributes) could silently grow
past their allocation and corrupt the following structure (stale
checksum via GCOL overlap): dataset header allocations are now
tracked so overflow routes through OCHK continuation blocks.
Also: TestVLenStringAttribute_H5dump un-hardcoded from a Windows path
(renamed TestCInterop_VLenStringAttribute), superblock v3 doc comments
corrected (v3 writing works and is verified by the suite).
rhaist
force-pushed
the
feat/c-interop-testbench
branch
from
July 17, 2026 09:02
5580730 to
f6dca7a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an interop test suite that validates Go-written files against the official HDF5 C tools (
h5dump/h5diff/h5repack), plus ac-interopCI job (Ubuntu,hdf5-tools). Tests self-skip when the tools are absent, so the existing jobs are unaffected.TestCInterop_WriteMatrix— 15 cases across the write feature matrix (all numeric types, multidim, chunked+gzip, shuffle+fletcher32, fixed/vlen strings, ragged vlen, groups+attributes, hard/soft links, enum, array types, superblock v0), each verified through realh5dump -poutput including data values.TestCInterop_RepackRoundTrip— Go write →h5repack(C rewrites) →h5diff(C compares) → Go reads both files back and verifies every value.Bugs the suite caught — all fixed
Open()path. The reader keeps a gzip-magic sniff fallback so files written by earlier releases stay readable.0 length enum namein h5dump). Version stamp corrected v3 → v1 to match the padded layout.delete_write.go's refcount path got the same fix.headerAllocs) so the existing OCHK continuation path actually fires.Also: the Windows-hardcoded
TestVLenStringAttribute_H5dumpnow uses shared helpers and runs everywhere (renamedTestCInterop_VLenStringAttribute); stale "superblock v3 not implemented for writing" doc comments corrected (v3 writing works and is covered by the suite).Test plan
go test -run TestCInterop -v .with HDF5 2.1.1 tools: 16/16 pass, no skipsFollow-ups (not in this PR)