Skip to content

feat: C-library interop testbench + 6 write-compatibility fixes - #59

Open
rhaist wants to merge 1 commit into
scigolib:mainfrom
shyrmapp:feat/c-interop-testbench
Open

feat: C-library interop testbench + 6 write-compatibility fixes#59
rhaist wants to merge 1 commit into
scigolib:mainfrom
shyrmapp:feat/c-interop-testbench

Conversation

@rhaist

@rhaist rhaist commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an interop test suite that validates Go-written files against the official HDF5 C tools (h5dump/h5diff/h5repack), plus a c-interop CI job (Ubuntu, hdf5-tools). Tests self-skip when the tools are absent, so the existing jobs are unaffected.

  • TestCInterop_WriteMatrix — 15 cases across the write feature matrix (all numeric types, multidim, chunked+gzip, shuffle+fletcher32, fixed/vlen strings, ragged vlen, groups+attributes, hard/soft links, enum, array types, superblock v0), each verified through real h5dump -p output including data values.
  • TestCInterop_RepackRoundTrip — Go write → h5repack (C rewrites) → h5diff (C compares) → Go reads both files back and verifies every value.

Bugs the suite caught — all fixed

  1. Deflate filter wrote gzip (RFC 1952) instead of zlib (RFC 1950) — every compressed dataset was unreadable by the C library, h5py, and our own Open() path. The reader keeps a gzip-magic sniff fallback so files written by earlier releases stay readable.
  2. Filter pipeline message: version 2 stamp with a version 1 body — h5dump parsed filter ID 0 ("unknown filter"). Now emits spec-correct v1 (padded null-terminated names, 4 pad bytes after odd client-data counts).
  3. Enum datatype message interleaved name/value pairs — the spec stores all names, then all values (0 length enum name in h5dump). Version stamp corrected v3 → v1 to match the padded layout.
  4. Hard links: RefCount message missing its version byte + EOA left 8 bytes short — fixed the message layout (reader accepts the old bare-uint32 layout for back-compat) and routed header rewrites through the bounds-checked writer; delete_write.go's refcount path got the same fix.
  5. Soft links written as standalone object headers holding a Link message — the C library cannot classify such an object. Now written per spec: cache-type-2 symbol table entry with the target path in the parent's local heap. SNOD entries now also serialize scratch-pad data they previously zeroed (cached STAB addresses were silently dropped on rewrite).
  6. In-place header rewrites could overflow their allocation — a dataset header growing past its 256-byte allocation overlapped the following GCOL by 2 bytes, corrupting the header checksum. Dataset header allocations are now tracked (headerAllocs) so the existing OCHK continuation path actually fires.

Also: the Windows-hardcoded TestVLenStringAttribute_H5dump now uses shared helpers and runs everywhere (renamed TestCInterop_VLenStringAttribute); stale "superblock v3 not implemented for writing" doc comments corrected (v3 writing works and is covered by the suite).

Test plan

  • go test -run TestCInterop -v . with HDF5 2.1.1 tools: 16/16 pass, no skips
  • Full suite: 3712 pass across 18 packages
  • New CI job runs the interop tests on every push/PR

Follow-ups (not in this PR)

  • HDF5 2.x "latest"-format read gaps: layout-v4 chunk indexes, complex datatypes, float16
  • FP8/bfloat16 helpers exist but are not wired into the read path
  • README compatibility claims need tempering until the above land

@rhaist
rhaist requested a review from kolkov as a code owner July 17, 2026 08:35
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/writer/filter_gzip.go 57.14% 3 Missing ⚠️
group_write.go 83.33% 1 Missing and 1 partial ⚠️
link_write.go 71.42% 0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@rhaist
rhaist force-pushed the feat/c-interop-testbench branch 5 times, most recently from 8d44447 to 5580730 Compare July 17, 2026 08:59
Add an interop test suite that validates Go-written files against the
official HDF5 C tools (h5dump/h5diff/h5repack), plus a CI job that
installs hdf5-tools on Ubuntu and runs it. Tests self-skip when the
tools are absent.

- TestCInterop_WriteMatrix: 15 cases across the write feature matrix
  (all numeric types, multidim, chunked+gzip, shuffle+fletcher32,
  fixed/vlen strings, ragged vlen, groups+attributes, hard/soft links,
  enum, array types, superblock v0), each verified via h5dump output
  including data values.
- TestCInterop_RepackRoundTrip: Go write -> h5repack (C rewrite) ->
  h5diff -> Go reads both files back and compares every value.

The suite immediately caught six write-side format bugs, all fixed:

1. Deflate filter wrote gzip (RFC 1952) streams instead of zlib
   (RFC 1950): every compressed dataset was unreadable by the C
   library, h5py, and our own Open() path. Reader keeps a gzip-magic
   sniff fallback so files written by earlier releases stay readable.
2. Filter pipeline message stamped version 2 with a version-1 body;
   h5dump parsed filter ID 0. Now emits spec-correct v1 (padded
   null-terminated names, pad after odd CD counts).
3. Enum datatype message interleaved name/value pairs; the spec stores
   all names then all values ("0 length enum name" in h5dump). Also
   corrected the version stamp (v3 -> v1, matching the padded layout).
4. RefCount message (hard links) lacked its version byte and the
   grown header write left the superblock EOA 8 bytes short.
5. Soft links were written as standalone object headers holding a
   Link message; the C library cannot classify such an object. Now
   written per spec as cache-type-2 symbol table entries with the
   target path in the parent's local heap. SNOD entries now serialize
   scratch-pad data (cached STAB addresses, soft-link offsets).
6. Object headers rewritten in place (attributes) could silently grow
   past their allocation and corrupt the following structure (stale
   checksum via GCOL overlap): dataset header allocations are now
   tracked so overflow routes through OCHK continuation blocks.

Also: TestVLenStringAttribute_H5dump un-hardcoded from a Windows path
(renamed TestCInterop_VLenStringAttribute), superblock v3 doc comments
corrected (v3 writing works and is verified by the suite).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant