Skip to content

UPSTREAM.md

Latest commit

 

History

History
137 lines (107 loc) · 5.32 KB

UPSTREAM.md

File metadata and controls

137 lines (107 loc) · 5.32 KB
Raw

UPSTREAM - HDF5 Reference Tracking

This file tracks the upstream HDF5 C library and Format Specification that this Pure Go implementation is based on.

Primary References

HDF5 Format Specification

URL: https://docs.hdfgroup.org/hdf5/latest/_f_m_t3.html Version: Format Specification v4.0 (HDF5 Library 2.0.0) Date: 2025-05-01 Status: Fully supported (superblock v0, v2, v3)

HDF5 C Library (Reference Implementation)

Repository: https://github.com/HDFGroup/hdf5 Branch: develop Commit: bc8f6bb02a Date: 2025-12-19 (local sync) Local Copy: Clone repository for development reference

Upstream Sync History

Latest Sync: 2025-12-19

Commit: bc8f6bb02a Commits Analyzed: 34 new commits since last sync

New CVEs Identified

CVE Severity File Status
CVE-2025-2308 HIGH H5Zscaleoffset.c ✅ Not affected (filter not implemented)
CVE-2025-2309 HIGH H5Odtype.c ✅ Not affected (bitfield data conversion not implemented)

Other Notable Changes

  • R-tree optimizations (#6039)
  • Zero-element reads on virtual datasets fix (#6083)
  • HDF5 2.0.0 HISTORY files updated

Tasks Created

  • TASK-034: CVE-2025-2309 bitfield datatype security review (MEDIUM) ✅ DONE - Not affected
  • TASK-035: CVE-2025-2308 documentation (N/A - not affected) ✅ DONE
  • TASK-036: R-tree optimizations review (LOW)

Previous Sync: 2025-11-13 (v0.13.0 Release)

Commit: e99a49585760aeb34f2bb2144d8e80e207a68ec4 Version: HDF5 2.0.0 (Format Spec v4.0) Commits Analyzed: 54 commits

CVEs Incorporated (v0.13.0)

  • CVE-2025-7067: Buffer overflow in chunk reading (HIGH) ✅
  • CVE-2025-6269: Heap overflow in fractal heap (MEDIUM) ✅
  • CVE-2025-2926: Stack overflow in B-tree recursion (MEDIUM) ✅
  • CVE-2025-44905: Integer overflow in dataspace (MEDIUM) ✅

Features Added

  • 64-bit chunk dimensions (breaking change, internal API)
  • FP8 (E4M3, E5M2) and bfloat16 datatypes

Implementation Notes

This is a Pure Go implementation, not a CGo wrapper or line-by-line port.

Approach

  • Format Spec v4.0 as primary reference for binary format
  • C library source code consulted for edge cases and validation
  • Go idioms preferred over C patterns
  • Independent test suite with round-trip validation

Key Differences from C Library

  1. Memory Management: Go GC vs manual malloc/free
  2. Error Handling: Go errors vs C return codes + errno
  3. Threading: User responsibility vs built-in thread safety
  4. Buffer Pooling: sync.Pool for reduced GC pressure
  5. No MPI Support: Single-process only (parallel I/O planned)

Feature Parity Status

Feature C Library Go Library Notes
Superblock v0,v2,v3 ✅ ✅ Full support
Object Header v1,v2 ✅ ✅ With continuations
All Datatypes ✅ ✅ Including FP8, bfloat16
Bitfield Datatype ✅ ❌ Not supported (explicit rejection)
Chunked + Filters ✅ ✅ GZIP, Shuffle, Fletcher32, LZF
BZIP2 Filter ✅ ⚠️ Read only (stdlib)
Scale-Offset Filter ✅ ❌ Not implemented
Dense Attributes ✅ ✅ Fractal heap + B-tree v2
Soft/External Links ✅ ✅ Full support
SWMR Mode ✅ ❌ Planned v0.14.0+
Parallel I/O (MPI) ✅ ❌ Planned v0.14.0+
SZIP Compression ✅ ❌ Stub only (requires libaec)
Virtual Datasets ✅ ❌ Planned v0.14.0+

Sync Workflow

When syncing with upstream changes:

  1. Check HDF5 releases: https://github.com/HDFGroup/hdf5/releases
  2. Review security advisories: Check for CVEs affecting our supported formats
  3. Pull latest: git pull origin develop (in your local HDF5 clone)
  4. Analyze relevant commits: Focus on format changes, not C-specific code
  5. Update this file: Document what was synced and when
  6. Create tasks: Add implementation tasks to docs/dev/backlog/

Files to Monitor in C Library

src/H5Fsuper.c          # Superblock parsing
src/H5Oattribute.c      # Attribute handling
src/H5Dchunk.c          # Chunked dataset I/O
src/H5HFdblock.c        # Fractal heap direct blocks
src/H5B2*.c             # B-tree v2 implementation
src/H5Tconv.c           # Datatype conversions
src/H5Odtype.c          # Datatype object header messages
src/H5Zscaleoffset.c    # Scale-offset filter (not implemented)

Quality Validation

Official HDF5 Test Suite

  • Files tested: 433
  • Pass rate: 98.2%
  • Source: Various HDF5 test files and real-world datasets

Interoperability Verified

  • ✅ h5py (Python)
  • ✅ HDFView (Java)
  • ✅ h5dump (C library CLI)
  • ✅ MATLAB HDF5 functions

Last Updated: 2025-12-19 Maintainer: Claude (Autonomous Developer)