Skip to content

Validate metadata cache load sizes before allocation#6506

Open
Alb3e3 wants to merge 3 commits into
HDFGroup:developfrom
Alb3e3:harden-metadata-cache-load-size
Open

Validate metadata cache load sizes before allocation#6506
Alb3e3 wants to merge 3 commits into
HDFGroup:developfrom
Alb3e3:harden-metadata-cache-load-size

Conversation

@Alb3e3

@Alb3e3 Alb3e3 commented Jul 4, 2026

Copy link
Copy Markdown

Describe your changes

This hardens malformed-file handling for metadata cache entry loads:

  • Check non-speculative metadata cache entry image sizes against the file EOA before allocating the on-disk image buffer. Existing speculative load behavior is preserved: speculative reads can still be trimmed to EOA before retry.
  • Validate decoded object header continuation chunk address/size pairs before queuing continuation chunks.
  • Add an ohdr regression that creates an object header continuation chunk, corrupts the encoded continuation size on disk, and verifies the malformed object header is rejected before the cache allocates that size.

This prevents malformed metadata from driving very large allocations before the later file-read/EOA checks run.

AI assistance disclosure: this patch was prepared with assistance from OpenAI Codex. I reviewed the changes and ran the verification below locally.

Issue ticket number (GitHub or JIRA)

N/A

Checklist before requesting a review

  • My code conforms to the guidelines in CONTRIBUTING.md
  • I made an entry in release_docs/CHANGELOG.md (bug fixes, new features)
  • I added a test (bug fixes, new features)

Local verification

  • git diff --check origin/develop..HEAD
  • cmake --build build-asan-o0 --target ohdr h5dump -j$(nproc)
  • ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=0:abort_on_error=1 timeout 180s build-asan-o0/bin/ohdr
  • 17 malformed h5dump crash-corpus files under ASan: no AddressSanitizer/UBSan/OOM output and no timeouts; all exit as ordinary open failures.
  • Valid-file sanity check: build-asan-o0/bin/h5dump -H test/testfiles/tmtimeo.h5

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Review Checklist

This PR touches the following areas. Each needs a sign-off
from its listed owners before merging.

@github-actions github-actions Bot removed the request for review from vchoi-hdfgroup July 4, 2026 20:24
@Alb3e3

Alb3e3 commented Jul 6, 2026

Copy link
Copy Markdown
Author

Pushed a follow-up for the CI failures. The first issue was formatting in test/ohdr.c. The Linux SWMR failures came from the new metadata load-size EOA guard rejecting reads against a SWMR reader's stale local EOA before the existing lower read path could handle writer-flushed metadata.

The update keeps the non-SWMR metadata size checks in place, removes the too-early continuation-message EOA check, and skips this cache-entry EOA validation only for H5F_ACC_SWMR_READ, matching the existing VFD/SWMR read behavior.

Local verification:

  • git diff --check
  • cmake --build /tmp/hdf5build-pr6506 --target flushrefresh swmr_check_compat_vfd ohdr use_disable_mdc_flushes twriteorder use_append_chunk use_append_mchunks -j2
  • ctest -V -R '^H5SHELL-test_flush_refresh$'
  • ctest -V -R '^H5SHELL-test_use_cases$'
  • ./bin/ohdr

@vchoi-hdfgroup vchoi-hdfgroup added the Component - C Library Core C library issues (usually in the src directory) label Jul 6, 2026

@hyoklee hyoklee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

release_docs looks good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component - C Library Core C library issues (usually in the src directory)

Projects

Status: To be triaged

Development

Successfully merging this pull request may close these issues.

3 participants