Harden serialized selection decoding

[Alb3e3]

Summary

This hardens serialized dataspace selection decoding against malformed or truncated selection buffers.

The patch adds explicit length validation before decoding the common serialized selection type, passes the remaining selection-type-specific payload size to the concrete deserializers, rejects zero-length concrete selection payloads, and validates the decoded hyperslab rank before it is used to patch or allocate a dataspace.

Motivation and context

Several selection deserializers computed *p + p_size - 1 before proving that p_size was non-zero. A malformed serialized selection can therefore underflow the end pointer and let later bounds checks reason about an invalid buffer range. Hyperslab selection decoding also accepted an unchecked serialized rank before patching the dataspace rank.

Testing

Built and tested with Clang 18 ASan/UBSan on develop:

• cmake -S . -B build-asan -G "Unix Makefiles" -DCMAKE_C_COMPILER=clang-18 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_TESTING=ON -DHDF5_BUILD_TOOLS=ON -DHDF5_BUILD_CPP_LIB=OFF -DHDF5_BUILD_FORTRAN=OFF -DHDF5_BUILD_JAVA=OFF -DHDF5_ENABLE_PARALLEL=OFF -DCMAKE_C_FLAGS_RELWITHDEBINFO="-O1 -g -DNDEBUG -fsanitize=address,undefined -fno-omit-frame-pointer" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined"
• cmake --build build-asan --target testhdf5 h5dump h5ls -j2
• ASAN_OPTIONS=allocator_may_return_null=1:detect_leaks=0:abort_on_error=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1 build-asan/bin/testhdf5 -only select
• 57-file test/testfiles tool baseline across 456 h5dump/h5ls invocations: 0 sanitizer failures. The existing slow tlayouto.h5 default h5dump case still timed out and was treated separately from sanitizer findings.
• Targeted mutation fuzz pass over the same 57-file corpus: 6,526 runs, 0 new sanitizer findings, 0 skipped known-submitted findings.

AI assistance was used to find and prepare this patch. I reviewed the changes and verified them locally with the commands above.

[github-actions]

Review Checklist

This PR touches the following areas. Each needs a sign-off
from its listed owners before merging.

• src — @mattjala
• test — @mattjala