reject SOHM list message count exceeding list_max

[naruto-lgtm]

Describe your changes

When a shared object header message index is stored as a list, both H5SM__cache_list_verify_chksum() and H5SM__cache_list_deserialize() take the num_messages count straight from the on-disk index header. The image buffer and the list->messages array are both sized for list_max entries, so a file whose num_messages is larger than list_max makes verify_chksum size the checksum region past the buffer and makes the deserialize loop write past the array and read past the buffer. A list holds at most list_max messages on disk before it is promoted to a B-tree, so any file that violates that is corrupt.

Reject num_messages > list_max at both callbacks before the count is used to size a read or index the array. That header field is the trust boundary here, and the rest of the SOHM code already assumes the count fits within list_max, so validating it once at load time keeps every later list->messages access sound.

Issue ticket number (GitHub or JIRA)

#6501

Checklist before requesting a review

• My code conforms to the guidelines in CONTRIBUTING.md
• I made an entry in release_docs/CHANGELOG.md (bug fixes, new features)
• I added a test (bug fixes, new features)

[github-actions]

Review Checklist

This PR touches the following areas. Each needs a sign-off
from its listed owners before merging.

• src — @mattjala
• test — @mattjala
• release_docs — @hyoklee

[hyoklee]

release_docs looks good.

[lrknox]

A test should be included if possible.

[naruto-lgtm]

Added a test in tsohm.c (test_sohm_reject_bad_count). It creates a file with a list-index SOHM, pushes the on-disk message count past list_max and repairs the table checksum, then reopens and shares a message so the list gets protected. With the fix that load is rejected cleanly; on the pre-fix code the same path is the heap-buffer-overflow read in H5F_get_checksums via H5SM__cache_list_verify_chksum, so it trips under ASan. It's guarded to the default sec2 driver since it edits the file image directly.