record zero-length files in the shard serialization manifest - #653
Open
Support-itecz wants to merge 1 commit into
Open
record zero-length files in the shard serialization manifest#653Support-itecz wants to merge 1 commit into
Support-itecz wants to merge 1 commit into
Conversation
_get_shards guarded its loop with `if path_size > 0`, so a zero-length file produced no shards and never became a manifest item. Shard serialization left every empty file in the model out of the signature, even though the file was there when the model was signed. The verifier re-serializes the model the same way, so the omission is symmetric and nothing reports a difference. Against a model signed with use_shard_serialization(), deleting a signed empty file verifies clean, adding new empty files verifies clean, and with ignore_unsigned_files an empty file can be filled with content and still verify clean. Empty marker files decide real loader behavior (__init__.py, py.typed, feature-flag files), so this is a gap in what the signature covers. Drop the guard so _endpoints(shard_size, 0) yields one (path, 0, 0) shard, and let ShardedFileHasher.set_shard accept end == start for it. File serialization already recorded empty files, which is why file_test.py asserts test_folder_model_empty_file_gets_included while the shard suite asserted the opposite and the shard goldens for both empty-file fixtures were zero-byte. The two serializers now agree. Shard signatures over models that contain empty files have to be regenerated. Signed-off-by: Itecz Solution <support@itecz.au>
Support-itecz
force-pushed
the
shard-empty-files
branch
from
July 30, 2026 12:01
3fad9c3 to
381e3e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_get_shardsguards its loop withif path_size > 0, so a zero-length file produces no shards and never becomes a manifest item. Shard serialization leaves every empty file in the model out of the signature, even though the file was there when the model was signed.The verifier re-serializes the model the same way, so the omission is symmetric and nothing reports a difference. Against a model signed with
use_shard_serialization():sub/__init__.py)ignore_unsigned_files, a signed-as-empty file can be filled with content and still verifies cleanEmpty marker files decide real loader behavior (
__init__.py,py.typed, feature-flag files), so this is a gap in what the signature covers rather than a cosmetic difference. File serialization already recorded empty files, which is whyfile_test.pyassertstest_folder_model_empty_file_gets_includedwhile the shard suite asserted the opposite and the shard goldens for both empty-file fixtures were zero-byte. Dropping the guard makes_endpoints(shard_size, 0)yield one(path, 0, 0)shard, andShardedFileHasher.set_shardacceptsend == startfor it, so the two serializers agree.Shard signatures over models that contain empty files have to be regenerated. The unit suite passes (198), and the version-compat verify scripts under
scripts/testsstill pass for v0.2.0 through v1.1.0, since those signatures are file-serialized.Checklist