Skip to content

record zero-length files in the shard serialization manifest - #653

Open
Support-itecz wants to merge 1 commit into
sigstore:mainfrom
Support-itecz:shard-empty-files
Open

record zero-length files in the shard serialization manifest#653
Support-itecz wants to merge 1 commit into
sigstore:mainfrom
Support-itecz:shard-empty-files

Conversation

@Support-itecz

Copy link
Copy Markdown
Contributor

Summary

_get_shards guards its loop with if path_size > 0, so a zero-length file produces no shards and never becomes a manifest item. Shard serialization leaves every empty file in the model out of the signature, even though the file was there when the model was signed.

The verifier re-serializes the model the same way, so the omission is symmetric and nothing reports a difference. Against a model signed with use_shard_serialization():

  • deleting a signed empty file verifies clean
  • adding new empty files verifies clean, including one that creates a new directory (sub/__init__.py)
  • with ignore_unsigned_files, a signed-as-empty file can be filled with content and still verifies clean

Empty marker files decide real loader behavior (__init__.py, py.typed, feature-flag files), so this is a gap in what the signature covers rather than a cosmetic difference. File serialization already recorded empty files, which is why file_test.py asserts test_folder_model_empty_file_gets_included while the shard suite asserted the opposite and the shard goldens for both empty-file fixtures were zero-byte. Dropping the guard makes _endpoints(shard_size, 0) yield one (path, 0, 0) shard, and ShardedFileHasher.set_shard accepts end == start for it, so the two serializers agree.

Shard signatures over models that contain empty files have to be regenerated. The unit suite passes (198), and the version-compat verify scripts under scripts/tests still pass for v0.2.0 through v1.1.0, since those signatures are file-serialized.

Checklist
  • All commits are signed-off, using DCO
  • All new code has docstrings and type annotations
  • All new code is covered by tests. Aim for at least 90% coverage. CI is configured to highlight lines not covered by tests.
  • Public facing changes are paired with documentation changes
  • Release note has been added to CHANGELOG.md if needed

@Support-itecz
Support-itecz requested review from a team as code owners July 30, 2026 12:01
_get_shards guarded its loop with `if path_size > 0`, so a zero-length
file produced no shards and never became a manifest item. Shard
serialization left every empty file in the model out of the signature,
even though the file was there when the model was signed.

The verifier re-serializes the model the same way, so the omission is
symmetric and nothing reports a difference. Against a model signed with
use_shard_serialization(), deleting a signed empty file verifies clean,
adding new empty files verifies clean, and with ignore_unsigned_files an
empty file can be filled with content and still verify clean. Empty
marker files decide real loader behavior (__init__.py, py.typed,
feature-flag files), so this is a gap in what the signature covers.

Drop the guard so _endpoints(shard_size, 0) yields one (path, 0, 0)
shard, and let ShardedFileHasher.set_shard accept end == start for it.
File serialization already recorded empty files, which is why
file_test.py asserts test_folder_model_empty_file_gets_included while
the shard suite asserted the opposite and the shard goldens for both
empty-file fixtures were zero-byte. The two serializers now agree.

Shard signatures over models that contain empty files have to be
regenerated.

Signed-off-by: Itecz Solution <support@itecz.au>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant