feat: expose include_metadata hashing option through Python API (PLT-1734)

[kurodo3]

Summary

• Adds include_metadata: bool = False (keyword-only) to ArrowDigester.__init__, hash_schema, hash_record_batch, and hash_table
• Implements the same two-phase SHA-256 algorithm as the Rust starfix crate (PLT-1733): Phase 1 hashes the structural schema unchanged; Phase 2 (when include_metadata=True) feeds compact JSON of all field/schema metadata into the same hasher
• hash_array intentionally has no include_metadata — standalone arrays carry no schema/field metadata context, matching the Rust API
• Default False preserves hash format 0.0.1 stability — all existing golden tests pass unchanged
• Empty-metadata invariant: a schema with no metadata produces the same hash regardless of the flag

Test plan

• All 165 tests pass: uv run pytest tests/ -q (129 pre-existing + 36 new in tests/test_metadata_hashing.py)
• New test classes cover: TestSortMetadata, TestCollectNestedFieldMetadata, TestFieldPathSorting, TestEmptyMetadataInvariant, TestFieldMetadataChangesHash, TestMetadataExcludedByDefault, TestSchemaMetadataChangesHash, TestMetadataDeterminism, TestEmptyMetadataInvariantFull, TestRoundTrip
• Golden value regression test confirms include_metadata=False produces identical output to hash format 0.0.1
• Cross-language byte-for-byte parity with Rust deferred to PLT-1735

Closes PLT-1734

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR exposes an include_metadata: bool = False option across the Python ArrowDigester API to optionally incorporate Arrow schema- and field-level metadata into the schema hashing phase, aligning with the Rust crate’s two-phase hashing approach while preserving hash format 0.0.1 stability by default.

Changes:

• Added Phase 2 schema hashing that (optionally) incorporates deterministically ordered schema + nested field metadata into the same SHA-256 hasher.
• Extended ArrowDigester.__init__, hash_schema, hash_record_batch, and hash_table with a keyword-only include_metadata flag (default False).
• Added a comprehensive new test suite and updated README + design docs describing metadata hashing behavior and invariants.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/starfix/arrow_digester.py	Implements metadata collection/sorting and optional Phase 2 schema hashing; exposes include_metadata across public entry points.
tests/test_metadata_hashing.py	Adds coverage for determinism, invariants, and behavioral differences when metadata is included/excluded.
README.md	Documents how to opt into metadata hashing and the empty-metadata invariant.
docs/metamorphic/specs/2026-06-18-include-metadata-hashing-design.md	Captures the intended algorithm, path conventions, and API changes for the feature.
docs/metamorphic/plans/2026-06-18-include-metadata-hashing.md	Provides an implementation plan corresponding to the design/spec.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kurodo3]

Fixed. Removed the from hashlib import _Hash import from the TYPE_CHECKING block and replaced it with a small module-level _Hasher Protocol that declares the single update(data: bytes | bytearray) -> None method. No private stdlib types referenced anywhere now.

[kurodo3]

Review round summary

One change made in response to Copilot's two comments (both pointing at the same issue):

Replace hashlib._Hash with a local _Hasher Protocol

hashlib._Hash is a private CPython implementation detail — not guaranteed stable across Python versions or type checkers. Replaced it with a small public Protocol defined at module level:

class _Hasher(Protocol):
    """Minimal protocol for a hash accumulator (e.g. ``hashlib.sha256()``)."""

    def update(self, data: bytes | bytearray, /) -> None: ...

• Removed from hashlib import _Hash from the TYPE_CHECKING block entirely.
• Updated _update_metadata_hash(hasher: _Hash, ...) → _update_metadata_hash(hasher: _Hasher, ...).
• _Hasher uses only public stdlib types (Protocol, bytes, bytearray) — no private imports remain.
• All 165 tests still pass.