Switch SMT to big-endian bit ordering

[jait91]

Syncs the SMT with the yellowpaper "big-endian bit strings" convention: key
bit d is the MSB-first bit (key[d/8] >> (7-d%8)) & 1, regions and inclusion-
cert bitmaps are big-endian bit vectors, and rsmt_sort_key(k) = k — so disk
traversal order is plain lexicographic byte order (keyPathLess collapses to
bytes.Compare). Wire byte order, hash domain separators, node/leaf hash
layouts, and CBOR/JSON-RPC shapes are unchanged.

All bit math routes through three helpers in pkg/api (KeyBitBE, SetBitBE,
ClearSuffixBE). The in-memory big.Int path machinery is untouched — only
FixedBytesToPath/PathToFixedBytes change, from a byte-order reversal to a full
256-bit reversal (bits.Reverse8 + byte swap), so path bit i still equals the
routing decision at depth i. Leaf hashes and the empty/one-leaf roots stay
byte-identical, but every internal node hash, root, cert bitmap and tree shape
changes.

No back-compat: TreeLayout bumps to yellowpaper-rsmt-sha256-v6a-be — pre-flip
data refuses to open, so all trees (incl. perf snapshots) need a rebuild from
finalized history, and shard routing (key→shard) must be re-derived at
rollout.

Vectors (internal/smt/v6a_interop_vectors_test.go): empty, one-leaf, shallow
split, deep split (diverge at bit 255), multi-leaf, and 6-leaf parity —
regenerated under big-endian with asymmetric canaries. The pkg/api helper
tests pin boundary depths 0/1/7/8/200/255, and an independent big-endian
reference (be_reference_anchor_test.go, raw SHA-256) anchors production to the
spec. Cross-verified both ways against the JS SDK
(unicitynetwork/state-transition-sdk-js#135): identical root c854db11… for the
shared 6-leaf parity tree, and their BitString uses the same MSB-first
convention. Java and Rust must reproduce at least the deep-split and
multi-leaf roots (multi-byte region packing) before the coordinated rollover.

Closes #169

[gemini-code-assist]

Code Review

This pull request transitions the Sparse Merkle Tree (SMT) implementation from LSB-first bit addressing to big-endian bit addressing, aligning it with the yellowpaper specification. The changes introduce new big-endian bit-string helpers in pkg/api/bits.go and update SMT traversal, hashing, region packing, and shard matching across the codebase. Additionally, a self-contained big-endian reference test is added to anchor the production implementation. A compilation issue was identified in internal/smt/disk/tree.go where the bytes package is used in keyPathLess but is missing from the import block.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[MastaP]

Two findings from a review of the big-endian switch. The conversion itself looks uniform and internally consistent across the in-memory, disk, and cert-verify paths (independent BE reference + JS-SDK parity vectors both anchor it), and the buildable suites pass. These are the two items worth acting on before merge.