From d3699ea18afdaf28100dc50054f00ca85b89a859 Mon Sep 17 00:00:00 2001 From: helly25 <6420169+helly25@users.noreply.github.com> Date: Sun, 12 Jul 2026 23:41:23 +0100 Subject: [PATCH 1/2] feat(hash): port mumbo to Starlark (hash.mumbo) Port the default 64-bit hash mumbo::GetHash64 to hash.bzl next to dumbo/fnv1a: the kSecret bank, the seed absorb, the small-key (<=16) loader, the 16-byte MUM chain, the 8-lane 128-byte bulk tier, and the two-multiply finalizer - all as masked 64-bit Starlark arithmetic. Its default seed is the library-wide kDefaultSeed (5381), unlike dumbo (0) and fnv1a (offset basis). Verified byte-for-byte against the C++ prime via hash_bzl_vs_cpp_mumbo_test (hash_tool already registered mumbo), across lengths 0..300 spanning all three tiers and every tail size. Only the one-shot 64-bit form is ported; the native 128-bit jumbo and streaming stay C++-only. Updates the Algorithm-overview Starlark column and documents each port's canonical default seed. --- CHANGELOG.md | 1 + mbo/hash/README.md | 32 +++++--- mbo/hash/hash.bzl | 112 +++++++++++++++++++++++++- mbo/hash/internal/hash_bzl_verify.bzl | 4 +- 4 files changed, 132 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ad2ae6..9b44c71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ - Added the `quality` command to `hash_benchmark_report.py`: it generates the SMHasher3 "Results" table in `mbo/hash/README.md` from a curated source (verify with `--check`, source measured verdict/score from a fresh run with `--smhasher `). - Fixed SMHasher3 result parsing so an invalid hash name or crash reads as `ERROR` instead of a false `PASS`, and the pass/fail score and failing families now parse from the run Summary; legacy/short SMHasher3 names are aliased to their registered form when a measured dataset is loaded. +- Ported the default `mumbo` 64-bit hash to Starlark (`//mbo/hash:hash.bzl` `hash.mumbo`), byte-identical to C++ and verified against it; `hash.bzl` now offers `mumbo`, `dumbo`, and `fnv1a`. # 0.13.1 diff --git a/mbo/hash/README.md b/mbo/hash/README.md index b510445..d6f9a67 100644 --- a/mbo/hash/README.md +++ b/mbo/hash/README.md @@ -72,7 +72,7 @@ For the exact score and the failing families see [Quality: SMHasher3](#quality-s | Algorithm | Widths | Available via | Starlark | NOTICE | Seeded | Streaming | SMHasher3 | | ----------- | ------ | --------------------------------- | -------- | ----------------------- | ------ | --------- | --------- | -| `mumbo` | 64 | `hash.h` (default 64/32) | no | none (in-house) | yes | yes | PASS | +| `mumbo` | 64 | `hash.h` (default 64/32) | yes | none (in-house) | yes | yes | PASS | | `jumbo` | 128 | `hash.h` (default 128) | no | none (in-house) | yes | yes (64) | PASS | | `murmur3` | 64/128 | `hash.h` | no | none (public domain) | yes | no | FAIL | | `siphash` | 64 | `hash.h` | no | none (CC0) | keyed | yes | PASS | @@ -83,9 +83,10 @@ For the exact score and the failing families see [Quality: SMHasher3](#quality-s | `xxh3` | 64/128 | `hash_extra.h` + `:hash_extra_cc` | no | **BSD-2 - ship NOTICE** | yes | no | FAIL | Notes: the **Starlark** column marks the hashes also implemented at build time -in [`hash.bzl`](hash.bzl) (`hash.dumbo` and `hash.fnv1a`), kept byte-for-byte -identical to the C++ prime and verified against it (`hash_tool`); mumbo and -jumbo stay C++-only. `fnv1a` is the algorithm family many `std::hash` +in [`hash.bzl`](hash.bzl) (`hash.mumbo`, `hash.dumbo`, `hash.fnv1a`), kept +byte-for-byte identical to the C++ prime and verified against it (`hash_tool`); +only the one-shot 64-bit form is ported, so the native 128-bit `jumbo` and +streaming stay C++-only. `fnv1a` is the algorithm family many `std::hash` implementations use (e.g. MSVC) - included as the familiar baseline. `siphash` is a keyed PRF: the DoS-resistant choice when the seed is a secret. `dumbo` is the compact single-lane member of the MUM family: the fastest hash here for tiny @@ -160,19 +161,28 @@ The version-and-seed fold uses the in-house `dumbo` hash (SMHasher3-proven - and the shipped library agree on the algorithm. C++ is the prime implementation; the Starlark port is kept byte-for-byte identical to it, verified by `//mbo/hash:hash_bzl_vs_cpp_dumbo_test` (the bzl output diffed -against the `hash_tool` C++ binary). `dumbo` is the only in-house hash available -in Starlark - mumbo and jumbo stay C++-only - and the standard `fnv1a` is -offered there as well, likewise verified against C++. +against the `hash_tool` C++ binary). `mumbo` (the default 64-bit hash) and +`dumbo` are both offered in Starlark alongside the standard `fnv1a`, each +likewise verified against C++; only the one-shot 64-bit form is ported (the +native 128-bit `jumbo` and streaming stay C++-only). Call the ports from your own rules via `@helly25_mbo//mbo/hash:hash.bzl` (at load -time; input is a printable-ASCII string or a list of byte values `0..255`, plus -an optional seed; returns the 64-bit hash as an `int`): +time; input is a printable-ASCII string or a list of byte values `0..255`). The +seed is an **optional** second argument: omit it and each hash uses its own +canonical default - the same one the matching C++ `GetHash64` uses, so the +values agree by default. `mumbo` defaults to the library-wide `kDefaultSeed` +(`5381`), `dumbo` to `0`, and `fnv1a` to the FNV-1a **offset basis** +(`0xCBF29CE484222325`, the standard FNV-1a starting constant). Pass a seed +explicitly to match a specific seeded C++ call. Each returns the 64-bit hash as +an `int`: ```starlark load("@helly25_mbo//mbo/hash:hash.bzl", "hash") -_key = hash.dumbo("my build-time key") # dumbo, seed 0 -_fnv = hash.fnv1a([0x61, 0x62, 0x63]) # fnv1a, seed = FNV offset basis +_a = hash.mumbo("my build-time key") # default 64-bit hash; default seed kDefaultSeed (5381) +_b = hash.dumbo("my build-time key") # dumbo; default seed 0 +_c = hash.fnv1a([0x61, 0x62, 0x63]) # fnv1a over bytes; default seed = FNV offset basis +_d = hash.mumbo("my build-time key", 1234) # explicit seed == mbo::hash::mumbo::GetHash64(key, 1234) ``` ## Configuration diff --git a/mbo/hash/hash.bzl b/mbo/hash/hash.bzl index 501a0d0..1a87b53 100644 --- a/mbo/hash/hash.bzl +++ b/mbo/hash/hash.bzl @@ -22,10 +22,11 @@ identical to its C++ counterpart, verified by `//mbo/hash:hash_bzl_vs_cpp_*_test algorithm-count agnostic: offering more (or fewer) bzl hashes only widens or narrows that comparison. -Of the in-house mumbo/jumbo and dumbo family only `dumbo` is ported (single -accumulator, no lanes/streaming/128-bit form - see `hash_dumbo.h`); mumbo and -jumbo stay C++-only. The standard `fnv1a` is also provided (it is what the mangle -seed generation historically folded). +Of the in-house mumbo/jumbo and dumbo family, `mumbo` (the default 64-bit hash) +and `dumbo` (the compact companion) are ported; only their one-shot 64-bit +`GetHash64` is offered here - the native 128-bit `jumbo` form and streaming stay +C++-only. The standard `fnv1a` is also provided (it is what the mangle seed +generation historically folded). Inputs are either a printable-ASCII string or a list of byte values (0..255); strings are converted with the printable-ASCII table below (Starlark has no @@ -103,6 +104,108 @@ def _dumbo(data, seed = 0): hash_value = _dumbo_mum_step(hash_value, _load_le(data, ptr, length - ptr)) return _dumbo_finalize(hash_value, seed, length) +# mumbo (see `hash_mumbo.h`): the library's default 64-bit hash. Its secret bank +# is the 64-bit fractional parts of the square roots of the first sixteen primes +# (the SHA-512 / SHA-384 initial hash values, FIPS 180-4). Unlike dumbo (default +# seed 0) and fnv1a (offset basis), mumbo's canonical default seed is the +# library-wide `kDefaultSeed`. Only the one-shot 64-bit `GetHash64` is ported; +# the native 128-bit `jumbo` form and streaming stay C++-only. +_KDEFAULT_SEED = 5381 # `mbo::hash::kDefaultSeed` (hash_types.h). +_MUMBO_SECRET = [ + 0x6A09E667F3BCC908, + 0xBB67AE8584CAA73B, + 0x3C6EF372FE94F82B, + 0xA54FF53A5F1D36F1, + 0x510E527FADE682D1, + 0x9B05688C2B3E6C1F, + 0x1F83D9ABFB41BD6B, + 0x5BE0CD19137E2179, + 0xCBBB9D5DC1059ED8, + 0x629A292A367CD507, + 0x9159015A3070DD17, + 0x152FECD8F70E5939, + 0x67332667FFC00B31, + 0x8EB44A8768581511, + 0xDB0C2E0D64F98FA7, + 0x47B5481DBEFA4FA4, +] +_MUMBO_BULK_WINDOW = 128 + +def _mumbo_load_small(data, ptr, length): + """Loads a 0..16 byte key into two words `(a, b)` (see `mumbo_internal::LoadSmall`).""" + if length >= 4: + if length >= 9: # 9..16: two 64-bit loads overlapping the end. + return _load_le(data, ptr, 8), _load_le(data, ptr + length - 8, 8) + if length == 8: + value = _load_le(data, ptr, 8) + return value, value + return _load_le(data, ptr, 4), _load_le(data, ptr + length - 4, 4) # 4..7 (len 4 -> equal) + if length == 3: + value = ((data[ptr] << 45) | (data[ptr + 1] << 8) | data[ptr + 2]) & _MASK64 + return value, value + if length == 2: + value = ((data[ptr] << 45) | (data[ptr + 1] << 8) | data[ptr]) & _MASK64 + return value, value + if length == 1: + value = ((data[ptr] << 45) | data[ptr]) & _MASK64 + return value, value + return 0, 0 + +def _mumbo_finish(val_a, val_b, seed, length): + """The shared two-multiply finalizer (see `mumbo_internal::Finish`).""" + low, high = _mult128((val_a ^ _MUMBO_SECRET[2] ^ length) & _MASK64, (val_b ^ seed) & _MASK64) + return _mul128_fold64((low ^ _MUMBO_SECRET[3] ^ length) & _MASK64, (high ^ _MUMBO_SECRET[1]) & _MASK64) + +def _mumbo(data, seed = _KDEFAULT_SEED): + """mumbo 64-bit hash of `data`; identical to C++ `mbo::hash::mumbo::GetHash64`.""" + data = _to_bytes(data) + length = len(data) + + # Absorb + finalize the seed (structured seeds must not correlate with input). + seed = _mul128_fold64((seed ^ _MUMBO_SECRET[0]) & _MASK64, _MUMBO_SECRET[1]) + + if length <= 16: + val_a, val_b = _mumbo_load_small(data, 0, length) + return _mumbo_finish(val_a, val_b, seed, length) + + ptr = 0 + remaining = length + + # >= 128 bytes: eight independent MUM chains over a 128-byte fetch window. + if length >= _MUMBO_BULK_WINDOW: + chain = [(seed ^ _MUMBO_SECRET[8 + i]) & _MASK64 for i in range(8)] + for _ in range(length // _MUMBO_BULK_WINDOW): + if remaining < _MUMBO_BULK_WINDOW: + break + for i in range(8): + chain[i] = _mul128_fold64( + (_load_le(data, ptr + 16 * i, 8) ^ _MUMBO_SECRET[4 + i]) & _MASK64, + (_load_le(data, ptr + 16 * i + 8, 8) ^ chain[i]) & _MASK64, + ) + ptr += _MUMBO_BULK_WINDOW + remaining -= _MUMBO_BULK_WINDOW + merged = 0 + for value in chain: + merged ^= value + seed = merged & _MASK64 + + # 17..127 bytes (and any bulk remainder): one MUM chain, 16 bytes per step. + # C++ loops `while remaining > 16`; Starlark has no `while`, so bound + break. + for _ in range(length // 16 + 1): + if remaining <= 16: + break + seed = _mul128_fold64( + (_load_le(data, ptr, 8) ^ _MUMBO_SECRET[1]) & _MASK64, + (_load_le(data, ptr + 8, 8) ^ seed) & _MASK64, + ) + ptr += 16 + remaining -= 16 + + # Final 1..16 bytes as two loads overlapping the end (len > 16, so in-bounds). + val_a = _load_le(data, ptr + remaining - 16, 8) + val_b = _load_le(data, ptr + remaining - 8, 8) + return _mumbo_finish(val_a, val_b, seed, length) + # fnv1a constants (see `hash_fnv1a.h`): the offset basis is the canonical default # seed, so `hash.fnv1a(data)` matches published FNV-1a 64 reference values. _FNV_OFFSET_BASIS = 0xCBF29CE484222325 @@ -121,4 +224,5 @@ def _fnv1a(data, seed = _FNV_OFFSET_BASIS): hash = struct( dumbo = _dumbo, fnv1a = _fnv1a, + mumbo = _mumbo, ) diff --git a/mbo/hash/internal/hash_bzl_verify.bzl b/mbo/hash/internal/hash_bzl_verify.bzl index d057111..59ddd4c 100644 --- a/mbo/hash/internal/hash_bzl_verify.bzl +++ b/mbo/hash/internal/hash_bzl_verify.bzl @@ -36,7 +36,7 @@ _LENGTHS = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 16, 17, 23, 24, 25, 31, 32, 33, 63 # A handful of concrete strings, including mangle fold inputs. _EXTRA = ["0.13.0|", "0.13.1|", "hello world, this is dumbo!", "|"] -_FNS = {"dumbo": hash.dumbo, "fnv1a": hash.fnv1a} +_FNS = {"dumbo": hash.dumbo, "fnv1a": hash.fnv1a, "mumbo": hash.mumbo} def _pattern(length): return "".join([_PRINTABLE[i % len(_PRINTABLE)] for i in range(length)]) @@ -46,7 +46,7 @@ def _hex16(value): digits = "%X" % value return ("0" * (16 - len(digits))) + digits -def hash_bzl_verify(name, algos = ["dumbo", "fnv1a"], tool = "//mbo/hash:hash_tool"): +def hash_bzl_verify(name, algos = ["dumbo", "fnv1a", "mumbo"], tool = "//mbo/hash:hash_tool"): """Wires up the bzl-vs-C++ diff tests for each `algo` in `algos`. Args: From b19772c319e442c43b8478fee3de447ea94ff717 Mon Sep 17 00:00:00 2001 From: helly25 <6420169+helly25@users.noreply.github.com> Date: Mon, 13 Jul 2026 19:01:15 +0100 Subject: [PATCH 2/2] Drop measurement bundle wrongly added to this PR (untracked WIP swept in by git add -A); file kept on disk --- ...x-16-core-processor_10c_gcc-15_8facf9fd_20260713_005125.tgz | 3 --- 1 file changed, 3 deletions(-) delete mode 100644 mbo/hash/measurements/data/linux-x86-64-amd-ryzen-9-9950x-16-core-processor_10c_gcc-15_8facf9fd_20260713_005125.tgz diff --git a/mbo/hash/measurements/data/linux-x86-64-amd-ryzen-9-9950x-16-core-processor_10c_gcc-15_8facf9fd_20260713_005125.tgz b/mbo/hash/measurements/data/linux-x86-64-amd-ryzen-9-9950x-16-core-processor_10c_gcc-15_8facf9fd_20260713_005125.tgz deleted file mode 100644 index c199dc1..0000000 --- a/mbo/hash/measurements/data/linux-x86-64-amd-ryzen-9-9950x-16-core-processor_10c_gcc-15_8facf9fd_20260713_005125.tgz +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e5f7c5ebdde48ee5ef8c250a7daa6cacdd96657bf81abab9561126978ae1affa -size 522407