From 59e856af697661a7b8272d190da247602a5836b3 Mon Sep 17 00:00:00 2001 From: helly25 <6420169+helly25@users.noreply.github.com> Date: Sat, 11 Jul 2026 18:47:04 +0100 Subject: [PATCH 1/2] mbo/hash/measurements: decouple measurement from publishing; labeled multi-machine README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The measurement run now writes ONLY a data bundle - no committed charts/tables into the tree - so it stays authoritative and per-compiler/per-machine runs go back-to-back (fixes the "dirty=True" non-authoritative warning caused by writing SVGs up front). A new `publish` command renders the README perf section from a SELECTED set of committed bundles: one labeled `###` block per machine (its charts + curated tables), replacing a marker region it cuts and re-appends. Charts carry the machine identifier as a subtitle (cpu-brand · arch · cores · compiler · sha), and each block heading covers both charts and tables. `verify` (no args) re-renders every featured bundle from the manifest publish leaves in the README and diffs the committed charts. Single measurement, not two: the benchmark emits its README size subset (kReadmeSizes) as `readme_sizes` in the dataset context, so the curated ~22-row tables are EXTRACTED from the one FULL run - no separate fast run, and no second size list to drift (the C++ list is the sole source, emitted into the data and read back). Fast mode stays only as a quick dev loop. --- mbo/hash/README.md | 7 +- mbo/hash/hash_benchmark.cc | 8 + mbo/hash/measurements/README.md | 95 +++++---- .../measurements/hash_benchmark_report.py | 181 +++++++++++++----- mbo/hash/measurements/run_measurements.py | 40 ++-- 5 files changed, 206 insertions(+), 125 deletions(-) diff --git a/mbo/hash/README.md b/mbo/hash/README.md index c6b6986..aee636b 100644 --- a/mbo/hash/README.md +++ b/mbo/hash/README.md @@ -226,7 +226,10 @@ the libc++ SSO cap, 47/48 and 63/64 the short-chain steps). The tool's full mode sweeps a denser exponential curve. Full-sweep curves (log-log axes, `run_measurements.py`; the tables below are the -dense README subset): +dense README subset). Everything between the markers is regenerated per machine +by `hash_benchmark_report.py publish` from the committed data bundles: + + ![mbo/hash 64-bit one-shot throughput vs key length, log-log](measurements/hash_throughput_64.svg) @@ -290,6 +293,8 @@ actually pays (as opposed to the hot, size-predictable throughput loop above). | 64 | 11.78 | 11.47 | 11.93 | **9.90** | 19.73 | 27.83 | 11.41 | 17.20 | | 1024 | **26.46** | 26.74 | 37.26 | 65.11 | 69.52 | 229.4 | 604.2 | 97.80 | + + Reading the results: `rapidhash` leads small keys, but after the if-ladder load path (see the design iterations) `mumbo` sits ~2.0 ns through 16 bytes - within ~0.3 ns of rapidhash across the inline-`std::string` range (2.01 ns diff --git a/mbo/hash/hash_benchmark.cc b/mbo/hash/hash_benchmark.cc index 6b49b8c..963106a 100644 --- a/mbo/hash/hash_benchmark.cc +++ b/mbo/hash/hash_benchmark.cc @@ -188,6 +188,14 @@ int main(int argc, char** argv) { benchmark::AddCustomContext("compiler", "gcc-" + std::to_string(__GNUC__)); benchmark::AddCustomContext("compiler_version", __VERSION__); #endif + // Emit the curated README size subset (kReadmeSizes) so the report tool extracts + // the small table straight from a FULL dataset - no separate fast run, and no + // second size list to drift (this C++ list is the single source of truth). + std::string readme_sizes; + for (const int size : mbo::hash::kReadmeSizes) { + readme_sizes += (readme_sizes.empty() ? "" : ",") + std::to_string(size); + } + benchmark::AddCustomContext("readme_sizes", readme_sizes); benchmark::RunSpecifiedBenchmarks(); benchmark::Shutdown(); return 0; diff --git a/mbo/hash/measurements/README.md b/mbo/hash/measurements/README.md index 9764e5e..78b5727 100644 --- a/mbo/hash/measurements/README.md +++ b/mbo/hash/measurements/README.md @@ -32,12 +32,11 @@ even invert the true ordering. So we: mbo/hash/measurements/ README.md # this design doc MODULE.bazel # separate dev module (isolates plotting deps) - run_measurements.py # one-shot authoritative runner (perf + chart + parallel smhasher) - hash_benchmark_report.py # run / store / tables / plot / smhasher (stdlib only) + run_measurements.py # per-machine runner: one full sweep + smhasher -> a data bundle (nothing else) + hash_benchmark_report.py # run / store / tables / plot / bundle / publish / verify (stdlib only) build_smhasher3.sh # reproducible SMHasher3 build (clone + fixes + install plugin + container gcc) smhasher3/mbohash.cpp # in-house mumbo/jumbo and dumbo SMHasher3 registration (includes the real headers) - hash_throughput_64.svg # published 64-bit throughput chart (committed; verifiable from data/) - hash_throughput_128.svg # published 128-bit throughput chart (committed; verifiable from data/) + charts/__*.svg # published per-machine charts (committed; rendered by `publish`, verifiable) data//*.tgz # per-machine measurement bundles, Git LFS (see "Data storage") ``` @@ -107,14 +106,17 @@ logs. Across several machines that accumulates, so: `smhasher.json` + per-algorithm logs. It is Git-LFS-tracked (`.gitattributes`), so the blobs stay out of the main pack; unpack to inspect or A/B. Keep the current bundle per machine (replace on re-measure); LFS history retains olds. -- **Published charts, plain git** (`hash_throughput_64.svg` / `_128.svg`): the - human-facing output embedded in the hash README. Small, and - the point - - _verifiable_: `hash_benchmark_report.py verify --bundle ` regenerates them - from the bundled canonical data and fails on any mismatch, so every published - figure is provably derived from a committed dataset and cannot be hand-edited - (`_svg_plot` is deterministic: geometry keyed only off the data, no wall-clock). - The README perf tables are likewise rendered from the canonical data (unpack a - bundle and run `tables`). +- **Published charts + tables, plain git** (`charts/__{64,128}.svg` + - the hash README perf section): rendered by `hash_benchmark_report.py publish` + from a SELECTED set of committed bundles - one labeled block per machine. Charts + carry the machine identifier (subtitle); the tables are the curated `kReadmeSizes` + subset extracted from the FULL dataset via the `readme_sizes` the benchmark emits + into its own context (no separate fast run, no second size list to drift). And - + the point - _verifiable_: `verify` re-renders from the same bundles (a manifest + publish leaves in the README) and diffs, so every figure is provably from + committed data and cannot be hand-edited (`_svg_plot` is deterministic). This is + the ONLY thing that writes to the tree; the measurement run writes nothing but + the bundle, so it stays authoritative and several runs can go back-to-back. - The loose run outputs in `data/` are staging only and git-ignored; `bundle` packs them into the `.tgz`. This directory is dev-only and `.bazelignore`'d, so it never enters the module or BCR (the release archive carries LFS pointer @@ -122,55 +124,48 @@ logs. Across several machines that accumulates, so: ## Regenerate / contribute a machine -Run the one-shot runner on the machine you want to add, from the repo root and a -clean `main` checkout (so the provenance is authoritative). It runs the perf -sweep, renders + verifies the charts, runs the SMHasher3 battery, packs the -per-machine LFS bundle, and prints exactly what to commit: +Two decoupled steps: each machine produces a data bundle (that is ALL the run +writes to the tree, so it stays authoritative and runs can go back-to-back), then +a single `publish` renders the README from the machines you choose. + +**1. Measure, per machine** (repo root, clean `main` for authoritative provenance): ```sh -# Everything, SMHasher3 batteries 4-at-a-time (perf runs first and alone): -mbo/hash/measurements/run_measurements.py --jobs 4 -# Pick the toolchain (and thus the recorded compiler): --config clang uses the -# hermetic LLVM clang; omit --config for the native toolchain (gcc on Linux). -# Run once per compiler to compare them - the bundle filenames won't collide: -mbo/hash/measurements/run_measurements.py --config clang --jobs 4 -# In-house family only, or perf/chart only: +mbo/hash/measurements/run_measurements.py --config clang --jobs 4 # clang; omit --config for native gcc +# faster: in-house family only, or perf only mbo/hash/measurements/run_measurements.py --algos mumbo,jumbo,dumbo --jobs 1 mbo/hash/measurements/run_measurements.py --skip-smhasher ``` -The performance sweep runs first and alone (SMHasher3 would contend for CPU and -skew the sub-ns numbers); the batteries are independent and their pass/fail is -load-independent, so `--jobs` runs several concurrently - trading cores for -wall-clock, not accuracy. SMHasher3 is built and run inside a container -(`build_smhasher3.sh`), so the runner invokes its Linux binary via `docker run`. +ONE full perf sweep runs first and alone (SMHasher3 would contend for CPU and +skew the sub-ns numbers), then the battery (`--jobs` at once; pass/fail is +load-independent, so that only trades cores for wall-clock). SMHasher3 is built +and run in a container (`build_smhasher3.sh`, via `docker run`). The run packs a +per-machine Git-LFS bundle and prints its path - commit it: + +```sh +git add mbo/hash/measurements/data//.tgz +git lfs push origin HEAD && git push +``` -Or drive the steps individually (`bundle` packs a run; `verify` audits the -published charts against a bundle; unpack a bundle for its canonical JSON): +**2. Publish, once** - render the README from the bundles you want to feature: ```sh -# Perf sweep -> staged canonical results.json + raw.json.gz (from the repo root): -python3 mbo/hash/measurements/hash_benchmark_report.py run \ - --reps 20 --warmup 0.05 \ - --raw mbo/hash/measurements/data/raw.json.gz \ - --out mbo/hash/measurements/data/results.json --tables - -# Pack a run's staged outputs into the per-machine Git-LFS bundle: -python3 mbo/hash/measurements/hash_benchmark_report.py bundle \ - --results mbo/hash/measurements/data/_results.json \ - --include mbo/hash/measurements/data/_raw.json.gz \ - mbo/hash/measurements/data/_smhasher.json - -# Verify the committed charts match a bundle's data (unpacks, re-renders, diffs): -python3 mbo/hash/measurements/hash_benchmark_report.py verify \ - --bundle mbo/hash/measurements/data//.tgz - -# Re-render the README tables / charts from a bundle's canonical JSON: -tar xzOf .tgz results.json > /tmp/results.json -python3 mbo/hash/measurements/hash_benchmark_report.py tables --results /tmp/results.json -python3 mbo/hash/measurements/hash_benchmark_report.py plot --results /tmp/results.json --out /tmp/hash.svg +mbo/hash/measurements/hash_benchmark_report.py publish \ + --bundles data//*.tgz data//*.tgz data//*.tgz +git add mbo/hash/README.md mbo/hash/measurements/charts ``` +`publish` cuts the marker region in the hash README and re-appends one labeled +block per bundle (charts + curated tables), in the listed order; `--bundles` is +the selection. `verify` (no args, from the repo root) re-checks every featured +bundle against the committed charts, reading the machine list back from the +README's manifest. + +The pipeline steps are also usable individually: `run` (perf -> canonical JSON), +`bundle` (pack a run), `tables` / `plot` (render from any canonical JSON - +`tar xzOf .tgz results.json` pulls one out), `publish`, `verify`. + ## SMHasher3 quality (`smhasher`) `hash_benchmark_report.py smhasher --algos all --smhasher3 ` runs the diff --git a/mbo/hash/measurements/hash_benchmark_report.py b/mbo/hash/measurements/hash_benchmark_report.py index 961b044..6eb67b8 100644 --- a/mbo/hash/measurements/hash_benchmark_report.py +++ b/mbo/hash/measurements/hash_benchmark_report.py @@ -247,6 +247,21 @@ def _platform_slug(ctx): return f"{os_name}-{_slug(machine) or 'arch'}-{_slug(ctx.get('cpu_brand') or '') or 'cpu'}" +def _machine_label(ctx): + """Readable machine identifier for headings + chart subtitles, e.g. + 'Apple M5 Pro · arm64 · 18-core · clang-22 · a1b2c3d4'.""" + parts = (ctx.get("uname") or "").split() # `uname -srm`: + bits = [str(ctx.get("cpu_brand") or "?"), parts[-1] if len(parts) >= 3 else "?"] + if ctx.get("num_cpus"): + bits.append(f"{ctx['num_cpus']}-core") + if ctx.get("compiler"): + bits.append(str(ctx["compiler"])) + sha = ((ctx.get("source") or {}).get("git_sha") or "")[:8] + if sha: + bits.append(sha) + return " · ".join(bits) + + # SMHasher3's per-test failures and final verdict. Tolerant: SMHasher3 output # varies by version, so we capture the raw log and parse best-effort. # Verdict + optional score, e.g. "Overall result: FAIL ( 181 / 188 )". @@ -395,13 +410,16 @@ def line(cells): return "\n".join([line(headers), "| " + " | ".join(sep) + " |"] + [line(r) for r in rows]) -def _throughput_table(data, preferred, relabel): +def _throughput_table(data, preferred, relabel, sizes=None): # Length-per-row, algorithm-per-column; each row's bold marks the fastest - # algorithm at that length. The sizes are read from the data's own buckets - # (the single source of truth), so there is no second size list to drift from - # the C++ kReadmeSizes - feed this the README (fast-mode) dataset. + # algorithm at that length. `sizes` is the curated README subset, taken from + # the dataset's own `readme_sizes` context (emitted by the benchmark = the C++ + # kReadmeSizes), so the small table is extracted from the FULL run with no + # second size list to drift. Without it, every measured length is shown. algos = _order(list(data), preferred) - sizes = sorted({int(s) for a in data.values() for s in a}) + if sizes is None: + sizes = sorted({int(s) for a in data.values() for s in a}) + sizes = [s for s in sizes if any(str(s) in data[a] for a in algos)] headers = ["Length"] + [relabel.get(a, a) for a in algos] rows = [] for size in sizes: @@ -440,40 +458,41 @@ def _latency_table(data): def render_tables(results): ctx = results.get("context", {}) - host = ctx.get("host_name", "?") - sha = (ctx.get("source") or {}).get("git_sha") or "?" meas = ctx.get("measurement", {}) agg = f"mean of the {meas.get('best_k', '?')} fastest of {meas.get('reps', '?')} reps" + raw = ctx.get("readme_sizes") # curated subset emitted by the benchmark (kReadmeSizes) + sizes = [int(x) for x in str(raw).split(",") if x.strip().isdigit()] if raw else None out = [ - f"", + f"", "", - f"### 64-bit one-shot throughput (ns/op, {agg}; lower is better)", + f"#### 64-bit one-shot throughput (ns/op, {agg}; lower is better)", "", - _throughput_table(results["throughput64"], _ORDER_64, {}), + _throughput_table(results["throughput64"], _ORDER_64, {}, sizes), "", - f"### 128-bit one-shot throughput (ns/op, {agg}; native-128 algorithms only)", + f"#### 128-bit one-shot throughput (ns/op, {agg}; native-128 algorithms only)", "", - _throughput_table(results["throughput128"], _ORDER_128, _LABEL_128), + _throughput_table(results["throughput128"], _ORDER_128, _LABEL_128, sizes), "", - f"### Mixed-length latency (ns/hash, {agg}; lower is better)", + f"#### Mixed-length latency (ns/hash, {agg}; lower is better)", "", _latency_table(results["latency"]), ] return "\n".join(out) -def _svg_plot(data, algos, title, path, label_map=None): +def _svg_plot(data, algos, title, path, label_map=None, subtitle=None): """Dependency-free log-log SVG: best-k-mean ns vs length, one line per algorithm. Both axes are log-scaled - the ns range spans ~4 decades (sub-ns small keys to microseconds of the byte-at-a-time hashes on bulk), so - a linear y-axis would crush every fast algorithm onto the baseline.""" + a linear y-axis would crush every fast algorithm onto the baseline. `subtitle` + (the machine identifier) is drawn under the title so a chart is self-labeling.""" import math label_map = label_map or {} sizes = sorted({int(s) for a in data.values() for s in a}) if not sizes or not algos: return - width, height, pad_l, pad_r, pad_t, pad_b = 900, 520, 66, 132, 46, 52 + width, height, pad_l, pad_r, pad_t, pad_b = 900, 536, 66, 132, 62, 52 plot_w, plot_h = width - pad_l - pad_r, height - pad_t - pad_b x0, x1 = math.log10(sizes[0]), math.log10(sizes[-1]) vals = [_val(data[a][str(s)]) for a in algos for s in sizes if str(s) in data[a] and _val(data[a][str(s)]) > 0] @@ -490,8 +509,12 @@ def py(ns): svg = [ f'', f'', - f'{title}', + f'{title}', ] + if subtitle: + svg.append( + f'{subtitle}' + ) exp = math.floor(ly0) # y gridlines + labels at each power of ten in range while exp <= math.ceil(ly1): ns = 10.0**exp @@ -542,6 +565,44 @@ def _dump_canonical(results, path): handle.write("\n") +# --- publish: labeled per-machine perf section rendered FROM the data bundles --- + +_PERF_BEGIN = "" +_PERF_END = "" + + +def _extract_bundle(path, dest): + """Extract a bundle .tgz into dest (hardened on py>=3.12; our own data).""" + with tarfile.open(path, "r:gz") as tar: + try: + tar.extractall(dest, filter="data") + except TypeError: + tar.extractall(dest) # noqa: S202 - our own bundle, older Python + + +def _render_charts(full, stem, charts_dir, subtitle): + """Write the 64/128 labeled SVGs for one machine; return [(tag, filename), ...].""" + written = [] + for key, order, tag, labels in ( + ("throughput64", _ORDER_64, "64", None), + ("throughput128", _ORDER_128, "128", _LABEL_128), + ): + if not full.get(key): + continue + name = f"{stem}_{tag}.svg" + _svg_plot( + full[key], _order(list(full[key]), order), f"mbo/hash - {tag}-bit one-shot throughput", + os.path.join(charts_dir, name), labels, subtitle=subtitle, + ) + written.append((tag, name)) + return written + + +def _bundle_stem(ctx): + """Chart-file stem for a machine: '_'.""" + return f"{_platform_slug(ctx)}_{_slug(ctx.get('compiler') or 'cc')}" + + def main(argv): parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) sub = parser.add_subparsers(dest="command", required=True) @@ -592,13 +653,18 @@ def main(argv): ) p_bundle = sub.add_parser("bundle", help="pack a run's artifacts into a per-machine .tgz under data/ (LFS-tracked)") - p_bundle.add_argument("--results", required=True, help="canonical results JSON; its context names the machine") + p_bundle.add_argument("--results", required=True, help="canonical results JSON (drives chart + tables); its context names the machine") p_bundle.add_argument("--include", nargs="*", default=[], help="extra files to pack (raw.json.gz, smhasher.json, logs)") p_bundle.add_argument("--data-dir", default="mbo/hash/measurements/data") - p_verify = sub.add_parser("verify", help="regenerate the SVG charts from a bundle and diff against the committed charts") - p_verify.add_argument("--bundle", required=True, help="the .tgz to verify") - p_verify.add_argument("--charts-dir", default="mbo/hash/measurements", help="dir holding the committed hash_throughput_*.svg") + p_publish = sub.add_parser("publish", help="render the labeled per-machine perf section (charts + tables) into the README from selected bundles") + p_publish.add_argument("--bundles", nargs="+", required=True, help="the .tgz bundles to feature, in display order") + p_publish.add_argument("--charts-dir", default="mbo/hash/measurements/charts") + p_publish.add_argument("--readme", default="mbo/hash/README.md") + + p_verify = sub.add_parser("verify", help="re-render charts from the README's bundle manifest and diff the committed charts") + p_verify.add_argument("--readme", default="mbo/hash/README.md") + p_verify.add_argument("--charts-dir", default="mbo/hash/measurements/charts") args = parser.parse_args(argv) # One stamp per invocation, so all files a run writes share it. Every @@ -686,7 +752,7 @@ def main(argv): os.makedirs(dest_dir, exist_ok=True) dest = os.path.join(dest_dir, f"{slug}_{cores}c_{compiler}_{sha}_{stamp}.tgz") with tarfile.open(dest, "w:gz") as tar: - tar.add(args.results, arcname="results.json") # stable name so `verify` finds it + tar.add(args.results, arcname="results.json") # canonical: chart + tables + verify for path in args.include: if path and os.path.exists(path): tar.add(path, arcname=os.path.basename(path)) @@ -694,31 +760,54 @@ def main(argv): print(f"wrote {dest}", file=sys.stderr) return 0 + if args.command == "publish": + os.makedirs(args.charts_dir, exist_ok=True) + rel = os.path.relpath(args.charts_dir, os.path.dirname(os.path.abspath(args.readme))) + sections = [] + for bundle_path in args.bundles: + with tempfile.TemporaryDirectory() as tmp: + _extract_bundle(bundle_path, tmp) + full = _load_json(os.path.join(tmp, "results.json")) + ctx = full.get("context", {}) + label = _machine_label(ctx) + embeds = [ + f"![mbo/hash {tag}-bit throughput vs key length, log-log]({rel}/{name})" + for tag, name in _render_charts(full, _bundle_stem(ctx), args.charts_dir, label) + ] + # `### {label}` heads the whole block (machine · compiler · sha), covering + # both the charts (which also carry it as a subtitle) and the tables. + sections.append("\n".join([f"### {label}", "", *embeds, "", render_tables(full)])) + print(f"published {label}", file=sys.stderr) + manifest = "" + region = "\n".join([_PERF_BEGIN, manifest, "", "\n\n".join(sections), _PERF_END]) + text = open(args.readme).read() + if _PERF_BEGIN not in text or _PERF_END not in text: + raise SystemExit(f"markers not found in {args.readme}; add a {_PERF_BEGIN} ... {_PERF_END} region") + text = text[: text.index(_PERF_BEGIN)] + region + text[text.index(_PERF_END) + len(_PERF_END) :] + with open(args.readme, "w") as handle: + handle.write(text) + print(f"wrote perf section into {args.readme} ({len(args.bundles)} machine(s))", file=sys.stderr) + return 0 + if args.command == "verify": - with tempfile.TemporaryDirectory() as tmp: - with tarfile.open(args.bundle, "r:gz") as tar: - try: - tar.extractall(tmp, filter="data") # py>=3.12 hardened extraction - except TypeError: - tar.extractall(tmp) # noqa: S202 - our own bundle, older Python - results = _load_json(os.path.join(tmp, "results.json")) - charts = [("throughput64", _ORDER_64, "mbo/hash - 64-bit one-shot throughput", "hash_throughput_64.svg", None)] - if results.get("throughput128"): - charts.append( - ("throughput128", _ORDER_128, "mbo/hash - 128-bit one-shot throughput", "hash_throughput_128.svg", _LABEL_128) - ) - mismatches = [] - for key, order, title, svg_name, labels in charts: - regen = os.path.join(tmp, svg_name) - _svg_plot(results[key], _order(list(results[key]), order), title, regen, labels) - committed = os.path.join(args.charts_dir, svg_name) - if not os.path.exists(committed) or not filecmp.cmp(regen, committed, shallow=False): - mismatches.append(svg_name) - if mismatches: - print(f"VERIFY FAILED: committed charts do not match the bundle data: {', '.join(mismatches)}", file=sys.stderr) - return 1 - print(f"VERIFY OK: committed charts match {os.path.basename(args.bundle)}", file=sys.stderr) - return 0 + match = re.search(r"", open(args.readme).read()) + if not match: + raise SystemExit(f"no bundle manifest in {args.readme}; run `publish` first") + bundles = match.group(1).split() + mismatches = [] + for bundle_path in bundles: + with tempfile.TemporaryDirectory() as tmp: + _extract_bundle(bundle_path, tmp) + full = _load_json(os.path.join(tmp, "results.json")) + for _tag, name in _render_charts(full, _bundle_stem(full.get("context", {})), tmp, _machine_label(full.get("context", {}))): + committed = os.path.join(args.charts_dir, name) + if not os.path.exists(committed) or not filecmp.cmp(os.path.join(tmp, name), committed, shallow=False): + mismatches.append(name) + if mismatches: + print(f"VERIFY FAILED: committed charts differ from the bundle data: {', '.join(sorted(set(mismatches)))}", file=sys.stderr) + return 1 + print(f"VERIFY OK: committed charts match all {len(bundles)} bundle(s)", file=sys.stderr) + return 0 return 1 diff --git a/mbo/hash/measurements/run_measurements.py b/mbo/hash/measurements/run_measurements.py index 664a16c..e87e928 100755 --- a/mbo/hash/measurements/run_measurements.py +++ b/mbo/hash/measurements/run_measurements.py @@ -46,7 +46,6 @@ import argparse import glob import os -import shutil import subprocess import sys @@ -99,8 +98,9 @@ def main(argv): if not args.skip_perf: cfg = ["--config", args.config] if args.config else [] - print(">>> [perf] full sweep (curve) + fast sweep (README tables), solo for clean numbers", file=sys.stderr) - # FULL sweep: the dense ns-vs-length curve + the authoritative raw (compare.py). + print(">>> [perf] full performance sweep (runs solo for clean numbers)", file=sys.stderr) + # A single FULL run: the dense curve AND (via readme_sizes in its context) the + # curated README table are both extracted from it downstream by `publish`. subprocess.run( [*report, "run", "--mode", "full", "--reps", str(args.reps), "--raw", os.path.join(data, "raw.json.gz"), "--out", os.path.join(data, "results.json")] + cfg, @@ -108,22 +108,6 @@ def main(argv): ) canonical = newest(os.path.join(data, "*_results.json")) extras.append(newest(os.path.join(data, "*_raw.json.gz"))) - # FAST sweep: the README tables. Its buckets ARE kReadmeSizes, and the tool - # reads the sizes from THIS data (no hardcoded list to drift from the C++). - with open(os.path.join(data, "README_tables.md"), "w") as tables: - subprocess.run( - [*report, "run", "--mode", "fast", "--reps", str(args.reps), - "--out", os.path.join(data, "results_readme.json"), "--tables"] + cfg, - stdout=tables, check=True, - ) - extras.append(newest(os.path.join(data, "*_results_readme.json"))) - print(">>> [perf] rendering ns-vs-length charts from the full sweep (64/128, log-log)", file=sys.stderr) - subprocess.run([*report, "plot", "--results", canonical, "--out", os.path.join(data, "hash_throughput.svg")], check=True) - for width in ("64", "128"): - shutil.copy( - newest(os.path.join(data, f"*_hash_throughput_{width}.svg")), - os.path.join(meas, f"hash_throughput_{width}.svg"), - ) if not args.skip_smhasher: print(f">>> [smhasher] building SMHasher3 (workdir {args.workdir})", file=sys.stderr) @@ -152,26 +136,26 @@ def main(argv): [*report, "bundle", "--results", canonical, "--include", *extras, "--data-dir", data], capture_output=True, text=True, check=True, ).stdout.strip() - print(">>> [verify] published charts vs the bundle data", file=sys.stderr) - subprocess.run([*report, "verify", "--bundle", bundle, "--charts-dir", meas], check=False) elif not args.skip_smhasher: print("NOTE: smhasher-only run has no perf canonical to key a bundle on; re-run with perf to bundle.", file=sys.stderr) + report_py = os.path.join(meas, "hash_benchmark_report.py") print( "\n".join( [ "", "=== done ===", - "Commit the published charts (plain git) and the per-machine data bundle (Git LFS):", - f" git add {meas}/hash_throughput_64.svg {meas}/hash_throughput_128.svg", + "This run wrote NOTHING to the tree except the bundle below, so it stays", + "authoritative and you can run other compilers/machines back-to-back.", + "Commit the per-machine data bundle (Git LFS):", ] + ([f" git add {bundle}"] if bundle else []) + [ - f"Refresh the README perf tables by pasting {data}/README_tables.md, and re-embed the charts:", - " ![mbo/hash 64-bit throughput](measurements/hash_throughput_64.svg)", - " ![mbo/hash 128-bit throughput](measurements/hash_throughput_128.svg)", - "Anyone can re-verify the published charts against the committed data:", - f" {os.path.join(meas, 'hash_benchmark_report.py')} verify --bundle {bundle or '/.tgz>'}", + "Then render the README charts + tables from the machines you want to feature:", + f" {report_py} publish --bundles ...", + " git add mbo/hash/README.md mbo/hash/measurements/charts", + "and re-verify the committed charts against their data any time:", + f" {report_py} verify", ] ) ) From a6c241e8d7c4046d9971747ebbba6059f4088bcc Mon Sep 17 00:00:00 2001 From: helly25 <6420169+helly25@users.noreply.github.com> Date: Sat, 11 Jul 2026 18:50:47 +0100 Subject: [PATCH 2/2] mbo/hash/measurements: flat data/ bundle layout The bundle filename already carries the full machine identity (_c___.tgz), so the per-machine subdirectory was redundant and only added a level to fight with. Bundles now live flat under data/; .gitignore/.gitattributes and the docs follow. --- .gitattributes | 2 +- mbo/hash/measurements/README.md | 10 +++++----- mbo/hash/measurements/data/.gitignore | 5 ++--- mbo/hash/measurements/hash_benchmark_report.py | 6 +++--- 4 files changed, 11 insertions(+), 12 deletions(-) diff --git a/.gitattributes b/.gitattributes index 14b8ed6..441513d 100644 --- a/.gitattributes +++ b/.gitattributes @@ -4,4 +4,4 @@ # helly25_mbo module or any BCR offering; the generated BCR source archive # carries LFS pointer files here, which is harmless (nothing builds them). # See mbo/hash/measurements/README.md. -mbo/hash/measurements/data/**/*.tgz filter=lfs diff=lfs merge=lfs -text +mbo/hash/measurements/data/*.tgz filter=lfs diff=lfs merge=lfs -text diff --git a/mbo/hash/measurements/README.md b/mbo/hash/measurements/README.md index 78b5727..769c41f 100644 --- a/mbo/hash/measurements/README.md +++ b/mbo/hash/measurements/README.md @@ -37,7 +37,7 @@ mbo/hash/measurements/ build_smhasher3.sh # reproducible SMHasher3 build (clone + fixes + install plugin + container gcc) smhasher3/mbohash.cpp # in-house mumbo/jumbo and dumbo SMHasher3 registration (includes the real headers) charts/__*.svg # published per-machine charts (committed; rendered by `publish`, verifiable) - data//*.tgz # per-machine measurement bundles, Git LFS (see "Data storage") + data/__*.tgz # per-machine measurement bundles, Git LFS, flat (see "Data storage") ``` The C++ benchmark itself (`mbo/hash/hash_benchmark.cc`, @@ -100,7 +100,7 @@ distilled canonical JSON is ~38 KB / ~100 KB, plus the per-algorithm SMHasher3 logs. Across several machines that accumulates, so: - **Per-machine bundle, Git LFS** - (`data/--/_c___.tgz`): one + (`data/_c___.tgz`): one gzipped tarball per run holds the _whole_ dataset - the canonical `results.json`, the raw `*_raw.json.gz` (for `compare.py` U-tests), and the `smhasher.json` + per-algorithm logs. It is Git-LFS-tracked (`.gitattributes`), @@ -144,7 +144,7 @@ and run in a container (`build_smhasher3.sh`, via `docker run`). The run packs a per-machine Git-LFS bundle and prints its path - commit it: ```sh -git add mbo/hash/measurements/data//.tgz +git add mbo/hash/measurements/data/.tgz git lfs push origin HEAD && git push ``` @@ -152,7 +152,7 @@ git lfs push origin HEAD && git push ```sh mbo/hash/measurements/hash_benchmark_report.py publish \ - --bundles data//*.tgz data//*.tgz data//*.tgz + --bundles data/.tgz data/.tgz data/.tgz git add mbo/hash/README.md mbo/hash/measurements/charts ``` @@ -195,7 +195,7 @@ Last verified run (2026-07): **mumbo-64/jumbo-128** and **dumbo-64** all PASS Loose staging files the tool writes are prefixed `YYYYMMDD_HHMMSS_` (local wall clock, one stamp per invocation) so runs never overwrite each other. The committed artifact is the per-machine bundle -`data/--/_c___.tgz` - the slug +`data/_c___.tgz` - the slug derived from the dataset's own `uname` + CPU brand, the `compiler` reported by the benchmark binary (`clang-NN` / `gcc-NN`, so GCC and Clang builds on one machine stay distinct), the SHA from its provenance - so a bundle is self-identifying and diff --git a/mbo/hash/measurements/data/.gitignore b/mbo/hash/measurements/data/.gitignore index fe0ef27..5ea9cd4 100644 --- a/mbo/hash/measurements/data/.gitignore +++ b/mbo/hash/measurements/data/.gitignore @@ -1,9 +1,8 @@ # Loose run outputs (results.json, raw.json.gz, smhasher.json, logs) are staging -# only - they get packed into a per-machine LFS bundle `/<...>.tgz` by +# only - they get packed into a per-machine LFS bundle `_..._.tgz` by # `hash_benchmark_report.py bundle`. Commit the bundles (LFS-tracked, see # ../../../.gitattributes), not the loose files. See ../README.md. * !.gitkeep !.gitignore -!*/ -!*/*.tgz +!*.tgz diff --git a/mbo/hash/measurements/hash_benchmark_report.py b/mbo/hash/measurements/hash_benchmark_report.py index 6eb67b8..8b878f8 100644 --- a/mbo/hash/measurements/hash_benchmark_report.py +++ b/mbo/hash/measurements/hash_benchmark_report.py @@ -748,9 +748,9 @@ def main(argv): cores = ctx.get("num_cpus", "?") compiler = _slug(ctx.get("compiler") or "cc") sha = ((ctx.get("source") or {}).get("git_sha") or "nogit")[:8] - dest_dir = os.path.join(args.data_dir, slug) - os.makedirs(dest_dir, exist_ok=True) - dest = os.path.join(dest_dir, f"{slug}_{cores}c_{compiler}_{sha}_{stamp}.tgz") + # Flat: the filename already carries the full machine identity, so no subdir. + os.makedirs(args.data_dir, exist_ok=True) + dest = os.path.join(args.data_dir, f"{slug}_{cores}c_{compiler}_{sha}_{stamp}.tgz") with tarfile.open(dest, "w:gz") as tar: tar.add(args.results, arcname="results.json") # canonical: chart + tables + verify for path in args.include: