bench: pony_bench-based benchmark suite for the 0.2.0 surface - #10
Merged
Conversation
Establishes a performance baseline before encoding work begins in 0.3.0
and so that implementation tweaks have attributable numbers.
Layout mirrors our existing build-tool split:
unicode_bench/ package: harness, corpora, 15 topic primitives
_bench.pony BenchSizes / _Overhead / _Bench / _ValBench
/ _CtorBench / _BenchPair / _BenchU32
_corpora.pony 7 generators: ascii / latin_precomposed /
latin_decomposed / cjk / mixed / emoji /
combining_marks
bench_*.pony 15 topic registrations (Bytes, Text,
Codepoints, Graphemes, Words, Sentences,
Lines, Normalize, Case, Compares, Search,
Split, Trim, Replace, Scripts)
unicode_bench_main/ binary entry point — actor Main is
main.pony BenchmarkList; registers every topic against
PonyBench in create.
The harness adapts the working pieces of the sibling
`string_benchmark` project — size buckets (16B → 160M), the per-bucket
`cfg(size)` that scales samples and iteration caps so large sizes
don't generate gigabytes of garbage per sample, the config-aware
overhead bench. New wrappers: `_BenchPair` for binary ops
(`Compares.equal_*`, `Search.contains`), `_BenchU32` for single-cp
property lookups.
Each `bench_*` topic exposes a public primitive with a
`register(bench: PonyBench, sizes: Array[USize] val)` method. Sizes
are passed in from main so smoke (single small bucket) and full-suite
runs share the same registration code.
Makefile targets (NOT part of `ci`; benchmark runtimes are far too
long for every PR):
make bench-build release-mode compile
make bench full suite
make bench-smoke --smoke single small bucket
make bench-csv -csv into build/bench-<sha>.csv
Validated end-to-end on the smoke bucket. Spot-check from the run:
Bytes.is_valid_utf8/ascii/1K ≈ 4.7µs vs cjk/1K ≈ 5.5µs — the 15%
delta from multi-byte UTF-8 work confirms the harness measures real
cost rather than noise.
Pony / LLVM gotchas worked around during bring-up: `DoNotOptimise`
generic specialization fails on `(USize, USize)` tuples (LLVM crash)
and on unions containing `String iso^` ephemeral types. Tuple cases
extract `(let lo, let hi)` then pass the sum; iso cases use
`(String iso | InvalidUtf8)` without the ephemeral marker.
Two operations cost minutes per single iteration at the 160M bucket, which makes the suite intractable for routine before/after diffing: - `Text.from_string[indexed]` builds an O(n) grapheme bitmap on top of validation. From the partial CSV: ~750 ns/byte at the 15M bucket, projecting to ~2 min per iter at 160M; with 5 samples + the paired overhead bench, ~30-45 min per Text-indexed bench triple. - `Normalize.nfc/combining` runs the canonical reorder over a 6-codepoint combining run per base letter, then composes. Same shape — likely 10+ min per bench at 160M. `BenchSizes.heavy_op_cap() = 16_000_000` is the new ceiling for these registrations. Other ops still run at every bucket; tighten the cap later if we find more offenders, or loosen it once an implementation gets fast enough to make 160M tractable. Also keeps the `tee`-based `bench-csv` target from the merge so CSV output streams to the terminal as well as the file.
`BenchSizes.label` previously rounded down to a K or M suffix, which lost precision: the 1_600-byte bucket showed up as "1K" (not 1 KiB); the 160_000_000-byte bucket as "152M" (not 152 MiB). Bench output now reads `Bytes.is_valid_utf8/ascii/1600B` and `Text.from_string/emoji/160000000B` — uglier but unambiguous, and the CSV column sorts numerically by suffix-stripped digits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Establishes a performance baseline before encoding work begins in 0.3.0 and so that implementation tweaks have attributable numbers.
Layout mirrors our existing build-tool split:
unicode_bench/ package: harness, corpora, 15 topic primitives
_bench.pony BenchSizes / _Overhead / _Bench / _ValBench
/ _CtorBench / _BenchPair / _BenchU32
corpora.pony 7 generators: ascii / latin_precomposed /
latin_decomposed / cjk / mixed / emoji /
combining_marks
bench*.pony 15 topic registrations (Bytes, Text,
Codepoints, Graphemes, Words, Sentences,
Lines, Normalize, Case, Compares, Search,
Split, Trim, Replace, Scripts)
unicode_bench_main/ binary entry point — actor Main is
main.pony BenchmarkList; registers every topic against
PonyBench in create.
The harness adapts the working pieces of the sibling
string_benchmarkproject — size buckets (16B → 160M), the per-bucketcfg(size)that scales samples and iteration caps so large sizes don't generate gigabytes of garbage per sample, the config-aware overhead bench. New wrappers:_BenchPairfor binary ops (Compares.equal_*,Search.contains),_BenchU32for single-cp property lookups.Each
bench_*topic exposes a public primitive with aregister(bench: PonyBench, sizes: Array[USize] val)method. Sizes are passed in from main so smoke (single small bucket) and full-suite runs share the same registration code.Makefile targets (NOT part of
ci; benchmark runtimes are far too long for every PR):make bench-build release-mode compile
make bench full suite
make bench-smoke --smoke single small bucket
make bench-csv -csv into build/bench-.csv
Validated end-to-end on the smoke bucket. Spot-check from the run: Bytes.is_valid_utf8/ascii/1K ≈ 4.7µs vs cjk/1K ≈ 5.5µs — the 15% delta from multi-byte UTF-8 work confirms the harness measures real cost rather than noise.
Pony / LLVM gotchas worked around during bring-up:
DoNotOptimisegeneric specialization fails on(USize, USize)tuples (LLVM crash) and on unions containingString iso^ephemeral types. Tuple cases extract(let lo, let hi)then pass the sum; iso cases use(String iso | InvalidUtf8)without the ephemeral marker.