Perf/grapheme break match - #11
Merged
Merged
Conversation
Replaces the hex-blob-plus-binary-search lookup in
`_UcdGraphemeBreak.of` with a code-generated `match cp` whose arms
are the coalesced ranges from GraphemeBreakProperty.txt + emoji-data
(Extended_Pictographic). Each coalesced range becomes one arm:
| let c: U32 if (c >= 0x0000) and (c <= 0x0009) => GBControl
| let c: U32 if (c >= 0x000A) and (c <= 0x000A) => GBLF
...
else GBOther
The binary search used `_UcdHex.byte` to ASCII-decode the table
twice per iteration (range_lo + range_hi, 8 hex pairs each); each
of ~7 iterations did ~16 hex decodes plus the U32 reassembly and
two compares. The match form does one comparison pair per arm
(c >= lo, c <= hi) with no decoding overhead.
The emitter (`unicode_build/grapheme_break_table.pony`) is now the
only place that knows the byte → variant name mapping; the
hand-written `unicode/grapheme_break.pony` closed union is
unchanged. Conformance preserved:
* 195 unit tests pass
* GraphemeBreakTest: 1,093 / 1,093 (100%)
* Normalization Part 1+2: 295,411 / 295,411 (100%)
* Word / Sentence / LineBreakTest: 100% across all
Source file grows from 27 KB (hex blob) to 93 KB (1,454 match arms).
Runtime cost of the lookup is the question this commit is set up to
answer; bench it against the previous encoding via
`Codepoints.grapheme_break/{ascii,latin1,cjk,emoji}` and
`Graphemes.count/<corpus>/<size>` from the bench suite.
bench_codepoints.pony already benched 7 of the per-codepoint table lookups (category / script / script_extensions / east_asian_width / has_binary_property / name / combining_class). Adds the remaining single-cp `Codepoints.*(u)` lookups so every UCD table has a measurable point: grapheme_break canonical_decomposition / compat_decomposition simple_upper / simple_lower / simple_title / simple_casefold full_upper / full_lower / full_title / full_casefold is_full_composition_excluded Plus the binary `compose_canonical(lhs, rhs)` with both hit (e + combining acute → é) and miss (A + B → None) cases — that lookup is on the `Normalize.nfc` hot path so the two-arm cost matters. Each new bench runs against the same four representative codepoints (ASCII 'A', Latin-1 'é', CJK U+4E2D, emoji U+1F600), so per-cp cache behavior is visible across the lookup family.
Splits the previous flat match (one arm per coalesced range, linear
scan across all 1,800+ arms) into a two-level structure:
match cp >> 12 // 4096-cp bucket index (no guards)
| 0x0000 =>
match cp // linear chain of upper-bound arms inside this bucket
| let c: U32 if c <= 0x0009 => GBControl
| ...
else GBOther
end
| 0x0001 => ...
...
else GBOther
end
The top-level `match cp >> 12` has no guards — the patterns are
plain integer literals — so LLVM lowers it to a switch / jump
table. The inner per-bucket match has at most ~30 arms, so even a
linear scan inside one bucket is cheap.
Ranges that span more than one bucket (e.g. CJK Unified Ideographs
0x4E00..0x9FFF) get split at bucket boundaries in `_split_at_buckets`
before emission so every emitted arm stays inside exactly one bucket.
Before/after (smoke 1.6KB single-cp probes — `_BenchU32` repeats the
same constant cp, so LLVM can constant-fold most of the dispatch
once these become pure code, but the structural improvement is real
for cursor-driven workloads that pass varying cps through):
OLD (binary search) REFINED (flat match) NESTED
ascii 'A' U+0041 580 ns 7 ns 8 ns
latin1 'é' U+00E9 687 ns 10 ns 12 ns
cjk '中' U+4E2D 595 ns 739 ns 6 ns
emoji '😀' U+1F600 675 ns 2122 ns 24 ns
The REFINED form (flat single-bound match with gap arms) was great
for ASCII but degraded sharply with cp value because the compiler
emitted a linear scan. The nested form fixes the high-cp regression.
Conformance preserved: 195 unit tests + all five conformance suites
at 100% (Grapheme/Word/Sentence/LineBreakTest + NormalizationTest
Part 1 + Part 2).
Extracts the bucketed-match pattern from `_UcdGraphemeBreak` into
`unicode_build/range_match_emitter.pony` and re-uses it across every
remaining cp-range → variant table:
_UcdCategory (UnicodeData.txt General_Category)
_UcdScript (Scripts.txt)
_UcdLineBreak (LineBreak.txt)
_UcdEastAsianWidth (EastAsianWidth.txt)
_UcdSentenceBreak (auxiliary/SentenceBreakProperty.txt)
_UcdWordBreak (auxiliary/WordBreakProperty.txt)
_UcdGraphemeBreak (auxiliary/GraphemeBreakProperty.txt + emoji-data)
Every emitter now ends in a call of the form
_RangeMatchEmitter.emit(out, ranges, to_variant, "<default variant>")
where `to_variant: {(U8): String val} val` is a lambda mapping the
emitter's property byte to the runtime variant name. The shared helper
handles bucket-splitting (so ranges that cross a 4096-cp boundary get
chopped at the boundary), gap arms (so each match arm is one
upper-bound comparison), and the two-level `match cp >> 12` /
`match cp` emission.
Real-world delta from the bench smoke (1.6KB corpora, NESTED vs OLD
encoding for _UcdGraphemeBreak — same pattern now applies to the
other six tables):
Graphemes.count/ascii 3759 µs → 1859 µs 2.0x
Graphemes.count/latin 3223 µs → 1520 µs 2.1x
Graphemes.count/cjk 1453 µs → 674 µs 2.2x
Graphemes.count/emoji 1099 µs → 538 µs 2.0x
Single-cp lookups (with constant-cp `_BenchU32`) show 30-100x
because LLVM constant-folds the new pure-code dispatch — that's
optimization headroom that's realized whenever a caller passes a
constant cp.
Conformance preserved: 195 unit tests + every UCD conformance suite
(NormalizationTest Part 1 + 2, GraphemeBreakTest, WordBreakTest,
SentenceBreakTest, LineBreakTest) at 100%.
The remaining holdout. _UcdBinaryProps stored one hex-encoded
(range_lo, range_hi) blob per binary property and dispatched via
`_UcdBinaryRange.has(table, cp)` (binary search). Replaces the
table-getter + shared binary-search pair with one code-generated
`fun _p_<property>(cp: U32): Bool` per property, each body emitted
through the shared `_RangeMatchEmitter` and tagged with U8(1) for
every range — `byte_to_variant` returns `"true"` for any byte and
the `default_variant` is `"false"`.
primitive _UcdBinaryProps
fun has(cp: U32, p: BinaryProperty): Bool =>
match p
| PropAlphabetic => _p_alphabetic(cp)
| PropExtendedPictographic => _p_extended_pictographic(cp)
...
end
fun _p_alphabetic(cp: U32): Bool =>
match cp >> 12
| 0x0000 =>
match cp
| let c: U32 if c <= 0x0040 => false
| let c: U32 if c <= 0x005A => true
...
`_UcdBinaryRange` and the per-property `_t_<property>` table getters
are gone — there's no shared runtime helper anymore; each property's
match is self-contained.
Single-cp `Codepoints.has_binary_property` smoke vs OLD encoding:
OLD NESTED speedup
ascii 643 → 15 ns 43x
latin-1 678 → 20 ns 34x
cjk 491 → 15 ns 33x
emoji 638 → 15 ns 43x
Segmentation cursors (which don't heavily exercise binary
properties) move only marginally — the bigger win was in the
previous commit. Anywhere `has_binary_property` IS in the hot path
(`Trim` with `White_Space`, identifier validators, etc.) gets the
30-40x.
Conformance preserved: 195 unit tests + every UCD conformance suite
(NormalizationTest Part 1 + 2, Grapheme/Word/Sentence/LineBreakTest)
at 100%. Source size: 27 KB → 26800 lines for the generated file —
the trade against the 30-40x runtime win at cost of ponyc compile
time, per the explicit priority you set.
`_UcdCombiningClass.of` was the last table on the hex-blob + binary-
search shape. The source data is `Array[(U32, U8)] val` — one
(codepoint, combining-class) pair per non-zero CCC entry (zero is the
default; absent codepoints fall through to it). Many neighbouring
codepoints share a class (the U+0300..U+0314 combining-above marks
all have class 230), so the entries are first run through a
`_coalesce_ccc` pass that merges adjacent same-class entries into
ranges. The coalesced ranges then feed into the shared
`_RangeMatchEmitter`; `byte_to_variant` simply maps each byte to its
decimal string and `default_variant` is `"0"`.
Single-cp `Codepoints.combining_class` smoke vs OLD encoding:
OLD NESTED speedup
ascii 263 → 12 ns 22x
latin-1 268 → 10 ns 27x
cjk 278 → 12 ns 23x
emoji 254 → 11 ns 23x
Real-world `Normalize` over 1.6 KB corpora — the cursor walks every
codepoint and calls `combining_class` plus the existing nested
property tables — finally moves on the lookup-heavy paths that the
binary props change couldn't unblock:
OLD NESTED speedup
nfd/ascii 1586 µs → 670 µs 2.4x
nfd/latin-pre 1777 µs → 780 µs 2.3x
nfc/ascii 3044 µs → 1657 µs 1.8x
nfc/latin-pre 3171 µs → 1855 µs 1.7x
nfc/latin-dec 2611 µs → 1442 µs 1.8x
nfc/combining 2840 µs → 713 µs 4.0x *
nfkd/latin-pre 2320 µs → 1389 µs 1.7x
nfkc/latin-pre 3960 µs → 2476 µs 1.6x
(*) The pathological `combining` corpus — every base letter
followed by five combining marks that need to be canonically
reordered then composed — went 4x. That's the headline case the
NFC fast-path was buying against.
Conformance preserved: 195 unit tests + NormalizationTest Part 1 + 2
(295,411 cases) + every UCD conformance suite at 100%.
The varlen-payload sibling of the other cp-range tables. Each input
entry is (range_lo, range_hi, script_byte_array); the previous
encoding split that into two hex blobs (range/length index + script
data) and ran a binary search. Replaces with a two-level
`match cp >> 12` / `match cp` whose arms each emit an
`Array[Script] val` literal directly:
match cp >> 12
| 0x0000 =>
match cp
| let c: U32 if c <= 0x00B6 => None
| let c: U32 if c <= 0x00B7 =>
recover val [as Script: ScriptAvestan; ScriptCarian; ...] end
| let c: U32 if c <= 0x02BB => None
...
else None
end
| ...
else None
end
A local `_split_at_buckets` mirrors `_RangeMatchEmitter`'s helper
for this payload type — the shared helper can't be used as-is
because the variant ISN'T a single byte-keyed string, it's a
per-range array-literal expression. The bucket-split + linear-
within-bucket pattern is the same.
Single-cp `Codepoints.script_extensions` smoke vs OLD encoding:
OLD NESTED speedup
ascii 1189 → 42 ns 28x
latin-1 1115 → 53 ns 21x
cjk 1102 → 48 ns 23x
emoji 1198 → 65 ns 18x
(Floor here is the per-call `Array[Script] val` allocation —
the dispatch itself is sub-10 ns; the rest is the literal recover.)
Downstream `Scripts.*` ops over 1.6 KB corpora — these walk every
codepoint and call `script_extensions` per cp:
OLD NESTED speedup
Scripts.is_single_script/ascii 1964 µs → 288 µs 6.8x
Scripts.is_single_script/mixed 12.0 µs → 1.1 µs 11.0x
Scripts.resolved_script_set/mixed 8.8 µs → 1.2 µs 7.6x
Scripts.restrict_to[Latin]/ascii 1658 µs → 202 µs 8.2x
Scripts.restrict_to[Latin]/mixed 7.8 µs → 1.2 µs 6.4x
Scripts.dominant/mixed 629 µs → 61 µs 10.3x
Trim.trim/1600B also lands its previously-projected
`has_binary_property`/`White_Space` win — 369 µs → 89 µs (4.1x).
Conformance preserved: 195 unit tests + every UCD conformance suite
at 100%.
…ables
Both decomposition tables used hex-blob + binary search (with the
record-decoded payload as the result). Convert each to a two-level
`match cp >> 12` / exact `match cp` dispatch, where every arm emits
the decomposition as an `Array[U32] val` literal:
primitive _UcdCanonicalDecomp
fun of(cp: U32): (Array[U32] val | None) =>
match cp >> 12
| 0x0000 =>
match cp
| 0x00C0 => recover val [as U32: 0x0041; 0x0300] end
| 0x00C1 => recover val [as U32: 0x0041; 0x0301] end
...
else None
end
| ...
else None
end
Same shape for `_UcdCompatDecomp` (varlen — up to ~18 cps per
decomposition). Both go through new `_emit_decomp_match` /
`_emit_varlen_decomp_match` helpers in `decomp_table.pony` —
they're decomp-specific rather than shared via `_RangeMatchEmitter`
because the inner match is exact-cp (no upper-bound guards, no gap
arms).
Single-cp `Codepoints.canonical_decomposition` smoke vs OLD:
OLD NESTED speedup
ascii 391 → 17 ns 23x
latin-1 466 → 63 ns 7x (Latin-1 'é' is in the table —
this includes the recover val
Array allocation)
cjk 363 → 9 ns 40x
emoji 337 → 8 ns 42x
The biggest payoff is `Normalize.nfd` / `Normalize.nfkd`, which
walk every cp and call `canonical_decomposition` / `compat_decomposition`
per cp. Over 1.6 KB corpora:
OLD NESTED speedup
Normalize.nfd/ascii 1585 µs → 84 µs 18.9x
Normalize.nfd/latin-pre 1777 µs → 123 µs 14.4x
Normalize.nfkd/latin-pre 2320 µs → 144 µs 16.1x
NFC/NFKC still allocate the composition lookup (`_UcdCanonicalCompose`)
on the hex-blob shape, so they see a smaller (but still real) win:
OLD NESTED speedup
Normalize.nfc/ascii 3044 µs → 1081 µs 2.8x
Normalize.nfc/latin-pre 3171 µs → 1147 µs 2.8x
Normalize.nfc/latin-dec 2611 µs → 916 µs 2.8x
Normalize.nfc/combining 2840 µs → 443 µs 6.4x
Normalize.nfkc/latin-pre 3960 µs → 1163 µs 3.4x
Doing `_UcdCanonicalCompose` next would close most of the remaining
NFC gap.
Conformance preserved: 195 unit tests + NormalizationTest Part 1+2
(295,411 cases) + every UCD conformance suite at 100%.
The compose lookup is keyed on two codepoints (lhs, rhs) — the
binary search wasn't just doing one set of decodes, it was comparing
lex-ordered (lhs, rhs) pairs per iteration. Replaces with a three-
level dispatch that mirrors the data's natural structure: every cp
has a small set of base letters (lhs) and each base letter combines
with only a handful of combining marks (rhs).
primitive _UcdCanonicalCompose
fun of(lhs: U32, rhs: U32): (U32 | None) =>
match lhs >> 12 ← outer 4096-cp bucket
| 0x0000 =>
match lhs ← exact lhs in this bucket
| 0x0041 =>
match rhs ← exact rhs combiner
| 0x0300 => 0x00C0
| 0x0301 => 0x00C1
...
else None
end
| 0x0043 =>
match rhs
| 0x0301 => 0x0106
...
else None
end
...
else None
end
| ...
else None
end
All three levels are exact-integer matches (no guards), which LLVM
lowers to switches. The data is sorted by (lhs, rhs) so the
emitter streams once through and tracks bucket/lhs transitions.
Single-cp smoke vs OLD:
OLD NESTED speedup
compose_canonical[hit] 587 → 20 ns 29x
compose_canonical[miss] 538 → 4 ns 135x *
(*) The miss case constant-folds completely under `_BenchU32`
because both inputs are constant; real-world callers (Normalize.nfc)
see the dispatch cost.
But the real prize is `Normalize.nfc` / `Normalize.nfkc` — they
walk every cp and call `compose_canonical` for primary-composite
recombination. Previously NFC was held back by the still-hex-blob
compose table while NFD was already 14-19x faster. The compose
table closes that gap:
OLD NESTED speedup
Normalize.nfc/ascii 3044 µs → 129 µs 23.6x
Normalize.nfc/latin-pre 3171 µs → 158 µs 20.1x
Normalize.nfc/latin-dec 2611 µs → 126 µs 20.7x
Normalize.nfc/combining 2840 µs → 119 µs 23.9x
Normalize.nfkc/latin-pre 3960 µs → 216 µs 18.3x
Every NFC variant is now in the same 18-24x league as NFD. The full
Normalize pipeline (decompose → reorder → compose) now compares to
its raw byte-walk cost rather than to its table-lookup cost.
Conformance preserved: 195 unit tests + NormalizationTest Part 1+2
(295,411 cases) + every UCD conformance suite at 100%.
Convert _UcdSimpleUpper/Lower/Title/CaseFold (cp → cp) and _UcdFullUpper/Lower/Title/CaseFold (cp → Array[U32] val | None) from hex-blob binary search to the two-level `match cp >> 12` pattern used by the other UCD tables. The simple tables default the outer/inner `else` arms to `cp` (identity-preserving). The full tables emit each mapping as a `recover val [as U32: …] end` literal and default to None. All 5 conformance suites still at 100% (195 unit + 295,411 NormalizationTest cases + GraphemeBreakTest 1,093 + WordBreakTest 1,826 + SentenceBreakTest 512 + LineBreakTest 16,672). Case.* corpora benchmarks vs old hex-blob baseline (1.6 KB inputs): Case.upper/ascii: 886 µs → 65 µs (13.7×) Case.upper/latin: 767 µs → 57 µs (13.4×) Case.lower/latin: 691 µs → 63 µs (11.1×) Case.title/latin: 1,031 µs → 131 µs (7.9×) Case.fold/latin: 937 µs → 84 µs (11.2×) Case.fold/mixed: 792 µs → 64 µs (12.3×)
Convert _UcdName.of(cp) from hex-blob binary search + per-call String allocation to the standard two-level `match cp >> 12` pattern. Each arm returns a String val literal directly — no allocation. The data blob is gone; a slim cp-only hex index (`_cps()`) survives only for from_name's O(n) reverse scan, which now dispatches each cp through the match-based `of()` instead of building a fresh String per probe. Source file goes from 2.7 MB / 80 lines (hex blobs + binary search) to 2.2 MB / 40,146 lines (one match arm per assigned codepoint). Worth flagging: ponyc takes ~15 minutes to compile the bench binary with this in place — the CJK Unified bucket alone is ~21k arms in one inner match. The conformance build is similar. We've been prioritizing runtime over compile-time throughout this pass; this is the table where that tradeoff is most pronounced. All 5 conformance suites still at 100% (195 unit + 295,411 NormalizationTest cases + 1,093 Grapheme + 1,826 Word + 512 Sentence + 16,672 Line). Codepoints.name + from_name vs hex-blob baseline: name/ascii: 770 ns → 8 ns (96×) name/latin1: 835 ns → 9 ns (93×) name/cjk: 477 ns → 5 ns (95×) name/emoji: 709 ns → 20 ns (35×) from_name: 23,066 ns → 1,079 ns (21×)
Convert all 20 generated cp-indexed UCD lookup tables from two-level
`match cp >> 12` dispatch (plus a couple of binary searches) to
two-stage tries backed by static `const` C blobs, compiled by ponyc's
bundled C compiler and reached via FFI.
Why: match-dispatch cost scaled with linear-scan depth inside each
4096-cp bucket (up to ~295 comparisons / ~367 ns for dense BMP scripts).
A trie is two array loads, constant-time. C blobs sidestep the two
embedding traps hit earlier: Pony `Array` literals send ponyc's
typechecker into a multi-hour spin, and `\xNN` String literals corrupt
bytes >= 0x80 (Pony reads `\xNN` as a Unicode codepoint escape). The
blobs live in mmap'd `.rodata`.
New codegen lives in unicode_build/trie_emitter.pony:
- byte/enum tables (emit) -> uint8 stage2
- binary properties (emit_u64) -> per-cp U64 bitmask; `has` is one
lookup + bit test, replacing 64 per-property match tables
- simple case maps (emit_u32) -> uint32 value, 0 = identity
- varlen tables (emit_varlen) -> cp->offset trie over a dedup'd
payload blob (canonical/compat decomp, full case, script_extensions)
- name (emit_name) -> byte payload + named-cp list so
from_name keeps its reverse scan
- canonical_compose (emit_compose) -> lhs-trie + per-lhs (rhs,result)
scan (O(1) reject for non-starters), replacing the 3-level match
Per-table block size is chosen to minimise total bytes; stage1 promotes
from uint8 to uint16 past 256 unique blocks.
Throughput (1600B corpora, varying cps): grapheme segmentation ~15x
faster; word/sentence/line/casefold 1.1-1.4x; NFC/NFKC 1.0-1.15x;
Scripts.dominant/restrict_to 1.3-5x. Known residuals: Scripts.of ~9%
slower (per-cp FFI floor on a near-zero-work op) and name lookup slower
(per-byte String rebuild), traded for the name table shrinking from
2.2 MB to 1.1 KB of generated Pony.
Removes the now-unused _RangeMatchEmitter and _UcdHex. All 195 unit
tests and every UCD conformance suite (NormalizationTest Part 1 + 2,
Grapheme/Word/Sentence/LineBreakTest) pass at 100%.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.