Skip to content

perf: reuse a thread-local scratch buffer for new_from_utf8 transcode - #2027

Open
bartlomieju wants to merge 5 commits into
mainfrom
perf/reuse-encode-scratch
Open

perf: reuse a thread-local scratch buffer for new_from_utf8 transcode#2027
bartlomieju wants to merge 5 commits into
mainfrom
perf/reuse-encode-scratch

Conversation

@bartlomieju

@bartlomieju bartlomieju commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

#2026's simdutf transcode path in new_from_utf8 allocated a fresh Vec on
every non-ASCII string creation. This reuses a thread-local scratch buffer
instead — taken out of the thread-local and put back around the V8 call, so its
borrow isn't held across a potential GC. The buffer is still copied into V8 by
NewFromOneByte/NewFromTwoByte, so only the per-call allocation is removed.

Impact

Incremental over #2026, new_from_utf8 (median of 6):

kind 16 64 256 4096
latin1 −13% −13% −15% −3%
twobyte −24% −24% −18% −3%

The allocation was the dominant fixed cost at small-medium sizes; it fades
against the copy/transcode for large strings.

Note on the alternative

This is the low-risk half of proposal "3B". A NewRawOneByteString /
NewRawTwoByteString binding could additionally eliminate the copy by
transcoding straight into V8's heap string, but that only helps large strings
by a few percent and requires unsafe V8-internal Factory access + writing
under a DisallowGarbageCollection scope, so it's left as a possible follow-up.

Correctness

All 16 test_api string tests pass (with and without simdutf).

@bartlomieju
bartlomieju force-pushed the perf/reuse-encode-scratch branch from c79b84f to 5faa602 Compare July 17, 2026 20:21
@bartlomieju bartlomieju changed the title perf(string): reuse a thread-local scratch buffer for new_from_utf8 transcode perf: reuse a thread-local scratch buffer for new_from_utf8 transcode Jul 18, 2026
V8's NewFromUtf8 runs a scalar UTF-8 decoder (twice: once for width/length,
once to write), which is extremely slow for non-ASCII input. When simdutf is
available, decode the input ourselves and hand V8 a pre-decoded buffer it can
memcpy:

- Pure ASCII -> NewFromOneByte directly (bytes are already valid Latin-1).
- Non-ASCII, >= 16 bytes -> try convert_utf8_to_latin1_with_errors (one-byte)
  then convert_utf8_to_utf16le_with_errors (two-byte); on invalid UTF-8, fall
  back to V8's lossy NewFromUtf8.
- Tiny non-ASCII (< 16 bytes) stays on V8's decoder (the two simdutf FFI calls
  cost more there).
- Inputs whose byte length exceeds MAX_LENGTH are left to V8, which rejects
  them (preserving the conservative None behavior; our shorter decoded length
  would otherwise accept some).

new_from_utf8 (median of 5):
  ascii:    -25% (4) .. -10% (256), ~0 at 4096
  latin1:   -44% (16), -75% (64), -87% (256), -93% (4096)
  twobyte:  -33% (16), -74% (64), -88% (256), -93% (4096)

Non-ASCII string creation (accented text, CJK, emoji, unicode JSON) is up to
~14x faster. ASCII and short non-ASCII are unaffected/faster. Adds a test for
the transcode paths + invalid-UTF-8 fallback and benches/string_encode.rs.

Supersedes #2020 (which was based on a stale main and did only the ASCII half
with std is_ascii).
Match benches/function.rs: a harness=false bench must produce no output when
nextest enumerates test binaries.
…ranscode

The simdutf transcode path in new_from_utf8 allocated a fresh Vec on every
non-ASCII string creation. Reuse a thread-local scratch buffer instead (taken
out and put back around the V8 call so its borrow isn't held across a possible
GC). The buffer is copied into V8 by NewFromOneByte/NewFromTwoByte as before, so
only the per-call allocation is removed.

Incremental over #2026 for non-ASCII creation:
  latin1:  -13% (16), -13% (64), -15% (256), -3% (4096)
  twobyte: -24% (16), -24% (64), -18% (256), -3% (4096)

The allocation was the dominant fixed cost at small-medium sizes. (This is the
low-risk alternative to a NewRawOneByte/TwoByte binding that would also
eliminate the copy: that only helps large strings by a few percent and needs
unsafe V8-internal Factory access + GC-safety care, so it is left as a possible
follow-up.)

Stacked on #2026.
@bartlomieju
bartlomieju force-pushed the perf/new-from-utf8-simd branch from be7b918 to cac1895 Compare July 18, 2026 15:45
@bartlomieju
bartlomieju force-pushed the perf/reuse-encode-scratch branch from 5faa602 to fb5a4a8 Compare July 18, 2026 15:46
Base automatically changed from perf/new-from-utf8-simd to main July 18, 2026 17:16
@bartlomieju

Copy link
Copy Markdown
Member Author

This needs to be rewritten to have a limit on size - otherwise allocating an 100Mb string will permamently leave 100Mb laying around

Address review: the thread-local scratch grew to the input length and was put
back uncapped, so a one-off huge string (e.g. 100 MB) left that allocation live
in the thread-local forever. Drop capacity back to ENCODE_SCRATCH_MAX_CAP (64Ki
elements = 64 KiB Latin-1 / 128 KiB UTF-16) on put-back via put_encode_scratch;
a larger string still transcodes, it just reallocates that one time. Extends the
transcode test with >cap Latin-1/UTF-16 strings plus a small string afterwards
to cover the shrink-then-reuse path.
@bartlomieju

Copy link
Copy Markdown
Member Author

Addressed in 19eac93. The scratch buffer's retained capacity is now capped: on put-back, put_encode_scratch drops any capacity above ENCODE_SCRATCH_MAX_CAP (64Ki elements — 64 KiB for Latin-1, 128 KiB for UTF-16) with clear() + shrink_to(), so a one-off 100 MB string no longer leaves 100 MB parked in the thread-local. A larger string still transcodes fine; it just reallocates that one time rather than retaining the buffer.

The cap sits comfortably past the sizes where reuse actually helps — per the PR's own numbers the allocation saving has faded to ~−3% by 4 KiB and is dominated by the copy/transcode well before 64 KiB.

Extended new_from_utf8_simd_transcode with >cap Latin-1 (é×100k) and UTF-16 (×100k) strings followed by a small string, to cover the shrink-then-reuse path. Passes with and without simdutf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant