encode_ordinary_batch — reproducible multi-second tail stalls on 32-core box (o200k_base, num_threads=8)

[daechoi]

Running encode_ordinary_batch with its default num_threads=8 on a 32-core Sapphire Rapids box, I'm seeing large tail spikes on the slowest run of every 10-run batch. Same input, same build, the worst run is 1.1× to 7.6× slower than the median depending on corpus. Multilingual text is the worst offender: two back-to-back runs produced worst-of-10 times of 9.3s and 14.6s, against medians of 5.0s and 4.8s.

I read the performance comment in src/lib.rs around line 232 that documents the fancy_regex/regex scratch-buffer contention and the thread-local clone workaround. That's clearly the right fix for the contention it describes — but the tail spikes I'm seeing are on top of that workaround, so something else is going on.

Repro

import time, tiktoken

enc = tiktoken.get_encoding("o200k_base")
BATCH_SIZE, RUNS = 256, 10

with open("corpora/multilingual.txt") as f:
    text = f.read()
docs = [text] * BATCH_SIZE

enc.encode_ordinary_batch([text])  # warm up

times = []
for _ in range(RUNS):
    t0 = time.perf_counter()
    enc.encode_ordinary_batch(docs)
    times.append(time.perf_counter() - t0)

times.sort()
print(f"median: {times[RUNS // 2] * 1000:.0f} ms")
print(f"worst : {times[-1] * 1000:.0f} ms  ({times[-1] / times[RUNS // 2]:.1f}x median)")

Full harness with five corpora, parity-checked against riptoken: https://github.com/daechoi/riptoken/blob/main/scripts/bench.py

Numbers

Two consecutive full-harness runs. All times in ms. docs is [text] * 256 per corpus, num_threads=8 default.

Run 1:

corpus	tokens/batch	median	worst-of-10	worst/median
english prose	10,240,256	953	7018	7.4×
python source	18,560,256	3015	7044	2.3×
rust source	22,528,256	3596	10015	2.8×
multilingual + emoji	21,913,600	5027	9287	1.8×
random-ish bytes	30,720,000	2856	3247	1.1×

Run 2 (immediate re-run, same machine, same build):

corpus	median	worst-of-10	worst/median
english prose	1227	9285	7.6×
python source	3059	8663	2.8×
rust source	5231	8775	1.7×
multilingual + emoji	4819	14567	3.0×
random-ish bytes	2856	3266	1.1×

Multilingual worst-of-10 moved from 9.3s to 14.6s between runs. The variance doesn't track corpus size — random-ish bytes has the most tokens but the tightest ratio, and english prose (smallest by token count) has the worst ratio. So whatever's going on, it's not a straight "more work, more variance" story.

Environment

• 32-core Intel Sapphire Rapids, 164 GB RAM, Ubuntu 24.04, Linux 6.8.0-1030-ibm
• Python 3.12.3
• tiktoken 0.12.0
• o200k_base

Where I am

I don't have a root cause yet. The regex_tls array at src/lib.rs:325 is 128 pre-cloned regex instances indexed by hash_current_thread() % 128; with 8 threads, slot collisions should be rare, so the residual variance is presumably not the same scratch-buffer contention the comment already addresses. Candidates I'd want to rule out: GIL scheduling under ThreadPoolExecutor, allocator pressure on the batch output path, something in the merge path. Would need to profile a worst run with py-spy / perf to say.

If this is a known issue you've already looked at, feel free to close. If it's useful I'm happy to profile and come back with a more specific report, or PR the benchmark harness if you want it in tree.

[alobroke]

I've put up a benchmark harness based on the methodology you described in this issue — PR #548.
It measures median and worst-of-N wall-clock time across four synthetic corpora (english prose, python source, multilingual+emoji, random ascii) with configurable runs, batch size, encoding, and thread counts. No external data files required — corpora are generated at runtime.
Happy to extend it if you have specific corpora or thread counts you'd like covered, or if you want to add it to CI as a regression guard once a root cause is found.