Parse speed round 4: 1.56× default / 1.74× single-thread, output-identical

[JSv4]

Description of the change

Makes the whole PDF parse 1.5×+ faster with byte-identical output, via systematic architectural changes to both halves of the pipeline. Measured on the standard timing harness (dense set, no-OCR, 3-run warm median, 4-CPU box, Python 3.11). The default-workers before/after was measured as three interleaved baseline/current pairs (baseline main @ bd76cc6 in a git worktree, alternating with this branch in one session) so box noise can't flatter either side — pairwise ratios 1.61× / 1.54× / 1.58×:

config	front-end	engine	total	pages/s	vs baseline
main @ bd76cc6, default workers (median of 3)	6701 ms	5938 ms	12639 ms	22.7	1.00×
this PR, default workers (median of 3)	3522 ms	4423 ms	7945 ms	35.5	1.56× (worst pair 1.54×)
main @ bd76cc6, workers=1	19745 ms	6099 ms	25844 ms	11.1	1.00×
this PR, workers=1	10501 ms	4379 ms	14880 ms	19.3	1.74×

Per-half medians (default workers): front-end 1.90×, engine 1.34×.

The levers (all output-identical)

1. Fast content-stream tokenizer (file_parser/fast_content_parser.py) — pdfminer lexes content streams through a per-chunk state machine that was ~half of front-end time. FastContentParser produces the same (pos, token) stream in one pass over the concatenated stream bytes and feeds pdfminer's unmodified stack machine (nextobject, do_keyword, BI/ID/EI inline images). Deliberately replicated quirks: the \CR LF-at-chunk-edge string-escape artifact (original per-stream 4096-byte chunk geometry reconstructed), the lazy octal-overflow AssertionError (poisoned token tail, so inline-image bytes the original never lexes can't crash the eager scan), one-shot EOF flush drops, silent number/keyword drops, and the stream-boundary whitespace flush (pdfminer #1157). FastPageInterpreter.execute adds an interned-keyword operator-dispatch cache.
2. Char-dict device (_CharDictDevice) — a PDFPageAggregator whose render_string_horizontal emits warp's char dicts directly, with LTChar.__init__'s exact bbox math inlined (same float ops and association through apply_matrix_rect's corner expressions) and per-text-run constants hoisted. No per-glyph LTChar, no layout-tree aggregation, no leaf walk. Rule lines/rects still come from the inherited, unmodified paint_path. Fallback chain preserved and tested under forced failure: device → layout-tree fast path → pdfplumber → OCR routing (all three produce identical XHTML).
3. GC parse guard (ingestor_utils/gc_guard.py) — a parse allocates millions of containers, so CPython's full (gen-2) collections fire repeatedly inside every parse and scan the whole heap: measured 15–30% of engine wall time on a warm process. The reentrant, thread-safe guard raises the gen-2 threshold for the parse and, at outermost exit, runs one full collection only if the pending gen-2 counter crossed the stock threshold — i.e., it purely reschedules the same collections to parse boundaries (a small parse that wouldn't have triggered one doesn't pay for one). Young-generation collection continues throughout, so memory stays bounded. WARP_GC_GUARD=0 disables.
4. sent_tokenize prefilter v3 — the ~350 abbreviation-protection rules ran ~27k surviving regex subs per S-1 exhibit though only ~3% fire. One pass per text now builds chunk-candidate sets — for purely-alphanumeric abbs, \s{abb}.\s can only match a whitespace-delimited chunk equal to abb or abb+c (\s ≡ str.isspace verified over all of Unicode); dotted abbs (u.s) are verified by literal-run offsets at chunk-start anchors, guarded on casefold length-alignment. Sub scans drop ~30×. Soundness is unit-tested per rule shape, plus a differential against a replica of the old loop over 1.3k corpus texts.
5. Small-document parallel gate — WARP_FE_PARALLEL_MIN_PAGES default 8 → 3: the persistent spawn pool makes per-document startup cheap, so 3–6 page docs gain ~25% front-end wall time from striping (parallel-path byte-identity re-verified on the small contracts); 1–2 page docs stay serial.
6. Engine micro-restructures — get_class exact-hit dict shortcut (provably identical: keys append-only, insertion-ordered scan); parse_line hot loop with loop-local counters, one casefold per token, frozenset punctuation tables, a tracked has-page-word flag (kills a 38k-call genexpr), and the noun-chunk-ending join skipped when the ending-token set is empty; digit-count short-circuit in indent_parser; one BS4 .text walk per <p> per phase.

How output-identity was verified

• 36-doc corpus hash-identity: SHA-256 of the XHTML, the all render's html+json file_data, and block texts — dense benchmark set + FortWorth contracts + 20 hetero-100 enterprise pages — identical before/after every change.
• Token-stream differential: FastContentParser vs pdfminer's PDFContentParser on every content stream of every fixture PDF — 5,338 pages / 42M tokens, values and positions, zero mismatches — plus ~12k adversarial fuzz cases over the tokenizer's edge branches (nested/escaped strings, chunk-edge \CR LF, octal overflows, hexstrings, literal hex escapes, EOF states).
• Full pytest suite: default suite 638 passed / 99 skipped / 0 failed; --runslow 735 passed / 1 failed — the one failure was test_min_pages_env_parsing asserting the old parallel-gate default of 8, updated for the intentional 8→3 change (that test file re-run green after). All committed baselines (S-1 cross-engine, OC export/batch, docsling, hetero-100, legal-100, semantic units) are untouched and pass unchanged — as expected, since the parse output is byte-identical.
• Fallback-chain check: forced failures of the device and layout-tree paths produce byte-identical XHTML through the pdfplumber path.

pdfminer upgrade guards (committed as tests/test_fast_content_parser.py)

The fast paths replicate three small pdfminer surfaces (tokenizer semantics, LTChar bbox math, the execute() body), which could silently fossilize when a pdfminer bump changes the originals. Two always-on test layers (~3s) turn that into a red X on the upgrade PR instead:

• Equivalence tests with pdfminer's live code as the oracle: token-stream equality (values + positions) vs PDFContentParser, char/line/rect page-object equality vs the layout-tree walk, over three diverse fixtures, plus a 500-case deterministic tokenizer fuzz including the 4096-chunk-edge \CR LF cases.
• Vendored-source pins: SHA-256 of inspect.getsource() for every replicated upstream surface (execute, LTChar.__init__, render_string*, render_char, the _parse_* family, the matrix helpers) plus direct assertions on the tokenizer's regex classes and escape table — failing with a re-sync procedure whenever upstream edits code we vendored, even if behavior looks unchanged on our fixtures.

Notes

• benchmarks/PERF_OPTIMIZATION.md documents the round with methodology and the not-pursued options (native word extraction, lxml, deepcopy specialization).
• CLAUDE.md's perf-knobs section is updated (new modules + WARP_GC_GUARD).
• No dependency changes; Python 3.10–3.14 compatible (measured here on 3.11; the shipped 3.14 runtime multiplies on top, as in round 3).

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Related issues

None — standalone performance round (follows the rounds documented in benchmarks/PERF_OPTIMIZATION.md).

Checklists

Development

• Lint rules pass locally (make check)
• The code changed/added as part of this pull request has been covered with tests (pdfminer upgrade guards in tests/test_fast_content_parser.py; prefilter soundness suites extended in tests/test_sent_tokenizer.py; min-pages default assertion updated; correctness of the new front-end paths is additionally pinned by the existing end-to-end suites since output is byte-identical)
• All tests related to the changed code pass in development (default suite 638 passed / 99 skipped / 0 failed before the guard module; guard module 30 passed; full --runslow 735 passed with the single gate-default assertion failure, and that test file re-run green after the assertion update)

Code review

• This pull request has a descriptive title and information useful to a reviewer
• "Ready for review" label attached to the PR and reviewers mentioned in a comment
• Changes have been reviewed by at least one other engineer
• Issue from task tracker has a link to this pull request