Parse speed round 4: 1.56× default / 1.74× single-thread, output-identical

CLAUDE.md

@@ -115,7 +115,7 @@ One `<p>` per visual-line segment; `text.split()` length must equal the number o
 - **Formatting is enforced**: black + isort (`profile=black`, line length 88). Run `make format` before committing or CI lint fails.
 - **Preserve the XHTML contract in the *front-end*** (extraction/OCR): the pure-Python parser must emit the exact word-box XHTML the legacy Tika produced so `visual_ingestor` sees byte-identical input. That invariant is about the front-end↔engine boundary — it is **not** a blanket ban on editing the engine. **The layout engine is fair game, and a *systematic* engine flaw — a defect that recurs across a structural class rather than a one-off — should be fixed *in the engine*, at its root, not papered over in a downstream consumer.** The bar for an engine change is **no systematic degradation**: validate against the S-1 cross-engine regression (the engine's own json/block output — improvements pass, regressions fail) and the OC baselines, and **re-floor/regenerate a baseline when a correct fix legitimately shifts it** (that is the expected, sanctioned response, not something to avoid). Reach for a downstream/exporter-scoped correction (see the OpenContracts section) only when the fix genuinely belongs to that consumer, or when an engine change's blast radius isn't yet worth paying — and when you do, say so explicitly rather than defaulting to it because the engine "feels" off-limits.
 - **OCR is optional**: `ocr_parser.ocr_available()` gates it; if `rapidocr-onnxruntime` isn't installed, scanned pages degrade to little/no text instead of crashing. `WARP_DISABLE_OCR=1` hard-disables OCR (env var, so it also reaches front-end worker processes). Default OCR render is 200 DPI (`ocr_parser.OCR_DPI`). rapidocr sometimes glues words on dense scans — a known limitation.
-- **Parallelism & perf knobs** (all output-identical; see `benchmarks/PERF_OPTIMIZATION.md`): the front-end stripes pages of docs ≥8 pages across a persistent spawn process pool (`WARP_FE_WORKERS`, ≤1 = serial; `WARP_FE_PARALLEL_MIN_PAGES`); the engine shares pure `Line`/`Word` analyses via bounded LRU caches (`line_parser.bare_line`/`make_word`; `WARP_LINE_CACHE=0` bypasses). Engine callsites must use `line_parser.bare_line(text)` for bare single-string construction. Note monkeypatches don't reach spawned workers — use the env vars in benchmarks/tests (the timing harness does). The default runtime is **Python 3.14** (`python:3.14-bookworm` images; opt in locally with an untracked `.python-version`; 1.5× single-thread vs 3.12, hash-identical output); supported range stays 3.10–3.14 and the experimental JIT stays off.
+- **Parallelism & perf knobs** (all output-identical; see `benchmarks/PERF_OPTIMIZATION.md`): the front-end stripes pages of docs ≥8 pages across a persistent spawn process pool (`WARP_FE_WORKERS`, ≤1 = serial; `WARP_FE_PARALLEL_MIN_PAGES`); the engine shares pure `Line`/`Word` analyses via bounded LRU caches (`line_parser.bare_line`/`make_word`; `WARP_LINE_CACHE=0` bypasses). Engine callsites must use `line_parser.bare_line(text)` for bare single-string construction. Note monkeypatches don't reach spawned workers — use the env vars in benchmarks/tests (the timing harness does). The default runtime is **Python 3.14** (`python:3.14-bookworm` images; opt in locally with an untracked `.python-version`; 1.5× single-thread vs 3.12, hash-identical output); supported range stays 3.10–3.14 and the experimental JIT stays off. Round-4 additions (2026-07-25, also output-identical): the front-end lexes content streams via `file_parser/fast_content_parser.py` (a one-pass tokenizer feeding pdfminer's unmodified stack machine — token-differential + fuzz tested against `PSBaseParser`; falls back to the layout-tree path, then pdfplumber, on any error) and builds char dicts in `_CharDictDevice.render_string_horizontal` (LTChar's exact bbox math inlined, per-run constants hoisted); whole-document parses run under `ingestor_utils/gc_guard.parse_gc_guard` (defers gen-2 GC to parse boundaries, same collection cadence; `WARP_GC_GUARD=0` disables); `sent_tokenize`'s abbreviation rules are pruned by per-text chunk-candidate sets (prefilter v3 — necessary-condition soundness unit-tested per rule shape).
 - **Tuned magic constants** live at the top of `visual_ingestor.py` (e.g. `header_margin`, `line_height_threshold`, `table_col_threshold`) — several marked "don't touch this!". Changing them shifts parsing behavior globally.
 - **Debug tracing**: `visual_ingestor.py`/`table_parser.py` have module-level `*_DEBUG` flags and a noisy `PROGRESS_DEBUG=True` (prints per-page progress) — flip rather than add prints.
 - **`num_pages` is `len(pages) - 1`** in `parse_blocks` (an off-by-one inherited from the original; preserved for compatibility). Don't assert on it for single-page docs.

benchmarks/PERF_OPTIMIZATION.md

@@ -1,8 +1,104 @@
-# Warp parse-throughput optimization (2026-06-28; rounds 2–3 2026-07-02)
+# Warp parse-throughput optimization (2026-06-28; rounds 2–3 2026-07-02; round 4 2026-07-25)
 
 Goal: make warp's parse speed competitive without regressing quality or tests.
 Methodology and levers, with measured effect and regression risk.
 
+## Round 4 (2026-07-25): content tokenizer, char-dict device, GC scheduling, engine prefilters — 1.56× default / 1.74× single-thread
+
+All levers are **output-identical** and verified three ways: byte-identity of
+XHTML + `all`/`json`/`html` renders + block texts over a 36-doc corpus (the
+dense benchmark set, the FortWorth contracts, and 20 hetero-100 enterprise
+pages); a token-stream differential of the new content tokenizer against
+pdfminer's on every content stream of every fixture PDF (5.3k pages, 42M
+tokens, positions included) plus ~12k adversarial fuzz cases; and the full
+pytest suite.
+
+Headline (dense set, no-OCR, 3-run warm median per harness invocation,
+4-CPU box, Python 3.11).  The default-workers before/after was measured as
+**three interleaved baseline/current pairs** (baseline = `main` @ bd76cc6 in
+a git worktree, alternating with this tree in one session) so box noise
+cannot flatter either side; pairwise ratios were 1.61× / 1.54× / 1.58×:
+
+| config | front-end | engine | total | **pages/s** |
+|---|---|---|---|---|
+| round-3 tip, default workers (median of 3) | 6701 ms | 5938 ms | 12639 ms | **22.7** |
+| + round 4, default workers (median of 3) | 3522 ms | 4423 ms | 7945 ms | **35.5** (**1.56×**; worst pair 1.54×) |
+| round-3 tip, workers=1 | 19745 ms | 6099 ms | 25844 ms | **11.1** |
+| + round 4, workers=1 | 10501 ms | 4379 ms | 14880 ms | **19.3** (**1.74×**) |
+
+Per-half medians (default workers): front-end **1.90×**, engine **1.34×**.
+
+1. **Fast content-stream tokenizer** (`file_parser/fast_content_parser.py`,
+   ~55% of front-end time): pdfminer tokenizes content streams through a
+   per-chunk state machine (`PSBaseParser._parse_main` + one method dispatch
+   per token, 4096-byte refills).  `FastContentParser` produces the same
+   `(pos, token)` stream in one pass over the concatenated stream bytes and
+   feeds pdfminer's **unmodified** `PSStackParser`/`PDFContentParser` stack
+   machine — `nextobject()`, `do_keyword`, and the BI/ID/EI inline-image
+   protocol (which re-tokenizes from the post-image offset) behave exactly as
+   before.  Deliberately replicated quirks: the `\CR LF`-at-chunk-edge string
+   escape artifact (original chunk geometry is reconstructed per stream), the
+   lazy `AssertionError` on `\4xx`+ octal escapes (poisoned token tail — an
+   eager raise would crash on inline-image bytes the original never lexes),
+   EOF one-shot-flush token drops, silent number/keyword drops, and the
+   stream-boundary "tack on whitespace" flush (pdfminer #1157) via a joiner
+   byte.  `FastPageInterpreter.execute` also caches operator-method dispatch
+   (keywords are interned singletons).
+2. **Char-dict device** (`_CharDictDevice` in `pdf_plumber_parser`): a
+   `PDFPageAggregator` whose `render_string_horizontal` emits warp's char
+   dicts directly — `LTChar.__init__`'s exact bbox math (same float ops,
+   same association, through to the corner expressions of
+   `apply_matrix_rect`) inlined with per-text-run constants hoisted (descent
+   line, matrix a/b/c/d products, upright flag, fontname).  No per-glyph
+   `LTChar` objects, no layout-tree aggregation, no leaf walk; rule
+   lines/rects still come from the inherited, unmodified `paint_path`.
+   Fallback chain preserved: device → layout-tree fast path → pdfplumber →
+   OCR routing.
+3. **GC parse guard** (`ingestor_utils/gc_guard.py`, both pipeline halves +
+   FE pool workers): a document parse allocates millions of containers, so
+   CPython's gen-2 collections fire repeatedly *inside* every parse and scan
+   the entire heap — ~15–30% of engine wall time on a warm process (measured
+   787→551 ms on a grown heap).  The reentrant, thread-safe guard raises the
+   gen-2 threshold for the duration of a parse (young gens keep collecting,
+   so memory stays bounded) and, on outermost exit, runs one full collection
+   *only if* the pending gen-2 counter crossed the threshold the stock
+   scheduler would have fired at — i.e. it purely reschedules the same
+   collections to parse boundaries.  `WARP_GC_GUARD=0` disables.
+4. **`sent_tokenize` prefilter v3** (`ingestor_utils/utils.py`): the ~350
+   abbreviation-protection rules ran ~27k surviving regex subs per S-1
+   exhibit though only ~3% fire.  Per text, one pass builds (a) the
+   word_exact candidate set — for a purely-alphanumeric abb, `\s{abb}.\s`
+   can only match a whitespace-delimited chunk equal to `abb` or `abb+c`
+   (`\s` ≡ `str.isspace` over all of Unicode, verified; non-ASCII chunks
+   contribute bounded prefixes since casefold can shift lengths) — and (b)
+   the dotted-rule set — `u.s`-style abbs have literal runs at fixed offsets
+   from a chunk-start anchor (anchors located by a compiled `(?<!\S)[…]`
+   class; positions trusted only when casefold preserved text length).
+   Candidate sets persist across rule fires (a fire rewrites its match
+   same-length/ASCII, so stale sets are only over-inclusive).  Sub scans
+   drop ~30× (27k → <1k per exhibit); soundness is unit-tested per rule
+   shape and the full pass is differential-tested against a replica of the
+   old loop on 1.3k corpus texts.
+5. **Engine micro-restructures**: `get_class` exact-hit dict shortcut
+   (provably returns what the fuzzy scan would — keys are append-only and
+   insertion-ordered); `parse_line` hot loop rewritten with loop-local
+   counters, one casefold per token, frozenset punctuation tables, a
+   tracked has-page-word flag (kills a 38k-call genexpr in `check_header`),
+   and the noun-chunk-ending join skipped when the ending-token set is
+   empty (always, for engine bare lines); `indent_parser` digit count
+   short-circuits at its threshold; one BS4 `.text` walk per `<p>` per
+   phase (`should_ignore_line` now takes the hoisted text).
+6. **Small-document parallel gate**: `WARP_FE_PARALLEL_MIN_PAGES` default
+   8 → 3.  The persistent spawn pool makes per-document startup cheap, so
+   3–6 page documents gain ~25% front-end wall time from striping
+   (byte-identity of the parallel path re-verified on the small
+   contracts); 1–2 page documents stay serial.
+
+Not pursued in round 4: native word extraction (replacing pdfplumber's
+`WordExtractor`; ~10-15% of FE, high divergence risk in `cluster_objects`
+mean-clustering), lxml/BS4 replacement (parser-dependent output risk),
+`table_parser` deepcopy specialization (~4% ceiling on table-heavy docs).
+
 ## Round 3 (2026-07-02): single-thread — engine micro-opts + Python 3.14 runtime
 
 Round 2's parallel front-end doesn't help single-page documents (ParseBench's