Close top ParseBench content-faithfulness drivers at the root (multi-column order, prose-as-table grids, unmapped-font text layers)

[JSv4]

Description of the change

Diagnoses and root-cause-fixes the top three drivers of Warp's ParseBench Content Faithfulness losses. Final measured result (official evaluator, all 506 text_content pages): 68.33 → 70.81 (+2.48) — correctness 75.97 → 77.24, reading-order 52.34 → 57.30; 112 pages improved / 27 regressed. Engine/front-end fixes only, no benchmark-boundary post-processing.

Diagnosis tooling. New scripts/parsebench_content_fasteval.py (mirrors the table/format fastevals; official ParseEvaluator, zero scoring re-implemented) scores every text_content page and dumps per-rule failure diagnostics. Every worst page was traced through the pipeline (front-end XHTML stream → engine blocks → rendered Markdown) to attribute its loss to the stage that caused it. Baseline loss pools: text_ocr 53.2 CF-units, text_multicolumns 34.8, text_simple 23.1, text_multilang 21.4.

Driver 1 — multi-column reading order (order component 52.3 overall; 30.7 on OCR pages):

• _group_words_into_lines_columns: a row now flushes the column buffers only when a word genuinely bridges a gutter (fully crosses it or sits inside it) or at a page-wide band break — a hyphenated overhang no longer degenerates narrow-gutter pages to row-major interleave.
• The OCR path runs the same conservative multi-column prose routing as the text path (scanned newspapers/newsletters used to interleave columns row-by-row and then weld into fake table rows).
• OCR det boxes are trimmed horizontally to compensate the detector's unclip dilation (_trim_det_box — the vertical counterpart was always compensated via OCR_FONT_RATIO), so word geometry stops bleeding into gutters.
• 3+-column layouts use a relaxed words-per-line floor guarded by a new line-pitch-regularity gate (prose is dense, TOC/data-card columns are gappy — measured ≥0.65 vs ≤0.45), and word density is CJK-aware (~2 chars/word).

Driver 2 — prose mis-inferred as tables (table_engine._looks_prose_grid): the alignment-band proposer sees two book-justified columns as one huge aligned band and inferred page-wide "grids" of sentence-length cells, which then replaced real, correctly-ordered prose blocks at the render boundary (scanned pages lost 15–20% of their words to sparse noise-grids; duplication + sentence fragmentation everywhere). An inferred (unruled) grid whose non-empty cells are ≥50% prose cells (≥6 words, mostly alphabetic — digit-dense TOC runs / numeric rows don't count) is now rejected. Structural token shape only; ruled grids untouched. Full ParseBench Tables fasteval (503 pages) holds at 56.95 GriTS (was 56.93).

Driver 3 — unreadable text layers (_strip_cid_words): fonts with no usable ToUnicode CMap made pdfminer emit literal (cid:NN) markers per glyph — 16 pages (incl. 6 of the 7 worst text_simple pages) "had text" that was thousands of garbage tokens. The markers are stripped (they are unextractable glyphs, not content); a fully-unmapped page collapses to ~0 words and the existing sparse-page detector routes it to OCR, which recovers the real text from pixels.

OCR quality knobs, measured and kept stock: WARP_OCR_DPI / WARP_OCR_MAX_SIDE_LEN / WARP_OCR_DET_LIMIT are now env-tunable; a 4-config sweep showed DPI 300 hurts the PP-OCR mobile models and max_side_len=3000 is marginal with outlier regressions, so defaults are unchanged.

Largest wins: the multi-column class (mckinsey +47.8, 3colpres +43.0, 3colceo +41.4, forword2cols +40.1, visa +39.5) and the cid class (staffrepport +44.6, contract +23.3). Two accepted regressions remain (coorp −7.8, scanbook −15.7) where the removed fake provider-grid had been papering over engine-level fusion of infographic labels / book-spread scans — documented as deferred engine work in the design doc: docs/superpowers/specs/2026-07-06-parsebench-content-faithfulness-gap-closure-design.md.

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Related issues

ParseBench content-faithfulness gap (see benchmarks/parsebench/RESULTS.md — Content Faith. 68.3 baseline)

Checklists

Development

• Lint rules pass locally (black --check + isort --check)
• The code changed/added as part of this pull request has been covered with tests (tests/test_multicolumn_routing.py, tests/test_table_engine.py::TestLooksProseGrid)
• All tests related to the changed code pass in development (full suite: 631 passed, 0 failed; hetero-100 baseline regenerated per the sanctioned re-floor policy — aggregate improves on every floored metric: macro-F1 +0.008, heading F1 +0.012, reading-order +0.003, both smells down; S-1 / OC / legal-100 / docsling / semunit untouched)

Code review

• This pull request has a descriptive title and information useful to a reviewer
• "Ready for review" label attached to the PR and reviewers mentioned in a comment
• Changes have been reviewed by at least one other engineer
• Issue from task tracker has a link to this pull request

🤖 Generated with Claude Code

https://claude.ai/code/session_018Db2HoocctVP1XZpmNmEsM