Exhaustive text-faithfulness evaluation: oracle extractors vs Warp export surfaces

[JSv4]

Description of the change

Exhaustive text-faithfulness evaluation answering one question: does Warp-Ingest ever LOSE text? (Structure/relationships out of scope.) Every PDF in the repo — 179 documents / 5,850 pages / 2.59M oracle tokens — was compared against two independent extractors (pypdfium2 = PDFium/Chrome's text stack; blind pdfplumber flat extract) on Warp's two exported text surfaces, and every flagged page was triaged.

Verdict: no material body-text loss.

• PAWLS token surface (the OC export's no-loss layer): effective recall 0.999855 vs PDFium / 0.999893 vs pdfplumber. All 61 flagged pages (375 tokens, 0.0145%) decompose into: oracle-side table-cell gluing (not Warp's fault), fake-bold overprint char-doubling and letter-spacing char-explosion (text present but garbled), unmapped CID glyphs, soft-hyphen splits, sub/superscript re-tokenization — plus the only genuine losses: ~10 tokens of 90°-rotated margin text (arXiv sidebar, Federal Register margin keys; a pdfminer-stack limitation) and ~6 embedded-layer tokens on scan pages rerouted to OCR (which simultaneously gained hundreds of tokens per page).
• Content/block_text surface (json/html render): 0.9975 — the deficit is the engine's deliberate repeated-furniture stripping (TOC banners, EDGAR footers, folios), plus one notable finding: repeated signature blocks (/s/ …, By:, Title:) are stripped as furniture on mass-consent exhibits (neutron_lime: 901 tokens across ~177 pages) — text retained in PAWLS, removed from the renders.

Deliverables:

• scripts/text_faithfulness_eval.py — the evaluator (multiset token recall + re-tokenization forgiveness, doc-parallel, checkpointed).
• docs/text_faithfulness_eval.md — full findings report with per-class triage table and recommended follow-ups (overprint dedupe, signature-block exemption, embedded+OCR merge).
• docs/text_faithfulness_eval_results.jsonl — raw per-doc/per-page results.
• tests/test_text_faithfulness.py — CI floor: PAWLS effective recall ≥ 0.999 vs the PDFium oracle on canonical fixtures.

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Related issues

None

Checklists

Development

• Lint rules pass locally
• The code changed/added as part of this pull request has been covered with tests
• All tests related to the changed code pass in development

Code review

• This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached
• "Ready for review" label attached to the PR and reviewers mentioned in a comment
• Changes have been reviewed by at least one other engineer
• Issue from task tracker has a link to this pull request

🤖 Generated with Claude Code

https://claude.ai/code/session_017ctAeFX3qYwAs1fhupd8xR