extract_pdf's doc comment describes a condition the code doesn't implement.
crates/webclaw-pdf/src/lib.rs:45-46:
/// Uses pdf-extract for text extraction and lopdf (transitive dep) for
/// metadata and page count. In `Auto` mode, returns `PdfError::EmptyPdf`
/// if no text is found (likely a scanned/image-only PDF).
The condition at line 73 is text.is_empty() after normalize_text, which trims and collapses blank lines and nothing else. EmptyPdf needs zero non-whitespace characters in the entire document.
A scanned PDF is not that. Page numbers, footer stamps, headers and partial OCR layers all survive normalize_text, so the common scanned document returns Ok with a handful of junk characters. The variant fires on a much narrower case than "scanned/image-only".
Why it matters
This misled an outside contributor. #104 and #105 propose an opt-in API returning the original PDF bytes on EmptyPdf so callers can run OCR without refetching. The premise is reasonable, and the parenthetical above is where it came from. The seam they built misses most of the documents they were targeting.
Fix
Reword the doc comment to state the real condition, and say what it excludes:
In Auto mode, returns PdfError::EmptyPdf when the extracted text contains no non-whitespace characters. Note this is narrower than "scanned document": a scanned page carrying a page number or footer stamp yields a small amount of text and returns Ok.
Worth deciding separately whether Auto should key on text density rather than emptiness. A 40-page PDF yielding 6 characters is the same failure for a caller as one yielding zero, and only the second is detectable today. That's a behaviour change, so it belongs in its own issue if we want it.
extract_pdf's doc comment describes a condition the code doesn't implement.crates/webclaw-pdf/src/lib.rs:45-46:The condition at line 73 is
text.is_empty()afternormalize_text, which trims and collapses blank lines and nothing else.EmptyPdfneeds zero non-whitespace characters in the entire document.A scanned PDF is not that. Page numbers, footer stamps, headers and partial OCR layers all survive
normalize_text, so the common scanned document returnsOkwith a handful of junk characters. The variant fires on a much narrower case than "scanned/image-only".Why it matters
This misled an outside contributor. #104 and #105 propose an opt-in API returning the original PDF bytes on
EmptyPdfso callers can run OCR without refetching. The premise is reasonable, and the parenthetical above is where it came from. The seam they built misses most of the documents they were targeting.Fix
Reword the doc comment to state the real condition, and say what it excludes:
Worth deciding separately whether
Autoshould key on text density rather than emptiness. A 40-page PDF yielding 6 characters is the same failure for a caller as one yielding zero, and only the second is detectable today. That's a behaviour change, so it belongs in its own issue if we want it.