Skip to content

Harden main-PDF validation and response recovery - #39

Open
q734738781 wants to merge 2 commits into
Rimagination:masterfrom
q734738781:agent/reject-springer-supplementary-pdfs
Open

Harden main-PDF validation and response recovery#39
q734738781 wants to merge 2 commits into
Rimagination:masterfrom
q734738781:agent/reject-springer-supplementary-pdfs

Conversation

@q734738781

@q734738781 q734738781 commented Jul 28, 2026

Copy link
Copy Markdown

Summary

  • reject Springer PDFs labeled Electronic Supplementary Material
  • reject files labeled Supporting information to https://doi.org
  • detect PDF response bodies that were lossily decoded and re-encoded as UTF-8
  • retry corrupted browser event bodies through the original PDF URL with the active browser state
  • apply both safeguards in the standard and institutional publisher batch implementations
  • add regression coverage for both workflows

Why

Springer supplementary PDFs can contain the article DOI. The previous verifier
therefore accepted some supplementary files as main articles when the DOI
matched.

Separately, some Wiley PDF responses obtained through the browser event API had
binary bytes replaced by UTF-8 U+FFFD sequences. They still began with
%PDF-, so the downloader accepted them early, but their Flate streams were
corrupted and text extraction or rendering failed. The new check rejects that
lossy event body and allows the existing browser-state HTTP fallback to retrieve
the original bytes.

Impact

Supplementary files are no longer counted as verified main articles, and
lossy browser response bodies are not delivered as valid PDFs. Normal PDFs,
including files with a small amount of legitimate UTF-8 metadata, remain
accepted.

Validation

  • python -m pytest -q — 51 passed
  • real Wiley regression — 30/30 PDFs that previously failed with zlib errors were retrieved and verified after the fix

@q734738781
q734738781 marked this pull request as ready for review July 28, 2026 07:52
@q734738781 q734738781 changed the title Reject Springer supplementary PDFs as main articles Harden main-PDF validation and response recovery Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant