Harden Semantic Scholar PDF downloads - #65
Conversation
There was a problem hiding this comment.
Pull request overview
This PR hardens the Semantic Scholar connector’s PDF download and read paths by treating openAccessPdf.url as a candidate (not guaranteed to be a direct PDF), validating responses before caching/parsing, and adding Europe PMC/PMC fallbacks using PMCID metadata.
Changes:
- Add candidate URL selection and robust PDF validation (header/content-type checks), plus invalid-cache removal and retry logic.
- Add PMCID-based fallbacks (Europe PMC rendered PDF, PMC PDF endpoint) and Europe PMC full-text XML fallback for
read_paper. - Add regression tests covering 403/HTML responses, PMC URL handling, and invalid cached PDF replacement.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
paper_search_mcp/academic_platforms/semantic.py |
Implements candidate URL download flow with validation, PMCID fallbacks, cache cleanup, and XML full-text fallback for reads. |
tests/test_semantic.py |
Adds mocked regression tests for forbidden/HTML download responses, PMC fallbacks, and invalid cache replacement. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Summary
openAccessPdf.urlas a candidate URL instead of assuming it is always a direct PDF.Root Cause
Semantic Scholar can return publisher, DOI, or PMC article URLs in
openAccessPdf.url. Some of those URLs return HTML challenge/landing pages or HTTP 403 responses rather than PDF bytes. The previous Semantic connector wrote successful HTTP 200 responses directly to.pdffiles and then handed them to PyPDF, which produced errors such asinvalid pdf header: b"<!doc"andStream has ended unexpectedly.Impact
The Semantic connector now fails cleanly when a URL is not a PDF, avoids leaving bogus cached PDFs behind, and recovers common open-access cases by using PMCID metadata to fetch PDFs from Europe PMC.
Validation
readextracts text successfully.