feat: matilde_fetch_fulltext — legal open-access full-text locator#9
Merged
Merged
Conversation
…ator Adds a 7th tool that resolves the best *legal* open-access location for a paper by DOI — a direct PDF where one exists, else an OA landing page — via OpenAlex (no key), Unpaywall (email-gated), and an arXiv-DOI synthesis fallback. This upgrades Matilde from metadata-only verification toward content-grounding: the agent can now retrieve the actual source to check whether a passage supports a claim. Deliberately OA-only: a paywalled work resolves to is_oa=false with no URL. It does not route around access controls (no Sci-Hub / piracy) — the point is legal full text for citation grounding, not paywall bypass. Fits the clean-commons posture of the public package. Engine (matilde_plugin/engine/fulltext.py) mirrors the citations engine: all network I/O injected, stdlib-only, fully unit-tested offline. Built test-first; pdf_url is only ever a genuine direct PDF (OpenAlex oa_url landing pages are reported as landing_url, never mislabeled as a PDF). - engine/fulltext.py + tools handler + registration (plugin.yaml, __init__) - tests: test_fulltext.py (8), test_fulltext_tool.py (4); registry test updated - docs: README tool table, SKILL.md claim-grounding workflow, plugin manifest - live-smoked against real DOIs (eLife gold-OA, arXiv synthesis) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
matilde_fetch_fulltext, a 7th tool that resolves the best legal open-access location for a paper by DOI — a direct PDF where one exists, otherwise an OA landing page — via OpenAlex (no key), Unpaywall (email-gated), and an arXiv-DOI synthesis fallback.This upgrades Matilde from metadata-only verification toward content-grounding: the agent can now retrieve the actual source to check whether a passage supports a claim — the exact gap SKILL.md previously flagged as "read the source yourself."
Why OA-only (not Sci-Hub)
A paywalled work resolves to
is_oa: falsewith no URL. The tool surfaces only legal open-access sources and will not route around a paywall. Putting a piracy resolver into a public, org-branded distribution repo would attach real legal/DMCA exposure and cut against the clean-commons posture we just hardened the sanitization gate to protect. The legal OA layer (Unpaywall + OpenAlexbest_oa_location+ arXiv) covers a large fraction of the literature and keeps the package shippable.How
matilde_plugin/engine/fulltext.pymirrors the citations engine: all network I/O injected, stdlib-only, fully unit-testable offline. Built test-first (TDD).A correctness note worth calling out:
pdf_urlis only ever a genuine direct PDF. OpenAlex'soa_urlis often a landing page, so it's reported aslanding_url— never mislabeled as a PDF (regression-tested).Tests
tests/test_fulltext.py(8) — engine, with fakefetch: OpenAlex OA, closed-access, Unpaywall fallback, email-gating, arXiv synthesis, DOI normalization, landing-vs-PDF labeling, JSON-serializable.tests/test_fulltext_tool.py(4) — handler envelope/validation/never-raises.🤖 Generated with Claude Code