feat(scholar): Phase 18 C2 tail — PMC references, ar5iv full text, Crossref#188
Merged
Merged
Conversation
…ossref
The deferred C2 slice (COMPLEX_CONTENT_DESIGN.md §4.3): three pure
modules plus wiring, all node-tested, no build required to land.
- platforms/pmc.js — PubMed Central enrich: the reference list parsed
into structure via shared/scholar-refs.js (year/DOI/PMID per entry;
title and authors ONLY when the DOM marks them — never guessed from
prose), figure captions, PMCID/PMID/DOI ids. `references` is a LOCAL
capture record per the design's §7 — no wire change.
- platforms/arxiv.js — an /abs/ capture is abstract-only; the handler
prefers the ar5iv full-text rendition when it is meaningfully longer
(>=2x and >=2000 chars). Identity stays the /abs/ URL; capture_url
records the ar5iv address actually fetched (a machine conversion of
the LaTeX source, not the PDF of record — hence the provenance).
Wire-bound derived fields re-derive with the body: wordCount /
readingTimeMinutes recomputed, links re-extracted from the adopted
text (or nulled — "not captured" is not "zero links").
- crossref.js — validated-DOI request shape + fill-only-missing
response mapping ("the captured text is always what the page said").
The reader-side trigger is DEFERRED: review showed the naive session
-stash write nulls sourceTabId and silently reroutes NIP-07 publish
through the worker signer, and no scholar display exists to refresh.
- Wiring: scholar-meta pass hoisted ABOVE handler dispatch (the arxiv
handler reads scholar.arxiv_id; with the old ordering it was a
permanent no-op), pmc/arxiv HANDLERS entries, detector cases (arxiv
exact-host so ar5iv tabs are not claimed; isPmcPage so gene/pubmed
pages fall through), ContentExtractor.extractFromHtmlString (the
Readability-over-fetched-HTML seam, links included), and two SW
messages — xray:scholar:fetch (HOST-ALLOWLISTED to ar5iv/arxiv; must
never become a generic fetch proxy) and xray:scholar:crossref
(accepts a DOI, never a URL).
An adversarial verify pass (3 lenses, executed repros) caught five
honesty violations in the first cut of the reference parser — Google
Scholar search links as entry.url, Elsevier-DOI years beating the true
year, SICI DOIs truncated at '<', italicized journal names adopted as
titles, inverted author names mis-split — plus the stale wire-bound
wordCount/links on the arXiv body swap and the two Crossref wiring
hazards. Every fix is pinned by a regression test whose fixture is the
repro.
No wire-format change. 1789 tests green (30 new); lint 0 errors.
Deliberately NOT built: dist/ is untouched so the maintainer's loaded
extension is unaffected until they rebuild.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…both JOURNAL entries
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The deferred C2 slice from
COMPLEX_CONTENT_DESIGN.md§4.3 — the "independent quick wins" that never landed. Built while you were mid-analysis, sodist/is deliberately untouched: everything is verified withnode --testoversrc/directly (1789 pass), and your loaded extension is unaffected until you rebuild.Directly serves the corpus you're building — PMC and arXiv are where the origins literature lives.
What you get
PMC (
pmc.ncbi.nlm.nih.gov, legacyncbi.nlm.nih.gov/pmc/too): captures now carry the reference list as structure — year/DOI/PMID per entry, title/authors only when the DOM marks them (never guessed from prose — the parser has an explicit honesty rule: wrong structure is worse than none), figure captions, and PMCID/PMID ids.referencesis a local capture record per the design's §7 — no wire change, and the natural feed for citation-graph work later.arXiv (
arxiv.org/abs/…): the abs page is abstract-only, so the capture prefers the ar5iv full-text rendition when it's meaningfully longer (≥2× and ≥2000 chars). Identity stays the/abs/URL;capture_urlrecords the ar5iv address actually fetched — ar5iv is a machine conversion of the LaTeX source, not the PDF of record, which is exactly why the provenance is explicit. Fails open to the abstract capture on any miss.Crossref (
shared/crossref.js+ the SW message): validated-DOI lookup that fills only fields the page didn't provide ("the captured text is always what the page said"). The reader-side trigger is deliberately deferred — see below.What adversarial review caught before you saw this
A 3-lens verify pass with executed repros broke the first cut seven ways; every fix is pinned by a regression test whose fixture is the repro:
entry.urlon essentially every modern-PMC reference (the[Google Scholar]decoration anchor). Citation-service chrome is now excluded.10.1016/j.vaccine.2015…→ 2015 instead of the citation's 2021). Year-matching now runs over DOI/URL-stripped text.<and shipped as wrong identifiers — now omitted entirely (absent beats wrong).'Lovelace, Ada, and Charles Babbage'mis-split into three authors. Single-token comma parts now bail to absent.wordCount/linksdescribing the abstract — and both are wire-bound (word_countandlinktags on the 30023). They now re-derive from the adopted body; links come from the same parsed DOM, or go null ("not captured" ≠ "zero links").sourceTabId, silently rerouting NIP-07 publishes through the worker signer. Deferred until there's a surface and a read-modify-write; documented in ROADMAP + JOURNAL.Wiring notes
scholar.arxiv_id, and with the old ordering it was a permanent no-op. Safe: the pass is side-effect-free and substack never readsscholar.xray:scholar:fetchis host-allowlisted (ar5iv/arxiv only) andxray:scholar:crossrefaccepts a DOI, never a URL — the SW must not grow a generic fetch proxy one message at a time.isPmcPage(so gene/pubmed pages fall through to generic).Verification
references/pmc/scholar.renditionare local capture fields.npm test— 1789 pass, 0 fail (30 new) ·node --checkon every touched file · lint 0 errorsnpm run buildwas NOT run — after merging, rebuild + reload the extension whenever you're done with the analysis; nothing here is live until then.Independent of the archive stack (#185–#187) — based on
main, no shared files with those PRs except none.🤖 Generated with Claude Code