Skip to content

feat(scholar): Phase 18 C2 tail — PMC references, ar5iv full text, Crossref#188

Merged
bryanmatthewsimonson merged 2 commits into
mainfrom
claude/phase-18-c2-scholar-tail
Jul 18, 2026
Merged

feat(scholar): Phase 18 C2 tail — PMC references, ar5iv full text, Crossref#188
bryanmatthewsimonson merged 2 commits into
mainfrom
claude/phase-18-c2-scholar-tail

Conversation

@bryanmatthewsimonson

Copy link
Copy Markdown
Owner

The deferred C2 slice from COMPLEX_CONTENT_DESIGN.md §4.3 — the "independent quick wins" that never landed. Built while you were mid-analysis, so dist/ is deliberately untouched: everything is verified with node --test over src/ directly (1789 pass), and your loaded extension is unaffected until you rebuild.

Directly serves the corpus you're building — PMC and arXiv are where the origins literature lives.

What you get

PMC (pmc.ncbi.nlm.nih.gov, legacy ncbi.nlm.nih.gov/pmc/ too): captures now carry the reference list as structure — year/DOI/PMID per entry, title/authors only when the DOM marks them (never guessed from prose — the parser has an explicit honesty rule: wrong structure is worse than none), figure captions, and PMCID/PMID ids. references is a local capture record per the design's §7 — no wire change, and the natural feed for citation-graph work later.

arXiv (arxiv.org/abs/…): the abs page is abstract-only, so the capture prefers the ar5iv full-text rendition when it's meaningfully longer (≥2× and ≥2000 chars). Identity stays the /abs/ URL; capture_url records the ar5iv address actually fetched — ar5iv is a machine conversion of the LaTeX source, not the PDF of record, which is exactly why the provenance is explicit. Fails open to the abstract capture on any miss.

Crossref (shared/crossref.js + the SW message): validated-DOI lookup that fills only fields the page didn't provide ("the captured text is always what the page said"). The reader-side trigger is deliberately deferred — see below.

What adversarial review caught before you saw this

A 3-lens verify pass with executed repros broke the first cut seven ways; every fix is pinned by a regression test whose fixture is the repro:

  • Google Scholar search links landed in entry.url on essentially every modern-PMC reference (the [Google Scholar] decoration anchor). Citation-service chrome is now excluded.
  • The year inside an Elsevier DOI beat the true year (10.1016/j.vaccine.2015… → 2015 instead of the citation's 2021). Year-matching now runs over DOI/URL-stripped text.
  • SICI DOIs truncated at their literal < and shipped as wrong identifiers — now omitted entirely (absent beats wrong).
  • Italicized journal names adopted as titles (APA style italicizes journals). The journal is the segment the year follows directly — that positional check rejects it while a genuine italicized title still lands.
  • 'Lovelace, Ada, and Charles Babbage' mis-split into three authors. Single-token comma parts now bail to absent.
  • The ar5iv body swap left wordCount/links describing the abstract — and both are wire-bound (word_count and link tags on the 30023). They now re-derive from the adopted body; links come from the same parsed DOM, or go null ("not captured" ≠ "zero links").
  • The proposed Crossref reader trigger was hazardous twice: no scholar display exists to refresh, and the naive session-stash write nulls sourceTabId, silently rerouting NIP-07 publishes through the worker signer. Deferred until there's a surface and a read-modify-write; documented in ROADMAP + JOURNAL.

Wiring notes

  • The scholar-meta pass is hoisted above handler dispatch — the arXiv handler reads scholar.arxiv_id, and with the old ordering it was a permanent no-op. Safe: the pass is side-effect-free and substack never reads scholar.
  • xray:scholar:fetch is host-allowlisted (ar5iv/arxiv only) and xray:scholar:crossref accepts a DOI, never a URL — the SW must not grow a generic fetch proxy one message at a time.
  • Detector: arXiv is exact-host (so ar5iv tabs aren't claimed); PMC goes through isPmcPage (so gene/pubmed pages fall through to generic).

Verification

  • No wire-format change — no kind, tag, or content field moves; references/pmc/scholar.rendition are local capture fields.
  • npm test1789 pass, 0 fail (30 new) · node --check on every touched file · lint 0 errors
  • npm run build was NOT run — after merging, rebuild + reload the extension whenever you're done with the analysis; nothing here is live until then.

Independent of the archive stack (#185#187) — based on main, no shared files with those PRs except none.

🤖 Generated with Claude Code

…ossref

The deferred C2 slice (COMPLEX_CONTENT_DESIGN.md §4.3): three pure
modules plus wiring, all node-tested, no build required to land.

- platforms/pmc.js — PubMed Central enrich: the reference list parsed
  into structure via shared/scholar-refs.js (year/DOI/PMID per entry;
  title and authors ONLY when the DOM marks them — never guessed from
  prose), figure captions, PMCID/PMID/DOI ids. `references` is a LOCAL
  capture record per the design's §7 — no wire change.
- platforms/arxiv.js — an /abs/ capture is abstract-only; the handler
  prefers the ar5iv full-text rendition when it is meaningfully longer
  (>=2x and >=2000 chars). Identity stays the /abs/ URL; capture_url
  records the ar5iv address actually fetched (a machine conversion of
  the LaTeX source, not the PDF of record — hence the provenance).
  Wire-bound derived fields re-derive with the body: wordCount /
  readingTimeMinutes recomputed, links re-extracted from the adopted
  text (or nulled — "not captured" is not "zero links").
- crossref.js — validated-DOI request shape + fill-only-missing
  response mapping ("the captured text is always what the page said").
  The reader-side trigger is DEFERRED: review showed the naive session
  -stash write nulls sourceTabId and silently reroutes NIP-07 publish
  through the worker signer, and no scholar display exists to refresh.
- Wiring: scholar-meta pass hoisted ABOVE handler dispatch (the arxiv
  handler reads scholar.arxiv_id; with the old ordering it was a
  permanent no-op), pmc/arxiv HANDLERS entries, detector cases (arxiv
  exact-host so ar5iv tabs are not claimed; isPmcPage so gene/pubmed
  pages fall through), ContentExtractor.extractFromHtmlString (the
  Readability-over-fetched-HTML seam, links included), and two SW
  messages — xray:scholar:fetch (HOST-ALLOWLISTED to ar5iv/arxiv; must
  never become a generic fetch proxy) and xray:scholar:crossref
  (accepts a DOI, never a URL).

An adversarial verify pass (3 lenses, executed repros) caught five
honesty violations in the first cut of the reference parser — Google
Scholar search links as entry.url, Elsevier-DOI years beating the true
year, SICI DOIs truncated at '<', italicized journal names adopted as
titles, inverted author names mis-split — plus the stale wire-bound
wordCount/links on the arXiv body swap and the two Crossref wiring
hazards. Every fix is pinned by a regression test whose fixture is the
repro.

No wire-format change. 1789 tests green (30 new); lint 0 errors.
Deliberately NOT built: dist/ is untouched so the maintainer's loaded
extension is unaffected until they rebuild.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bryanmatthewsimonson
bryanmatthewsimonson merged commit e9f1bcb into main Jul 18, 2026
1 check passed
@bryanmatthewsimonson
bryanmatthewsimonson deleted the claude/phase-18-c2-scholar-tail branch July 18, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant