diff --git a/docs/CAPTURE_GUIDE.md b/docs/CAPTURE_GUIDE.md index ee00494..42879f1 100644 --- a/docs/CAPTURE_GUIDE.md +++ b/docs/CAPTURE_GUIDE.md @@ -206,6 +206,23 @@ These "just work" without special handling: Substacks) — article body plus comments if you opt in. - **Any article page** — Readability extracts the main content. Works on most news sites, blogs, documentation, Wikipedia, etc. +- **PubMed Central** (`pmc.ncbi.nlm.nih.gov/articles/PMC/`, legacy + `www.ncbi.nlm.nih.gov/pmc/...` too) — plain server-rendered HTML, no + timing quirks. On top of the article body, the capture carries the + **reference list as structure** (year/DOI/PMID per entry, title and + authors only where the page marks them — never guessed), figure + captions, and the PMCID/PMID/DOI ids. References are a local capture + record; nothing new publishes. +- **arXiv** (`arxiv.org/abs/`) — the abs page shows only the + abstract, so the capture **prefers the ar5iv full-text rendition** + (`ar5iv.labs.arxiv.org`) when it's meaningfully fuller: the body is + swapped for the full text, `capture_url` records the ar5iv address + actually fetched, and the article's identity stays the `/abs/` URL. + Honesty note: ar5iv is a machine conversion of the LaTeX source, not + the PDF of record — figures/equations can differ; that's exactly why + the provenance is recorded. If ar5iv has no conversion (or it's + stub-short), you keep the abstract capture unchanged. `/pdf/` links + route through the PDF capture path instead. --- diff --git a/docs/JOURNAL.md b/docs/JOURNAL.md index 16799d5..896930e 100644 --- a/docs/JOURNAL.md +++ b/docs/JOURNAL.md @@ -19,6 +19,54 @@ or files, and the "so-what" for future readers. --- +## 2026-07-17 — Phase 18 C2 tail: scholarly handlers, and what adversarial review caught + +Tags: `design`, `pattern`. + +The C2 deferred tail (PMC handler, ar5iv-preferring arXiv handler, +Crossref) landed as pure modules + wiring. Decisions worth recording: + +**The scholar-meta hoist.** `enrichArticleForPlatform` ran the +scholar-meta pass *after* handler dispatch, so any handler reading +`article.scholar` read nothing. Harmless while no handler did; the +arXiv handler's whole trigger is `scholar.arxiv_id`, so the pass now +runs first. Safe: it is side-effect-free and substack (the only other +enrich handler) never reads `scholar`. + +**References parse under an honesty rule** — wrong structure is worse +than no structure — and the review pass proved the first cut violated +it five ways, each with an executed repro: Google-Scholar *search* +links landing in `entry.url` on essentially every modern-PMC reference +(the `[Google Scholar]` decoration anchor); the year inside an +Elsevier-style DOI (`10.1016/j.vaccine.2015…`) outranking the true +year; SICI DOIs truncated at their literal `<` and emitted as wrong +identifiers (now omitted — scholar-meta's private copy still truncates, +unify later); italicized *journal names* adopted as titles (the journal +is the segment the year follows directly — that positional check is the +fix); and `'Lovelace, Ada'` mis-split into two authors (single-token +comma parts now bail). Each fix is pinned by a regression test whose +fixture IS the repro. + +**The arXiv body swap re-derives the wire-bound fields.** Adopting the +ar5iv full text while keeping the abstract's `wordCount` and `links` +would publish a `word_count` tag off by ~100× and `link` tags for a +body no longer shipped — the same class of stale-derived-state bug as +the archive-reload work, caught by the same review pattern. +`extractFromHtmlString` extracts links from the adopted body so they +ride along; absent that, `links` goes null ("not captured" ≠ "zero"). + +**`xray:scholar:fetch` is host-allowlisted** (ar5iv/arxiv hosts only) +and `xray:scholar:crossref` accepts a DOI, never a URL — the SW must +not grow a generic fetch proxy one message at a time. + +**The Crossref reader trigger is deliberately deferred.** The obvious +wiring is wrong twice: there is no scholar display in the reader to +refresh, and persisting the patched article back to the session stash +by writing a fresh record nulls `sourceTabId` — which silently reroutes +NIP-07 publishes through the worker signer (`handleCapturePublish`'s +tabless branch). It needs a read-modify-write and a surface; parked +until there is one. + ## 2026-07-17 — Load archive → republish drifted the x tag; the fix is a proof, not a flag Tags: `bug`, `design`. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 8d07dc3..bd882c2 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1506,7 +1506,20 @@ grounding substrate*). - ✅ **C2 (light)** scholarly metadata — DOI/arXiv/journal/authors from citation meta tags on every capture (`platforms/scholar-meta.js`); additive `doi` + NIP-73 `i` + `arxiv` tags on 30023. - *(Deferred: ar5iv-preferring arXiv handler, PMC handler, Crossref.)* +- ✅ **C2 (tail)** the deferred scholarly handlers — `platforms/pmc.js` + (references as structure via the shared `scholar-refs.js` parser — + title/authors only when DOM-marked, never guessed; figure captions; + PMCID/PMID ids) and `platforms/arxiv.js` (ar5iv full-text preference + on `/abs/` captures: body swap gated on a meaningfully-longer check, + `capture_url` records the rendition fetched, wire-bound + wordCount/links re-derived from the adopted body); `crossref.js` + (validated DOI → `api.crossref.org` request + fill-only-missing + mapping) behind the host-allowlisted `xray:scholar:fetch` / + `xray:scholar:crossref` SW messages. `references` stays a **local + capture record** (§7) — no wire change. *(Deferred: the reader-side + Crossref trigger — needs a scholar display surface and a careful + read-modify-write of the session stash; the naive write nulls + `sourceTabId` and breaks NIP-07 publish routing.)* - ✅ **C3** PDF routing + acquisition — background routes PDF tabs to the reader's `?pdf=` path (`shared/pdf-detect.js` + Content-Type sniff); Import-file fallback; IndexedDB v3 `source_documents` store diff --git a/src/background/index.js b/src/background/index.js index 52d969f..993d3e0 100644 --- a/src/background/index.js +++ b/src/background/index.js @@ -28,6 +28,7 @@ import { fetchSubstackPost, fetchSubstackComments } from '../shared/platforms/su import { handleScreenshotCapture } from '../shared/screenshot.js'; import { runSuggestionPass, runAuditPass, runAuditModulePass, getLlmConfig, runLensPass, getLensConfig, runCorpusMapPass, runCorpusReducePass, runHypothesisEdgePass, getCorpusConfig } from '../shared/llm-client.js'; import { pdfDocumentUrl } from '../shared/pdf-detect.js'; +import { crossrefRequestFor, mapCrossrefWork } from '../shared/crossref.js'; import { articleAnswersTo } from '../shared/url-identity.js'; import { Signer } from '../shared/signer.js'; import { loadFlags, isEnabled } from '../shared/metadata/feature-flags.js'; @@ -763,6 +764,59 @@ chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { return true; // async } + // Content script → worker: fetch a scholarly rendition's HTML on + // the capture pipeline's behalf (Phase 18 C2 tail — the arXiv + // handler's ar5iv full-text preference). Lives here because MV3 + // content scripts have no cross-origin fetch; the SW does, via + // host_permissions. HOST-ALLOWLISTED: this message must never + // become a generic fetch proxy — only the scholarly hosts the C2 + // handlers actually consume. credentials:'omit' — public pages, + // no cookie riding. + if (message.type === 'xray:scholar:fetch') { + const SCHOLAR_FETCH_HOSTS = new Set([ + 'ar5iv.labs.arxiv.org', 'ar5iv.org', 'arxiv.org' + ]); + let target = null; + try { target = new URL(String(message.url || '')); } catch (_) { /* rejected below */ } + if (!target || target.protocol !== 'https:' + || !SCHOLAR_FETCH_HOSTS.has(target.hostname.toLowerCase().replace(/^www\./, ''))) { + sendResponse({ ok: false, error: 'host not allowed' }); + return false; + } + (async () => { + try { + const resp = await fetch(target.href, { credentials: 'omit' }); + if (!resp.ok) { sendResponse({ ok: false, error: 'HTTP ' + resp.status }); return; } + sendResponse({ ok: true, html: await resp.text() }); + } catch (err) { + sendResponse({ ok: false, error: err && err.message ? err.message : String(err) }); + } + })(); + return true; // async + } + + // Reader → worker: Crossref DOI lookup (Phase 18 C2 tail). + // Accepts a DOI ONLY — the URL is built exclusively by + // crossrefRequestFor, which hard-validates the shape; a + // caller-supplied URL would make this an open proxy. The patch is + // metadata-only and applyCrossref (shared/crossref.js) fills only + // fields the page itself did not provide. + if (message.type === 'xray:scholar:crossref') { + const req = crossrefRequestFor(message.doi); + if (!req) { sendResponse({ ok: false, error: 'invalid doi' }); return false; } + (async () => { + try { + const resp = await fetch(req.url, { headers: { Accept: 'application/json' } }); + if (!resp.ok) { sendResponse({ ok: false, error: 'crossref HTTP ' + resp.status }); return; } + const patch = mapCrossrefWork(await resp.json()); + sendResponse(patch ? { ok: true, patch } : { ok: false, error: 'unmappable response' }); + } catch (err) { + sendResponse({ ok: false, error: err && err.message ? err.message : String(err) }); + } + })(); + return true; // async + } + // Reader → worker: archive-reader flow. Given a URL, query the // configured relay pool for kind-30023 events tagged with that // URL, pick the most recent, reconstruct the article, and hand diff --git a/src/shared/content-detector.js b/src/shared/content-detector.js index 87041cf..44b2153 100644 --- a/src/shared/content-detector.js +++ b/src/shared/content-detector.js @@ -1,3 +1,5 @@ +import { isPmcPage } from './platforms/pmc.js'; + export const ContentDetector = { /** * Analyze current page and return content type info @@ -26,6 +28,18 @@ export const ContentDetector = { if (hostname.includes('substack.com') || isSubstack()) { return { type: 'article', platform: 'substack', confidence: 0.9, metadata: {} }; } + // Phase 18 C2 tail — scholarly platforms. arXiv is exact-host + // on purpose: .includes('arxiv.org') would also claim + // ar5iv.labs.arxiv.org tabs (a rendition, not the platform). + // PMC goes through isPmcPage(url), not a bare hostname check — + // gene/pubmed/other NCBI pages must fall through to generic + // article detection (wrong platform label otherwise). + if (hostname === 'arxiv.org' || hostname === 'www.arxiv.org') { + return { type: 'article', platform: 'arxiv', confidence: 0.9, metadata: {} }; + } + if (isPmcPage(url)) { + return { type: 'article', platform: 'pmc', confidence: 1.0, metadata: {} }; + } if (hostname.includes('reddit.com')) { return { type: 'social_post', platform: 'reddit', confidence: 1.0, metadata: {} }; } diff --git a/src/shared/content-extractor.js b/src/shared/content-extractor.js index 6ec8172..82ac311 100644 --- a/src/shared/content-extractor.js +++ b/src/shared/content-extractor.js @@ -357,6 +357,47 @@ export const ContentExtractor = { } }, + // Extraction over a FETCHED HTML string (Phase 18 C2 tail) — the + // seam the arXiv handler's ar5iv full-text preference runs through. + // Same Readability contract as the live-DOM path: `content` stays + // HTML at capture time (markdown happens downstream in + // event-builder). A element makes the fetched page's relative + // image/link URLs resolve against its own origin, and outbound links + // are extracted from the SAME parsed body so the `link` tags a later + // publish emits describe the adopted text, not the page it replaced + // (wire-bound — see arxiv.js's adopt block). `ownHost` is the fetched + // URL's host, so a rendition's internal anchors mark `internal` and + // never publish. Null on any failure — callers fail open. + extractFromHtmlString: (html, baseUrl) => { + try { + const parsedDoc = new DOMParser().parseFromString(String(html || ''), 'text/html'); + if (baseUrl && parsedDoc.head) { + const base = parsedDoc.createElement('base'); + base.setAttribute('href', baseUrl); + parsedDoc.head.insertBefore(base, parsedDoc.head.firstChild); + } + const parsed = new Readability(parsedDoc).parse(); + if (!parsed || !parsed.content || !parsed.textContent) return null; + const out = { + content: parsed.content, + textContent: parsed.textContent, + title: parsed.title || '' + }; + try { + const bodyDiv = parsedDoc.createElement('div'); + bodyDiv.innerHTML = parsed.content; + let ownHost = ''; + try { ownHost = new URL(baseUrl).hostname; } catch (_) { /* no host */ } + const outbound = ContentExtractor.extractOutboundLinks(bodyDiv, baseUrl, ownHost); + out.links = outbound.links; + if (outbound.truncated) out.links_truncated = true; + } catch (_) { /* links stay absent — arxiv.js nulls them honestly */ } + return out; + } catch (_) { + return null; + } + }, + // Outbound links (docs/NIP_DRAFT.md `link` tags): every // http(s) anchor in the extracted body, deduped through the unified // normalizer so a link and its tracking-param variant count as ONE diff --git a/src/shared/crossref.js b/src/shared/crossref.js new file mode 100644 index 0000000..4337dca --- /dev/null +++ b/src/shared/crossref.js @@ -0,0 +1,180 @@ +// Crossref DOI enrichment — Phase 18 C2 +// (docs/COMPLEX_CONTENT_DESIGN.md §4.3, "DOI enrichment"). +// +// When a DOI is detected on a capture (scholar-meta.js), a background +// Crossref lookup fills canonical title/authors/date. Metadata only; +// the captured text is always what the page said. +// +// This module is the pure half: it builds the request shape and maps +// the response. The fetch itself lives in the background (wired via an +// xray:scholar:crossref message — see the wiring notes in the PR). No +// chrome.*, no network, no DOM; callers inject the parsed JSON. + +// The registrant-prefix shape every real DOI has: `10.<4-9 digits>/` +// plus a suffix. The suffix may not contain whitespace, quotes, or +// angle brackets — this value feeds a background fetch and must never +// build a URL from arbitrary input, so anything off-shape is rejected +// outright rather than "cleaned". +const DOI_SHAPE_RE = /^10\.\d{4,9}\/[^\s"'<>]+$/; + +const CROSSREF_WORKS_BASE = 'https://api.crossref.org/works/'; + +// The scholar-record fields a Crossref patch is allowed to carry / +// fill. Field names match scholar-meta.js where they overlap +// (journal / authors / published); title / publisher / type are +// Crossref-only additions to the local capture record. +const PATCH_FIELDS = ['title', 'authors', 'published', 'journal', 'publisher', 'type']; + +/** + * Build the Crossref works request for a DOI. + * + * @param {string} doi a bare DOI, e.g. "10.1234/abc.567" + * @returns {{url: string}|null} the request shape, or null when the + * input is not a plausibly-shaped DOI (never a URL built + * from junk). + */ +export function crossrefRequestFor(doi) { + if (typeof doi !== 'string') return null; + const trimmed = doi.trim(); + if (!DOI_SHAPE_RE.test(trimmed)) return null; + return { url: CROSSREF_WORKS_BASE + encodeURIComponent(trimmed) }; +} + +function firstNonEmptyString(value) { + if (!Array.isArray(value)) return null; + const first = value[0]; + if (typeof first !== 'string') return null; + const trimmed = first.trim(); + return trimmed || null; +} + +function mapAuthors(list) { + if (!Array.isArray(list)) return null; + const out = []; + for (const a of list) { + if (!a || typeof a !== 'object') continue; + const given = typeof a.given === 'string' ? a.given.trim() : ''; + const family = typeof a.family === 'string' ? a.family.trim() : ''; + const personal = [given, family].filter(Boolean).join(' '); + if (personal) { + out.push(personal); + } else if (typeof a.name === 'string' && a.name.trim()) { + // Organizational authors ("literal" names) carry `name` + // instead of family/given. + out.push(a.name.trim()); + } + } + return out.length ? out : null; +} + +// published-print, then published, then issued — the print date is the +// canonical publication date when Crossref has one; `issued` is the +// earliest-known date and always present on real works. +const DATE_KEYS = ['published-print', 'published', 'issued']; + +function mapDateParts(msg) { + for (const key of DATE_KEYS) { + const field = msg[key]; + const dateParts = field && typeof field === 'object' ? field['date-parts'] : null; + if (!Array.isArray(dateParts) || !Array.isArray(dateParts[0])) continue; + const [y, m, d] = dateParts[0]; + if (!Number.isInteger(y) || y <= 0) continue; + let out = String(y).padStart(4, '0'); + if (Number.isInteger(m) && m >= 1 && m <= 12) { + out += '-' + String(m).padStart(2, '0'); + if (Number.isInteger(d) && d >= 1 && d <= 31) { + out += '-' + String(d).padStart(2, '0'); + } + } + return out; + } + return null; +} + +function pickMessage(json) { + if (!json || typeof json !== 'object' || Array.isArray(json)) return null; + const msg = json.message && typeof json.message === 'object' && !Array.isArray(json.message) + ? json.message + : json; + return msg; +} + +/** + * Map a Crossref works response to a scholar patch. + * + * Accepts either the full response envelope ({ message: {...} }) or + * the message object itself. Every access is null-safe: junk or + * partial input yields null (or a partial patch), never a throw. + * + * @param {object} json parsed Crossref response + * @returns {object|null} { title?, authors?: string[], published?, + * journal?, publisher?, type? } — absent fields absent — + * or null when nothing mappable was found. + */ +export function mapCrossrefWork(json) { + try { + const msg = pickMessage(json); + if (!msg) return null; + const out = {}; + + const title = firstNonEmptyString(msg.title); + if (title) out.title = title; + + const authors = mapAuthors(msg.author); + if (authors) out.authors = authors; + + const published = mapDateParts(msg); + if (published) out.published = published; + + const journal = firstNonEmptyString(msg['container-title']); + if (journal) out.journal = journal; + + if (typeof msg.publisher === 'string' && msg.publisher.trim()) { + out.publisher = msg.publisher.trim(); + } + if (typeof msg.type === 'string' && msg.type.trim()) { + out.type = msg.type.trim(); + } + + return Object.keys(out).length ? out : null; + } catch (_) { + return null; + } +} + +function isMissing(value) { + if (value === undefined || value === null) return true; + if (typeof value === 'string') return value.trim() === ''; + if (Array.isArray(value)) return value.length === 0; + return false; +} + +/** + * Apply a Crossref patch to a scholar record, FILL-ONLY-MISSING. + * + * Never overwrites a field the page itself provided — "Metadata only; + * the captured text is always what the page said" (§4.3) is the + * contract. Crossref supplies canon where the page was silent, never + * a correction to what the page said. + * + * Sets `scholar.crossref = true` when anything was filled, so the + * provenance of every field stays inspectable. + * + * @param {object} scholar the capture's scholar record (mutated) + * @param {object|null} patch from mapCrossrefWork + * @returns {object} the scholar record (unchanged when there was + * nothing to fill) + */ +export function applyCrossref(scholar, patch) { + if (!scholar || typeof scholar !== 'object') return scholar; + if (!patch || typeof patch !== 'object') return scholar; + let filled = false; + for (const key of PATCH_FIELDS) { + if (isMissing(patch[key])) continue; + if (!isMissing(scholar[key])) continue; // the page wins, always + scholar[key] = patch[key]; + filled = true; + } + if (filled) scholar.crossref = true; + return scholar; +} diff --git a/src/shared/platforms/arxiv.js b/src/shared/platforms/arxiv.js new file mode 100644 index 0000000..347994a --- /dev/null +++ b/src/shared/platforms/arxiv.js @@ -0,0 +1,167 @@ +// arXiv enrich handler — Phase 18 C2 +// (docs/COMPLEX_CONTENT_DESIGN.md §4.3). +// +// The problem: capturing an arxiv.org/abs/ page yields only the +// ABSTRACT — Readability faithfully extracts what the page shows, and +// the abs page shows a summary. ar5iv (ar5iv.labs.arxiv.org/html/) +// serves a full-text HTML rendition of the same paper. This handler +// upgrades the capture to full text when it can, with honest +// provenance when it does. +// +// HONESTY NOTE — provenance matters here: the ar5iv rendition is a +// MACHINE CONVERSION of the paper's LaTeX source (LaTeXML), NOT the +// arXiv PDF of record. Figures, tables, and occasionally equations can +// render differently or drop out. That is exactly why an adopted +// capture records the ar5iv URL in `capture_url` (what was actually +// fetched) while `article.url` stays the /abs/ address — the +// original-as-identity pattern. url-identity.js already canonicalizes +// ar5iv hosts (ar5iv.org, ar5iv.labs.arxiv.org) back to +// arxiv.org/abs/, so the abs page IS the stable citable identity +// for every rendition of the paper. +// +// Contract (src/shared/platforms/index.js): enrich(article) → article, +// fail-open — any internal failure returns the article UNCHANGED, with +// no partial writes. Pure module: no chrome.*, no window/document — +// the caller injects url/fetchHtml/extract. In the extension the +// orchestrator backs `fetchHtml` with a background fetch message +// (content scripts cannot cross-origin fetch under MV3) and `extract` +// with the existing Readability pipeline. +// +// arXiv id + version are NOT re-derived here: scholar-meta.js already +// extracted `arxiv_id` / `arxiv_version` into `article.scholar` on +// every capture. This handler only reads them. The `references` / +// `scholar` structures are LOCAL capture records — no wire change +// (§7 of the design doc); event-builder is untouched. + +// Same two id generations as scholar-meta.js: new-style `2401.12345` +// and old-style `math.GT/0309136` (subject classes 2–12 letters, +// possibly hyphenated, SEVEN digits). Anchored to the whole /abs/ +// path so listing pages and malformed ids don't match. +const ABS_PATH_RE = /^\/abs\/((?:[a-z-]+(?:\.[a-z-]{2,12})?\/\d{7})|\d{4}\.\d{4,5})(v\d+)?\/?$/i; + +// Adoption thresholds: the abs page is short (title + authors + +// abstract — typically well under 2,500 chars), a paper body is not. +// "Meaningfully longer" = at least double the current capture AND an +// absolute floor, so a broken/empty ar5iv page can never displace a +// real abstract capture. +const MIN_FULLTEXT_CHARS = 2000; +const FULLTEXT_RATIO = 2; + +/** + * True when `url` is an arXiv abstract page — arxiv.org/abs/, + * www. tolerated, query/fragment ignored. /pdf/ and /html/ pages, + * ar5iv hosts, and listing pages are NOT abs pages. + * + * @param {string} url + * @returns {boolean} + */ +export function isArxivAbsPage(url) { + let u; + try { u = new URL(String(url || '')); } catch (_) { return false; } + if (u.protocol !== 'https:' && u.protocol !== 'http:') return false; + const host = u.hostname.toLowerCase().replace(/^www\./, ''); + if (host !== 'arxiv.org') return false; + return ABS_PATH_RE.test(u.pathname); +} + +/** + * Build the ar5iv full-text rendition URL for an arXiv id. + * Old-style ids (`math/0309136`, `math.GT/0309136`) are preserved + * verbatim — the embedded `/` is part of the id, never encoded. + * + * @param {string} id bare arXiv id (no version suffix) + * @param {number|string} [version] version number → `v` suffix + * @returns {string} + */ +export function ar5ivUrlFor(id, version) { + const v = version ? `v${version}` : ''; + return `https://ar5iv.labs.arxiv.org/html/${id}${v}`; +} + +/** + * Upgrade an abstract-only /abs/ capture to the ar5iv full text. + * + * All effects are injected — the module stays pure and testable: + * + * @param {object} article the Readability-extracted capture + * @param {object} deps + * @param {object} [deps.doc] accepted for handler-contract symmetry; unused + * @param {string} [deps.url] the page URL (falls back to article.url) + * @param {(url: string) => Promise} deps.fetchHtml + * cross-origin HTML fetch (backed by a background message) + * @param {(html: string, baseUrl: string) => {content, textContent, title, links?}|null} deps.extract + * the Readability pipeline over a fetched HTML string; `links` + * (the outbound-link extraction over the same DOM) is optional + * but wire-relevant — see the adopt block + * @returns {Promise} the same article — enriched on the adopt + * path, byte-identical on every failure path + */ +export async function enrichArticle(article, { url, fetchHtml, extract } = {}) { + if (!article) return article; + try { + const pageUrl = url || article.url || ''; + // Never touch a non-abs page: /pdf/ tabs render in the browser + // viewer (no content script), everything else isn't ours. + if (!isArxivAbsPage(pageUrl)) return article; + + // scholar-meta.js already extracted the id/version — read, don't + // re-derive. No id means nothing to fetch. + const scholar = article.scholar; + const id = scholar && scholar.arxiv_id; + if (!id) return article; + if (typeof fetchHtml !== 'function' || typeof extract !== 'function') return article; + + const ar5ivUrl = ar5ivUrlFor(id, scholar.arxiv_version); + const html = await fetchHtml(ar5ivUrl); + if (!html || typeof html !== 'string') return article; + + const extracted = await extract(html, ar5ivUrl); + if (!extracted || !extracted.content || !extracted.textContent) return article; + + // Only adopt a MEANINGFULLY longer body — the ar5iv page for a + // withdrawn/unconverted paper is itself stub-short, and must + // never displace a real abstract capture. + const curLen = String(article.textContent || '').length; + const newLen = String(extracted.textContent).length; + if (newLen < MIN_FULLTEXT_CHARS || newLen < FULLTEXT_RATIO * curLen) return article; + + // Adopt — a single write block after every check has passed, so + // no failure path can leave a partially-upgraded article. + // Identity stays the /abs/ URL (article.url untouched); + // capture_url records exactly what was fetched, because the + // ar5iv text is a machine conversion of the LaTeX source and + // can differ from the PDF of record. + article.content = extracted.content; + article.textContent = extracted.textContent; + article.capture_url = ar5ivUrl; + article.scholar = { ...scholar, rendition: 'ar5iv' }; + // Everything DERIVED from the old body re-derives with it — + // these are wire-bound values, not cosmetics: event-builder + // publishes wordCount as the 30023 `word_count` tag and + // article.links as `link` tags. Left alone, a full-text paper + // would ship the abstract's ~50-word count and the abs page's + // outbound links (same formulas as content-extractor.js). + article.wordCount = String(extracted.textContent) + .split(/\s+/).filter((w) => w.length > 0).length; + article.readingTimeMinutes = Math.ceil(article.wordCount / 225); + if (Array.isArray(extracted.links)) { + article.links = extracted.links; + // Set-when-true only, the capture convention + // (content-extractor.js:183); a stale true must not survive + // the swap either. + if (extracted.links_truncated) article.links_truncated = true; + else delete article.links_truncated; + } else { + // No link extraction over the adopted body → the old links + // describe a body we no longer ship. Null is the established + // "not captured" value ("not captured" is not "zero links"). + article.links = null; + delete article.links_truncated; + } + return article; + } catch (_) { + // Fail-open: enrichment is best-effort; the abstract capture + // still makes it to the reader. + return article; + } +} diff --git a/src/shared/platforms/index.js b/src/shared/platforms/index.js index d29c36d..4b3ee8d 100644 --- a/src/shared/platforms/index.js +++ b/src/shared/platforms/index.js @@ -25,6 +25,8 @@ // tiktok ✓ Phase 8b (#19) — synthesize-only, screenshot-evidence // instagram ✓ Phase 8c (#19) — synthesize-only, og-meta + screenshot // facebook ✓ Phase 8d (#19) — synthesize-only, graphql + og-meta + screenshot +// pmc ✓ Phase 18 C2 tail — enrich-only, references + figure captions +// arxiv ✓ Phase 18 C2 tail — enrich-only, ar5iv full-text preference import * as substack from './substack.js'; import * as youtube from './youtube.js'; @@ -32,8 +34,11 @@ import * as twitter from './twitter.js'; import * as tiktok from './tiktok.js'; import * as instagram from './instagram.js'; import * as facebook from './facebook.js'; +import * as pmc from './pmc.js'; +import * as arxiv from './arxiv.js'; import { extractGenericComments } from './comment-extractor.js'; import { extractScholarlyMeta } from './scholar-meta.js'; +import { ContentExtractor } from '../content-extractor.js'; import { resolveUrlIdentity, rewriteArchivedLinks } from '../url-identity.js'; import { recordAlias } from '../url-aliases.js'; import { Utils } from '../utils.js'; @@ -59,6 +64,30 @@ const HANDLERS = { }, facebook: { synthesize: () => facebook.synthesizeArticle() + }, + // Phase 18 C2 tail — scholarly enrich handlers. Both are pure + // modules; the real DOM / URL / fetch are closed over HERE, so the + // handlers stay node-testable with stubs. + pmc: { + enrich: (article) => pmc.enrichArticle( + article, + typeof document !== 'undefined' ? document : null, + (typeof window !== 'undefined' && window.location && window.location.href) || '' + ) + }, + arxiv: { + enrich: (article) => arxiv.enrichArticle(article, { + url: (typeof window !== 'undefined' && window.location && window.location.href) || '', + // Content scripts cannot cross-origin fetch under MV3 — + // the SW fetches on our behalf (host-allowlisted there). + fetchHtml: async (u) => { + try { + const resp = await chrome.runtime.sendMessage({ type: 'xray:scholar:fetch', url: u }); + return (resp && resp.ok && typeof resp.html === 'string') ? resp.html : null; + } catch (_) { return null; } + }, + extract: (html, baseUrl) => ContentExtractor.extractFromHtmlString(html, baseUrl) + }) } }; @@ -93,6 +122,24 @@ export async function captureForPlatform(platform) { export async function enrichArticleForPlatform(article, platform) { if (!article) return article; let enriched = article; + // Phase 18 C2 — scholarly metadata (DOI / arXiv / journal / + // citation authors) from standard meta tags. Generic like the + // comment pass below: a no-op on non-scholarly pages. Runs BEFORE + // handler dispatch on purpose (C2 tail): the arxiv handler reads + // article.scholar.arxiv_id to decide what to fetch, and pmc reads + // scholar ids to avoid re-deriving them — with the old ordering + // (scholar after dispatch) the arxiv enrich was a permanent no-op. + // Safe to hoist: this pass is side-effect-free and substack, the + // only other enrich handler, never reads article.scholar. + if (!enriched.scholar && typeof document !== 'undefined') { + try { + const scholar = extractScholarlyMeta(document, + (typeof window !== 'undefined' && window.location && window.location.href) || ''); + if (scholar) enriched.scholar = scholar; + } catch (err) { + console.warn('[X-Ray] Scholarly metadata extraction failed:', err); + } + } if (platform) { const h = HANDLERS[platform]; if (h && typeof h.enrich === 'function') { @@ -116,18 +163,6 @@ export async function enrichArticleForPlatform(article, platform) { console.warn('[X-Ray] Generic comment extraction failed:', err); } } - // Phase 18 C2 — scholarly metadata (DOI / arXiv / journal / - // citation authors) from standard meta tags. Generic like the - // comment pass: a no-op on non-scholarly pages. - if (!enriched.scholar && typeof document !== 'undefined') { - try { - const scholar = extractScholarlyMeta(document, - (typeof window !== 'undefined' && window.location && window.location.href) || ''); - if (scholar) enriched.scholar = scholar; - } catch (err) { - console.warn('[X-Ray] Scholarly metadata extraction failed:', err); - } - } // URL identity — a capture made on an archive/mirror re-keys to the // recovered ORIGINAL (original-as-identity; JOURNAL 2026-07-09) and // keeps the fetched address as provenance. Fail-open: when the diff --git a/src/shared/platforms/pmc.js b/src/shared/platforms/pmc.js new file mode 100644 index 0000000..5b6bc4d --- /dev/null +++ b/src/shared/platforms/pmc.js @@ -0,0 +1,163 @@ +// PubMed Central enrich handler — Phase 18 C2 tail +// (docs/COMPLEX_CONTENT_DESIGN.md §4.3). +// +// PMC serves unusually clean scholarly HTML: citation_* metas, a +// structured reference list, and labeled figures. Readability gets the +// body; this handler layers PMC-specific structure on top: +// +// article.pmc = { pmcid, pmid?, doi?, figures? } +// article.references = parsed reference entries (+ the honest +// article.references_truncated flag at the cap) +// +// `references` is a LOCAL capture record only (§7: "local capture +// record") — explicitly NO wire change; event-builder is untouched. +// +// Pure module: doc and url are INJECTED (no window/document/chrome +// globals) so the content-script wiring closes over the real DOM and +// tests pass stubs. Fail-open everywhere — any internal failure +// returns the article unchanged (the enrich contract). Reads +// article.scholar when present and only fills gaps from PMC-specific +// metas like citation_pmid. NOTE: the scholar-meta pass must run +// BEFORE handler dispatch for that read to see anything — the +// platforms/index.js hoist (Phase 18 C2 tail) guarantees it; without +// it this self-heals by reading the same metas directly. + +import { parseReferenceList, cleanDoi } from '../scholar-refs.js'; + +const MAX_FIGURES = 40; + +// Both PMC hosts: the 2024+ pmc.ncbi.nlm.nih.gov/articles/PMC/ +// and the legacy www.ncbi.nlm.nih.gov/pmc/articles/PMC/. +const PMC_URL_RE = /^https?:\/\/(?:pmc\.ncbi\.nlm\.nih\.gov\/articles|www\.ncbi\.nlm\.nih\.gov\/pmc\/articles)\/PMC\d+(?:[/?#]|$)/i; + +const PMCID_RE = /\bPMC(\d+)\b/i; + +// Reference-list roots across PMC generations; first hit wins. +const REF_ROOT_SELECTORS = [ + 'section.ref-list', + 'div.ref-list', + 'ol.ref-list', + 'ul.ref-list', + '#ref-list', + 'section[id^="ref-list"]', + 'section.references', + '#references' +]; + +/** Pure URL test for a PMC article page (both hosts). */ +export function isPmcPage(url) { + return PMC_URL_RE.test(String(url || '')); +} + +function collapse(text) { + return String(text == null ? '' : text).replace(/\s+/g, ' ').trim(); +} + +function qsa(el, selector) { + if (!el || typeof el.querySelectorAll !== 'function') return []; + try { + const nodes = el.querySelectorAll(selector); + return nodes ? Array.from(nodes) : []; + } catch (_) { + return []; + } +} + +function metaContent(doc, names) { + const wanted = new Set(names.map((n) => n.toLowerCase())); + for (const m of qsa(doc, 'meta')) { + const name = String((m.getAttribute && (m.getAttribute('name') || m.getAttribute('property'))) || '').toLowerCase(); + if (!wanted.has(name)) continue; + const content = m.getAttribute && m.getAttribute('content'); + if (content && content.trim()) return content.trim(); + } + return ''; +} + +function pmcidFrom(text) { + const m = PMCID_RE.exec(String(text || '')); + return m ? 'PMC' + m[1] : null; +} + +function first(el, selectors) { + for (const sel of selectors) { + const nodes = qsa(el, sel); + if (nodes.length) return nodes[0]; + } + return null; +} + +function applyIds(article, doc, url) { + const scholar = article.scholar || {}; + const pmcid = pmcidFrom(url) || pmcidFrom(metaContent(doc, ['og:url'])); + const pmid = metaContent(doc, ['citation_pmid']); + // scholar-meta already extracted citation_doi when present — read + // its key first, only fall back to the meta for the gap case. + const doi = scholar.doi || cleanDoi(metaContent(doc, ['citation_doi'])); + if (!pmcid && !pmid && !doi) return; + const pmc = article.pmc || (article.pmc = {}); + if (pmcid && !pmc.pmcid) pmc.pmcid = pmcid; + if (pmid && !pmc.pmid) pmc.pmid = pmid; + if (doi && !pmc.doi) pmc.doi = doi; +} + +function findRefRoot(doc) { + for (const sel of REF_ROOT_SELECTORS) { + const nodes = qsa(doc, sel); + if (nodes.length) return nodes[0]; + } + // Legacy PMC: bare .ref-cit-blk blocks with no single list root — + // hand parseReferenceList a synthetic root over those items. + const blocks = qsa(doc, '.ref-cit-blk'); + if (blocks.length) { + return { querySelectorAll: (sel) => (sel === '.ref-cit-blk' ? blocks : []) }; + } + return null; +} + +function applyReferences(article, doc) { + if (article.references != null) return; // another extractor won + const root = findRefRoot(doc); + if (!root) return; + const { references, truncated } = parseReferenceList(root); + if (!references.length) return; // absent, not empty + article.references = references; + if (truncated) article.references_truncated = true; +} + +function applyFigures(article, doc) { + const figures = []; + for (const fig of qsa(doc, 'figure, .fig')) { + if (figures.length >= MAX_FIGURES) break; + const captionNode = first(fig, ['figcaption', '.caption']); + const caption = captionNode ? collapse(captionNode.textContent) : ''; + if (!caption) continue; // text only, or nothing + const labelNode = first(fig, ['.label', '.fig-label', '.obj_head', 'label']); + const label = labelNode ? collapse(labelNode.textContent) : ''; + const entry = {}; + if (label && label !== caption) entry.label = label; + entry.caption = caption; + figures.push(entry); + } + if (!figures.length) return; // absent, not empty + const pmc = article.pmc || (article.pmc = {}); + pmc.figures = figures; +} + +/** + * Layer PMC-specific structure onto a Readability-extracted article. + * article.platform stays whatever it is. Never throws; always returns + * the same article object. + * + * @param {object} article + * @param {Document|object|null} doc injected document (real or stub) + * @param {string} [url] injected page URL + * @returns {object} the same article + */ +export function enrichArticle(article, doc, url = '') { + if (!article) return article; + try { applyIds(article, doc, url); } catch (_) { /* fail-open */ } + try { applyReferences(article, doc); } catch (_) { /* fail-open */ } + try { applyFigures(article, doc); } catch (_) { /* fail-open */ } + return article; +} diff --git a/src/shared/scholar-refs.js b/src/shared/scholar-refs.js new file mode 100644 index 0000000..9c0b9a7 --- /dev/null +++ b/src/shared/scholar-refs.js @@ -0,0 +1,245 @@ +// Reference-list parsing — Phase 18 C2 tail +// (docs/COMPLEX_CONTENT_DESIGN.md §4.3). +// +// Parses a reference-list DOM subtree into structured entries destined +// for the LOCAL capture record (`article.references` — §7: "local +// capture record"; explicitly NO wire change, event-builder untouched). +// +// Pure module: callers inject the list root (a real element or a stub +// — objects exposing querySelectorAll / getAttribute / textContent). +// No chrome.*, no window/document globals. +// +// HONESTY RULE: wrong structure is worse than no structure. Fields are +// emitted only when they confidently parse; a segment that doesn't +// parse yields { raw } alone. Absent fields are ABSENT, never ''. +// Titles are never guessed from prose — only DOM-marked title nodes +// count (.ref-title, or an italic/cite node that plausibly spans a +// title and not the whole citation). + +const MAX_REFERENCES = 200; + +// Same DOI shape as platforms/scholar-meta.js (kept in lockstep). +const DOI_RE = /\b(10\.\d{4,9}\/[^\s"'<>]+)/; +// Global variant for stripping every DOI span out of year-matching text. +const DOI_ALL_RE = /\b10\.\d{4,9}\/[^\s"'<>]+/g; +const PMID_RE = /\bPMID:?\s*(\d{1,9})\b/i; +// Publication year in citation position — right after a sentence / +// journal / publisher separator ("J Exp Med. 2015;" / "(2015)") — so +// a year inside the title ("The 1918 influenza…") isn't mistaken for +// the publication year. Falls back to the first bare year token. +const YEAR_CITED_RE = /[.;(]\s*((?:19|20)\d{2})\b/; +const YEAR_ANY_RE = /\b((?:19|20)\d{2})\b/; +const URL_RE = /\bhttps?:\/\/[^\s"'<>]+/gi; +// Trailing punctuation from prose is never part of a DOI or URL. +const TRAILING_PUNCT_RE = /[).,;\]]+$/; + +function collapse(text) { + return String(text == null ? '' : text).replace(/\s+/g, ' ').trim(); +} + +function qsa(el, selector) { + if (!el || typeof el.querySelectorAll !== 'function') return []; + try { + const nodes = el.querySelectorAll(selector); + return nodes ? Array.from(nodes) : []; + } catch (_) { + return []; + } +} + +/** Extract and clean a DOI from arbitrary text. Null when absent. */ +export function cleanDoi(raw) { + const text = String(raw || ''); + const m = DOI_RE.exec(text); + if (!m) return null; + // SICI-style DOIs (older Wiley/Cancer journals) contain literal '<' + // — '10.1002/1097-0142(19960815)78:4<747::AID-CNCR9>3.0.CO;2-D' — + // which DOI_RE cannot cross. A truncated DOI is a wrong, non-resolving + // identifier presented as real; the honest answer is no DOI at all. + // (scholar-meta.js's private copy still truncates — its inputs are + // meta-tag values; noted there when these unify.) + if (text[m.index + m[0].length] === '<') return null; + return m[1].replace(TRAILING_PUNCT_RE, ''); +} + +/** + * Best-effort structure from ONE citation string. Regex-only — never + * guesses a title (that needs DOM marking, see parseReferenceList). + * + * @param {string} raw + * @returns {{ raw: string, year?: number, doi?: string, pmid?: string, + * url?: string }} + */ +export function parseReferenceString(raw) { + const text = collapse(raw); + const entry = { raw: text }; + if (!text) return entry; + + const doi = cleanDoi(text); + if (doi) entry.doi = doi; + + const pmid = PMID_RE.exec(text); + if (pmid) entry.pmid = pmid[1]; + + // Year-match against text with DOI/URL spans removed: an Elsevier + // DOI ('10.1016/j.vaccine.2015.03.022') puts a dot-preceded year + // token inside the identifier, and the citation-position regex + // preferred it over the true year whenever the visible year lacked + // a [.;(] separator ('Nature 2021;384'). + const yearText = text.replace(DOI_ALL_RE, ' ').replace(URL_RE, ' '); + const year = YEAR_CITED_RE.exec(yearText) || YEAR_ANY_RE.exec(yearText); + if (year) entry.year = Number(year[1]); + + // First URL that isn't the DOI's own resolver address. + URL_RE.lastIndex = 0; + let u; + while ((u = URL_RE.exec(text))) { + const cleaned = u[0].replace(TRAILING_PUNCT_RE, ''); + if (!/doi\.org\//i.test(cleaned)) { + entry.url = cleaned; + break; + } + } + + return entry; +} + +// Title markers, most-trusted first. `.ref-title` is an explicit +// publisher marking; i/em/cite are conventions that ALSO wrap journal +// names, species names, and (PMC) whole citations, so they only count +// when the text is title-shaped and a strict subset of the citation. +const TITLE_SELECTORS = ['.ref-title', 'i', 'em', 'cite']; + +function plausibleTitle(t, raw) { + if (t.length < 15 || t.split(' ').length < 3) return false; + // Spans (nearly) the whole citation → a wrapper, not a title. + if (t.length >= raw.length * 0.8) return false; + // Carries citation plumbing → full-citation text, not a title. + if (DOI_RE.test(t) || PMID_RE.test(t)) return false; + // Sits in the JOURNAL slot → a journal name, not a title. APA-style + // lists italicize journal names ('… Journal of Widget Studies, + // 12(3), 45-67'), and those pass every guard above. The journal is + // the segment the year/volume follows directly; a real title is + // followed by the journal, never by the year. + const at = raw.indexOf(t); + if (at >= 0 && /^[.,]?\s*\(?(?:19|20)\d{2}|^[.,]?\s*\d{1,4}\s*\(/.test(raw.slice(at + t.length))) { + return false; + } + return true; +} + +function markedTitle(item, raw) { + for (const sel of TITLE_SELECTORS) { + for (const node of qsa(item, sel)) { + const t = collapse(node.textContent); + if (!t) continue; + if (sel === '.ref-title') return t; + if (plausibleTitle(t, raw)) return t; + } + } + return null; +} + +// Authors only when the DOM marks them — prose-splitting a citation +// head is guesswork and the honesty rule forbids it. +const AUTHOR_SELECTORS = ['.ref-authors', '.citation-authors', '.authors']; +// A bare-initials comma part ("Smith, J.") means the commas separate +// surname from initials, not author from author — ambiguous, bail. +const INITIALS_RE = /^[A-Z][A-Z.]{0,2}$/; +const FILLER_RE = /^(?:et al\.?|and|&)$/i; + +function markedAuthors(item) { + for (const sel of AUTHOR_SELECTORS) { + for (const node of qsa(item, sel)) { + const t = collapse(node.textContent).replace(/[.;,\s]+$/, ''); + if (!t) continue; + const bySemicolon = t.includes(';'); + const parts = t.split(bySemicolon ? ';' : ',') + .map((p) => collapse(p).replace(/\.$/, '')) + // 'and Charles Babbage' as a comma part is a conjunction + // rider, not an author named "and …" — strip the joiner. + .map((p) => p.replace(/^(?:and|&)\s+/i, '')) + .filter((p) => p && !FILLER_RE.test(p)); + if (!parts.length) continue; + if (!bySemicolon && parts.some((p) => INITIALS_RE.test(p))) continue; + // Comma-split ambiguity, the fuller shape: 'Lovelace, Ada' + // is one inverted name, not two authors. A single-token + // part ('Lovelace') can only come from that inversion — + // real comma-separated author lists carry full names. Bail + // (absent) rather than mis-split; the honesty rule. + if (!bySemicolon && parts.some((p) => !p.includes(' '))) continue; + return parts; + } + } + return null; +} + +function fillFromAnchors(item, entry) { + for (const a of qsa(item, 'a')) { + const href = (a.getAttribute && a.getAttribute('href')) || ''; + if (!href) continue; + if (!entry.doi && /doi\.org\//i.test(href)) { + const d = cleanDoi(href.split(/[?#]/)[0]); + if (d) entry.doi = d; + continue; + } + const pm = /(?:pubmed\.ncbi\.nlm\.nih\.gov|ncbi\.nlm\.nih\.gov\/pubmed)\/(\d{1,9})/i.exec(href); + if (pm) { + if (!entry.pmid) entry.pmid = pm[1]; + continue; + } + // Citation-service chrome is not the referenced work's address. + // Modern PMC decorates every reference with [DOI] [PubMed] + // [PMC free article] [Google Scholar] anchors — without the + // scholar.google exclusion, the Scholar *search link* landed in + // entry.url on essentially every reference. + if (!entry.url && /^https?:\/\//i.test(href) + && !/doi\.org\//i.test(href) && !/ncbi\.nlm\.nih\.gov/i.test(href) + && !/scholar\.google\./i.test(href) && !/google\.[a-z.]+\/scholar/i.test(href)) { + entry.url = href; + } + } +} + +/** + * Parse a reference-list subtree into structured entries. + * + * Accepts the usual list shapes: an ol/ul (or a section containing + * one) whose items are
  • , and PMC's legacy .ref-cit-blk blocks. + * + * @param {Element|object} listEl injected list root (real or stub) + * @returns {{ references: Array<{ raw: string, title?: string, + * authors?: string[], year?: number, doi?: string, + * pmid?: string, url?: string }>, truncated: boolean }} + */ +export function parseReferenceList(listEl) { + let items = qsa(listEl, 'li'); + if (!items.length) items = qsa(listEl, '.ref-cit-blk'); + + // querySelectorAll('li') is a DEEP search: a list nested inside a + // reference item returns both the outer item (whose textContent + // concatenates every nested citation — corrupt) and each inner one + // (again — double-counted). Keep only LEAF items: an item that + // contains another collected item is a container, not a citation. + // Stub documents without .contains are unaffected (fail-open). + if (items.length > 1 && items.some((it) => it && typeof it.contains === 'function')) { + items = items.filter((it) => + !(it && typeof it.contains === 'function' + && items.some((other) => other !== it && it.contains(other)))); + } + + const truncated = items.length > MAX_REFERENCES; + const references = []; + for (const item of items.slice(0, MAX_REFERENCES)) { + const raw = collapse(item && item.textContent); + if (!raw) continue; + const entry = parseReferenceString(raw); + const title = markedTitle(item, raw); + if (title) entry.title = title; + const authors = markedAuthors(item); + if (authors) entry.authors = authors; + fillFromAnchors(item, entry); + references.push(entry); + } + return { references, truncated }; +} diff --git a/tests/arxiv.test.mjs b/tests/arxiv.test.mjs new file mode 100644 index 0000000..4ebde67 --- /dev/null +++ b/tests/arxiv.test.mjs @@ -0,0 +1,293 @@ +// arXiv enrich handler tests — Phase 18 C2 +// (docs/COMPLEX_CONTENT_DESIGN.md §4.3). Pure-module pattern: every +// effect (fetchHtml, extract) is a stub; no DOM, no chrome.*. + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { isArxivAbsPage, ar5ivUrlFor, enrichArticle } from '../src/shared/platforms/arxiv.js'; + +const ABS_URL = 'https://arxiv.org/abs/2401.12345'; + +function absArticle(overrides = {}) { + return { + url: ABS_URL, + title: 'A Paper', + content: '

    Abstract: we prove a thing.

    ', + textContent: 'Abstract: we prove a thing.', + scholar: { arxiv_id: '2401.12345', arxiv_version: 2 }, + ...overrides + }; +} + +const FULL_TEXT = 'Full body of the paper. '.repeat(200); // ~4,800 chars + +function goodDeps(calls = {}) { + return { + url: ABS_URL, + fetchHtml: async (u) => { calls.fetched = u; return 'ar5iv body'; }, + extract: (html, baseUrl) => { + calls.extracted = { html, baseUrl }; + return { content: '

    ' + FULL_TEXT + '

    ', textContent: FULL_TEXT, title: 'A Paper' }; + } + }; +} + +// ------------------------------------------------------------------ +// isArxivAbsPage +// ------------------------------------------------------------------ + +test('isArxivAbsPage: abs shapes match, www and version and query tolerated', () => { + assert.equal(isArxivAbsPage('https://arxiv.org/abs/2401.12345'), true); + assert.equal(isArxivAbsPage('https://www.arxiv.org/abs/2401.12345'), true); + assert.equal(isArxivAbsPage('https://arxiv.org/abs/2401.12345v3'), true); + assert.equal(isArxivAbsPage('https://arxiv.org/abs/2401.12345?context=cs.LG'), true); + assert.equal(isArxivAbsPage('https://arxiv.org/abs/math/0309136'), true); + assert.equal(isArxivAbsPage('https://arxiv.org/abs/math.GT/0309136'), true); + assert.equal(isArxivAbsPage('https://arxiv.org/abs/cond-mat.mes-hall/0212413'), true); +}); + +test('isArxivAbsPage: non-abs shapes do not match', () => { + assert.equal(isArxivAbsPage('https://arxiv.org/pdf/2401.12345'), false); + assert.equal(isArxivAbsPage('https://arxiv.org/html/2401.12345'), false); + assert.equal(isArxivAbsPage('https://ar5iv.labs.arxiv.org/html/2401.12345'), false); + assert.equal(isArxivAbsPage('https://arxiv.org/list/cs.LG/recent'), false); + assert.equal(isArxivAbsPage('https://arxiv.org/abs/'), false); + assert.equal(isArxivAbsPage('https://example.com/abs/2401.12345'), false); + assert.equal(isArxivAbsPage('not a url'), false); + assert.equal(isArxivAbsPage(''), false); + assert.equal(isArxivAbsPage(null), false); +}); + +// ------------------------------------------------------------------ +// ar5ivUrlFor +// ------------------------------------------------------------------ + +test('ar5ivUrlFor: new-style id, with and without version', () => { + assert.equal(ar5ivUrlFor('2401.12345'), 'https://ar5iv.labs.arxiv.org/html/2401.12345'); + assert.equal(ar5ivUrlFor('2401.12345', 3), 'https://ar5iv.labs.arxiv.org/html/2401.12345v3'); +}); + +test('ar5ivUrlFor: old-style ids preserved verbatim (slash not encoded)', () => { + assert.equal(ar5ivUrlFor('math/0309136'), 'https://ar5iv.labs.arxiv.org/html/math/0309136'); + assert.equal(ar5ivUrlFor('math.GT/0309136', 2), 'https://ar5iv.labs.arxiv.org/html/math.GT/0309136v2'); +}); + +// ------------------------------------------------------------------ +// enrichArticle — adopt path +// ------------------------------------------------------------------ + +test('adopts the ar5iv rendition: content swapped, capture_url + rendition set', async () => { + const calls = {}; + const article = absArticle(); + const out = await enrichArticle(article, goodDeps(calls)); + + assert.equal(out, article, 'same object back (enrich contract)'); + assert.equal(calls.fetched, 'https://ar5iv.labs.arxiv.org/html/2401.12345v2', 'version rides into the fetch URL'); + assert.equal(calls.extracted.baseUrl, calls.fetched, 'extract sees the ar5iv base URL'); + assert.equal(out.content, '

    ' + FULL_TEXT + '

    '); + assert.equal(out.textContent, FULL_TEXT); + assert.equal(out.capture_url, 'https://ar5iv.labs.arxiv.org/html/2401.12345v2', 'provenance: what was actually fetched'); + assert.equal(out.url, ABS_URL, 'identity stays the /abs/ URL'); + assert.equal(out.scholar.rendition, 'ar5iv'); + assert.equal(out.scholar.arxiv_id, '2401.12345', 'scholar fields preserved'); + assert.equal(out.scholar.arxiv_version, 2); + assert.equal(out.title, 'A Paper', 'title not adopted from the rendition'); +}); + +test('adopts at the exact thresholds (>= 2x and >= 2000 chars)', async () => { + const cur = 'a'.repeat(1000); + const next = 'b'.repeat(2000); + const article = absArticle({ textContent: cur, content: '

    ' + cur + '

    ' }); + const out = await enrichArticle(article, { + url: ABS_URL, + fetchHtml: async () => '', + extract: () => ({ content: '

    ' + next + '

    ', textContent: next, title: 'T' }) + }); + assert.equal(out.textContent, next); + assert.equal(out.scholar.rendition, 'ar5iv'); +}); + +test('adopts when the current capture has no textContent at all', async () => { + const article = absArticle({ textContent: undefined, content: undefined }); + const out = await enrichArticle(article, goodDeps()); + assert.equal(out.textContent, FULL_TEXT); + assert.equal(out.scholar.rendition, 'ar5iv'); +}); + +test('async extract is awaited', async () => { + const article = absArticle(); + const out = await enrichArticle(article, { + url: ABS_URL, + fetchHtml: async () => '', + extract: async () => ({ content: '

    ' + FULL_TEXT + '

    ', textContent: FULL_TEXT, title: 'T' }) + }); + assert.equal(out.textContent, FULL_TEXT); +}); + +// ------------------------------------------------------------------ +// enrichArticle — fail-open paths (article byte-identical) +// ------------------------------------------------------------------ + +async function assertUnchanged(article, deps, label) { + const snapshot = structuredClone(article); + const out = await enrichArticle(article, deps); + assert.equal(out, article, `${label}: same object back`); + assert.deepEqual(out, snapshot, `${label}: article byte-identical`); +} + +test('fail-open: fetchHtml returns null', async () => { + await assertUnchanged(absArticle(), { + url: ABS_URL, + fetchHtml: async () => null, + extract: () => { throw new Error('must not be called'); } + }, 'fetch null'); +}); + +test('fail-open: fetchHtml throws', async () => { + await assertUnchanged(absArticle(), { + url: ABS_URL, + fetchHtml: async () => { throw new Error('network down'); }, + extract: () => { throw new Error('must not be called'); } + }, 'fetch throw'); +}); + +test('fail-open: extract returns null', async () => { + await assertUnchanged(absArticle(), { + url: ABS_URL, + fetchHtml: async () => '', + extract: () => null + }, 'extract null'); +}); + +test('fail-open: extract throws', async () => { + await assertUnchanged(absArticle(), { + url: ABS_URL, + fetchHtml: async () => '', + extract: () => { throw new Error('parse failed'); } + }, 'extract throw'); +}); + +test('fail-open: body below the absolute floor (< 2000 chars)', async () => { + const short = 'c'.repeat(1999); + await assertUnchanged(absArticle({ textContent: 'tiny abstract' }), { + url: ABS_URL, + fetchHtml: async () => '', + extract: () => ({ content: '

    ' + short + '

    ', textContent: short, title: 'T' }) + }, 'below floor'); +}); + +test('fail-open: body not >= 2x the current capture', async () => { + const cur = 'a'.repeat(3000); + const next = 'b'.repeat(5999); // >= 2000 but < 2x + await assertUnchanged(absArticle({ textContent: cur }), { + url: ABS_URL, + fetchHtml: async () => '', + extract: () => ({ content: '

    ' + next + '

    ', textContent: next, title: 'T' }) + }, 'below ratio'); +}); + +test('never touches a non-abs page (fetchHtml not called)', async () => { + for (const url of [ + 'https://arxiv.org/pdf/2401.12345', + 'https://ar5iv.labs.arxiv.org/html/2401.12345', + 'https://example.com/abs/2401.12345' + ]) { + let fetched = false; + await assertUnchanged(absArticle({ url }), { + url, + fetchHtml: async () => { fetched = true; return ''; }, + extract: () => ({ content: '

    x

    ', textContent: FULL_TEXT, title: 'T' }) + }, `non-abs ${url}`); + assert.equal(fetched, false, `no fetch for ${url}`); + } +}); + +test('fail-open: no scholar / no arxiv_id (fetchHtml not called)', async () => { + let fetched = false; + const deps = { + url: ABS_URL, + fetchHtml: async () => { fetched = true; return ''; }, + extract: () => ({ content: '

    x

    ', textContent: FULL_TEXT, title: 'T' }) + }; + await assertUnchanged(absArticle({ scholar: undefined }), deps, 'no scholar'); + await assertUnchanged(absArticle({ scholar: { doi: '10.1/x' } }), deps, 'no arxiv_id'); + assert.equal(fetched, false); +}); + +test('fail-open: missing injected deps', async () => { + await assertUnchanged(absArticle(), { url: ABS_URL }, 'no fetchHtml/extract'); + await assertUnchanged(absArticle(), undefined, 'no deps bag'); +}); + +test('null article passes through', async () => { + assert.equal(await enrichArticle(null, goodDeps()), null); +}); + +test('url falls back to article.url when not injected', async () => { + const article = absArticle(); + const out = await enrichArticle(article, { + fetchHtml: async () => '', + extract: () => ({ content: '

    ' + FULL_TEXT + '

    ', textContent: FULL_TEXT, title: 'T' }) + }); + assert.equal(out.scholar.rendition, 'ar5iv'); +}); + +test('old-style id builds an old-style ar5iv URL on the adopt path', async () => { + let fetched = null; + const article = absArticle({ + url: 'https://arxiv.org/abs/math.GT/0309136', + scholar: { arxiv_id: 'math.GT/0309136' } + }); + const out = await enrichArticle(article, { + url: 'https://arxiv.org/abs/math.GT/0309136', + fetchHtml: async (u) => { fetched = u; return ''; }, + extract: () => ({ content: '

    ' + FULL_TEXT + '

    ', textContent: FULL_TEXT, title: 'T' }) + }); + assert.equal(fetched, 'https://ar5iv.labs.arxiv.org/html/math.GT/0309136'); + assert.equal(out.capture_url, 'https://ar5iv.labs.arxiv.org/html/math.GT/0309136'); + assert.equal(out.scholar.rendition, 'ar5iv'); +}); + +// ------------------------------------------------------------------ +// Adversarial-review regressions (Phase 18 C2 verify pass): the adopt +// block must re-derive everything DERIVED from the old body. wordCount +// and links are WIRE-BOUND — event-builder publishes word_count and +// link tags from them — so leaving them describing the abstract ships +// wrong metadata on a full-text capture. +// ------------------------------------------------------------------ + +test('REGRESSION: adoption recomputes wordCount and readingTimeMinutes from the new body', async () => { + const article = absArticle({ wordCount: 5, readingTimeMinutes: 1 }); + await enrichArticle(article, goodDeps()); + assert.equal(article.scholar.rendition, 'ar5iv', 'sanity: adoption happened'); + const expectedWords = FULL_TEXT.split(/\s+/).filter((w) => w.length > 0).length; + assert.equal(article.wordCount, expectedWords, + 'word_count is a 30023 wire tag — it must describe the shipped body'); + assert.equal(article.readingTimeMinutes, Math.ceil(expectedWords / 225)); +}); + +test('REGRESSION: adoption replaces links when extract provides them, nulls when it cannot', async () => { + // Provided: the outbound links of the ADOPTED body ride along. + const newLinks = [{ url: 'https://example.com/cited', text: 'cited', count: 1, internal: false }]; + const withLinks = absArticle({ links: [{ url: 'https://arxiv.org/list/cs.LG', text: 'listing', count: 1, internal: true }] }); + const deps = goodDeps(); + const baseExtract = deps.extract; + deps.extract = (h, b) => ({ ...baseExtract(h, b), links: newLinks }); + await enrichArticle(withLinks, deps); + assert.deepEqual(withLinks.links, newLinks, 'link tags publish from article.links'); + + // Not provided: the abs page's links describe a body we no longer + // ship — null is the established "not captured" value. + const withoutLinks = absArticle({ links: [{ url: 'https://arxiv.org/list/cs.LG', text: 'listing', count: 1, internal: true }], links_truncated: true }); + await enrichArticle(withoutLinks, goodDeps()); + assert.equal(withoutLinks.links, null); + assert.equal('links_truncated' in withoutLinks, false, 'a stale truncation flag must not survive the swap'); +}); + +test('REGRESSION: the fail-open paths still leave wordCount/links byte-identical', async () => { + const article = absArticle({ wordCount: 5, readingTimeMinutes: 1, links: [{ url: 'https://x.example/a', text: 'a', count: 1, internal: false }] }); + const before = structuredClone(article); + await enrichArticle(article, { url: ABS_URL, fetchHtml: async () => null, extract: () => null }); + assert.deepEqual(article, before, 'no partial writes on a failed fetch — including the derived fields'); +}); diff --git a/tests/crossref.test.mjs b/tests/crossref.test.mjs new file mode 100644 index 0000000..5848b5e --- /dev/null +++ b/tests/crossref.test.mjs @@ -0,0 +1,177 @@ +// Crossref DOI-enrichment tests — Phase 18 C2 +// (docs/COMPLEX_CONTENT_DESIGN.md §4.3). Pure module: no fetch, no +// chrome.*, no DOM — the fixture is a hand-built Crossref works +// response in the documented shape. + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { crossrefRequestFor, mapCrossrefWork, applyCrossref } from '../src/shared/crossref.js'; + +// A realistic /works/ response, per the documented Crossref +// shape: envelope { status, message-type, message }, message.title[], +// author[{family,given}], issued['date-parts'], container-title[]. +const WORKS_RESPONSE = { + status: 'ok', + 'message-type': 'work', + 'message-version': '1.0.0', + message: { + DOI: '10.1234/abc.567', + type: 'journal-article', + title: ['A Canonical Paper Title'], + author: [ + { given: 'Ada', family: 'Lovelace', sequence: 'first', affiliation: [] }, + { given: 'Charles', family: 'Babbage', sequence: 'additional', affiliation: [] } + ], + 'container-title': ['Journal of Examples'], + publisher: 'Example University Press', + 'published-print': { 'date-parts': [[2026, 1, 15]] }, + issued: { 'date-parts': [[2025, 11]] } + } +}; + +test('crossrefRequestFor: valid DOI -> percent-encoded works URL', () => { + const req = crossrefRequestFor('10.1234/abc.567'); + assert.deepEqual(req, { url: 'https://api.crossref.org/works/10.1234%2Fabc.567' }); + + // Characters that would break or truncate a raw URL get encoded. + const hashy = crossrefRequestFor('10.1234/a#b(2026)'); + assert.ok(hashy.url.includes('%23'), 'hash is percent-encoded'); + assert.ok(hashy.url.startsWith('https://api.crossref.org/works/'), 'fixed base'); + + // Surrounding whitespace is trimmed, not fatal. + assert.ok(crossrefRequestFor(' 10.1234/abc.567\n')); +}); + +test('crossrefRequestFor: junk -> null, never a URL from arbitrary input', () => { + for (const junk of [ + null, undefined, 42, {}, '', + 'not a doi', + '10.12/short-prefix', // registrant prefix needs 4-9 digits + '10.1234/', // no suffix + '10.1234/abc def', // interior whitespace + '10.1234/abc"quote', // quotes + "10.1234/abc'quote", + '10.1234/', // angle brackets + 'https://doi.org/10.1234/abc.567' // a URL is not a bare DOI + ]) { + assert.equal(crossrefRequestFor(junk), null, `rejects ${JSON.stringify(junk)}`); + } +}); + +test('mapCrossrefWork: realistic works response maps fully', () => { + const patch = mapCrossrefWork(WORKS_RESPONSE); + assert.deepEqual(patch, { + title: 'A Canonical Paper Title', + authors: ['Ada Lovelace', 'Charles Babbage'], + published: '2026-01-15', // published-print wins over issued + journal: 'Journal of Examples', + publisher: 'Example University Press', + type: 'journal-article' + }); + + // The bare message object (no envelope) is accepted too. + assert.deepEqual(mapCrossrefWork(WORKS_RESPONSE.message), patch); +}); + +test('mapCrossrefWork: date-parts precision — year only, year+month', () => { + const yearOnly = mapCrossrefWork({ message: { issued: { 'date-parts': [[2019]] } } }); + assert.deepEqual(yearOnly, { published: '2019' }); + + const yearMonth = mapCrossrefWork({ message: { issued: { 'date-parts': [[2019, 3]] } } }); + assert.deepEqual(yearMonth, { published: '2019-03' }); + + // published (online) is preferred over issued when print is absent. + const online = mapCrossrefWork({ + message: { + published: { 'date-parts': [[2020, 6, 2]] }, + issued: { 'date-parts': [[2020]] } + } + }); + assert.equal(online.published, '2020-06-02'); +}); + +test('mapCrossrefWork: authors — family-only and organizational names', () => { + const patch = mapCrossrefWork({ + message: { + author: [ + { family: 'Plato' }, + { name: 'CERN Collaboration' }, + { affiliation: [] }, // neither name nor family: skipped + 'junk-entry' + ] + } + }); + assert.deepEqual(patch, { authors: ['Plato', 'CERN Collaboration'] }); +}); + +test('mapCrossrefWork: junk and partial input never throws', () => { + for (const junk of [null, undefined, 'string', 42, [], { message: null }, + { message: 'nope' }, { message: [] }, {}]) { + assert.equal(mapCrossrefWork(junk), null, `null for ${JSON.stringify(junk)}`); + } + + // Partial message -> partial patch, absent fields absent. + const partial = mapCrossrefWork({ message: { title: ['Only A Title'] } }); + assert.deepEqual(partial, { title: 'Only A Title' }); + + // Wrong-typed fields are ignored, not fatal. + const wrongTypes = mapCrossrefWork({ + message: { + title: 'not-an-array', + author: { family: 'not-an-array' }, + issued: { 'date-parts': 'nope' }, + 'container-title': [42], + publisher: ['not-a-string'], + type: 'journal-article' + } + }); + assert.deepEqual(wrongTypes, { type: 'journal-article' }); + + // Nothing mappable at all -> null, not {}. + assert.equal(mapCrossrefWork({ message: { title: [] } }), null); +}); + +test('applyCrossref: fill-only-missing — the page always wins', () => { + const scholar = { + doi: '10.1234/abc.567', + title: 'The Title As The Page Said It', + authors: ['Page Author'] + }; + const patch = mapCrossrefWork(WORKS_RESPONSE); + const out = applyCrossref(scholar, patch); + + assert.equal(out, scholar, 'same record back (mutated in place)'); + assert.equal(out.title, 'The Title As The Page Said It', 'page-provided title survives'); + assert.deepEqual(out.authors, ['Page Author'], 'page-provided authors survive'); + assert.equal(out.journal, 'Journal of Examples', 'missing journal filled'); + assert.equal(out.published, '2026-01-15', 'missing date filled'); + assert.equal(out.publisher, 'Example University Press'); + assert.equal(out.type, 'journal-article'); + assert.equal(out.crossref, true, 'provenance flag set when anything filled'); + assert.equal(out.doi, '10.1234/abc.567', 'doi untouched'); +}); + +test('applyCrossref: empty-ish page fields count as missing', () => { + const scholar = { doi: '10.1234/abc.567', journal: ' ', authors: [] }; + applyCrossref(scholar, { journal: 'Journal of Examples', authors: ['Ada Lovelace'] }); + assert.equal(scholar.journal, 'Journal of Examples'); + assert.deepEqual(scholar.authors, ['Ada Lovelace']); + assert.equal(scholar.crossref, true); +}); + +test('applyCrossref: nothing filled -> no crossref flag; null patch -> unchanged', () => { + const full = { doi: '10.1234/abc.567', title: 'T', authors: ['A'], published: '2026', journal: 'J', publisher: 'P', type: 'journal-article' }; + const before = { ...full }; + applyCrossref(full, { title: 'Other', journal: 'Other J' }); + assert.deepEqual(full, before, 'fully-provided record untouched'); + assert.equal(full.crossref, undefined, 'no provenance flag when nothing filled'); + + const scholar = { doi: '10.1234/abc.567' }; + assert.equal(applyCrossref(scholar, null), scholar); + assert.deepEqual(scholar, { doi: '10.1234/abc.567' }); + + // Defensive: a falsy scholar record comes back as-is (the lookup + // should never have run without one). + assert.equal(applyCrossref(null, { title: 'T' }), null); +}); diff --git a/tests/pmc.test.mjs b/tests/pmc.test.mjs new file mode 100644 index 0000000..7e07cac --- /dev/null +++ b/tests/pmc.test.mjs @@ -0,0 +1,223 @@ +// PubMed Central enrich handler tests — Phase 18 C2 tail +// (docs/COMPLEX_CONTENT_DESIGN.md §4.3). Stub-document pattern (see +// tests/scholar-meta.test.mjs) — no DOM, no jsdom. + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { isPmcPage, enrichArticle } from '../src/shared/platforms/pmc.js'; + +function el(text, children = {}, attrs = {}) { + return { + textContent: text, + getAttribute: (n) => (n in attrs ? attrs[n] : null), + querySelectorAll: (sel) => children[sel] || [] + }; +} +function meta(name, content) { + return { getAttribute: (n) => (n === 'name' ? name : n === 'content' ? content : null) }; +} +function metaProp(property, content) { + return { getAttribute: (n) => (n === 'property' ? property : n === 'content' ? content : null) }; +} +function doc(map) { + return { querySelectorAll: (sel) => map[sel] || [] }; +} + +const PMC_URL = 'https://pmc.ncbi.nlm.nih.gov/articles/PMC7095418/'; +const LEGACY_URL = 'https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1234567/'; + +// Realistic current-PMC shape: section.ref-list > ul > li, each li a +// wrapping the whole citation plus a DOI anchor; PMID in text. +function pmcDoc() { + const cite1 = 'Zhu N, Zhang D, Wang W, et al. A novel coronavirus from patients ' + + 'with pneumonia in China, 2019. N Engl J Med. 2020;382(8):727-733. PMID: 31978945.'; + const li1 = el(cite1, { + cite: [el(cite1)], + a: [el('DOI', {}, { href: 'https://doi.org/10.1056/NEJMoa2001017' })] + }); + const cite2 = 'Huang C, Wang Y, Li X, et al. Clinical features of patients infected ' + + 'with 2019 novel coronavirus in Wuhan, China. Lancet. 2020;395:497-506.'; + const li2 = el(cite2, { cite: [el(cite2)] }); + const refRoot = el('', { li: [li1, li2] }); + + const fig = el('', { + figcaption: [el('Fig. 1. Viral load over time in upper respiratory specimens.')], + '.obj_head': [el('Fig. 1.')] + }); + + return doc({ + meta: [ + meta('citation_pmid', '32109013'), + meta('citation_doi', '10.1056/NEJMoa2001017'), + meta('citation_journal_title', 'The New England Journal of Medicine'), + metaProp('og:url', PMC_URL), + meta('viewport', 'width=device-width') + ], + 'section.ref-list': [refRoot], + 'figure, .fig': [fig] + }); +} + +test('isPmcPage: both hosts, only article paths', () => { + assert.equal(isPmcPage(PMC_URL), true); + assert.equal(isPmcPage('https://pmc.ncbi.nlm.nih.gov/articles/PMC7095418'), true); + assert.equal(isPmcPage('https://pmc.ncbi.nlm.nih.gov/articles/PMC7095418/?report=classic'), true); + assert.equal(isPmcPage(LEGACY_URL), true); + + assert.equal(isPmcPage('https://pubmed.ncbi.nlm.nih.gov/32109013/'), false); + assert.equal(isPmcPage('https://www.ncbi.nlm.nih.gov/gene/1489680'), false); + assert.equal(isPmcPage('https://pmc.ncbi.nlm.nih.gov/about/'), false); + assert.equal(isPmcPage('https://evil.example/pmc.ncbi.nlm.nih.gov/articles/PMC1/'), false); + assert.equal(isPmcPage(''), false); + assert.equal(isPmcPage(null), false); +}); + +test('enrichArticle: ids, references, and figure captions from a realistic PMC page', () => { + const article = { title: 'A novel coronavirus', platform: null }; + const out = enrichArticle(article, pmcDoc(), PMC_URL); + + assert.equal(out, article); // same object, enrich contract + assert.equal(out.platform, null); // platform untouched + + assert.equal(out.pmc.pmcid, 'PMC7095418'); + assert.equal(out.pmc.pmid, '32109013'); + assert.equal(out.pmc.doi, '10.1056/NEJMoa2001017'); + + assert.equal(out.references.length, 2); + assert.equal(out.references[0].doi, '10.1056/NEJMoa2001017'); // from the anchor + assert.equal(out.references[0].pmid, '31978945'); + assert.equal(out.references[0].year, 2020); + assert.equal(out.references[0].title, undefined); // whole-citation cite is not a title + assert.equal(out.references[1].year, 2020); + assert.equal(out.references_truncated, undefined); + + assert.deepEqual(out.pmc.figures, [ + { label: 'Fig. 1.', caption: 'Fig. 1. Viral load over time in upper respiratory specimens.' } + ]); +}); + +test('enrichArticle: article.scholar.doi wins over the citation_doi meta', () => { + const article = { scholar: { doi: '10.9999/from-scholar-meta' } }; + enrichArticle(article, pmcDoc(), PMC_URL); + assert.equal(article.pmc.doi, '10.9999/from-scholar-meta'); + assert.equal(article.scholar.doi, '10.9999/from-scholar-meta'); // untouched +}); + +test('enrichArticle: legacy host URL yields the pmcid', () => { + const article = {}; + enrichArticle(article, doc({}), LEGACY_URL); + assert.equal(article.pmc.pmcid, 'PMC1234567'); + assert.equal(article.pmc.pmid, undefined); + assert.equal(article.pmc.doi, undefined); +}); + +test('enrichArticle: page with no references and no figures — no empty arrays appear', () => { + const article = { title: 'T' }; + enrichArticle(article, doc({ meta: [meta('citation_pmid', '11111111')] }), PMC_URL); + assert.equal('references' in article, false); + assert.equal('references_truncated' in article, false); + assert.equal('figures' in article.pmc, false); + assert.equal(article.pmc.pmid, '11111111'); +}); + +test('enrichArticle: pre-existing article.references is left alone', () => { + const existing = [{ raw: 'existing entry' }]; + const article = { references: existing }; + enrichArticle(article, pmcDoc(), PMC_URL); + assert.equal(article.references, existing); // same array, not replaced + assert.equal(article.references.length, 1); +}); + +test('enrichArticle: legacy .ref-cit-blk shape with no list root parses', () => { + const blocks = [el('Doe J. Legacy citation. J Old. 1999;1:1. PMID: 10500123.')]; + const article = {}; + enrichArticle(article, doc({ '.ref-cit-blk': blocks }), PMC_URL); + assert.equal(article.references.length, 1); + assert.equal(article.references[0].pmid, '10500123'); +}); + +test('enrichArticle: reference cap (200 + truncated flag) and figure cap (40, no flag)', () => { + const lis = []; + for (let i = 0; i < 201; i++) lis.push(el(`Doe J. Paper ${i}. J Rep. 2020;1:${i}.`)); + const figs = []; + for (let i = 0; i < 45; i++) { + figs.push(el('', { figcaption: [el(`Figure ${i} caption text.`)] })); + } + const article = {}; + enrichArticle(article, doc({ + 'section.ref-list': [el('', { li: lis })], + 'figure, .fig': [figs].flat() + }), PMC_URL); + assert.equal(article.references.length, 200); + assert.equal(article.references_truncated, true); + assert.equal(article.pmc.figures.length, 40); +}); + +test('enrichArticle: figure without caption text is skipped, label-only is not enough', () => { + const article = {}; + enrichArticle(article, doc({ + 'figure, .fig': [ + el('', { '.obj_head': [el('Fig. 2.')] }), // no caption → skipped + el('', { '.caption': [el(' A real caption. ')] }) // legacy .caption shape + ] + }), PMC_URL); + assert.deepEqual(article.pmc.figures, [{ caption: 'A real caption.' }]); +}); + +test('enrichArticle: fail-open — throwing doc, null doc, null article', () => { + const boom = { querySelectorAll() { throw new Error('boom'); } }; + const article = { title: 'T' }; + assert.equal(enrichArticle(article, boom, PMC_URL), article); + assert.equal(article.pmc.pmcid, 'PMC7095418'); // URL id still recovered + assert.equal('references' in article, false); + + const bare = { title: 'B' }; + assert.equal(enrichArticle(bare, null, ''), bare); + assert.equal('pmc' in bare, false); // nothing found → nothing set + + assert.equal(enrichArticle(null, pmcDoc(), PMC_URL), null); +}); + +// ------------------------------------------------------------------ +// Adversarial-review regressions (Phase 18 C2 verify pass). +// ------------------------------------------------------------------ + +test('REGRESSION: a reference root with zero parseable entries sets NO references key', () => { + // The reachable case the earlier no-references test masked: the + // ref-list root EXISTS but yields nothing (empty/non-li items). + // "Absent, not empty" — an empty array would read as "this paper + // cites nothing", which is false. + const emptyRoot = el('', { li: [el(' ')] }); + const d = doc({ + meta: [meta('citation_pmid', '32109013')], + 'section.ref-list': [emptyRoot] + }); + const article = { title: 'T' }; + enrichArticle(article, d, PMC_URL); + assert.equal('references' in article, false); + assert.equal('references_truncated' in article, false); +}); + +test('REGRESSION: the modern-PMC anchor set does not put Google Scholar in entry.url', () => { + // The full real decoration: [DOI] [PubMed] [PMC free article] + // [Google Scholar]. Pre-fix, the Scholar SEARCH link became + // entry.url on essentially every reference. + const cite = 'Zhu N, et al. A novel coronavirus. N Engl J Med. 2020;382:727. PMID: 31978945.'; + const li = el(cite, { + cite: [el(cite)], + a: [ + el('DOI', {}, { href: 'https://doi.org/10.1056/NEJMoa2001017' }), + el('PubMed', {}, { href: 'https://pubmed.ncbi.nlm.nih.gov/31978945' }), + el('PMC free article', {}, { href: 'https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7092803/' }), + el('Google Scholar', {}, { href: 'https://scholar.google.com/scholar_lookup?journal=N+Engl+J+Med' }) + ] + }); + const d = doc({ 'section.ref-list': [el('', { li: [li] })] }); + const article = { title: 'T' }; + enrichArticle(article, d, PMC_URL); + assert.equal(article.references[0].url, undefined, + 'every anchor here is citation plumbing — no work URL exists on this entry'); + assert.equal(article.references[0].doi, '10.1056/NEJMoa2001017'); + assert.equal(article.references[0].pmid, '31978945'); +}); diff --git a/tests/scholar-refs.test.mjs b/tests/scholar-refs.test.mjs new file mode 100644 index 0000000..97d1cc9 --- /dev/null +++ b/tests/scholar-refs.test.mjs @@ -0,0 +1,289 @@ +// Reference-list parser tests — Phase 18 C2 tail +// (docs/COMPLEX_CONTENT_DESIGN.md §4.3). Stub-document pattern (see +// tests/scholar-meta.test.mjs) — no DOM, no jsdom. + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { parseReferenceList, parseReferenceString, cleanDoi } from '../src/shared/scholar-refs.js'; + +// Stub element: textContent + getAttribute + querySelectorAll keyed by +// the exact selector string the module asks for. +function el(text, children = {}, attrs = {}) { + return { + textContent: text, + getAttribute: (n) => (n in attrs ? attrs[n] : null), + querySelectorAll: (sel) => children[sel] || [] + }; +} + +const NLM = 'Zhu N, Zhang D, Wang W, et al. A novel coronavirus from patients ' + + 'with pneumonia in China, 2019. N Engl J Med. 2020;382(8):727-733. ' + + 'doi: 10.1056/NEJMoa2001017. PMID: 31978945.'; + +test('parseReferenceString: year, DOI, PMID from an NLM-style citation', () => { + const e = parseReferenceString(' Zhu N, Zhang D, Wang W, et al.\n A novel coronavirus from patients ' + + 'with pneumonia in China, 2019. N Engl J Med. 2020;382(8):727-733. ' + + 'doi: 10.1056/NEJMoa2001017. PMID: 31978945.'); + assert.equal(e.raw, NLM); // whitespace-collapsed + assert.equal(e.year, 2020); // not 2019 (title year) + assert.equal(e.doi, '10.1056/NEJMoa2001017'); // trailing period stripped + assert.equal(e.pmid, '31978945'); + assert.equal(e.title, undefined); // never guessed from prose +}); + +test('parseReferenceString: year in the title does not win over the cited year', () => { + const e = parseReferenceString('Smith J. The 1918 influenza pandemic revisited. J Hist Med. 2004;12:1-10.'); + assert.equal(e.year, 2004); +}); + +test('parseReferenceString: APA parenthesized year', () => { + const e = parseReferenceString('Smith, J. (2015). On widgets. Widget Press.'); + assert.equal(e.year, 2015); +}); + +test('parseReferenceString: unparseable segment yields { raw } alone — absent, not empty', () => { + const e = parseReferenceString('World Health Organization. Weekly epidemiological update.'); + assert.deepEqual(e, { raw: 'World Health Organization. Weekly epidemiological update.' }); +}); + +test('parseReferenceString: empty input yields { raw: "" } only', () => { + assert.deepEqual(parseReferenceString(''), { raw: '' }); + assert.deepEqual(parseReferenceString(null), { raw: '' }); +}); + +test('parseReferenceString: url extracted, trailing punctuation stripped, doi.org skipped', () => { + const e = parseReferenceString('Dataset. Available: https://doi.org/10.5555/xyz123. Mirror: https://example.org/data).'); + assert.equal(e.doi, '10.5555/xyz123'); + assert.equal(e.url, 'https://example.org/data'); +}); + +test('cleanDoi: extraction + trailing punctuation', () => { + assert.equal(cleanDoi('see doi:10.1234/abc.567).'), '10.1234/abc.567'); + assert.equal(cleanDoi('no doi here'), null); +}); + +test('parseReferenceList: li items — string fields plus DOM-marked title and anchor ids', () => { + const li1 = el(NLM, { + // PMC wraps the WHOLE citation in — not a title. + cite: [el(NLM)], + a: [ + el('DOI link', {}, { href: 'https://doi.org/10.1056/NEJMoa2001017?utm_source=x' }), + el('PubMed', {}, { href: 'https://pubmed.ncbi.nlm.nih.gov/31978945/' }) + ] + }); + const li2 = el('Doe J. Widget dynamics in mice. J Widgetry. 2015;1:1-9.', { + '.ref-title': [el('Widget dynamics in mice')] + }); + const root = el('', { li: [li1, li2] }); + + const { references, truncated } = parseReferenceList(root); + assert.equal(truncated, false); + assert.equal(references.length, 2); + + assert.equal(references[0].raw, NLM); + assert.equal(references[0].title, undefined); // whole-citation cite rejected + assert.equal(references[0].doi, '10.1056/NEJMoa2001017'); + assert.equal(references[0].pmid, '31978945'); + assert.equal(references[0].year, 2020); + + assert.equal(references[1].title, 'Widget dynamics in mice'); + assert.equal(references[1].year, 2015); +}); + +test('parseReferenceList: anchors fill gaps the string missed; external url captured', () => { + const li = el('Agency report, published online.', { + a: [ + el('doi', {}, { href: 'https://doi.org/10.5555/rep2020' }), + el('report', {}, { href: 'https://agency.example/report' }), + el('local anchor', {}, { href: '#ref-5' }) + ] + }); + const { references } = parseReferenceList(el('', { li: [li] })); + assert.equal(references[0].doi, '10.5555/rep2020'); + assert.equal(references[0].url, 'https://agency.example/report'); +}); + +test('title honesty: numbered-label + whole-citation cite is NOT a title; short italics are NOT titles', () => { + // "1. " label makes the cite text a strict subset of raw — the + // 0.8-of-raw + citation-plumbing guards must still reject it. + const li1 = el('1. ' + NLM, { cite: [el(NLM)] }); + // Italicized journal name / species name — too short / too few words. + const li2 = el('Doe J. On E. coli. Nature. 2001;410:1-2.', { + i: [el('E. coli'), el('Nature')] + }); + const { references } = parseReferenceList(el('', { li: [li1, li2] })); + assert.equal(references[0].title, undefined); + assert.equal(references[1].title, undefined); +}); + +test('title: a plausible italic title (subset of the citation, no plumbing) is accepted', () => { + const raw = 'Doe J. A grand unified theory of widgets. Widget Press; 2010.'; + const li = el(raw, { i: [el('A grand unified theory of widgets')] }); + const { references } = parseReferenceList(el('', { li: [li] })); + assert.equal(references[0].title, 'A grand unified theory of widgets'); +}); + +test('authors: only DOM-marked; Vancouver comma list splits, et al dropped', () => { + const li = el('Smith J, Jones B, et al. A paper. J. 2020;1:1.', { + '.ref-authors': [el('Smith J, Jones B, et al.')] + }); + const { references } = parseReferenceList(el('', { li: [li] })); + assert.deepEqual(references[0].authors, ['Smith J', 'Jones B']); +}); + +test('authors: ambiguous "Surname, I." comma shape is rejected; unmarked prose never parsed', () => { + const marked = el('Smith, J. A paper. 2020.', { '.ref-authors': [el('Smith, J.')] }); + const unmarked = el('Smith J, Jones B. A paper. J. 2020;1:1.'); + const { references } = parseReferenceList(el('', { li: [marked, unmarked] })); + assert.equal(references[0].authors, undefined); + assert.equal(references[1].authors, undefined); +}); + +test('parseReferenceList: .ref-cit-blk fallback when there are no li items', () => { + const blocks = [ + el('Doe J. Legacy citation. J Old. 1999;1:1. PMID: 10500123.'), + el('Roe R. Another one. J Old. 2001;2:2.') + ]; + const root = el('', { '.ref-cit-blk': blocks }); + const { references, truncated } = parseReferenceList(root); + assert.equal(truncated, false); + assert.equal(references.length, 2); + assert.equal(references[0].pmid, '10500123'); + assert.equal(references[0].year, 1999); +}); + +test('parseReferenceList: cap at 200 with an honest truncated flag', () => { + const items = []; + for (let i = 0; i < 205; i++) { + items.push(el(`Doe J. Paper number ${i}. J Rep. 2020;1:${i}.`)); + } + const { references, truncated } = parseReferenceList(el('', { li: items })); + assert.equal(references.length, 200); + assert.equal(truncated, true); +}); + +test('parseReferenceList: empty items skipped; junk items keep { raw } alone', () => { + const { references } = parseReferenceList(el('', { + li: [el(' \n '), el('An unstructured note without any ids or dates')] + })); + assert.equal(references.length, 1); + assert.deepEqual(references[0], { raw: 'An unstructured note without any ids or dates' }); +}); + +test('parseReferenceList: null / selector-less / listless roots yield empty, not throw', () => { + assert.deepEqual(parseReferenceList(null), { references: [], truncated: false }); + assert.deepEqual(parseReferenceList({}), { references: [], truncated: false }); + assert.deepEqual(parseReferenceList(el('some prose, no list items')), { references: [], truncated: false }); +}); + +// ------------------------------------------------------------------ +// Adversarial-review regressions (Phase 18 C2 verify pass). Each of +// these was a defect DEMONSTRATED against the first cut — the fixtures +// are the repros. Do not simplify them. +// ------------------------------------------------------------------ + +test('REGRESSION: a Google Scholar lookup anchor never becomes entry.url', () => { + // Modern PMC decorates every reference with [DOI] [PubMed] + // [PMC free article] [Google Scholar]; the Scholar SEARCH link was + // landing in entry.url on essentially every reference. + const anchor = (href) => el('link', {}, { href }); + const item = el('Zhu N, et al. A novel coronavirus. N Engl J Med. 2020;382:727.', { + a: [ + anchor('https://scholar.google.com/scholar_lookup?journal=N+Engl+J+Med'), + anchor('https://www.google.de/scholar?q=whatever'), + anchor('https://publisher.example/article/727') + ] + }); + const { references } = parseReferenceList(el('', { li: [item] })); + assert.equal(references[0].url, 'https://publisher.example/article/727', + 'citation-service chrome is not the referenced work\'s address'); +}); + +test('REGRESSION: a year inside an Elsevier-style DOI does not beat the cited year', () => { + const e = parseReferenceString( + 'Zhang X, et al. Widget results. Nature 2021;384:1-9. doi: 10.1016/j.vaccine.2015.03.022'); + assert.equal(e.year, 2021, 'the .2015 inside the DOI is an identifier, not a year'); + assert.equal(e.doi, '10.1016/j.vaccine.2015.03.022'); +}); + +test('REGRESSION: the year fallback fires when no [.;(] separator precedes it', () => { + // 'Nature 2021;384' — the year follows a SPACE, so the + // citation-position regex misses and the bare-token fallback must + // answer. This branch had zero coverage. + const e = parseReferenceString('Zhang X, et al. Widget results. Nature 2021;384:1-9'); + assert.equal(e.year, 2021); +}); + +test('REGRESSION: a SICI DOI is omitted, never truncated at the "<"', () => { + const e = parseReferenceString( + 'Smith A. Older paper. Cancer. 1996;78:747. doi:10.1002/1097-0142(19960815)78:4<747::AID-CNCR9>3.0.CO;2-D'); + assert.equal(e.doi, undefined, + 'a truncated DOI is a wrong identifier presented as real — absent is the honest answer'); + assert.equal(e.year, 1996); +}); + +test('REGRESSION: an italicized JOURNAL name is not adopted as the title', () => { + // APA style italicizes journal names; the journal is the segment + // the year/volume follows directly, and it was passing every guard. + const raw = 'Doe J. Short title. American Journal of Epidemiology. 2015;1:1.'; + const item = el(raw, { i: [el('American Journal of Epidemiology')] }); + const { references } = parseReferenceList(el('', { li: [item] })); + assert.equal(references[0].title, undefined); + // …and a genuine italicized TITLE (followed by the journal, not + // the year) still lands. + const raw2 = 'Doe J. On widgets and their discontents. J Widget Stud. 2015;1:1.'; + const item2 = el(raw2, { i: [el('On widgets and their discontents')] }); + const r2 = parseReferenceList(el('', { li: [item2] })); + assert.equal(r2.references[0].title, 'On widgets and their discontents'); +}); + +test('REGRESSION: the 0.8 whole-citation guard holds without DOI/PMID masking it', () => { + // Every earlier fixture also carried a DOI/PMID, so the plumbing + // guard rejected the wrapper regardless and the length guard was + // never actually load-bearing under test. + const raw = 'Huang C, Wang Y. Clinical features of patients. Lancet. 2020;395:497-506.'; + const item = el(raw, { cite: [el(raw)] }); // wraps the WHOLE citation + const { references } = parseReferenceList(el('', { li: [item] })); + assert.equal(references[0].title, undefined, + 'a node spanning (nearly) the whole citation is a wrapper, not a title'); +}); + +test('REGRESSION: inverted "Surname, Given" author lists bail instead of mis-splitting', () => { + const item = (authors) => + el('Some citation. J Test. 2001;1:1.', { '.ref-authors': [el(authors)] }); + // 'Lovelace, Ada, and Charles Babbage' → ['Lovelace','Ada','and Charles Babbage'] pre-fix. + const a = parseReferenceList(el('', { li: [item('Lovelace, Ada, and Charles Babbage')] })); + assert.equal(a.references[0].authors, undefined, 'ambiguous inversion — absent, not mis-split'); + const b = parseReferenceList(el('', { li: [item('Lovelace, Ada.')] })); + assert.equal(b.references[0].authors, undefined); + // Full names separated by commas are unambiguous and still parse. + const c = parseReferenceList(el('', { li: [item('Ada Lovelace, Charles Babbage')] })); + assert.deepEqual(c.references[0].authors, ['Ada Lovelace', 'Charles Babbage']); +}); + +test('REGRESSION: semicolon-separated author lists split on the semicolon', () => { + // The one shape where 'Surname, I.' IS safely splittable — the + // semicolons disambiguate. This branch had zero coverage. + const item = el('Some citation. J Test. 2001;1:1.', { + '.ref-authors': [el('Smith, J.; Jones, B.; Kim, C.')] + }); + const { references } = parseReferenceList(el('', { li: [item] })); + assert.deepEqual(references[0].authors, ['Smith, J', 'Jones, B', 'Kim, C']); +}); + +test('REGRESSION: a list nested inside a reference item is not double-counted', () => { + // Real querySelectorAll('li') is a DEEP search: the outer item's + // textContent concatenates the nested citations (corrupt) and the + // inner items repeat (double-count). Leaf items only. + const inner = el('Roe R. Inner citation. J In. 2001;2:2.'); + inner.contains = () => false; + const outer = el('Doe J. Outer citation. J Out. 1999;1:1. Roe R. Inner citation. J In. 2001;2:2.'); + outer.contains = (n) => n === inner; + const { references } = parseReferenceList({ + querySelectorAll: (sel) => (sel === 'li' ? [outer, inner] : []) + }); + assert.equal(references.length, 1); + assert.equal(references[0].raw, 'Roe R. Inner citation. J In. 2001;2:2.', + 'the container entry (concatenated nested text) is dropped, the leaf kept'); +});