feat(search): add auditable multi-source literature search

[jung233]

Summary

• keep the existing PubMed support and extend search to 10 scholarly APIs: PubMed, PMC, bioRxiv, medRxiv, arXiv, OpenAlex, Crossref, Semantic Scholar, CORE, and Unpaywall
• add a unified Boolean/field query contract, structured filters, bounded per-source pagination, deduplication, OA enrichment, and redacted retrieval provenance
• preserve the legacy Web /api/search result-array response by default; the advanced UI opts into detailed: true
• expose the same advanced contract through MCP with descriptions for all 23 search parameters
• add the advanced Web query builder and source controls while keeping the existing direct /api/download pipeline unchanged
• package the Web template in built wheels and support both old and new Starlette TemplateResponse signatures

Commit structure

• 31cce98 Add auditable multi-source literature search
• 9b65e63 Expose advanced search in MCP and Web UI
• 00b32b0 Document advanced literature search workflow

Each commit is usable on top of the previous one. No test files are included in the PR.

Compatibility

• existing keyword clients still receive a JSON array from /api/search
• direct DOI/arXiv responses retain the existing is_direct shape
• records without a DOI/arXiv identifier open their source page instead of being sent to the PDF downloader
• no SSE, cancellation callback, background task, or download-terminal-state changes are included
• offset is explicitly per-source; token/page limitations are reported as warnings

Verification

All project execution was performed in isolated remote containers.

• upstream base: 5e4a6f2 (master), GitHub mergeable=true
• Python 3.11 and 3.12 source compilation: passed
• targeted Ruff checks for the search/Web changes: passed
• full remote regression suite: 65 passed
• focused candidate search tests: 15 passed
• additional adaptation compatibility tests: 6 passed
• Web E2E: page 200, advanced UI present, legacy array response 200, detailed Crossref search with provenance 200
• MCP streamable HTTP E2E: 37 tools listed, all 23 search schema fields described, real Crossref tool call passed
• temporary Web/MCP containers: OOM killed false, restart count 0, approximately 45 MiB / 56 MiB memory

[Copilot]

Pull request overview

Adds PubMed as an additional search backend (via NCBI E-utilities) and increases default result limits to return more results by default, aligning web/API behavior with the expanded set of sources.

Changes:

• Add PubMed keyword search and include pmid + source metadata in merged results.
• Increase default limit from 10 to 50 in both the web request model and search_papers.
• Expand parallel search fan-out (now 4 sources) and adjust per-source fetch sizing.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/scansci_pdf/web.py	Raises default API search limit to 50 for the web endpoint.
src/scansci_pdf/search.py	Adds PubMed search integration and updates defaults/merge behavior for expanded multi-source searching.

Comments suppressed due to low confidence (1)

src/scansci_pdf/search.py:257

• as_completed(..., timeout=30) can raise concurrent.futures.TimeoutError if any source call takes longer than 30s, which would bubble up and fail the entire search. With an additional PubMed source (and an explicit sleep), this becomes more likely. Consider removing the as_completed timeout and relying on per-request HTTP timeouts instead.

            pool.submit(_search_pubmed, query, per_source, year_from, year_to): "pubmed",
        }
        for future in as_completed(futures, timeout=30):
            try:
                all_results.extend(future.result())

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.