Initial implementation: download, store, and export PubMed abstracts

[gaurav]

WIP

• We need to export the PMCIDs and DOIs as well -- these may be present in different files, but Babel knows how to download them, so it should be easy to add.

Summary

Initial implementation of pubmed2db: a uv/click tool that downloads all PubMed abstracts, loads them into a full-version-history DuckDB database, and exports the latest version of every abstract to JSON (for Node Annotator / ElasticSearch) and Parquet (for downloadable queries).

Pipeline download → load → export:

• download — reuses cthoyt/pubmed-downloader for bulk baseline/update fetching; adds .md5 sidecar tracking so new/changed checksums drive incremental reloads.
• parse — drives the XML iteration itself, calling @cthoyt's _extract_article for the rich record while additionally capturing raw PubDate components (full date fidelity) and <DeleteCitation> PMIDs.
• load — full history, every version tagged with source_file provenance; latest_article view selects the newest non-deleted version per PMID. Journal names come from the NLM Catalog.
• export — one configurable command: sharded NDJSON with DocumentMetadataAPI field names (empty string, never null; pub_month as a 3-letter abbreviation) and per-table Parquet (latest or full history).

The database uses PubMed's own field names; DocumentMetadataAPI names are applied only at JSON export.

Design decisions

• Reuse pubmed-downloader as-is (no upstream changes) for download + parse; DuckDB is our store (we don't use its JSONL cache).
• Keep full version history; fully normalized tables; DuckDB for native Parquet + scale.
• MD5 is low-priority (HTTP downloads are reliable, PubMed files immutable): store the checksum and reload only on new/changed files.

See CLAUDE.md for architecture and FUTURE.md for follow-ups (replacing Babel's downloader, scale/perf, etc.).

Known issue worked around

pubmed_downloader.catalog.process_journal_overview() (≤ 0.0.14) raises on the real J_Entrez.txt (its Journal model requires start_year/end_year the file omits), so we parse the overview file ourselves. Tracked in FUTURE.md to revert once fixed upstream.

Verification

• 34 tests pass (no network): parse fidelity, latest-version/delete logic, idempotent + MD5-change reloads, journal parsing, JSON spec fields, Parquet filtering, CLI end-to-end.
• Live run on one real update file: MD5-verified download → loaded 16,664 articles + 56 deletions → 41,923 journals → exported 16,664 docs across even NDJSON shards (zero nulls, journal names/abbrevs resolved, months normalized) and 15 consistent Parquet files.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Initial implementation of pubmed2db, a Python CLI tool that downloads PubMed baseline/update XMLs, loads them into a full-version-history DuckDB schema, and exports the latest non-deleted article set to NDJSON (DocumentMetadataAPI fields) and Parquet.

Changes:

• Added end-to-end pipeline modules (download → parse → load → export → status) plus a Click CLI wrapper.
• Introduced a normalized DuckDB schema with a latest_article view for newest-version selection and deletion handling.
• Added comprehensive pytest coverage with XML/journal fixtures, plus docs and Slurm run guidance.

Reviewed changes

Copilot reviewed 26 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/pubmed2db/__init__.py	Defines package version and module docstring.
src/pubmed2db/cli.py	Click CLI implementing download, journals, load, export, update, status.
src/pubmed2db/db.py	DuckDB connection + schema init + source-file registry helpers.
src/pubmed2db/download.py	PubMed sync via pubmed-downloader with .md5 sidecar tracking and optional verification.
src/pubmed2db/export.py	JSON (DocumentMetadataAPI fields, empty-string-not-null) and Parquet exports.
src/pubmed2db/load.py	Normalized loading with full history, idempotent reloads, and journal overview parsing/loading.
src/pubmed2db/parse.py	Self-driven XML iteration using _extract_article, plus raw PubDate + DeleteCitation capture.
src/pubmed2db/schema.sql	Full normalized schema + latest_article view implementing newest-version selection with deletions.
src/pubmed2db/status.py	Read-only pipeline readiness/state summarization derived from DB state.
src/pubmed2db/util.py	Shared helpers for peak RSS and duration formatting.
tests/conftest.py	Shared fixtures (gzipping XML fixtures, DuckDB connections, staged download layout).
tests/test_cli.py	CLI end-to-end tests (export, status, error messaging, directory scan, gzip export).
tests/test_db_download.py	Tests for filename ordering, registry upsert behavior, and MD5 parsing/helpers.
tests/test_export.py	Tests for month normalization, JSON field contract, sharding, and Parquet latest/all behavior.
tests/test_journals.py	Tests for journal overview parsing and journal load behavior.
tests/test_load.py	Tests for full history retention, latest selection, deletion behavior, and MD5-triggered reloads.
tests/test_parse.py	Tests for raw-date fidelity, rich extraction fields, and DeleteCitation capture.
tests/__init__.py	Marks tests as a package.
tests/fixtures/pubmed25n0001.xml	Baseline fixture XML used for parsing/load/export tests.
tests/fixtures/pubmed25n0002.xml	Update fixture XML including a revised article and a DeleteCitation.
tests/fixtures/J_Entrez_sample.txt	Sample NLM journal overview fixture for journal parser tests.
scripts/benchmark_load.py	Script to benchmark parsing vs insertion throughput and peak RSS.
slurm/README.md	Operational notes for running loads on Slurm (memory/time sizing, monitoring).
README.md	Project documentation: purpose, pipeline, usage examples, and development notes.
CLAUDE.md	Architecture/design orientation document for the repository.
FUTURE.md	Deferred work items and known limitations.
pyproject.toml	Project metadata, dependencies, and pytest configuration.
.gitignore	Ignores downloaded data while keeping directory skeleton.
data/.gitkeep	Keeps data/ directory present in the repository.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 30 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/pubmed2db/download.py:57

• hashlib.file_digest() is only available in newer Python versions (3.11+). Since this project declares requires-python = ">=3.10", this implementation will raise AttributeError on Python 3.10 when --verify is enabled.

def file_md5(path: Path) -> str:
    """Compute the MD5 hex digest of a local file."""
    with path.open("rb") as fh:
        return hashlib.file_digest(fh, "md5").hexdigest()

src/pubmed2db/download.py:89

• If the .md5 sidecar format changes (or is otherwise unparsable), parse_md5_text() returns None, which currently makes changed = prior != published_md5 flip to True and can spuriously bump downloaded_at (triggering unnecessary reloads). Treat an unparsable checksum like a transient failure and keep the prior checksum instead.

            response = requests.get(url + ".md5", timeout=60)
            response.raise_for_status()
            published_md5 = parse_md5_text(response.text)
            _save_md5_sidecar(file_name, response.text)

src/pubmed2db/download.py:103

• When --no-verify is used, a changed published checksum does not currently force a re-download. Because pubmed_downloader is documented here as “skip-by-name”, ensure() may keep the old local file even though the remote content changed, and the pipeline would reload stale bytes. A checksum change should trigger a redownload regardless of --verify.

        changed = prior is None or prior != published_md5

        path = Path(ensure_module.ensure(url=url))

        if verify and published_md5 is not None:
            actual = file_md5(path)