Add PubMed metadata source

[shuchenliu]

add support for PubMed

[Copilot]

Pull request overview

This PR adds a new pubmed_metadata plugin that ingests RENCI pubmed2db PubMed citation metadata (16 gzip NDJSON shards), including a streaming/validating parser, Elasticsearch mapping, and BioThings Hub dumper/uploader integration.

Changes:

• Added PubMed metadata plugin modules: dumper (download), uploader (parallel shard upload), parser (stream + validate + transform), mapping, and release-pinned constants.
• Added source documentation describing schema, validation guarantees, shard/concurrency behavior, and deployment/alias guidance.
• Added unit tests covering parser validation/date handling, mapping expectations, static release constants, and uploader shard partitioning.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_pubmed_metadata_uploader.py	Tests uploader job partitioning and load behavior with a stubbed BioThings environment.
tests/test_pubmed_metadata_static.py	Verifies pinned release constants and expected shard list/URLs.
tests/test_pubmed_metadata_parser.py	Validates record schema enforcement, date construction rules, and gzip/NDJSON streaming errors.
tests/test_pubmed_metadata_mapping.py	Confirms ES mapping choices (non-indexed abstract, sortable fields, date formats).
plugins/pubmed_metadata/uploader.py	Implements shard presence checks, 4-way partitioned jobs, and document streaming upload.
plugins/pubmed_metadata/static.py	Defines pinned upstream release, base URL, shard count, filenames, and terms URLs.
plugins/pubmed_metadata/README.md	Documents source, schema, validation behavior, and operational/deployment notes.
plugins/pubmed_metadata/parser.py	Provides strict streaming parser with schema/type/ID/date validation and contextual error reporting.
plugins/pubmed_metadata/mapping.py	Adds ES mapping for pubmed documents with sortable journal.name.raw and partial-date formats.
plugins/pubmed_metadata/dumper.py	Adds HTTP dumper configured for bounded parallel downloads and pinned release.
plugins/pubmed_metadata/init.py	Exposes dumper/uploader classes for plugin discovery/import.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

tests/test_pubmed_metadata_uploader.py:58

• The fixture removes existing plugins.pubmed_metadata* modules from sys.modules using direct pop() and then restores them manually. If the import fails before the yield, the teardown section will never run and the test session can be left with a corrupted sys.modules state. Use monkeypatch.delitem() for the removals so pytest reliably restores state even when setup fails, and simplify the teardown accordingly.

    original_plugin_modules = {
        name: module
        for name, module in tuple(sys.modules.items())
        if name == "plugins.pubmed_metadata" or name.startswith("plugins.pubmed_metadata.")
    }

plugins/pubmed_metadata/uploader.py:38

• The FileNotFoundError message hard-codes "16 shards" even though the expected shard list comes from PUBMED_METADATA_FILES. If the shard count changes (eg a new release), this message becomes inaccurate. Build the message from len(PUBMED_METADATA_FILES) instead of a magic number.

            raise FileNotFoundError(
                "PubMed metadata upload requires all 16 shards; missing: "
                + ", ".join(missing_paths)
            )