feat: add GTDB as a biolink:OrganismTaxon source in the taxon pipeline - #978
Open
SkyeAv wants to merge 2 commits into
Open
feat: add GTDB as a biolink:OrganismTaxon source in the taxon pipeline#978SkyeAv wants to merge 2 commits into
SkyeAv wants to merge 2 commits into
Conversation
GTDB (Genome Taxonomy Database) provides a standardized bacterial/archaeal taxonomy. Ingest it as a taxon vocabulary in the existing taxon pipeline: every rank in each genome's gtdb_taxonomy string becomes a GTDB CURIE (GTDB:s__Escherichia_coli), labeled with the taxon name, and species-rank taxa are linked eq to their ncbi_species_taxid so a GTDB species resolves into the NCBI taxon clique. - src/datahandlers/gtdb.py parses the bac120/ar53 metadata (columns read by name); write_gtdb_labels emits one label per unique taxon, build_gtdb_relationships emits species-rank GTDB<->NCBITaxon edges, and the concord provenance records the configured (pinnable) download URL. - GTDB is not in the Biolink Model's organism-taxon id_prefixes, so the taxon build passes extra_prefixes=[GTDB] (the documented escape hatch). GTDB is also a unique_prefixes entry: the GTDB<->NCBI mapping is many-to-one, so two GTDB species sharing an NCBI taxid are never merged (at most one joins the clique, the rest stay singletons). - New get_gtdb_bac120/get_gtdb_ar53 download rules (retries: 3) + gtdb_labels/gtdb_ids/ get_gtdb_relationships; GTDB added to taxon_labels/ids/concords. It joins the existing OrganismTaxon.txt (no new compendium file), so the util.py aggregators and report tables need no change. 8 offline unit tests over a verbatim 113-column GTDB fixture (incl. a glom-level test pinning the many-to-one guard) + 1 network test pinning extra_prefixes via create_node. Full unit suite green (410 passed). Deferred: source-impact report / clique diff (per maintainer); the taxon pipeline has no impact-report hook yet (PIPELINE_CONFIG registers only anatomy/disease).
astral-sh/ruff-action@v3 floats to the latest ruff, now 0.16.0 (pyproject pins only a floor, >=0.14.14). Ruff 0.16.0 formats Python code fences inside *.md, which overlaps with the dedicated rumdl Markdown check and was failing 'Check Python formatting with ruff' on every open PR. Exclude *.md from ruff format so ruff stays scoped to Python and Markdown formatting CI is deterministic across ruff releases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds GTDB (Genome Taxonomy Database, bacterial/archaeal taxonomy) as a
biolink:OrganismTaxonsource in the existingtaxonpipeline. Every rank in each genome'sgtdb_taxonomystring becomes a labeledGTDBCURIE (GTDB:s__Escherichia_coli), and species-ranktaxa are linked
eqto theirncbi_species_taxidso a GTDB species resolves into the NCBI taxon clique.Ingest
src/datahandlers/gtdb.pyparses thebac120/ar53metadata (columns read byname, robust to reordering).
write_gtdb_labelsemits one label per unique taxon (all ranks);build_gtdb_relationshipsemits species-rankGTDB↔NCBITaxonedges and records the configureddownload URL in the concord provenance.
get_gtdb_bac120/get_gtdb_ar53rules (retries: 3) viapull_via_wget;gtdb_metadata_url_prefixinconfig.yamldefaults to GTDBreleases/latest/and should be pinnedfor a reproducible build.
gtdb_labels/gtdb_ids/get_gtdb_relationshipsintaxon.snakefile;GTDBadded totaxon_labels/taxon_ids/taxon_concords. GTDB joins the existingOrganismTaxon.txt(no newcompendium file), so the
util.pyaggregators andreport_tables.pyneed no change.Design (load-bearing)
extra_prefixes=[GTDB]: GTDB is not in the Biolink Model'sorganism taxonid_prefixes(
[NCBITaxon, MESH, UMLS], verified against the pinned 4.4.3), sowrite_compendiumwould silently dropevery GTDB CURIE; the taxon build passes
extra_prefixes=[GTDB](the documented escape hatch). FilingGTDB with the Biolink team is the long-term fix.
GTDBinunique_prefixes: the GTDB↔NCBI species mapping is many-to-one, so two GTDB species sharingone
ncbi_species_taxidmust not merge. ListingGTDBas unique makesglomreject that merge — at mostone GTDB species joins a given NCBITaxon clique (first-in-file-order), the rest stay singletons.
GTDB; Bioregistry's preferred prefix is lowercasegtdb.GTDB is not yet in the Biolink Model, so there is no conflict — reconcile if it is registered upstream.
Docs
docs/sources/GTDB/README.md(new) + an entry in thedocs/sources/index.Testing
uv run pytest -m unit tests/datahandlers/test_gtdb.py -q --no-cov→8 passed— over a verbatim113-column GTDB fixture: taxonomy parsing (spaces→underscores, label spacing), one-label-per-taxon + dedup,
species-only
GTDB↔NCBITaxonconcord, by-name column lookup, gzipped input, missing-value skips, and aglom-level test pinning the many-to-one guard (two GTDB species sharing an NCBI taxid stay two cliques).
uv run pytest -m network --network tests/datahandlers/test_gtdb.py -q --no-cov→1 passed—create_nodekeeps a GTDB CURIE with
extra_prefixes=[GTDB]and drops it without (pins theextra_prefixesdecision).uv run pytest -m unit -q(full CI gate) →410 passed.uv run snakemake -c1 taxon -n --quiet→ DAG builds (all 5 GTDB rules wired);ruff/snakefmt/rumdl→ clean.
Caveats / Deferred
no impact-report hook (
PIPELINE_CONFIGregisters onlyanatomy/disease); registering one + a build-vs-buildclique diff is the recommended follow-up, since the many-to-one design is exactly what that scrutiny is for.
Questions for the reviewer
the right tradeoff, or should species-rank
GTDB↔NCBITaxonlinks be omitted until a 1:1 mapping is available?