Skip to content

feat: add GTDB as a biolink:OrganismTaxon source in the taxon pipeline - #978

Open
SkyeAv wants to merge 2 commits into
mainfrom
feat/gtdb-taxon
Open

feat: add GTDB as a biolink:OrganismTaxon source in the taxon pipeline#978
SkyeAv wants to merge 2 commits into
mainfrom
feat/gtdb-taxon

Conversation

@SkyeAv

@SkyeAv SkyeAv commented Jul 28, 2026

Copy link
Copy Markdown
Member

Adds GTDB (Genome Taxonomy Database, bacterial/archaeal taxonomy) as a
biolink:OrganismTaxon source in the existing taxon pipeline. Every rank in each genome's
gtdb_taxonomy string becomes a labeled GTDB CURIE (GTDB:s__Escherichia_coli), and species-rank
taxa are linked eq to their ncbi_species_taxid so a GTDB species resolves into the NCBI taxon clique.

Ingest

  • New handler: src/datahandlers/gtdb.py parses the bac120/ar53 metadata (columns read by
    name
    , robust to reordering). write_gtdb_labels emits one label per unique taxon (all ranks);
    build_gtdb_relationships emits species-rank GTDBNCBITaxon edges and records the configured
    download URL in the concord provenance.
  • Download: get_gtdb_bac120 / get_gtdb_ar53 rules (retries: 3) via pull_via_wget;
    gtdb_metadata_url_prefix in config.yaml defaults to GTDB releases/latest/ and should be pinned
    for a reproducible build.
  • Wiring: gtdb_labels / gtdb_ids / get_gtdb_relationships in taxon.snakefile; GTDB added to
    taxon_labels / taxon_ids / taxon_concords. GTDB joins the existing OrganismTaxon.txt (no new
    compendium file), so the util.py aggregators and report_tables.py need no change.

Design (load-bearing)

  • extra_prefixes=[GTDB]: GTDB is not in the Biolink Model's organism taxon id_prefixes
    ([NCBITaxon, MESH, UMLS], verified against the pinned 4.4.3), so write_compendium would silently drop
    every GTDB CURIE; the taxon build passes extra_prefixes=[GTDB] (the documented escape hatch). Filing
    GTDB with the Biolink team is the long-term fix.
  • GTDB in unique_prefixes: the GTDB↔NCBI species mapping is many-to-one, so two GTDB species sharing
    one ncbi_species_taxid must not merge. Listing GTDB as unique makes glom reject that merge — at most
    one GTDB species joins a given NCBITaxon clique (first-in-file-order), the rest stay singletons.
  • Prefix casing: Babel uses display-cased GTDB; Bioregistry's preferred prefix is lowercase gtdb.
    GTDB is not yet in the Biolink Model, so there is no conflict — reconcile if it is registered upstream.

Docs

  • docs/sources/GTDB/README.md (new) + an entry in the docs/sources/ index.

Testing

  • uv run pytest -m unit tests/datahandlers/test_gtdb.py -q --no-cov8 passed — over a verbatim
    113-column GTDB fixture: taxonomy parsing (spaces→underscores, label spacing), one-label-per-taxon + dedup,
    species-only GTDBNCBITaxon concord, by-name column lookup, gzipped input, missing-value skips, and a
    glom-level test pinning the many-to-one guard
    (two GTDB species sharing an NCBI taxid stay two cliques).
  • uv run pytest -m network --network tests/datahandlers/test_gtdb.py -q --no-cov1 passedcreate_node
    keeps a GTDB CURIE with extra_prefixes=[GTDB] and drops it without (pins the extra_prefixes decision).
  • uv run pytest -m unit -q (full CI gate) → 410 passed.
  • uv run snakemake -c1 taxon -n --quiet → DAG builds (all 5 GTDB rules wired); ruff / snakefmt / rumdl
    → clean.

Caveats / Deferred

  • License: GTDB is CC-BY-SA-4.0 (share-alike) — confirm acceptable for downstream redistribution.
  • Coverage: species with no NCBI equivalent and all higher ranks form labeled singletons.
  • Comparison deferred (per maintainer): no source-impact report / clique diff yet. The taxon pipeline has
    no impact-report hook (PIPELINE_CONFIG registers only anatomy/disease); registering one + a build-vs-build
    clique diff is the recommended follow-up, since the many-to-one design is exactly what that scrutiny is for.

Questions for the reviewer

  • Many-to-one handling. Is "at most one GTDB species per NCBI taxid (first-in-file-order), rest singletons"
    the right tradeoff, or should species-rank GTDBNCBITaxon links be omitted until a 1:1 mapping is available?

AI-drafted: This PR was drafted by an AI coding agent (pi) at the maintainer's direction and is pending
human review. Verification commands above were run by the agent.

GTDB (Genome Taxonomy Database) provides a standardized bacterial/archaeal taxonomy.
Ingest it as a taxon vocabulary in the existing taxon pipeline: every rank in each
genome's gtdb_taxonomy string becomes a GTDB CURIE (GTDB:s__Escherichia_coli), labeled
with the taxon name, and species-rank taxa are linked eq to their ncbi_species_taxid so
a GTDB species resolves into the NCBI taxon clique.

- src/datahandlers/gtdb.py parses the bac120/ar53 metadata (columns read by name);
  write_gtdb_labels emits one label per unique taxon, build_gtdb_relationships emits
  species-rank GTDB<->NCBITaxon edges, and the concord provenance records the configured
  (pinnable) download URL.
- GTDB is not in the Biolink Model's organism-taxon id_prefixes, so the taxon build passes
  extra_prefixes=[GTDB] (the documented escape hatch). GTDB is also a unique_prefixes entry:
  the GTDB<->NCBI mapping is many-to-one, so two GTDB species sharing an NCBI taxid are never
  merged (at most one joins the clique, the rest stay singletons).
- New get_gtdb_bac120/get_gtdb_ar53 download rules (retries: 3) + gtdb_labels/gtdb_ids/
  get_gtdb_relationships; GTDB added to taxon_labels/ids/concords. It joins the existing
  OrganismTaxon.txt (no new compendium file), so the util.py aggregators and report tables
  need no change.

8 offline unit tests over a verbatim 113-column GTDB fixture (incl. a glom-level test pinning
the many-to-one guard) + 1 network test pinning extra_prefixes via create_node. Full unit suite
green (410 passed).
Deferred: source-impact report / clique diff (per maintainer); the taxon pipeline has no
impact-report hook yet (PIPELINE_CONFIG registers only anatomy/disease).
@SkyeAv SkyeAv self-assigned this Jul 28, 2026
@SkyeAv SkyeAv added Priority: Low documentation Improvements or additions to documentation enhancement New feature or request new data source A new data source to include in Babel discussion needed testing Related to the test suite or testing infrastructure labels Jul 28, 2026
astral-sh/ruff-action@v3 floats to the latest ruff, now 0.16.0 (pyproject
pins only a floor, >=0.14.14). Ruff 0.16.0 formats Python code fences inside
*.md, which overlaps with the dedicated rumdl Markdown check and was failing
'Check Python formatting with ruff' on every open PR. Exclude *.md from ruff
format so ruff stays scoped to Python and Markdown formatting CI is
deterministic across ruff releases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

discussion needed documentation Improvements or additions to documentation enhancement New feature or request new data source A new data source to include in Babel Priority: Low testing Related to the test suite or testing infrastructure

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant