Unknown Unknowns (UU)

Unknown Unknowns is a bioinformatics research project that quantifies taxonomic bias in protein and genome databases — the gap between what is observed (sequenced, assembled, annotated) and what is expected given the described diversity of life. Across the tree of life it measures over- and under-representation of clades in UniProt/UniRef and in genome assemblies (GoaT/ToLQC), tracks protein-existence evidence (PE1–5), and assesses assembly quality (BUSCO completeness, Merqury QV, N50), all anchored to a shared NCBI taxonomy backbone. The whole project ships as one installable package (unknown_unknowns) exposed through a single uu command-line entry point.

Install & quickstart

The project uses uv with a pinned Python 3.13.

# from the repo root
uv sync                 # create .venv and install the project (editable); provides the `uu` command
uv run uu --help        # top-level command groups (one per science pillar + `run`)
uv run uu run demo      # smoke-test the whole toolchain on committed example data

uv run uu run demo parses the tiny committed idmapping dump under data/examples/, so it works without any of the multi-GB external datasets. Once the environment is active you can drop the uv run prefix and call uu … directly.

Optional heavier tooling (Jupyter, notebooks, and the PyQt6 GUI stack used for tree rendering, plus test/lint tools) lives in a separate group:

uv sync --group dev     # jupyter + notebook + PyQt6 + pytest + ruff
uv run pytest           # per-pillar import smoke tests

uv sync gives you the same unknown_unknowns package that the notebooks import, so library code and exploratory work never drift.

Command map

Everything is invoked as uu <pillar> <command>. Run uv run uu <pillar> <command> --help (or -h) for the authoritative flag list. Most commands need external multi-GB inputs (see Data policy); the notes below call out which ones run on committed data or a live API instead.

uu taxonomy — NCBI backbone, Newick trees, iTOL colouring

The shared substrate every other pillar denominates against (via taxopy/taxonkit).

Command	Purpose	Key options
extract-rank RANK	Emit every NCBI taxon of a rank to a text file	-t/--type names|ids, -o
newick	Build a Newick tree from the backbone down to a rank	--max-rank, --include-counts, -o
color-itol NEWICK	Write an iTOL TREE_COLORS file colouring a tree by rank	-r/--target-rank, -c/--color-file, --list-ranks, -o
extract-names NEWICK	List unique taxon/leaf names from a Newick file	--format lines|list, -o

uv run uu taxonomy extract-rank phylum --type names

Reads the NCBI taxdump backbone (gitignored, ~2.8 GB — fetch externally) and writes the phylum list under data/processed/.

uu representation — log2(Observed/Expected) census per rank

Command	Purpose	Key options
log2fc	log2 fold change (Observed/Expected) of protein counts per rank, with binomial 95% CI	--data, --counts, --database, --rank (required); --count-level species|genus, --parent-type/--parent-value, --compare/--compare-database, -o

uv run uu representation log2fc \
  --data data/processed/proteins_per_taxon.tsv \
  --counts data/processed/species_genus_counts.tsv \
  --database unique_uniref90 \
  --rank phylum

The two input TSVs are derived per-taxon tables that are gitignored — regenerate or fetch them before running. Output is a horizontal bar plot under reports/figures/.

uu proteomics — UniProt/UniRef mapping, PE evidence, protein counts

Command	Purpose	Key options
idmapping	Group UniProt accessions/taxa by UniRef cluster	-i/--input (required), -c/--cluster uniref90|uniref50, -o
histogram TSV	Log-scaled histogram of per-proteome protein counts, split by domain	-o, --processed-output
estimated-proteins	Concentric circles comparing sampled vs. estimated protein counts	--swissprot, --uniprotkb, --known-species, --estimated-species, -o
phylo-tree JSON	Publication-quality phylogenetic tree with described-vs-undescribed bars	--bar-mode percentage|absolute, --debug-tree, -o

uv run uu proteomics idmapping \
  -i data/examples/idmapping_selected.tab.example \
  --cluster uniref50

This example runs on committed data. On the full dataset, point -i at the ~9.7 GB idmapping_selected.tab dump (fetch externally).

uu assembly — GoaT/ToLQC inventory, quality metrics, fetchers

Command	Purpose	Key options
goat	Search GoaT for taxon assemblies → taxon/assembly TSV	-q/--query, --result, --fields, --size, -o
tolqc	Scrape the ToLQC catalogue for per-species metrics	(see --help)
genome-stats-collect	Collect per-taxon genome sequencing statistics from GoaT	(see --help)
genome-stats-plot	Render the status/quality/percentage summary plots from a stats CSV	(see --help)
fetch-assembly	List (and optionally download) NCBI Assembly FTP links for a taxon	--taxon, --email, --download/--no-download, -o
fetch-ena	Fetch assembly metadata, XML, and raw-read records from ENA	(see --help)
fetch-taxa	Fetch all NCBI taxa of a rank under a parent taxon (with lineage)	--parent, --rank, --email, -o

uv run uu assembly goat --query "tax_tree(Aves)"

The goat/tolqc/genome-stats-collect commands hit live public APIs (network required, no local bulk data). The NCBI fetchers require an --email address per NCBI policy.

uu data — inspect and fetch the large external datasets

The multi-GB inputs are gitignored (see Data policy); this group automates the download recipes so they can be pulled on demand.

Command	Purpose	Key options
list	Show every dataset with size, fetch method, and whether it's present locally	—
path KEY	Print the local path a dataset belongs at	—
fetch KEY	Download a dataset (KEY or all for the http ones)	--dry-run, --force

uv run uu data list                       # what's needed and what's already here
uv run uu data fetch taxdump              # stream + extract the NCBI taxdump (~2.8 GB)
uv run uu data fetch all --dry-run        # preview every http download without fetching

http datasets (taxdump, UniProt taxonomy/proteomes, idmapping) stream to their local path with a progress bar; genomeark-qv uses aws s3 sync (needs the AWS CLI); goat/tolqc point you at the uu assembly commands that produce them. Live endpoints serve the current release, not the pinned snapshot dates.

uu run pipelines

uu run chains pillar commands into end-to-end steps.

Command	Purpose	Key options
taxonomy-tree	Build an NCBI Newick tree and (optionally) its iTOL colouring in one step	--max-rank, -r/--target-rank, -o
demo	Smoke-test the toolchain end to end on committed data/examples/ inputs	—

uv run uu run taxonomy-tree --max-rank class --target-rank phylum

taxonomy-tree builds the tree from the NCBI backbone (needs the taxdump) and, when --target-rank is given, also writes the matching iTOL TREE_COLORS file. demo needs no external data.

Repository layout

Path	Contents
src/unknown_unknowns/	Installable package (import unknown_unknowns.*, editable via uv sync) with the uu CLI entry point (unknown_unknowns.cli:app). Cross-cutting infra in common/ (central data paths, shared I/O); one subpackage per science pillar: taxonomy/, representation/, proteomics/, assembly/; multi-step wiring in pipelines/. Each pillar has a README naming its Hero: module; non-working code is quarantined under experimental/.
notebooks/	Exploratory and figure-making notebooks, mirroring the pillars. Not import targets — they consume uu.
reports/	Curated, shareable, small results that are committed: figures/, tables/, and itol/ tree artifacts.
docs/	Narrative knowledge base: methods, references, provenance, the handover_inventory.md, and legacy READMEs preserved for attribution.
data/	Data root. Bulk inputs are gitignored; only curated small inputs are committed via a whitelist (data/reference/, data/examples/). See data/README.md for provenance and external-download recipes.
out/	Gitignored, regenerable pipeline scratch outputs.
tests/	CI-friendly smoke tests: per-pillar import checks (example-driven runs planned).

Root files: pyproject.toml + uv.lock (merged deps, hatchling packaging src/unknown_unknowns), .python-version (3.13), LICENSE (MIT, code), CITATION.cff, and .gitignore (whitelist data policy).

Snapshot dates

Every dataset in this project is a dated snapshot. Results are only interpretable against the snapshot they were computed from.

Dataset	Snapshot date	Notes
UniProt proteomes	2025-06-11	proteomes-per-taxon, protein counts
UniProt taxonomy stats	2024-05-30	per-taxon UniProt taxonomy table
GoaT assembly pulls	2025-05-26 & 2025-06-06	assembly inventory / genome-size API pulls
idmapping / UniRef mapping	undated (pre 2025-06-11)	idmapping → UniRef90/50 → taxon
NCBI taxdump	2025-09-03 / 2025-09-04	taxonomy backbone (main repo)
into_the_unknown (separate study)	UniProt 2024_06	out of scope for this repo — see below

Data policy

This is a public repository, so no multi-GB data and no secrets are ever committed. Bulk inputs — the ~9.7 GB UniRef mapping tables, the ~3.6 GB UniProt/UniRef count tables, the ~2.8 GB NCBI taxdump, bulk proteome TSVs, FASTAs, and third-party published trees (TimeTree v5, Hug-2016 New Tree of Life) — are all gitignored and must be fetched externally.

What is committed is a deliberately small, whitelisted set: the project's own hard-to-reproduce GoaT/ToLQC API snapshots, the Chapman-2009 diversity denominators, tiny toy inputs under data/examples/ (so uu run demo and the tests run without the originals), and curated figures/tables under reports/. Derived tables (e.g. *_withTaxPath*, lineage_cache.tsv) are gitignored because they regenerate from the pillar code.

See data/README.md for the full provenance table and the external-download recipes (snapshot URLs, aws s3 sync --no-sign-request s3://genomeark …, taxdump download, etc.).

Attribution & provenance

Authors: Constantin Carl, Tobias Senoner, Ivan Koludarov, Zeno Sewald.

Much of the representation (log2 Observed/Expected) analysis and the assembly/genomics analysis was authored by Constantin Carl during his time as a HiWi, and is consolidated here with attribution preserved. His contributions are attributable three ways: each pillar README folds in his original per-directory READMEs; docs/legacy/ROSTLAB_hiwi_README.md preserves his original top-level README verbatim; and docs/provenance.md + CITATION.cff carry the full per-area authorship and snapshot map. See docs/provenance.md for the file-by-file breakdown.

The separate into_the_unknown study (protein-language-model blind spots, UniProt 2024_06) is not part of this repository and lives at github.com/tsenoner/into_the_unknown.

Known gaps

These are acknowledged limitations of the current consolidation — intentionally surfaced so a collaborator does not mistake absence for oversight:

• No realized-observation layer. There is no PRIDE / PeptideAtlas integration; "observed" currently means present in a sequence database, not experimentally detected at the protein level.
• PE1–5 table is hand-entered and inconsistent. The protein-existence counts feeding the Sankey (notebooks/proteomics/sankey_data.yaml) were transcribed by hand and are not internally reconciled across sources (the SwissProt column is reversed between the YAML and the notebook).
• No unified assembly-quality table. BUSCO, Merqury QV, N50, and completeness live in separate notebooks; there is no single joined, canonical assembly-quality table, and L50 was never computed.
• log2(O/E) not produced at genus rank. The over/under-representation census is available at higher ranks (superkingdom/kingdom/phylum) but the genus-rank result has not been generated (uu representation log2fc supports --count-level genus, but the genus-rank figure has not been produced).

License

Code is released under the MIT License (LICENSE). Committed data snapshots retain their upstream sources' terms; third-party datasets are documented but not redistributed (see data/README.md).