Create summary tables

[erikrikarddaniel]

Description of feature

In magmap and metatdenovo we're producing summary tables in <OUTDIR>/summary_tables that are gzipped tsv tables with consistent column names (lower case only) ready for analysis in R, Python or whichever tool one prefers. For Ampliseq, this would mean producing two tables: <OUTDIR>/summary_tables/ampliseq.counts.tsv.gz, ASV counts in "long" format without zeroes, i.e. missing ASVs in a particular sample, and <OUTDIR>/summary_tables/ampliseq.taxonomy.<tool><db>.tsv.gz, where <tool> is the annotation tool, e.g. dada_add_species or phyloplace, <db> is the name (including version number) of the annotation database.

In an ideal world, I'd also like to consolidate taxonomies, i.e. create a table that for each ASV selects the best scoring taxonomy from a number of different databases run with the same tool. I can see two ways of doing this: Either allow the database flags, e.g. --dada_ref_taxonomy to take a comma-separated list of databases or let the pipeline check for existing taxonomies when invoked with --consolidate_taxonomies. I think the latter would be easiest if a bit unorthodox that the pipeline reads its own output directory.

I'm ready to work on a PR for this that we can take under consideration later.

[erikrikarddaniel]

For quite a while, I've wanted to have the possibility to specify multiple databases as a list, e.g. --dada_ref_taxonomy gtdb,silva,pr2. I've discussed with my programming partner Claude ;-), and it feels like it wouldn't be that bad in terms of refactoring. A thing I'm uncertain of however, is how this will affect the QIIME2 steps. I asked Claude again:

"""

• ch_tax_tsv (the multi-classifier channel) already coexists with multiple entries today — subworkflows/local/qiime2_export.nf:29's COMBINE_TABLE already does ch_tax_tsv.combine(...), fanning out once per classifier already present in the channel. No rework needed here; this is the channel your new taxonomy-table module would also key off, same pattern.
• ch_tax (a single QIIME2-native .qza artifact) is a different story: workflows/ampliseq.nf:1039-1079 picks it via an if/elif priority chain across classifiers — DADA2 beats SINTAX beats KRAKEN2 beats QIIME2 beats VSEARCH-LCA, first match wins, exactly one classifier's result becomes "the" taxonomy. Every real downstream QIIME2 analysis step hangs off this single value: taxon-based table filtering (QIIME2_TABLEFILTERTAXA, line 1083), rank-agglomerated exports (QIIME2_EXPORT_ABSOLUTE/RELTAX), barplots (QIIME2_BARPLOT/BARPLOTAVG), diversity (QIIME2_DIVERSITY), and ANCOM/ANCOMBC/ANCOMBC2 (QIIME2_ANCOM) — all of these fundamentally need one unambiguous per-ASV taxonomy to agglomerate/filter/test by, so they can't just "run N times" the way COMBINE_TABLE does.

That changes the sequencing question from last time: if multiple databases for the same classifier are allowed, there's currently no mechanism to pick which one feeds ch_tax — so consolidation (or an explicit "use this one for downstream analysis" selector) becomes a co-requisite for list-of-databases to reach the real QIIME2 analysis, not something that can trail along afterward for free. The "easy" version of list-of-databases is one that only enriches the new taxonomy summary tables and leaves ch_tax/QIIME2 analysis exactly as it works today (still one classifier, still one database) — that's a meaningfully smaller scope than making multiple databases actually drive filtering/ANCOM/diversity/barplot.
"""

One option would be to consolidate taxonomies, i.e. for each ASV choose the one that's best according to the method's score, and pass that to ch_tax.

[d4straub]

summary_tables

All for it.

consolidate taxonomies,

not sure how to score that, but why not, sounds not bad.

multiple databases

Yes, I think it would be good to be able to choose several databases for one classification tool. Until now only several tools with one database each, which is indeed limiting sometimes.

However downstream analysis with QIIME2 is then weird. By default, I think only one taxonomy should enter the QIIME2 step. Optimally, one taxonomy would be picked automatically when (as currently) only one database per tool is selected. We could change however that its possible with an additional param or such to override which taxon table enters downstream analysis, or a "best" for each ASV, as you describe. Not sure how that would look like though. Would need some more thoughts...

[erikrikarddaniel]

Updated scope (after discussion between maintainers, 2026-08-04)

Scope has grown from the original two-table ask to include multi-database support and taxonomy
consolidation — both now confirmed in-scope for this release, not stretch goals. Splitting the work
into two PRs, same release:

PR A — ampliseq.counts.tsv.gz. Long-format ASV counts (asv_id, sample, count, zero rows
dropped), sourced from DADA2's own raw merged/chimera-filtered output (not the final, heavily
filtered ASV table). No dependency on anything below — ships first.

PR B — list-of-databases + taxonomy consolidation + ampliseq.taxonomy.<classifier>.<database>.tsv.gz, combined into one PR. These three can't really be split further: the moment a classifier
(DADA2/SINTAX/KRAKEN2/QIIME2/VSEARCH-LCA; PPLACE is excluded, being tree- not database-based) can be
run against more than one reference database, something has to decide which result feeds the actual
downstream QIIME2 analysis (table filtering, ANCOM, diversity, barplots) — that's consolidation, so
it has to land alongside list-of-databases, not after it. Adding the per-classifier/per-database
taxonomy summary table on top of that same plumbing is then a small addition.

Both tables also get a Parquet sibling by default, alongside .tsv.gz (opt-out via a hidden
--skip_parquet_summary flag, for anyone who doesn't want the extra output).

KRAKEN2 is excluded from the taxonomy table (its output shape is the most different of the group,
and little-used); PPLACE ranks are inferred positionally (1st..7th field → kingdom..species) since
its taxonomy strings aren't rank-labeled.

Consolidation: three candidate algorithms — which one(s) do we want?

1. First-in-list wins — simplest, fully deterministic, just uses whichever database was listed
   first for a classifier. No real per-ASV logic; a reasonable placeholder if we want PR B to land
   before the other two algorithms are fully designed.
2. Best-scoring per ASV — for classifiers with a real confidence value (DADA2, SINTAX, QIIME2),
   highest confidence wins per ASV; for the rest (VSEARCH-LCA), deepest resolved rank wins, ties
   broken by database order. Note: this is currently weakened for DADA2 because only the
   most-specific rank's bootstrap confidence is exported today, not per-rank — see the separate issue
   below.
3. Decider database — one nominated database's classification at a coarse rank routes each ASV to
   a different database's result, rather than scoring. Example: use SILVA as the decider — if it
   calls an ASV eukaryotic, use PR2's result for that ASV; if bacterial/archaeal, use SBDI-GTDB's
   result. Motivated by databases like SILVA covering the full diversity breadth while others
   (SBDI-GTDB, GTDB, PR2) are lineage-restricted, so a broad-coverage database can be a good router
   even where it isn't the most precise classifier for every lineage.

Open question — needs explicit sign-off

Does the consolidated result become the new default input to the downstream QIIME2 analysis chain
(replacing today's implicit "first classifier in a fixed priority order wins" behavior), whenever
--consolidate_taxonomies is active? We think this was the gist of what we discussed, but want to
confirm explicitly before implementing, since it changes real analysis output, not just adds a new
file.

Related, separate issue

Filing a follow-up issue: DADA2's assignTaxonomy computes bootstrap confidence for every rank
internally, but the pipeline currently only exports the value for the single most-specific resolved
rank. Exposing all per-rank values would improve consolidation algorithm 2 above. Separable from this
work.

[erikrikarddaniel]

Follow-up: consolidation algorithm details, decided

Digging deeper into the three consolidation candidates above turned up a few sub-choices each one
actually needs before it's buildable. Decisions below — these will go into docs/usage.md once
implemented, since they materially affect analysis output, not just which file gets written.

1. First-in-list wins → priority cascade, not strict whole-row.
Strict whole-row first-wins would ignore every other listed database for any ASV the first database
classifies at all, even partially — wasting the point of listing more than one. Going with cascade
instead: per ASV, walk ranks kingdom→species, take the first-listed database's call at each rank if
it isn't NA, otherwise fall through to the next-listed database's call at that same rank. Still
fully deterministic, no confidence math — but now per-rank rather than per-row, so it needs the same
rank-harmonization/canonical-order handling as algorithm 2. This is what PR B will implement first.

2. Best-scoring per ASV → whole-row, not per-rank stitching.
Per-rank stitching (best-scoring call independently at each rank) risks producing an internally
inconsistent taxonomy string — e.g. a genus from database A that database A itself never placed
under the family database B reported. Going with whole-row: the entire winning database's row is
used for that ASV. Scoring: highest confidence wins for classifiers with a real confidence value
(DADA2, SINTAX, QIIME2); deepest resolved rank wins otherwise (VSEARCH-LCA), ties broken by database
declaration order. Caveat for the docs: bootstrap confidence isn't fully database-agnostic — a
smaller/coarser reference set tends toward systematically higher confidence purely from less internal
ambiguity, independent of whether it's actually the better call. Also still weakened today by DADA2
only exporting the most-specific rank's confidence, not per-rank (see the separate DADA2 issue).

3. Decider database → strictly within one classifier, never across classifiers.
We clearly agreed on this: never compare or consolidate anything between classifiers. So the
SILVA→PR2/SBDI-GTDB routing example only applies when all three databases are run through the same
classifier (e.g. all via DADA2) — it does not mean routing between, say, a QIIME2 run and a DADA2 run.
This algorithm also isn't fully independent of 1/2: it needs a fallback to one of them for ASVs the
decider itself can't place at the routing rank, or that the routed-to database has no entry for.
Lowest build priority of the three — planned as a genuine follow-up once 1 and 2 exist.

Build order for PR B: algorithm 1 (cascade) first — deterministic, no comparability caveats to
document, still uses every listed database. Algorithm 2 (whole-row) in the same PR if we want a real
second option at launch, since it reuses the rank-harmonization work cascade already needs. Algorithm
3 (decider, within-classifier only, with fallback) as a later follow-up.

[d4straub]

1. Above is sometimes written deterministic, but DADA2's taxonomic classifier isnt deterministic itself (see benjjneb/dada2#1115), therefore results downstream from it cannot be deterministic as well. This doesnt make anything invalid, just the docu should reflect that. Maybe in general we should add somewhere in the docu what classification tool is deterministic and which one isnt, I think thats not there yet.

2. Regarding

Does the consolidated result become the new default input to the downstream QIIME2 analysis chain
(replacing today's implicit "first classifier in a fixed priority order wins" behavior), whenever
--consolidate_taxonomies is active? We think this was the gist of what we discussed, but want to
confirm explicitly before implementing, since it changes real analysis output, not just adds a new
file.

Yes.
I think that wasnt explicitly mentioned above, but --consolidate_taxonomies shall take several options, right? i.e. --consolidate_taxonomies "cascade" default [= 1. First-in-list wins → priority cascade, not strict whole-row.], other options could be "score" [=2. Best-scoring per ASV → whole-row, not per-rank stitching. → strictly within one classifier, never across classifiers.], and "decider" [=3. Decider database]. I think the default should be the simplest of all solutions which seems "cascade".

3. Regarding

We clearly agreed on this: never compare or consolidate anything between classifiers.

I think all consolidation options could be working across classification tools and databases, except "score" that must be limited to across databases within a specific tool due to comparability issues of the scoring (but are we sure that its not comparable? I am not, was just a though without clear base on my side).

4. The consolidation algorithm details sound fine to me. But "score" [=2. Best-scoring per ASV → whole-row, not per-rank stitching.] is not entirely clear to me:

Going with whole-row: the entire winning database's row is
used for that ASV. Scoring: highest confidence wins for classifiers with a real confidence value
(DADA2, SINTAX, QIIME2); deepest resolved rank wins otherwise (VSEARCH-LCA), ties broken by database
declaration order.

Is this now "score" or deepest rank based? It seems to me that you propose a mixture? But wouldnt these be both valid independent approaches? I.e. either score-based (requires score comparability, i.e. likely within classification tools) or classification depth (requires same number/ranks)

Also, "decider" could refer to barrnap classification instead of to the first database of the classification tool, i.e. barrnap could be a classification-tool-independent decider? Also, it seems to me it doesnt need the other options as fall-back, but its acceptable if it does.