diff --git a/protocols/shannon-diversity/protocol.md b/protocols/shannon-diversity/protocol.md new file mode 100644 index 0000000..748f3c8 --- /dev/null +++ b/protocols/shannon-diversity/protocol.md @@ -0,0 +1,149 @@ +--- +name: "shannon-diversity" +description: "Calculate the Shannon diversity index for each sample in a feature-by-sample abundance table." +version: "1.0.0" +authors: + - name: "Otto Infield-Harm" + orcid: "0009-0009-8629-6124" + - name: "Levi Waldron" + orcid: "0000-0003-2725-0694" +reviews: + - name: "Levi Waldron" + orcid: "0000-0003-2725-0694" + date: "2026-10-27" + protocol_version: "1.0.0" + status: approved +date: "2026-10-26" +status: draft +type: "atomic" + +artifact_doi: ~ +collection_doi: "10.5281/zenodo.22731694" +protocol_citation: "10.1002/j.1538-7305.1948.tb01338.x" +method_origin_citation: "10.1002/j.1538-7305.1948.tb01338.x" +license: "CC-BY-4.0" +protocols_used: [] +upstream_repositories: + - "https://github.com/vegandevs/vegan" + - "https://github.com/scikit-bio/scikit-bio" +database_urls: [] +key_packages: + - "vegan" + - "scikit-bio" +category: "Statistical Analysis" +tags: + - alpha-diversity + - shannon + - microbiome +--- + +# Shannon diversity of a single sample + +Calculate one Shannon diversity value for each sample in a feature-by-sample abundance +table. The calculation is descriptive: this protocol does not rarefy, subsample, or +otherwise normalize samples to a common sequencing depth, and it does not compare groups, +test hypotheses, or interpret the resulting values. + +## Materials + +- **Input data:** + - `data` — a table with one row per feature and one column per sample. The first + column must contain unique, non-missing feature identifiers, and there must be at + least one sample column. Every other cell must be a non-negative number, and no cell + may be missing: a missing value must not be imputed, converted to zero, or omitted. + - When `input_scale` is `proportions`, each sample column must sum to 1 within an + absolute tolerance of $10^{-6}$. A sample outside that tolerance is not renormalized. + - When `input_scale` is `counts`, samples should be rarefied or otherwise normalized to + a common sequencing depth, so that every sample column has the same total. If the + totals differ, proceed, and warn that the Shannon values depend on sequencing depth + and are not comparable across samples, and advise rarefying to a common depth before + comparing them. Proportions carry no depth, so this condition cannot be checked on + them. +- **Parameters:** + - `input_scale` — whether `data` holds `counts` or `proportions`. Required, with no + default, because the scale changes Step 1 and cannot be told reliably from the values. + It must be one of those two words. + - `log_base` — the base $b$ of the logarithm in Step 2. The default is `e`, the natural + logarithm, which gives the index in nats; `2` gives bits. It must be `e` or a number + greater than 1. +- **Software & Repositories:** + - `vegan` (https://github.com/vegandevs/vegan) — R package; `diversity()` computes + Step 2. + - `scikit-bio` (https://github.com/scikit-bio/scikit-bio) — Python package; + `skbio.diversity.alpha.shannon()` computes Step 2. + - Either is optional. Any implementation that follows Steps 1 and 2 is acceptable. + +## Steps + +### Step 1: Convert abundances to proportions + +For each sample independently, let $x_{ij}$ be the abundance of feature $i$ in +sample $j$. + +For `counts`, calculate + +$$ +p_{ij} = \frac{x_{ij}}{\sum_{i} x_{ij}}. +$$ + +If a count sample's total is zero, its proportions are undefined: stop, report which +sample it is, and produce no output. For `proportions`, use the values as given: +$p_{ij} = x_{ij}$. + +Zero-valued features remain part of the table but make no contribution to the index. + +### Step 2: Calculate Shannon diversity + +For each sample $j$, calculate + +$$ +H_j = -\sum_{i:p_{ij}>0} p_{ij}\log_b(p_{ij}), +$$ + +where $b$ is `log_base`. Terms with $p_{ij}=0$ are omitted, because their limiting +contribution is zero. + +The same calculation, for a feature-by-sample matrix `x` with feature identifiers as row +names: + +- R: `vegan::diversity(x, index = "shannon", MARGIN = 2, base = exp(1))`. `MARGIN = 2` + treats columns as samples; the function's default, `MARGIN = 1`, treats rows as samples. +- Python: `skbio.diversity.alpha.shannon(x[:, j], base=math.e)` for each sample column `j`. + +Replace `exp(1)` or `math.e` with $b$ when `log_base` is not `e`. + +### Step 3: Return the result + +Create a table with exactly two columns, `sample_id` and `shannon`, and one row +for each input sample. Preserve the input sample-column order. Do not add feature +summaries, group labels, confidence intervals, tests, plots, or interpretations. +Record `input_scale`, `log_base`, and any warning given under Materials alongside the +execution metadata, not as additional rows or columns in this output table. + +## Notes + +The Shannon index was introduced by Shannon (1948), which is cited as both the +source of this protocol's calculation and the method origin. Manghi et al. +(*Nature Communications*, 2025, DOI: 10.1038/s41467-025-66888-1) is an application +of the method, not the method-origin citation. + +Both `vegan::diversity()` and `skbio.diversity.alpha.shannon()` divide each sample by its +own total, so they accept counts and proportions alike; neither checks the conditions in +Materials, which must be checked first. `vegan::diversity()` returns `NA` for a sample +with a missing value rather than stopping. Pass the logarithm base explicitly to both: +the default of `shannon()` has differed between scikit-bio releases. + +## History & Reviews + + +### Version 1.0.0 (2026-10-26) + +#### Changes +- Initial protocol for calculating per-sample Shannon diversity. + +#### Reviews + +**Review by Levi Waldron ([0000-0003-2725-0694](https://orcid.org/0000-0003-2725-0694))** +- **Date:** 2026-10-27 +- **Status:** `approved` +- **Notes:** Review given on waldronlab/agent-protocols#53. diff --git a/protocols/shannon-diversity/tests/counts_and_zeros/expected.csv b/protocols/shannon-diversity/tests/counts_and_zeros/expected.csv new file mode 100644 index 0000000..15d070b --- /dev/null +++ b/protocols/shannon-diversity/tests/counts_and_zeros/expected.csv @@ -0,0 +1,3 @@ +sample_id,shannon +sample_a,0.6931471805599453 +sample_b,0.5623351446188083 diff --git a/protocols/shannon-diversity/tests/counts_and_zeros/input.csv b/protocols/shannon-diversity/tests/counts_and_zeros/input.csv new file mode 100644 index 0000000..69ec556 --- /dev/null +++ b/protocols/shannon-diversity/tests/counts_and_zeros/input.csv @@ -0,0 +1,4 @@ +feature_id,sample_a,sample_b +feature_1,1,1 +feature_2,1,3 +feature_3,0,0 diff --git a/protocols/shannon-diversity/tests/counts_and_zeros/test.yaml b/protocols/shannon-diversity/tests/counts_and_zeros/test.yaml new file mode 100644 index 0000000..714ee41 --- /dev/null +++ b/protocols/shannon-diversity/tests/counts_and_zeros/test.yaml @@ -0,0 +1,14 @@ +name: counts_and_zeros +description: Counts with unequal sample totals give per-sample Shannon values, zero-valued features contribute zero, and the executor warns that the samples were not rarefied. +inputs: + data: {path: input.csv} + input_scale: counts + log_base: e +evaluation: + mode: diagnostic + numeric_tolerance: 1.0e-12 + accepted_outcomes: + - outcome: computed_with_depth_warning + expected_outputs: + result: {path: expected.csv} + expected_warning: raref # matches rarefy, rarefied, rarefying, rarefaction diff --git a/protocols/shannon-diversity/tests/missing_values/input.csv b/protocols/shannon-diversity/tests/missing_values/input.csv new file mode 100644 index 0000000..5a1e92c --- /dev/null +++ b/protocols/shannon-diversity/tests/missing_values/input.csv @@ -0,0 +1,4 @@ +feature_id,sample_a,sample_b +feature_1,1,1 +feature_2,,3 +feature_3,0,0 diff --git a/protocols/shannon-diversity/tests/missing_values/test.yaml b/protocols/shannon-diversity/tests/missing_values/test.yaml new file mode 100644 index 0000000..7562751 --- /dev/null +++ b/protocols/shannon-diversity/tests/missing_values/test.yaml @@ -0,0 +1,11 @@ +name: missing_values +description: Missing abundance values must stop the calculation rather than be imputed or treated as zero. +inputs: + data: {path: input.csv} + input_scale: counts + log_base: e +evaluation: + mode: diagnostic + accepted_outcomes: + - outcome: refused_for_missing_values + expected_absent: [result]