Document GEO and RNA-seq ingestion pipelines, fix stale v109 references

README.md

@@ -87,106 +87,23 @@ and selects 3 real experiment datasets plus 4 demo predictors and TargetScan.
 
 ### 1. Add Experiment Data
 
-The experiment-ingestion pipeline creates DE tables under `data/experiments/processed/` for local
-workflow use.
+Experiment DE tables are produced by a dedicated ingestion pipeline that runs DESeq2 on a count
+matrix or on FASTQs (via FastQC + fastp + STAR + featureCounts), then syncs the result into
+`metadata/mirna_experiment_info.tsv`. Raw data can also be fetched directly from GEO and turned
+into a ready-to-review YAML config automatically. See the dedicated pipelines for the full
+workflow:
+
+- [GEO download pipeline](pipelines/geo/README.md) — fetch FASTQs/count matrices from GEO and
+  auto-generate a YAML config
+- [Experiment ingestion pipeline](pipelines/experiments/README.md) — run DESeq2 (count matrix or
+  reads mode) and sync results into the registry
 
 For the curated benchmark datasets tracked in `metadata/mirna_experiment_info.tsv`, the expected
 workflow is different: those metadata rows stay versioned in the repo, and the corresponding DE
 tables live under the local `data/experiments/processed/18745741/` cache. The repo currently ships
 the 3 default benchmark TSVs there, and other curated tables are fetched from Zenodo when needed.
 
-Experiment config summary:
-
-- top-level:
-  `dataset_id`, `mirna_name`, `experiment_type`, optional `gse`
-- `source`:
-  `mode: count_matrix` or `mode: reads`
-- `comparison`:
-  control vs treated columns or explicit control vs treated samples
-- `metadata`:
-  fields that will later be synced into `metadata/mirna_experiment_info.tsv`
-
-Supported inputs:
-
-- count matrix: counts matrix + control columns + treated columns -> DESeq2
-- reads: local FASTQs + local reference files + explicit sample groups -> `FastQC + fastp + STAR + featureCounts + DESeq2`
-
-This path expects the `funmirbench-experiments` environment from `pipelines/experiments/environment.yml` to be
-active so `fastqc`, `fastp`, `STAR`, `featureCounts`, and `Rscript` are available on `PATH`.
-
-Download the shipped real example inputs:
-
-```bash
-uv run funmirbench-experiments-download-examples
-```
-
-That downloader fetches:
-
-- the real `GSE253003` count matrix
-- the real `GSE93717` FASTQ files
-- the shared Homo sapiens Ensembl v109 genome FASTA and GTF used by the reads example
-
-Run the real count-matrix example:
-
-```bash
-uv run funmirbench-experiments --config pipelines/experiments/configs/gse253003.count_matrix.example.yaml
-```
-
-Run the reads example the same way:
-
-```bash
-uv run funmirbench-experiments --config pipelines/experiments/configs/gse93717.reads.example.yaml
-```
-
-Tracked example configs:
-
-- `pipelines/experiments/configs/gse253003.count_matrix.example.yaml`
-- `pipelines/experiments/configs/gse93717.reads.example.yaml`
-
-Reads configs can either:
-
-- use local `reads_1` and optional `reads_2`
-- use local `genome_fasta_path` and `gtf_path`
-
-So the practical reads flow is:
-
-1. activate `funmirbench-experiments`
-2. run `uv run funmirbench-experiments-download-examples`
-3. run `uv run funmirbench-experiments --config pipelines/experiments/configs/gse93717.reads.example.yaml`
-
-The shipped reads example now points at the downloaded Ensembl v109 reference source files under
-`data/experiments/raw/refs/ensembl_v109/`, so it builds the derived STAR index automatically.
-
-Each run writes:
-
-- `data/experiments/processed/<dataset_id>.tsv`
-- `pipelines/experiments/runs/<timestamp>_<dataset_id>/candidate_metadata.tsv`
-- `pipelines/experiments/runs/<timestamp>_<dataset_id>/run_manifest.json`
-
-The reads example uses a reproduced dataset id, `GSE93717_OE_miR_941_deseq2`, so syncing it creates
-a separate variant instead of overwriting the curated `GSE93717_OE_miR_941` registry row.
-
-### 2. Sync Experiment Metadata
-
-The ingestion pipeline does not edit `metadata/mirna_experiment_info.tsv` by itself. It writes a
-`candidate_metadata.tsv` under `pipelines/experiments/runs/<timestamp>_<dataset_id>/` first.
-Then sync it into the registry with:
-
-```bash
-uv run funmirbench-sync-metadata
-```
-
-That command auto-discovers all `candidate_metadata.tsv` files under `pipelines/experiments/runs/`
-and upserts them into the registry. Re-running is safe — existing rows with matching `id` values are
-replaced, not duplicated.
-
-To sync a specific file instead:
-
-```bash
-uv run funmirbench-sync-metadata --input pipelines/experiments/runs/<run_dir>/candidate_metadata.tsv
-```
-
-### 3. Add Predictor Data
+### 2. Add Predictor Data
 
 Predictor score files live under `data/predictions/` and are discovered through
 `metadata/predictions_info.tsv`.
@@ -219,7 +136,7 @@ that those build thresholds match the current `evaluation` config before running
 
 The demo predictors already have registry rows in `metadata/predictions_info.tsv`.
 
-### 4. Run The Benchmark
+### 3. Run The Benchmark
 
 The default benchmark config is `benchmark.yaml`.
 

pipelines/experiments/README.md

@@ -0,0 +1,176 @@
+# Experiment Ingestion Pipeline
+
+This pipeline turns raw experiment data (a pre-existing count matrix or FASTQs) into a
+benchmark-ready differential-expression (DE) table under `data/experiments/processed/`, using
+DESeq2 (`funmirbench/experiments_pipeline.py`). It is typically the second stage of the full
+experiment workflow, after the [GEO download pipeline](../geo/README.md) has produced a YAML
+config — though a config can also be written by hand.
+
+---
+
+## Overview of the full pipeline
+
+```
+0. (Optional) Run the GEO download pipeline   →   auto-generates a YAML config
+          ↓
+1. Review / write pipelines/experiments/configs/{dataset_id}.yaml
+          ↓
+2. Run funmirbench-experiments --config ...   →   data/experiments/processed/{dataset_id}.tsv
+          ↓
+3. Run funmirbench-sync-metadata   →   upserts row into metadata/mirna_experiment_info.tsv
+```
+
+---
+
+## Step 0 (Optional) - Acquire data from GEO
+
+Raw FASTQs or a count matrix, plus a ready-to-review YAML config, can be generated automatically
+from a GEO accession. See the [GEO download pipeline](../geo/README.md) for the full workflow.
+
+## Step 1 - Write or review the YAML config
+
+Experiment config summary:
+
+- top-level:
+  `dataset_id`, `mirna_name`, `experiment_type`, optional `gse`
+- `source`:
+  `mode: count_matrix` or `mode: reads`
+- `comparison`:
+  control vs treated columns or explicit control vs treated samples
+- `metadata`:
+  fields that will later be synced into `metadata/mirna_experiment_info.tsv`
+
+Supported inputs:
+
+- count matrix: counts matrix + control columns + treated columns -> DESeq2
+- reads: local FASTQs + local reference files + explicit sample groups -> `FastQC + fastp + STAR + featureCounts + DESeq2`
+
+Tracked example configs:
+
+- `pipelines/experiments/configs/gse253003.count_matrix.example.yaml`
+- `pipelines/experiments/configs/gse93717.reads.example.yaml`
+
+Reads configs can either:
+
+- use local `reads_1` and optional `reads_2`
+- use local `genome_fasta_path` and `gtf_path`
+
+---
+
+## Step 2 - Set up the environment
+
+```bash
+conda env create -f pipelines/experiments/environment.yml
+conda activate funmirbench-experiments
+```
+
+That environment also includes `uv`, so `uv run ...` keeps working after activation, and provides
+`fastqc`, `fastp`, `STAR`, `featureCounts`, and `Rscript` on `PATH`.
+
+---
+
+## Step 3 - Run the pipeline
+
+RNA-seq pipeline summary:
+
+- `count_matrix` mode validates the configured count matrix and runs DESeq2 directly on the selected
+  control and treated columns.
+- `reads` mode runs read QC/trimming, aligns reads to the configured reference with STAR, counts
+  genes with featureCounts, then runs the same DESeq2 step.
+- DE outputs preserve the original method-level `PValue` and `FDR` values. Benchmarking derives
+  `benchmark_FDR` and `plot_FDR` internally without modifying those RNA-seq outputs.
+
+Download the shipped real example inputs:
+
+```bash
+uv run funmirbench-experiments-download-examples
+```
+
+That downloader fetches:
+
+- the real `GSE253003` count matrix
+- the real `GSE93717` FASTQ files
+- the shared Homo sapiens Ensembl v115 genome FASTA and GTF used by the reads example
+
+Run the real count-matrix example:
+
+```bash
+uv run funmirbench-experiments --config pipelines/experiments/configs/gse253003.count_matrix.example.yaml
+```
+
+Run the reads example the same way:
+
+```bash
+uv run funmirbench-experiments --config pipelines/experiments/configs/gse93717.reads.example.yaml
+```
+
+So the practical reads flow is:
+
+1. activate `funmirbench-experiments`
+2. run `uv run funmirbench-experiments-download-examples`
+3. run `uv run funmirbench-experiments --config pipelines/experiments/configs/gse93717.reads.example.yaml`
+
+The shipped reads example now points at the downloaded Ensembl v115 reference source files under
+`data/experiments/raw/refs/ensembl_v115/`, so it builds the derived STAR index automatically.
+
+Each run writes:
+
+- `data/experiments/processed/<dataset_id>.tsv`
+- `pipelines/experiments/runs/<timestamp>_<dataset_id>/candidate_metadata.tsv`
+- `pipelines/experiments/runs/<timestamp>_<dataset_id>/run_manifest.json`
+
+The reads example uses a reproduced dataset id, `GSE93717_OE_miR_941_deseq2`, so syncing it creates
+a separate variant instead of overwriting the curated `GSE93717_OE_miR_941` registry row.
+
+### Output DE table schema
+
+The processed DE TSV keeps `PValue` and `FDR` unchanged from the DE method, including missing
+adjusted p-values and exact zero values. During benchmarking, rows with `PValue` but no adjusted
+p-value get `benchmark_FDR = 1`, so they remain non-significant background genes. Rows with both
+`PValue` and `FDR` missing remain excluded from FDR-thresholded evaluation. Rows with `FDR = 0`
+use the smallest positive floating-point value for `plot_FDR`, computed with `np.nextafter(0, 1)`,
+for finite `-log10` plotting while keeping `benchmark_FDR = 0`.
+
+Benchmark DE tables use a canonical schema independent of the DE method:
+
+- `gene_id`: Ensembl gene identifier
+- `logFC`: signed log fold change
+- `FDR`: adjusted p-value or q-value used for FDR-thresholded evaluation
+- `control_mean_normalized_count`: mean of the DESeq2-normalized counts across the control samples
+- `PValue`: optional raw p-value used to classify rows with missing adjusted p-values
+
+Common native headers are normalized when benchmark tables are read. For example, `log2FoldChange`
+maps to `logFC`, `padj`, `adj.P.Val`, and `qvalue` map to `FDR`, and `pvalue` or `P.Value` map to
+`PValue`.
+
+---
+
+## Step 4 - Sync into the experiment registry
+
+The ingestion pipeline does not edit `metadata/mirna_experiment_info.tsv` by itself. It writes a
+`candidate_metadata.tsv` under `pipelines/experiments/runs/<timestamp>_<dataset_id>/` first.
+Then sync it into the registry with:
+
+```bash
+uv run funmirbench-sync-metadata
+```
+
+That command auto-discovers all `candidate_metadata.tsv` files under `pipelines/experiments/runs/`
+and upserts them into the registry. Re-running is safe — existing rows with matching `id` values are
+replaced, not duplicated.
+
+To sync a specific file instead:
+
+```bash
+uv run funmirbench-sync-metadata --input pipelines/experiments/runs/<run_dir>/candidate_metadata.tsv
+```
+
+---
+
+## Outputs
+
+| Path | Description |
+|---|---|
+| `data/experiments/processed/<dataset_id>.tsv` | Final DE table |
+| `pipelines/experiments/runs/<timestamp>_<dataset_id>/candidate_metadata.tsv` | Registry row candidate, synced via `funmirbench-sync-metadata` |
+| `pipelines/experiments/runs/<timestamp>_<dataset_id>/run_manifest.json` | Run manifest (commands, inputs, timestamps) |

pipelines/experiments/configs/GSE124530_OE_let_7c_5p.yaml

@@ -0,0 +1,30 @@
+dataset_id: GSE124530_OE_let_7c_5p
+mirna_name: hsa-let-7c-5p
+experiment_type: OE
+gse: GSE124530
+source:
+  mode: reads
+  genome_fasta_path: /data1/repos/FuNmiRBench/data/experiments/raw/refs/ensembl_v115/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz
+  gtf_path: /data1/repos/FuNmiRBench/data/experiments/raw/refs/ensembl_v115/Homo_sapiens.GRCh38.115.gtf.gz
+  fastqc_threads: 8
+  fastp_threads: 8
+  star_threads: 16
+  featurecounts_threads: 8
+comparison:
+  control_samples:
+  - sample_id: GSM3535919
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE124530/SRR8382242.fastq.gz
+  - sample_id: GSM3535920
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE124530/SRR8382243.fastq.gz
+  treated_samples:
+  - sample_id: GSM3535869
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE124530/SRR8382192.fastq.gz
+  - sample_id: GSM3535870
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE124530/SRR8382193.fastq.gz
+metadata:
+  organism: Homo sapiens
+  tested_cell_line: HeLa
+  treatment: Overexpression of hsa-let-7c-5p
+  tissue: Cervix
+  method: RNA-seq
+  article_pubmed_id: https://pubmed.ncbi.nlm.nih.gov/30670076

pipelines/experiments/configs/GSE129076_OE_miR_450a-5p_A2780.yaml

@@ -0,0 +1,37 @@
+dataset_id: GSE129076_OE_miR_450a-5p_A2780
+mirna_name: hsa-miR-450a-5p
+experiment_type: OE
+gse: GSE129076
+
+source:
+  mode: reads
+  genome_fasta_path: /data1/repos/FuNmiRBench/data/experiments/raw/refs/ensembl_v115/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz
+  gtf_path: /data1/repos/FuNmiRBench/data/experiments/raw/refs/ensembl_v115/Homo_sapiens.GRCh38.115.gtf.gz
+  fastqc_threads: 8
+  fastp_threads: 8
+  star_threads: 16
+  featurecounts_threads: 8
+
+comparison:
+  control_samples:
+  - sample_id: GSM3692987
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE129076/SRR8816234.fastq.gz
+  - sample_id: GSM3692988
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE129076/SRR8816235.fastq.gz
+  - sample_id: GSM3692989
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE129076/SRR8816236.fastq.gz
+  treated_samples:
+  - sample_id: GSM3692990
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE129076/SRR8816237.fastq.gz
+  - sample_id: GSM3692991
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE129076/SRR8816238.fastq.gz
+  - sample_id: GSM3692992
+    reads_1: /data1/repos/FuNmiRBench/data/experiments/raw/GSE129076/SRR8816239.fastq.gz
+
+metadata:
+  organism: Homo sapiens
+  tested_cell_line: A2780
+  treatment: Overexpression of miR-450a-5p
+  tissue: Ovarian
+  method: RNA-seq
+  article_pubmed_id: https://pubmed.ncbi.nlm.nih.gov/31101765/

pipelines/experiments/configs/gse129146.reads.example.yaml

@@ -1,6 +1,6 @@
 dataset_id: GSE129146_OE_miR_455-3p
 mirna_name: hsa-miR-455-3p
-experiment_type: OE
+experiment_type: Overexpression
 gse: GSE129146
 
 source:

pipelines/experiments/configs/gse253003.count_matrix.example.yaml

@@ -1,6 +1,6 @@
 dataset_id: GSE253003_OE_miR_323a_3p
 mirna_name: hsa-miR-323a-3p
-experiment_type: OE
+experiment_type: Overexpression
 gse: GSE253003
 
 source: