Predictors cleanup

.gitignore

@@ -13,8 +13,6 @@ miniconda3/
 
 # Local data outputs (NOT tracked)
 data/experiments/processed/*
-!data/experiments/processed/18745741/
-!data/experiments/processed/18745741/*.tsv
 data/experiments/raw/*
 data/predictions/*
 !data/predictions/random/
@@ -26,7 +24,9 @@ data/predictions/perfect/*
 data/joined/
 data/plots/
 data/experiments/raw/refs/*
-data/resources/ensembl/*
+data/resources/ # legacy local cache path
+data/common_resources/*
+!data/common_resources/README.md
 
 # Benchmark results
 results/
@@ -40,11 +40,11 @@ pipelines/geo/runs/*
 pipelines/experiments/runs/*
 
 # Predictor pipeline generated data (downloads + intermediate tables)
-pipelines/standardized_predictors/*/data/*
-pipelines/standardized_predictors/mirbind2/unified_human_mirbase_mane_select_3utrs_selected_18_mirnas_mirbind2_predictions.tsv
-pipelines/standardized_predictors/mirbind2/3utrs_mirbind2_predictions_no_sequences.tsv
-pipelines/standardized_predictors/mirbind2/3utrs_mirbind2_predictions.tsv
+pipelines/standardized_predictors/*/data/
+pipelines/standardized_predictors/*/resources/
 
-# Track standardized predictor pipeline logs
+# Ignore logs globally, but track standardized predictor pipeline logs alongside outputs
 *.log
-!pipelines/standardized_predictors/**/*_pipeline.log
+!data/predictions/*/
+data/predictions/*/*
+!data/predictions/*/*_pipeline.log

README.md

@@ -38,8 +38,9 @@ That environment also includes `uv`, so `uv run ...` keeps working after activat
 Main directories:
 
 - `data/experiments/processed/`: root directory for processed experiment DE tables
-- `data/experiments/processed/18745741/`: local cache for curated benchmark DE tables from Zenodo record `18745741`; the repo currently ships the 3 default benchmark TSVs here
+- `data/experiments/processed/21671476/`: local cache for curated benchmark DE tables from Zenodo record `21671476`; the repo currently ships the 3 default benchmark TSVs here
 - `data/experiments/raw/`: local raw GEO inputs such as count matrices and FASTQs
+- `data/common_resources/`: shared, downloaded miRBase and Ensembl annotation cache
 - `data/predictions/`: local generated predictor TSVs
 - `metadata/mirna_experiment_info.tsv`: experiment registry
 - `metadata/predictions_info.tsv`: predictor registry
@@ -65,7 +66,7 @@ uv run funmirbench --config benchmark.yaml
 ```
 
 Before benchmarking, `funmirbench` syncs the selected curated experiment DE tables from Zenodo
-into `data/experiments/processed/18745741/` as needed. The repo currently ships the 3 TSVs used by
+into `data/experiments/processed/21671476/` as needed. The repo currently ships the 3 TSVs used by
 the default benchmark config, while other curated benchmark DE tables are treated as fetched local cache.
 
 The default config already points at:
@@ -84,7 +85,7 @@ workflow use.
 
 For the curated benchmark datasets tracked in `metadata/mirna_experiment_info.tsv`, the expected
 workflow is different: those metadata rows stay versioned in the repo, and the corresponding DE
-tables live under the local `data/experiments/processed/18745741/` cache. The repo currently ships
+tables live under the local `data/experiments/processed/21671476/` cache. The repo currently ships
 the 3 default benchmark TSVs there, and other curated tables are fetched from Zenodo when needed.
 
 Experiment config summary:
@@ -237,9 +238,9 @@ uv run funmirbench --config benchmark.yaml
 ```
 
 That command automatically syncs only the experiment DE tables selected by your benchmark config
-from Zenodo into the local `data/experiments/processed/18745741/` cache before joining predictions.
+from Zenodo into the local `data/experiments/processed/21671476/` cache before joining predictions.
 On the first protein-coding run, it also downloads or reuses the Ensembl v115 GTF and caches the
-protein-coding gene set at `data/resources/ensembl/protein_coding_gene_ids.txt`.
+protein-coding gene set at `data/common_resources/ensembl/protein_coding_gene_ids.txt`.
 
 If you want to prefetch the full curated experiment cache yourself, you can also run:
 
@@ -285,8 +286,8 @@ evaluation:
   write_top_prediction_cdfs: true
   report_min_common_coverage: 0.10
   protein_coding_only: true
-  protein_coding_gtf: data/resources/ensembl/Homo_sapiens.GRCh38.115.gtf.gz
-  protein_coding_gene_cache: data/resources/ensembl/protein_coding_gene_ids.txt
+  protein_coding_gtf: data/common_resources/ensembl/Homo_sapiens.GRCh38.115.gtf.gz
+  protein_coding_gene_cache: data/common_resources/ensembl/protein_coding_gene_ids.txt
 
 out_dir: results/
 ```

benchmark.yaml

@@ -32,8 +32,8 @@ evaluation:
   # Primary benchmark universe: Ensembl protein-coding genes only.
   # The first run downloads/reuses the Ensembl v115 GTF and caches the gene set.
   protein_coding_only: true
-  protein_coding_gtf: data/resources/ensembl/Homo_sapiens.GRCh38.115.gtf.gz
-  protein_coding_gene_cache: data/resources/ensembl/protein_coding_gene_ids.txt
+  protein_coding_gtf: data/common_resources/ensembl/Homo_sapiens.GRCh38.115.gtf.gz
+  protein_coding_gene_cache: data/common_resources/ensembl/protein_coding_gene_ids.txt
   figure_dpi: 600
 
 out_dir: results/

data/common_resources/README.md

@@ -0,0 +1,29 @@
+# Common Resources
+
+This directory contains external biological resources shared by multiple standardized predictor
+pipelines and the benchmark.
+
+The downloaded resource files are not tracked by Git. A pipeline or benchmark command downloads a
+resource automatically when it is first required, then subsequent runs and other pipelines reuse
+the cached copy.
+
+Expected directory structure:
+
+```text
+data/common_resources/
+├── README.md
+├── mirbase/
+│   └── mature.fa
+└── ensembl/
+    ├── Homo_sapiens.GRCh38.115.gtf.gz
+    └── protein_coding_gene_ids.txt
+```
+
+- `mirbase/mature.fa` is the miRBase 22.1 mature-miRNA FASTA used by all standardized predictor
+  pipelines.
+- `ensembl/Homo_sapiens.GRCh38.115.gtf.gz` is the Ensembl release 115 annotation shared by
+  microT-CNN, miRAW, TargetScan, and protein-coding benchmark filtering.
+- `ensembl/protein_coding_gene_ids.txt` is derived from the shared GTF and cached by the benchmark.
+
+Deleting any downloaded file is safe; it will be downloaded or regenerated the next time it is
+needed.

data/predictions/microt_cnn/microt_cnn_pipeline.log

@@ -0,0 +1,22 @@
+2026-07-27 17:01:24,871 INFO pipeline: Starting microT-CNN standardization pipeline
+2026-07-27 17:01:24,871 INFO pipeline: Step 1/7: Resolve raw microT-CNN predictions and external resources
+2026-07-27 17:01:24,872 INFO utils: Downloading miRBase mature.fa resource: data/common_resources/mirbase/mature.fa
+2026-07-27 17:01:26,166 INFO utils: Saved miRBase mature.fa resource: data/common_resources/mirbase/mature.fa
+2026-07-27 17:01:26,167 INFO utils: Downloading microT-CNN all-score prediction file: pipelines/standardized_predictors/microt_cnn/data/microT_CNN_prediction_result_human_all_scores_gene_level.tsv.gz
+2026-07-27 17:02:41,903 INFO utils: Saved microT-CNN all-score prediction file: pipelines/standardized_predictors/microt_cnn/data/microT_CNN_prediction_result_human_all_scores_gene_level.tsv.gz
+2026-07-27 17:02:41,906 INFO pipeline: Step 2/7: Load raw microT-CNN predictions
+2026-07-27 17:03:08,473 INFO pipeline: Step 3/7: Create miRNA and transcript mapping resources
+2026-07-27 17:03:08,495 INFO utils: Downloading Ensembl v115 GTF resource: data/common_resources/ensembl/Homo_sapiens.GRCh38.115.gtf.gz
+2026-07-27 17:03:23,203 INFO utils: Saved Ensembl v115 GTF resource: data/common_resources/ensembl/Homo_sapiens.GRCh38.115.gtf.gz
+2026-07-27 17:03:40,133 INFO utils: Built Ensembl transcript-to-gene cache from GTF: transcript_rows=509650 | mapped=509650 | output=pipelines/standardized_predictors/microt_cnn/resources/ensembl115_tx2gene.tsv.gz
+2026-07-27 17:03:40,133 INFO pipeline: Step 4/7: Map prediction miRNA names to MIMAT IDs
+2026-07-27 17:03:48,679 INFO utils: Drop prediction rows with miRNA names that cannot map to MIMAT IDs: 48281464 -> 48281464 rows
+2026-07-27 17:03:49,284 INFO pipeline: Step 5/7: Map microT-CNN Ensembl transcript IDs to Ensembl gene IDs
+2026-07-27 17:04:22,975 INFO utils: Drop prediction rows with Ensembl transcripts that cannot map to genes: 48281464 -> 48281464 rows
+2026-07-27 17:04:23,770 INFO pipeline: Step 6/7: Build standardized output columns
+2026-07-27 17:04:32,652 INFO utils: Drop prediction rows with non-numeric scores: 48281464 -> 48281464 rows
+2026-07-27 17:05:06,643 INFO utils: Collapse transcript rows to one row per Ensembl gene-miRNA pair: 48281464 -> 48281464 rows
+2026-07-27 17:05:06,948 INFO pipeline: Step 7/7: Validate and write standardized output table
+2026-07-27 17:06:41,085 INFO pipeline: Output written to: data/predictions/microt_cnn/microt_cnn_standardized.tsv
+2026-07-27 17:06:41,085 INFO pipeline: Rows written: 48281464
+2026-07-27 17:06:41,085 INFO pipeline: Relative output path: data/predictions/microt_cnn/microt_cnn_standardized.tsv

data/predictions/miraw/miraw_pipeline.log

@@ -0,0 +1,27 @@
+2026-07-27 17:16:41,160 INFO miraw_pipeline: Starting miRAW standardization pipeline
+2026-07-27 17:16:41,160 INFO miraw_pipeline: Step 1/8: Download or reuse raw site-level miRAW predictions
+2026-07-27 17:16:41,533 INFO miraw_utils: Downloading miRAW predictions from DOI 10.6084/m9.figshare.32982218: pipelines/standardized_predictors/miraw/data/helios_summary.tsv.gz
+2026-07-27 17:17:30,976 INFO miraw_utils: Saved miRAW predictions from DOI 10.6084/m9.figshare.32982218: pipelines/standardized_predictors/miraw/data/helios_summary.tsv.gz (713928894 bytes)
+2026-07-27 17:17:30,980 INFO miraw_pipeline: Step 2/8: Resolve miRBase 22.1 and Ensembl release 115 resources
+2026-07-27 17:17:30,980 INFO miraw_utils: Using cached miRBase 22.1 mature.fa resource: data/common_resources/mirbase/mature.fa
+2026-07-27 17:17:30,981 INFO miraw_utils: Using cached Ensembl release 115 Homo sapiens GTF: data/common_resources/ensembl/Homo_sapiens.GRCh38.115.gtf.gz
+2026-07-27 17:17:30,981 INFO miraw_pipeline: Step 3/8: Load and validate raw miRAW prediction rows
+2026-07-27 17:19:40,927 INFO miraw_utils: Loaded 49223901 miRAW prediction rows generated against Ensembl release 112 and miRBase release 22.1 from pipelines/standardized_predictors/miraw/data/helios_summary.tsv.gz
+2026-07-27 17:20:29,363 INFO miraw_utils: Validate parsed miRAW rows: 49223901 -> 49223901 rows
+2026-07-27 17:20:52,821 INFO miraw_utils: Drop exact duplicate parsed miRAW rows: 49223901 -> 49223901 rows
+2026-07-27 17:20:56,741 INFO miraw_pipeline: Step 4/8: Build miRNA-name-to-MIMAT mapping from miRBase 22.1
+2026-07-27 17:20:56,766 INFO miraw_utils: Loaded 2656 unique human miRNA-name-to-MIMAT mappings from 2656 miRBase 22.1 FASTA headers
+2026-07-27 17:20:56,767 INFO miraw_pipeline: Step 5/8: Build ENSG-to-gene-name mapping from Ensembl release 115
+2026-07-27 17:21:08,297 INFO miraw_utils: Loaded 43048 Ensembl release 115 ENSG-to-gene-name mappings from 78899 gene rows; invalid_ids=0, missing_names=35851, conflicts=0
+2026-07-27 17:21:08,297 INFO miraw_pipeline: Step 6/8: Retain and annotate miRNAs present in miRBase 22.1
+2026-07-27 17:21:14,931 INFO miraw_utils: Retain miRNAs present in miRBase release 22.1: 49223901 -> 49223901 rows
+2026-07-27 17:21:26,274 INFO miraw_pipeline: Step 7/8: Retain and annotate genes present in Ensembl release 115
+2026-07-27 17:21:40,507 INFO miraw_utils: Retain genes present in Ensembl release 115: 49223901 -> 48041969 rows
+2026-07-27 17:21:40,507 INFO miraw_utils: Dropped 467 unique Ensembl release 112 source IDs absent from release 115; first examples=['ENSG00000100101', 'ENSG00000111780', 'ENSG00000124593', 'ENSG00000125695', 'ENSG00000131152', 'ENSG00000139656', 'ENSG00000141979', 'ENSG00000142539', 'ENSG00000148362', 'ENSG00000159239']
+2026-07-27 17:21:48,082 INFO miraw_pipeline: Step 8/8: Finalize one highest-scoring standardized row per Ensembl_ID-miRNA_ID pair
+2026-07-27 17:22:36,230 INFO miraw_utils: Validate standardized miRAW identifiers and values: 48041969 -> 48041969 rows
+2026-07-27 17:23:08,797 INFO miraw_utils: Finalize one highest-scoring row per Ensembl_ID-miRNA_ID pair: 48041969 -> 48041969 rows
+2026-07-27 17:23:15,843 INFO miraw_utils: Final miRAW table uses Ensembl release 115 gene IDs/names and miRBase release 22.1 mature miRNA names/accessions
+2026-07-27 17:24:55,586 INFO miraw_pipeline: Output written to: data/predictions/miraw/miraw_standardized.tsv
+2026-07-27 17:24:55,587 INFO miraw_pipeline: Rows written: 48041969
+2026-07-27 17:24:55,591 INFO miraw_pipeline: Relative output path: data/predictions/miraw/miraw_standardized.tsv

data/predictions/mirbind2/mirbind2_pipeline.log

@@ -0,0 +1,19 @@
+2026-07-27 17:25:38,489 INFO pipeline: Starting miRBind2 standardization pipeline
+2026-07-27 17:25:38,489 INFO pipeline: Step 1/6: Download or resolve shared miRBase and raw miRBind2 predictions
+2026-07-27 17:25:38,489 INFO utils: Using cached miRBase mature.fa resource: data/common_resources/mirbase/mature.fa
+2026-07-27 17:25:38,490 INFO utils: Downloading miRBind2 raw prediction file: pipelines/standardized_predictors/mirbind2/data/3utrs_mirbind2_predictions.tsv.gz
+2026-07-27 17:26:29,909 INFO utils: Saved miRBind2 raw prediction file: pipelines/standardized_predictors/mirbind2/data/3utrs_mirbind2_predictions.tsv.gz
+2026-07-27 17:26:29,933 INFO pipeline: Step 2/6: Load raw miRBind2 predictions
+2026-07-27 17:28:08,400 INFO utils: Drop invalid raw miRBind2 prediction rows: 50171840 -> 50171840 rows
+2026-07-27 17:28:40,128 INFO utils: Drop exact duplicate raw miRBind2 prediction rows: 50171840 -> 50171840 rows
+2026-07-27 17:28:40,564 INFO pipeline: Step 3/6: Create miRNA name-to-MIMAT mapping from miRBase mature.fa
+2026-07-27 17:28:40,588 INFO utils: Loaded 2656 human miRNA name-to-MIMAT mappings from miRBase 22.1
+2026-07-27 17:28:40,589 INFO pipeline: Step 4/6: Annotate raw miRNA names to MIMAT IDs
+2026-07-27 17:28:49,058 INFO utils: Drop prediction rows with miRNA names that cannot map to MIMAT IDs: 50171840 -> 50171840 rows
+2026-07-27 17:28:50,017 INFO pipeline: Step 5/6: Build standardized output columns
+2026-07-27 17:31:20,610 INFO utils: Drop standardized rows with invalid canonical values or scores: 50171840 -> 50171840 rows
+2026-07-27 17:31:56,980 INFO utils: Drop duplicate standardized Ensembl_ID-miRNA_ID pairs after MIMAT mapping: 50171840 -> 50171840 rows
+2026-07-27 17:31:57,217 INFO pipeline: Step 6/6: Validate and write standardized output table
+2026-07-27 17:33:19,214 INFO pipeline: Output written to: data/predictions/mirbind2/mirbind2_standardized.tsv
+2026-07-27 17:33:19,230 INFO pipeline: Rows written: 50171840
+2026-07-27 17:33:19,232 INFO pipeline: Relative output path: data/predictions/mirbind2/mirbind2_standardized.tsv

data/predictions/mirdb_mirtarget/mirdb_mirtarget_pipeline.log

@@ -0,0 +1,34 @@
+2026-07-27 17:34:39,690 INFO pipeline: Starting miRDB standardization pipeline
+2026-07-27 17:34:39,690 INFO pipeline: Step 1/9: Resolve raw miRDB predictions and external miRBase/BioMart resources
+2026-07-27 17:34:39,690 INFO utils: Using cached miRBase mature.fa resource: data/common_resources/mirbase/mature.fa
+2026-07-27 17:34:39,691 INFO utils: Downloading BioMart NCBI Gene ID/RefSeq-to-Ensembl mapping table: pipelines/standardized_predictors/mirdb_mirtarget/resources/hsapiens_ncbi_gene_id_refseq_to_ensembl.tsv
+2026-07-27 17:36:14,426 INFO utils: Saved BioMart NCBI Gene ID/RefSeq-to-Ensembl mapping table: pipelines/standardized_predictors/mirdb_mirtarget/resources/hsapiens_ncbi_gene_id_refseq_to_ensembl.tsv
+2026-07-27 17:36:14,428 INFO utils: Downloading miRDB v6.0 all-score prediction file: pipelines/standardized_predictors/mirdb_mirtarget/data/miRDB_v6.0_prediction_result_human_all_scores.txt.gz
+2026-07-27 17:36:21,741 INFO utils: Saved miRDB v6.0 all-score prediction file: pipelines/standardized_predictors/mirdb_mirtarget/data/miRDB_v6.0_prediction_result_human_all_scores.txt.gz
+2026-07-27 17:36:21,765 INFO pipeline: Step 2/9: Load raw miRDB predictions
+2026-07-27 17:36:38,706 INFO utils: Drop invalid raw miRDB prediction rows: 8255691 -> 8255691 rows
+2026-07-27 17:36:42,616 INFO utils: Drop exact duplicate raw miRDB prediction rows: 8255691 -> 8255691 rows
+2026-07-27 17:36:44,848 INFO utils: Check raw miRDB rows for duplicate miRNA-RefSeq transcript prediction pairs: 0 duplicate pairs involving 0 rows
+2026-07-27 17:36:50,184 INFO utils: Check raw miRDB rows for duplicate miRNA-NCBI Gene ID prediction pairs from raw column 4: 0 duplicate pairs involving 0 rows
+2026-07-27 17:36:53,372 INFO pipeline: Step 3/9: Create miRNA name-to-MIMAT mapping from miRBase mature.fa
+2026-07-27 17:36:53,394 INFO pipeline: Step 4/9: Create NCBI Gene ID-to-Ensembl gene mapping from BioMart table
+2026-07-27 17:36:53,633 INFO utils: Filter NCBI Gene ID-to-Ensembl mapping table to drop invalid rows: 168112 -> 110750 rows
+2026-07-27 17:36:53,661 INFO utils: Filter NCBI Gene ID-to-Ensembl mapping table to drop exact duplicate rows: 110750 -> 34460 rows
+2026-07-27 17:36:53,677 INFO utils: Filter NCBI Gene ID-to-Ensembl mapping table to drop NCBI Gene IDs with conflicting Ensembl gene mappings: 34460 -> 23869 rows
+2026-07-27 17:36:54,116 INFO pipeline: Step 5/9: Create RefSeq-to-Ensembl fallback gene mapping from BioMart table
+2026-07-27 17:36:54,349 INFO utils: Filter RefSeq-to-Ensembl mapping table to drop invalid rows: 168112 -> 79730 rows
+2026-07-27 17:36:54,376 INFO utils: Filter RefSeq-to-Ensembl mapping table to drop exact duplicate rows: 79730 -> 74568 rows
+2026-07-27 17:36:54,418 INFO utils: Filter RefSeq-to-Ensembl mapping table to drop RefSeq IDs with conflicting Ensembl gene mappings: 74568 -> 63157 rows
+2026-07-27 17:36:55,571 INFO pipeline: Step 6/9: Map prediction miRNA names to MIMAT IDs
+2026-07-27 17:36:56,819 INFO utils: Drop prediction rows with miRNA names that cannot map to MIMAT IDs: 8255691 -> 8255691 rows
+2026-07-27 17:36:56,988 INFO pipeline: Step 7/9: Map prediction NCBI Gene IDs to Ensembl gene IDs and gene names
+2026-07-27 17:37:00,426 INFO utils: Rows left for RefSeq fallback after NCBI Gene ID mapping: 8255691 -> 489329 rows
+2026-07-27 17:37:00,531 INFO pipeline: Step 8/9: Map remaining prediction RefSeq transcript IDs to Ensembl gene IDs and gene names
+2026-07-27 17:37:02,250 INFO utils: Rows still unmapped after RefSeq fallback: 489329 -> 451230 rows
+2026-07-27 17:37:03,225 INFO utils: Drop prediction rows that cannot map to Ensembl genes after NCBI Gene ID mapping and RefSeq fallback: 8255691 -> 7804461 rows
+2026-07-27 17:37:03,458 INFO pipeline: Step 9/9: Build, validate, and write standardized output table
+2026-07-27 17:37:11,369 INFO utils: Drop final Ensembl_ID-miRNA_ID pairs with conflicting scores after NCBI/RefSeq-to-gene mapping: 7804461 -> 7799023 rows
+2026-07-27 17:37:13,648 INFO utils: Drop duplicate final Ensembl_ID-miRNA_ID pairs with identical scores after NCBI/RefSeq-to-gene mapping: 7799023 -> 7795525 rows
+2026-07-27 17:37:27,314 INFO pipeline: Output written to: data/predictions/mirdb_mirtarget/mirdb_mirtarget_standardized.tsv
+2026-07-27 17:37:27,315 INFO pipeline: Rows written: 7795525
+2026-07-27 17:37:27,315 INFO pipeline: Relative output path: data/predictions/mirdb_mirtarget/mirdb_mirtarget_standardized.tsv