Rerun predictors

[stephaniesamm]

microtcnn

• still downloading resources to pipelines/standardized_predictors/microt_cnn/data/; why do we have resources under data/resources/?
• still outputting log to pipelines/standardized_predictors/microt_cnn/. i thought we agreed this should live in the same dir as the final std preds outputs?
• also mirbase download timeout error:

RuntimeError: Failed to download miRBase mature.fa resource from https://mirbase.org/download_version_files/22.1/mature.fa to pipelines/standardized_predictors/microt_cnn/data/resources/mirbase/mature.fa: HTTPSConnectionPool(host='mirbase.org', port=443): Max retries exceeded with url: /download_version_files/22.1/mature.fa (Caused by ConnectTimeoutError(<HTTPSConnection(host='mirbase.org', port=443) at 0x7ed0b4339700>, 'Connection to mirbase.org timed out. (connect timeout=120)'))

[stephaniesamm]

miraw

• also downloads resources to pipelines/standardized_predictors/miraw/data/
• also still outputting log to pipelines/standardized_predictors/miraw/
• also runs into same runtime error to download from mirbase

[stephaniesamm]

mirbind2

• correctly using /data/resources/mirbase/mature.fa
• also still outputting log to pipelines/standardized_predictors/mirbind2/

[stephaniesamm]

mirdb_mirtarget

• correctly using /data/resources/mirbase/mature.fa and /data/resources/biomart/; if the latter file is only used for mirdb, should be place it under pipelines/standardized_predictors/mirdb_mirtarget/data/resources/, like the rest?
• also still outputting log to pipelines/standardized_predictors/mirdb_mirtarget/

[stephaniesamm]

targetscan

• also still outputting log to pipelines/standardized_predictors/targetscan/
• also downloads mirbase resource to pipelines/standardized_predictors/targetscan/data/
• mature.fa download is not successful

[stephaniesamm]

Overall

@zacharopoulou KINDLY REVIEW THE FOLLOWING:

• Do we want logs in the script directory or in the output directory alongside the std preds output? Currently all logs are being outputted in script directory.
• I think we should use the common data/resources/mirbase/mature.fa for all (microt_cnn, miraw, targetscan need to be updated)
• I think data/resources/biomart/ is only being used by mirdb_mirtarget pipeline and should therefore reside in the mirdb_mirtarget/data/resources/ directory. Otherwise resources of all other predictors need to move to data/resources/ so everything is organised in the same way. Probably the former approach is better and data/resources/ can be renamed to data/common_resources/. Thoughts?
• Are there any other files besides mirbase mature.fa that are commonly used across all/most predictors? Yes I think Homo_sapiens.GRCh38.115.gtf.gz is used by targetscan, miraw, microt_cnn. I think this should also reside in data/common_resources/.

[zacharopoulou]

Ok lets start with overall rules:

• Every pipeline runs with uv run
• Shared biological resources are downloaded on the first run that needs them.
• Shared resources are reused by later pipelines and are ignored by Git.
• Predictor-specific downloads and intermediate files stay inside each predictor’s data/ directory and are ignored.
• Standardized prediction tsvs stay under data/predictions// and remain ignored.
• Pipeline logs also go under data/predictions//, but logs are tracked by Git.

Directory structure

FuNmiRBench/
├── data/
│ ├── common_resources/
│ │ ├── README.md # tracked
│ │ │
│ │ ├── mirbase/
│ │ │ └── 22.1/
│ │ │ └── mature.fa # ignored
│ │ │
│ │ └── ensembl/
│ │ └── release-115/
│ │ ├── Homo_sapiens.GRCh38.115.gtf.gz
│ │ └── ensembl115_tx2gene.tsv.gz
│ │ # both ignored
│ │
│ └── predictions/
│ ├── microt_cnn/
│ │ ├── microt_cnn_standardized.tsv # ignored
│ │ └── microt_cnn_pipeline.log # tracked
│ │
│ ├── miraw/
│ │ ├── miraw_standardized.tsv # ignored
│ │ └── miraw_pipeline.log # tracked
│ │
│ ├── mirbind2/
│ │ ├── mirbind2_standardized.tsv # ignored
│ │ └── mirbind2_pipeline.log # tracked
│ │
│ ├── mirdb_mirtarget/
│ │ ├── mirdb_mirtarget_standardized.tsv # ignored
│ │ └── mirdb_mirtarget_pipeline.log # tracked
│ │
│ └── targetscan/
│ ├── targetscan_standardized.tsv # ignored
│ └── targetscan_pipeline.log # tracked
│
├── pipelines/
│ └── standardized_predictors/
│ ├── README.md
│ ├── common.py
│ │
│ ├── microt_cnn/
│ │ ├── pipeline.py
│ │ ├── utils.py
│ │ ├── README.md
│ │ └── data/
│ │ └── microT_CNN_prediction_result_human_all_scores_gene_level.tsv.gz
│ │ # ignored
│ │
│ ├── miraw/
│ │ ├── pipeline.py
│ │ ├── utils.py
│ │ ├── README.md
│ │ └── data/
│ │ └── helios_summary.tsv.gz # ignored
│ │
│ ├── mirbind2/
│ │ ├── pipeline.py
│ │ ├── utils.py
│ │ ├── README.md
│ │ └── data/
│ │ └── 3utrs_mirbind2_predictions.tsv.gz # ignored
│ │
│ ├── mirdb_mirtarget/
│ │ ├── pipeline.py
│ │ ├── utils.py
│ │ ├── README.md
│ │ └── data/
│ │ ├── miRDB_v6.0_prediction_result_human_all_scores.txt.gz
│ │ └── resources/
│ │ └── biomart/
│ │ └── hsapiens_ncbi_gene_id_refseq_to_ensembl.tsv
│ │ # ignored
│ │
│ └── targetscan/
│ ├── pipeline.py
│ ├── utils.py
│ ├── README.md
│ └── data/
│ ├── Summary_Counts.all_predictions.txt.zip
│ ├── Summary_Counts.all_predictions.txt
│ ├── miR_Family_Info.txt.zip
│ ├── miR_Family_Info.txt
│ ├── Gene_info.txt.zip
│ ├── Gene_info.txt
│ └── targetscan-specific derived tables/caches
│ # all ignored
│
├── .gitignore
└── pyproject.toml

• Biomart mapping stays under mirDB. Changes definitely are needed at common.py script
• Every predictor needs changes, but Targetscan must also include all the cli commands
• Gitignore needs a thorough clean-up
• data/common_resources/README.md could explain the reason of existance of this directory. Something like: "This directory contains external biological resources that are shared by multiple standardized predictor pipelines. The actual downloaded resource files are not tracked by Git. They are downloaded automatically when a pipeline first requires them and are reused by subsequent pipelines. Directory structure: ...."