Pretraining domain determines whether deep learning embeddings outperform handcrafted radiomics — analysis code and data

Reproducibility package for the systematic review and meta-analysis

Pretraining Domain Determines Whether Deep Learning Embeddings Outperform Handcrafted Radiomics: A Systematic Review and Meta-Analysis W. A. I. C. Kumarananda, Kalani L. P. Kurumbala Pitige

PROSPERO registration: CRD420261393443. Manuscript currently under peer review.

What is in this repository

Everything needed to reproduce every figure and number in the manuscript: the analysis scripts, the screening logs, the extracted-data tables, the item-level quality-appraisal scores, and the supplementary materials. The manuscript text is held back until acceptance; a link to the published version will be added here on publication.

.
├── scripts/                Analysis pipeline (Python + R)
├── data/                   Screening logs, final corpus, quality-appraisal scores, bibliography
├── supplementary/          PRISMA-recommended supplementary files
├── docs/                   Statistical synthesis workflow and reproducibility notes
├── outputs/                Created at runtime — figures and tables write here (gitignored)
├── requirements.txt        Python dependencies
├── CHANGELOG.md            Corpus and analysis version history
├── CITATION.cff            How to cite this repository
└── LICENSE / LICENSE-data  MIT for code, CC BY 4.0 for data

Headline result

Across 32 head-to-head studies in which both paradigms were applied to the same patient cohort and the same clinical task, pretrained deep learning embeddings outperformed handcrafted radiomics by a pooled ΔAUC = +0.044 (95% CI +0.012 to +0.077; p = 0.009; I² = 65%).

The effect is concentrated entirely in models whose pretraining corpus matched the imaging domain:

Subgroup	k	Pooled ΔAUC	95% CI
Domain-matched pretraining (RadImageNet, MedicalNet, BiomedCLIP/PMC-15M, CLIP-WIT)	6	+0.104	+0.028 to +0.179
ImageNet-only pretraining	26	+0.024	−0.011 to +0.059 (crosses zero)

Between-subgroup difference p = 0.019.

Twenty-six of the 32 studies (81%) used ImageNet-pretrained convolutional networks, which the primary literature frequently describes as foundation-model work. In this corpus that class of model confers no reliable advantage over handcrafted radiomics.

Quality appraisal across all 32 studies: median RQS 12/36, median CLAIM completeness 81.7%, and 29 of 32 studies at high PROBAST risk of bias.

Reproduce every number above in one command

python scripts/34_replication_and_fragility.py

It recomputes the pooled estimates and quality summaries from data/12_Final_Included_Corpus_v2_32studies.csv alone, reports leave-one-out fragility for each subgroup, and finishes with self-tests that assert the published values. It has no dependency on the R pipeline.

Quickstart — reproduce the analyses

Requires Python 3.10+ and R 4.2+.

git clone https://github.com/Iranchamika/radiomics-vs-foundation-models.git
cd radiomics-vs-foundation-models
pip install -r requirements.txt
mkdir -p outputs

# Verification: pooled estimates, subgroups, quality summaries, fragility, self-tests
python scripts/34_replication_and_fragility.py

# Random-effects meta-analysis (DerSimonian-Laird + Knapp-Hartung)
Rscript scripts/19_meta_analysis.R

# Forest plot
Rscript scripts/20_forest_plot.R

# Quality-appraisal figures
python scripts/21_rqs_heatmap.py
python scripts/22_claim_heatmap.py
python scripts/23_probast_traffic_light.py

# PRISMA 2020 flow diagram
python scripts/24_prisma_flow.py

# Per-study characteristics table
python scripts/32_table1_generator.py

# PRISMA 27-item checklist
python scripts/33_prisma_checklist_generator.py

All outputs are written to outputs/, which is created at runtime and not tracked in git.

Script catalogue

Script	Language	Purpose
34_replication_and_fragility.py	Python	Standalone verification. Reproduces every pooled estimate and quality summary from the corpus CSV; leave-one-out and post-search fragility analysis; self-tests against the published values
19_meta_analysis.R	R	Random-effects DerSimonian–Laird meta-analysis with Knapp–Hartung adjustment; subgroup pooling by pretraining tier, modality, task type; sensitivity analyses
20_forest_plot.R	R	Forest plot
21_rqs_heatmap.py	Python	Radiomics Quality Score item-level heatmap
22_claim_heatmap.py	Python	CLAIM 42-item checklist heatmap
23_probast_traffic_light.py	Python	PROBAST domain-level risk-of-bias visualisation
24_prisma_flow.py	Python	PRISMA 2020 flow diagram
25_characteristics_table.py	Python	Supplementary characteristics table generator
30_fetch_pubmed_authors.py	Python	Helper — resolve first-author surnames from PMIDs
32_table1_generator.py	Python	Manuscript Table 1 (per-study characteristics longtable)
33_prisma_checklist_generator.py	Python	PRISMA 27-item checklist with page references

Data files

File	Rows	Description
data/12_Final_Included_Corpus_v2_32studies.csv	32	Primary corpus. 33 extracted fields per study: modality, anatomy, cohort sizes, pretrained-model details and pretraining corpus, radiomics package, ML head, per-arm metric values, Δ, and RQS / CLAIM / PROBAST ratings
data/12_Final_Included_Corpus.csv	28	First-round corpus (PubMed only). Superseded — retained so the v1.0 results remain reproducible. See CHANGELOG.md
data/15_RQS_Item_Scores.csv	28	Per-study Radiomics Quality Score, 16 items
data/16_CLAIM_Item_Scores.csv	28	Per-study CLAIM scores, 42 items
data/17_PROBAST_Signaling.csv	28	PROBAST signalling-question responses
data/28_Kappa_Sample_25.csv	25	Inter-rater reliability sample (AI triage vs human reviewer)
data/29_Kappa_Results.csv	—	κ point estimates and 95% CIs under three framings
data/31_Exclusion_Reasons_Detail.csv	46	Full-text exclusion taxonomy, per-PMID
data/09a_PubMed_AI_Prescreen.csv	245	Stage 1 AI pre-screen output
data/10_Stage2_MAYBE_Sample.csv	—	Stage 2 reviewer adjudication of MAYBE candidates
data/11_Stage3_Remaining_MAYBEs.csv	—	Stage 3 reconciliation log
data/07_Pilot_Adjudication.csv	—	Pilot κ calibration (first 20 abstracts)
data/references.bib	28	BibTeX entries, PMID-keyed

A column-level data dictionary is at data/README.md.

Item-level RQS, CLAIM and PROBAST scores for the four studies added in the second-database round are recorded in the primary corpus (summary columns); their item-level rows are pending addition to 15/16/17_*.csv. See CHANGELOG.md.

Methodology in one paragraph

Systematic search of PubMed/MEDLINE and Lens.org to 15 May 2026. Stage 1 title/abstract pre-screen was assisted by a large-language-model triage classifier calibrated against a human reviewer on a stratified 25-record sample (Cohen's κ = 0.41, 95% CI 0.08–0.75, binary advance-vs-exclude framing); Stage 2 and Stage 3 adjudication of MAYBE candidates was performed by the human reviewer, so the final corpus is human-determined. Full-text screening produced 28 included studies from PubMed; a second-database round via Lens.org added 4, for 32 in total. Quality appraisal applied RQS (Lambin 2017), CLAIM (Mongan 2020) and PROBAST (Wolff 2019) to every included study. Meta-analysis used the Hanley–McNeil SE(AUC) approximation under an independent-arms assumption (conservative), DerSimonian–Laird random-effects pooling with the Knapp–Hartung small-sample adjustment, pre-specified subgroup analyses (pretraining tier, modality, task type, validation status) and five pre-specified sensitivity analyses.

Caveats and reproducibility notes

• The domain-matched subgroup rests on k = 6 and is not robust. Leave-one-out analysis shows that removing a single study (PMID 40605934) moves the estimate to +0.090 (95% CI −0.002 to +0.181), which no longer excludes zero. This estimate should be read as hypothesis-generating. scripts/34_replication_and_fragility.py prints the full leave-one-out table. The overall +0.044 estimate, by contrast, is robust to removing any single study.
• Single-reviewer screening. Title/abstract screening used one human reviewer with LLM triage assistance rather than two independent reviewers. κ = 0.41 is the lowest-confidence number in the study and is reported as such.
• The Hanley–McNeil SE(AUC) approximation assumes balanced classes where prevalence was not reported. The independent-arms variance approximation, var(Δ) = var_FM + var_Rad, is conservative; the paired variance would be smaller. Sensitivity to this assumption is reported in the manuscript.
• Pooling spans 6 modalities, 17 anatomical sites and 4 task types (I² = 65%). Heterogeneity is substantial and is addressed by subgroup analysis rather than eliminated.
• PRISMA flow counts for the first round reconcile as 245 → 170 → 75 → 74 assessed → 46 excluded → 28 included; the second round added 4 from Lens.org.

Licensing

• Code (scripts/, requirements.txt, top-level config): MIT — see LICENSE.
• Data and documentation (data/, docs/, supplementary/, README.md): Creative Commons Attribution 4.0 International — see LICENSE-data. The 32 included primary studies remain copyrighted by their original publishers; this repository contains extracted summary data, not the source articles.

Citing this work

Until the manuscript is published, please cite the repository using the metadata in CITATION.cff, or the archived version at https://doi.org/10.5281/zenodo.20502586. After publication the manuscript becomes the preferred citation.

Contact

Repository maintainer: W. A. I. C. Kumarananda — iranchamika@gmail.com — ORCID 0009-0007-8063-0027.

Please open a GitHub issue for corrections and reproducibility questions rather than emailing.