patheval: Unified evaluation of pathology foundation models across eva, HEST, and PLISM

A unified benchmarking harness for pathology foundation models. It stitches three upstream benchmark suites behind a single CLI with a pluggable model registry. Add a model once, and evaluate on multiple benchmarks with a single command.

• eva by kaiko.ai; frozen-encoder linear probing and segmentation. Ships with six default benchmarks (BACH, CRC, PCam, MHIST, CoNSeP, MoNuSAC); eva itself supports many more pathology datasets and you can enable any of them by dropping a YAML into configs/eva/offline/ or configs/eva/online/.
• HEST by MahmoodLab; predict gene expression from H&E patches across nine spatial-transcriptomics tasks.
• PLISM by Owkin; measure how consistent frozen features stay for the same tissue across 7 scanner and staining combinations.

Install

Requires Python 3.11+.

git clone https://github.com/RamonKaspar/pathology-eval.git
cd pathology-eval

conda create -n pathology-eval python=3.11
conda activate pathology-eval

pip install -e .                  # core (eva benchmarks only)
pip install -e ".[hest,plism]"    # with optional HEST + PLISM support

Copy .env.example to .env and fill in DATA_ROOT, EMBEDDINGS_ROOT, OUTPUT_ROOT, and HF_TOKEN before running anything.

Quickstart

# Run one benchmark on one model
patheval eva run kaiko_vits16 bach

# Run every eva benchmark on one model
patheval eva run kaiko_vits16

# Gene expression prediction (HEST)
patheval hest run kaiko_vits16

# Cross-scanner robustness (PLISM)
patheval plism run kaiko_vits16

# Render the heatmap + CSV for any family
patheval eva plot
patheval hest plot
patheval plism plot

List every registered model:

patheval models list

Benchmarks

Eva — classification and segmentation

Linear probing on frozen embeddings (offline) and online segmentation with ConvDecoderWithImage, matching the eva pathology leaderboard protocol.

The six benchmarks below were chosen for their modest compute and storage footprint — the full suite fits on a single consumer GPU and under ~30 GB of disk. eva supports larger pathology datasets as well (e.g., Camelyon16 at ~700 GB, full TCGA splits, etc.); to enable one, drop a YAML into configs/eva/offline/ or configs/eva/online/ and add a corresponding entry to configs/eva/benchmarks.yaml (display name, primary metric, optional average-exclusion flag) so it renders correctly in tables and heatmaps.

Benchmark	Task	Cancer type	Classes	Train / Val	Magnification	Size	Dataset
PCam	Binary classification	Breast	2	262k / 33k	10× (1.0 μm/px)	~8 GB	Zenodo
CRC	9-class classification	Colorectal	9	100k / 7.2k	20× (0.5 μm/px)	~12 GB	Zenodo
BACH*	4-class classification	Breast	4	268 / 132	20× (0.42 μm/px)	~10 GB	Zenodo
MHIST	Binary classification	Colorectal polyp	2	2.2k / 977	5× (2.0 μm/px)	~350 MB	BMIRDS
CoNSeP	Nuclear segmentation	Colorectal	5 (incl. bg)	27 / 14	40× (0.25 μm/px)	~800 MB	Warwick
MoNuSAC	Nuclear segmentation	Multi-organ	5 (+ambiguous)	209 / 85	40× (0.25 μm/px)	~600 MB	GrandChallenge

* BACH is excluded from the average ranking because of microns-per-pixel inconsistencies in the dataset; see the eva leaderboard note and BACH dataset docs.

HEST — gene-expression prediction

Given a 112×112 μm H&E patch centred on a spatial-transcriptomics spot, predict expression of the 50 most variable genes. Evaluation uses Ridge regression with PCA (256 dims) on frozen embeddings; metric is Pearson correlation per gene, averaged across genes and folds.

Task	Cancer type	Samples	Technology	Size
IDC	Breast (invasive ductal)	4	Xenium	~5.6 GB
PRAD	Prostate	23	Visium	~11.0 GB
PAAD	Pancreas	3	Xenium	~1.2 GB
SKCM	Melanoma	2	Xenium	~0.5 GB
COAD	Colon	4	Xenium	~2.5 GB
READ	Rectum	4	Visium	~1.6 GB
CCRCC	Kidney (clear cell RCC)	24	Visium	~14.0 GB
LUNG	Lung	2	Xenium	~0.8 GB
LYMPH_IDC	Lymph node (breast)	4	Visium	~4.0 GB

Total benchmark data: ~42 GB (auto-downloaded from HuggingFace on first run).

PLISM — cross-scanner/staining robustness

Same tissue, 7 different scanner/staining combinations. A robust encoder should produce matching features across scans. We report median cosine similarity and top-10 retrieval accuracy, stratified by whether pairs differ in scanner, staining, or both.

Metric	Description
Cosine similarity	Feature consistency across matched tile pairs (all conditions)
Top-10 accuracy	Retrieval accuracy: is the matched tile in a different scan among the 10 nearest neighbours?
Leaderboard score	Mean of cosine-median (all) + top-10 median (scanner / staining / both)

The dataset contains 91 H5 files (~225 GB): 13 registered tissue sections × 7 scanner/staining combinations × 16,278 tiles per slide at 224 × 224 pixels. Download from HuggingFace to $DATA_ROOT/plism/ before running.

Models

15 backbones are registered out of the box (see configs/models.yaml):

Key	Architecture	Pretraining
bioptimus_h_optimus_0	ViT-G/14	500k WSIs
bioptimus_h_optimus_1	ViT-G/14	1M+ WSIs
bioptimus_h0_mini	ViT-B/14	Distilled from H-optimus-0
mahmood_uni2_h	ViT-H/14	350k+ WSIs
paige_virchow2	ViT-H/14	DINOv2, 3.1M WSIs
hkust_gpfm	ViT-L/14	DINOv2, 72k public WSIs
kaiko_vits16	ViT-S/16	DINO, TCGA
kaiko_midnight_12k	ViT-G/14	DINOv2, 12k WSIs
ctranspath	Swin-T + ConvStem	15M patches
retccl	ResNet-50	SimCLR, 32k WSIs
swin_tiny_in22k	Swin-T	ImageNet-22K (general-vision baseline)
swin_tiny_random	Swin-T	Random (null baseline)
vit_small_in21k	ViT-S/16	ImageNet-21K (general-vision baseline)
vit_small_random	ViT-S/16	Random (null baseline)
resnet50_imagenet	ResNet-50	ImageNet-1K (general-vision baseline)

Seven of these use eva's built-in backbone registry. The other eight (bioptimus_h_optimus_1, ctranspath, retccl, swin_tiny_in22k, swin_tiny_random, vit_small_in21k, vit_small_random, resnet50_imagenet) are custom adapters shipped in this repo under src/pathology_eval/models/ — either because the model isn't yet in eva (H-optimus-1), because it needs externally-supplied weights (CTransPath, RetCCL), or because it's a baseline eva doesn't provide (the Swin-T, ViT-S/16, and ResNet-50 rows).

Adding your own model

There are two paths, depending on whether you need custom Python or just new weights.

Path 1 — new weights, existing architecture (most common)

Add an entry to configs/models.yaml under custom_models: and point at your checkpoint. No Python required.

custom_models:
  my_custom_vit:
    display_name: "My custom ViT-S/16"
    model_name:   "pathology/my_custom_vit"
    backbone:     "pathology/vit_small"          # reuse the generic ViT-S adapter
    in_features:  384
    normalize_mean: "[0.5,0.5,0.5]"
    normalize_std:  "[0.5,0.5,0.5]"
    timm_model:   "vit_small_patch16_224"
    weights_file: "/absolute/path/to/my_weights.pth"

Then:

patheval eva run my_custom_vit

Path 2 — custom architecture

Add a wrapper under src/pathology_eval/models/, register it with eva's backbone_registry, and import it from src/pathology_eval/models/__init__.py so the registration fires on package import. A minimal wrapper:

# src/pathology_eval/models/my_fm.py
import timm
import torch.nn as nn
from eva.vision.models.networks.backbones.registry import backbone_registry


@backbone_registry.register("pathology/my_fm")
def my_fm(out_indices=None) -> nn.Module:
    return timm.create_model(
        "hf-hub:me/my-model", pretrained=True, num_classes=0,
        features_only=out_indices is not None, out_indices=out_indices,
    )

Register the import and add a matching models.yaml entry (with model_name: "pathology/my_fm") and you're done.

CLI

patheval eva    run  <model_key> [benchmark]    # one or all eva benchmarks
patheval eva    plot [--show-std]               # eva results table + heatmap
patheval hest   run  <model_key> [--all] [-d TASK ...]
patheval hest   plot [--show-std]
patheval plism  run  <model_key> [--all]
patheval plism  plot
patheval models list                            # registered models

Outputs land in $OUTPUT_ROOT; aggregated CSVs and PNGs are written to results/.

Running on a SLURM cluster

Templates under slurm/ wrap the CLI with sensible SBATCH defaults (1 GPU, 64 GB RAM, 12 h). Override partition, GPU type, and time at submit time:

# Eva benchmark (single model, single benchmark)
sbatch -p gpu --gres=gpu:rtx4090:1 slurm/submit_eva.sh kaiko_vits16 bach

# Fan out every model × benchmark combination
bash scripts/cluster/run_all.sh --all

Citation

BibTeX for the upstream benchmarks

@inproceedings{kaiko.ai2024eva,
    title={eva: Evaluation framework for pathology foundation models},
    author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
    booktitle={Medical Imaging with Deep Learning},
    year={2024},
    url={https://openreview.net/forum?id=FNBQOPj18N}
}

@inproceedings{jaume2024hest,
    author = {Guillaume Jaume and Paul Doucet and Andrew H. Song and Ming Y. Lu and Cristina Almagro-Perez and Sophia J. Wagner and Anurag J. Vaidya and Richard J. Chen and Drew F. K. Williamson and Ahrong Kim and Faisal Mahmood},
    title = {HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis},
    booktitle = {Advances in Neural Information Processing Systems},
    year = {2024},
    month = dec,
}

@misc{filiot2025distillingfoundationmodelsrobust,
      title={Distilling foundation models for robust and efficient models in digital pathology},
      author={Alexandre Filiot and Nicolas Dop and Oussama Tchita and Auriane Riou and Thomas Peeters and Daria Valter and Marin Scalbert and Charlie Saillard and Geneviève Robin and Antoine Olivier},
      year={2025},
      eprint={2501.16239},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.16239},
}