Skip to content

Add GPFM and GenBio-PathFM as supported tile encoders#212

Merged
clemsgrs merged 3 commits into
mainfrom
add-gpfm-genbio-tile-encoders
Jun 30, 2026
Merged

Add GPFM and GenBio-PathFM as supported tile encoders#212
clemsgrs merged 3 commits into
mainfrom
add-gpfm-genbio-tile-encoders

Conversation

@clemsgrs

@clemsgrs clemsgrs commented Jun 30, 2026

Copy link
Copy Markdown
Owner

Adds two open-weight histopathology tile encoders in a single PR (one shared docker-test run).

Closes #210
Closes #211

GPFM (gpfm)

majiabo/GPFM (MIT) — ViT-L/14 DINOv2. Weights ship as a standalone GPFM.pth rather than a timm hf-hub: config, so the timm arch is built unpretrained and the checkpoint is load_state_dict-ed (MOOZY pattern). A defensive unwrapper handles wrapped/prefixed re-exports but is a no-op on the published bare state_dict.

  • Verified on real weights: GPFM.pth is a bare DINOv2 state_dict, loads strict=True with 0 missing / 0 unexpected keys, pooled forward returns (1, 1024).
  • Metadata: level tile, dim 1024, input_size 224, patch_size 14, supported_spacing_um=0.5, precision fp32, source majiabo/GPFM. Dense-capable (in both dense coverage lists).
  • Spacing is 0.5 µm/px (20x-equivalent): upstream recommends 512px@0.25µm resized to 224, ≈ 0.5 µm/px effective, matching UNI — not the model's native acquisition magnification.

GenBio-PathFM (genbio-pathfm)

genbio-ai/genbio-pathfm — 1.1B-param custom HF AutoModel (trust_remote_code), registered as a custom TileEncoder (not timm) with non-ImageNet normalization (mean=(0.697,0.575,0.728), std=(0.188,0.240,0.187)).

  • Output dim verified on real weights: the backbone is a single-channel ViT (in_chans=1, embed_dim=1536); its canonical forward splits the RGB tile into 3 single-channel images and concatenates the per-channel CLS tokens, returning (1, 4608) (= embed_dim*3, matching the card's advertised feature dim). encode_tiles returns this default forward output.
  • Dense (2D-grid) support: via the model's forward_with_patches (DINOv2-style x_norm_patchtokens), which returns the three per-channel patch grids concatenated along the feature dim — folded into (B, 4608, 14, 14) at 224 (patch size 16, prefix tokens already stripped). So d (4608) equals the pooled dim. Added to the dense coverage list.
  • Attention-map extraction is not supported (deliberate opt-out): the backbone uses a fused scaled_dot_product_attention (no materialized weights, no output_attentions) and encodes the three colour channels independently, so there is no single coherent CLS-over-patches attention to recover — only three separate grayscale-channel attentions. Documented in the encoder docstring and the docs attention section.
  • Pooled transform aligned with the model card: Resize((224, 224)) (exact, tuple form) + custom Normalize, no CenterCrop (the card has none, and it is a no-op after a square resize). get_dense_transform is normalization-only.
  • Metadata: level tile, dim 4608, input_size 224, patch_size 16, supported_spacing_um=0.5 (20x), precision fp32, source genbio-ai/genbio-pathfm. Custom GenBio AI Community License noted in the docstring.

Tests

  • Metadata-contract additions: test_encoder_registry.py (EXPECTED_TILE_ENCODERS + focused test_gpfm_metadata_contract / test_genbio_pathfm_metadata_contract), test_patch_size_metadata.py (gpfm (14,14), genbio-pathfm (16,16); both dense-capable, NON_DENSE_TILE_ENCODERS now empty), test_regression_core.py (list_models("tile")).
  • tests/test_gpfm_genbio_heavy.py (@pytest.mark.heavy) asserts the real-weight forward shapes and dense grid shapes — GPFM (1, 1024, 16, 16), GenBio (1, 4608, 14, 14); skip cleanly if weights are unavailable.
  • Default suite stays network-free / weight-free.

Docs

README tile count 18 → 20; docs/models.rst tile-table rows for both; both added to the dense-grids coverage list (with a per-channel-concat note for GenBio), and GenBio flagged as not covered in the dense-attention section. Reference cells kept minimal (no license / normalization notes). Also dropped the stale "gated weights, requires HF_TOKEN" note from the mSTAR row.

Notes on divergence from the original plan

  • GPFM spacing landed at 0.5 (effective input spacing after resize), not the initially proposed 0.25 (which conflated training magnification with effective input spacing).
  • GenBio gained a dense (2D-grid) path; the original plan scoped it pooled-only. Attention grids were investigated against the real model source and deliberately left unsupported (reasoning above).

clemsgrs added 3 commits June 30, 2026 15:32
GPFM (majiabo/GPFM, MIT) is a ViT-L/14 DINOv2 tile encoder. Its weights
ship as a standalone GPFM.pth checkpoint rather than a timm hf-hub config,
so build the timm arch unpretrained and load_state_dict the checkpoint
(MOOZY pattern). Verified on the real weights: GPFM.pth is a bare DINOv2
state_dict that loads strict=True with zero missing/unexpected keys and a
1024-d pooled forward. Metadata: dim 1024, input_size 224, patch_size 14,
supported_spacing_um 0.25 (40x), precision fp32. Dense-capable.

GenBio-PathFM (genbio-ai/genbio-pathfm) is a 1.1B-param custom HF AutoModel
(trust_remote_code), so register it as a custom TileEncoder with non-ImageNet
normalization. The backbone is a single-channel ViT (in_chans=1, embed_dim
1536): its canonical forward splits RGB into 3 single-channel images and
concatenates the per-channel CLS tokens -> 4608-d (verified on real weights;
matches the card's advertised feature dim). No dense path, so no patch_size.
Metadata: dim 4608, input_size 224, supported_spacing_um 0.5, precision fp32.

Add focused metadata-contract tests plus heavy real-weight forward-shape
tests, and recognise non-dense tile encoders in the patch_size contract.
Also drop the stale gated-weights/HF_TOKEN note from the mSTAR docs row.
GPFM trains on 512px tiles at 0.25 um/px (128 um FOV) resized to 224,
so the model effectively sees ~0.5 um/px (20x), matching how UNI's
256px@0.5 -> 224 is registered. 0.25 would tile half the trained FOV.
GenBio-PathFM now supports dense (2D-grid) extraction via the model's
forward_with_patches path (per-channel patch grids concatenated along the
feature dim -> (B, 4608, 14, 14) at 224; patch size 16). Attention-map
extraction stays unsupported: the backbone uses fused SDPA with no exposed
weights and encodes the three colour channels independently, so there is no
single coherent CLS-over-patches attention to recover.

Also align the GenBio pooled transform with its model card (Resize((224, 224))
+ custom Normalize; drop the CenterCrop, a no-op after a square resize) and
inline the patch_size/encode_dim literals to match the house style.
@clemsgrs clemsgrs merged commit f988b80 into main Jun 30, 2026
3 of 4 checks passed
@clemsgrs clemsgrs deleted the add-gpfm-genbio-tile-encoders branch June 30, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add GenBio-PathFM as a supported tile encoder Add GPFM as a supported tile encoder

1 participant