Add GPFM and GenBio-PathFM as supported tile encoders#212
Merged
Conversation
GPFM (majiabo/GPFM, MIT) is a ViT-L/14 DINOv2 tile encoder. Its weights ship as a standalone GPFM.pth checkpoint rather than a timm hf-hub config, so build the timm arch unpretrained and load_state_dict the checkpoint (MOOZY pattern). Verified on the real weights: GPFM.pth is a bare DINOv2 state_dict that loads strict=True with zero missing/unexpected keys and a 1024-d pooled forward. Metadata: dim 1024, input_size 224, patch_size 14, supported_spacing_um 0.25 (40x), precision fp32. Dense-capable. GenBio-PathFM (genbio-ai/genbio-pathfm) is a 1.1B-param custom HF AutoModel (trust_remote_code), so register it as a custom TileEncoder with non-ImageNet normalization. The backbone is a single-channel ViT (in_chans=1, embed_dim 1536): its canonical forward splits RGB into 3 single-channel images and concatenates the per-channel CLS tokens -> 4608-d (verified on real weights; matches the card's advertised feature dim). No dense path, so no patch_size. Metadata: dim 4608, input_size 224, supported_spacing_um 0.5, precision fp32. Add focused metadata-contract tests plus heavy real-weight forward-shape tests, and recognise non-dense tile encoders in the patch_size contract. Also drop the stale gated-weights/HF_TOKEN note from the mSTAR docs row.
GPFM trains on 512px tiles at 0.25 um/px (128 um FOV) resized to 224, so the model effectively sees ~0.5 um/px (20x), matching how UNI's 256px@0.5 -> 224 is registered. 0.25 would tile half the trained FOV.
GenBio-PathFM now supports dense (2D-grid) extraction via the model's forward_with_patches path (per-channel patch grids concatenated along the feature dim -> (B, 4608, 14, 14) at 224; patch size 16). Attention-map extraction stays unsupported: the backbone uses fused SDPA with no exposed weights and encodes the three colour channels independently, so there is no single coherent CLS-over-patches attention to recover. Also align the GenBio pooled transform with its model card (Resize((224, 224)) + custom Normalize; drop the CenterCrop, a no-op after a square resize) and inline the patch_size/encode_dim literals to match the house style.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two open-weight histopathology tile encoders in a single PR (one shared
docker-testrun).Closes #210
Closes #211
GPFM (
gpfm)majiabo/GPFM(MIT) — ViT-L/14 DINOv2. Weights ship as a standaloneGPFM.pthrather than a timmhf-hub:config, so the timm arch is built unpretrained and the checkpoint isload_state_dict-ed (MOOZY pattern). A defensive unwrapper handles wrapped/prefixed re-exports but is a no-op on the published bare state_dict.GPFM.pthis a bare DINOv2state_dict, loadsstrict=Truewith 0 missing / 0 unexpected keys, pooled forward returns(1, 1024).supported_spacing_um=0.5, precision fp32, sourcemajiabo/GPFM. Dense-capable (in both dense coverage lists).0.5µm/px (20x-equivalent): upstream recommends 512px@0.25µm resized to 224, ≈ 0.5 µm/px effective, matching UNI — not the model's native acquisition magnification.GenBio-PathFM (
genbio-pathfm)genbio-ai/genbio-pathfm— 1.1B-param custom HFAutoModel(trust_remote_code), registered as a customTileEncoder(not timm) with non-ImageNet normalization (mean=(0.697,0.575,0.728),std=(0.188,0.240,0.187)).in_chans=1,embed_dim=1536); its canonicalforwardsplits the RGB tile into 3 single-channel images and concatenates the per-channel CLS tokens, returning(1, 4608)(=embed_dim*3, matching the card's advertised feature dim).encode_tilesreturns this default forward output.forward_with_patches(DINOv2-stylex_norm_patchtokens), which returns the three per-channel patch grids concatenated along the feature dim — folded into(B, 4608, 14, 14)at 224 (patch size 16, prefix tokens already stripped). Sod(4608) equals the pooled dim. Added to the dense coverage list.scaled_dot_product_attention(no materialized weights, nooutput_attentions) and encodes the three colour channels independently, so there is no single coherent CLS-over-patches attention to recover — only three separate grayscale-channel attentions. Documented in the encoder docstring and the docs attention section.Resize((224, 224))(exact, tuple form) + custom Normalize, no CenterCrop (the card has none, and it is a no-op after a square resize).get_dense_transformis normalization-only.supported_spacing_um=0.5(20x), precision fp32, sourcegenbio-ai/genbio-pathfm. Custom GenBio AI Community License noted in the docstring.Tests
test_encoder_registry.py(EXPECTED_TILE_ENCODERS+ focusedtest_gpfm_metadata_contract/test_genbio_pathfm_metadata_contract),test_patch_size_metadata.py(gpfm(14,14), genbio-pathfm(16,16); both dense-capable,NON_DENSE_TILE_ENCODERSnow empty),test_regression_core.py(list_models("tile")).tests/test_gpfm_genbio_heavy.py(@pytest.mark.heavy) asserts the real-weight forward shapes and dense grid shapes — GPFM(1, 1024, 16, 16), GenBio(1, 4608, 14, 14); skip cleanly if weights are unavailable.Docs
README tile count 18 → 20;
docs/models.rsttile-table rows for both; both added to the dense-grids coverage list (with a per-channel-concat note for GenBio), and GenBio flagged as not covered in the dense-attention section. Reference cells kept minimal (no license / normalization notes). Also dropped the stale "gated weights, requiresHF_TOKEN" note from the mSTAR row.Notes on divergence from the original plan
0.5(effective input spacing after resize), not the initially proposed0.25(which conflated training magnification with effective input spacing).