Add a natural-image DINOv2 ViT-B/14 control tile encoder#213
Merged
Conversation
Register `dinov2-vitb14`: the original DINOv2 ViT-B/14 self-supervised on LVD-142M natural images, shipped by timm as `vit_base_patch14_dinov2.lvd142m` (public weights on Hugging Face, no gated access). It mirrors the plain timm ViT tile encoders (lunit / prost40m / uni), so the dense (`encode_tiles_dense`) and attention (`encode_tiles_attention`) paths are inherited unchanged and it is dense-extraction-capable exactly like the pathology encoders. Almost every pathology tile encoder here (UNI, Virchow, GigaPath, H-optimus, Midnight, ...) is a DINOv2-family ViT, so a DINOv2 ViT trained on natural images holds architecture and objective fixed and varies only the pretraining domain -- the control needed to ask "does pathology-pretraining actually pay off?" for a downstream task. Metadata: dim 768, input_size 224, patch_size 14, precision fp16. A natural-image model has no intrinsic micron spacing; supported_spacing_um=0.5 is a convention that matches the pathology tile encoders' default task-spacing so selecting this encoder by name lands on identical tile geometry. Sweep other task-spacings via allow_non_recommended_settings=True (it is spacing-agnostic). `dynamic_img_size=True` lets the natively-518px backbone run at the 224px tile geometry (pos-embed interpolation, a no-op at native size). Add `dinov2` / `dinov2-base` / `dinov2-vitb` -> `dinov2-vitb14` aliases, docs, and registration + dense/attention parity tests mirroring the encoder suite.
3 tasks
Split the overloaded supported_spacing_um into its two real roles: a validation constraint and a tiling default. supported_spacing_um now accepts None to mark a spacing-agnostic encoder (validate_encoder_config skips the spacing check entirely), and default_spacing_um declares the single spacing a slide is tiled at when the encoder is selected by name without an explicit requested_spacing_um. default_spacing_um is optional and derived from supported_spacing_um when that is a single value, so single-spacing encoders need no edits. The multi-spacing guardrail is preserved: encoders advertising a list of spacings still require an explicit choice unless they declare default_spacing_um. Default resolution stays lazy so tile-size validation never trips over a list/agnostic encoder. The natural-image DINOv2 control declares supported_spacing_um=None, default_spacing_um=0.5: it tiles at 0.5 to match the pathology encoders' geometry but accepts any requested spacing without the allow_non_recommended escape hatch, since it has no intrinsic micron-per-pixel spacing to violate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Registers
dinov2-vitb14: the original DINOv2 ViT-B/14 self-supervised on LVD-142M natural images (Oquab et al., 2024), shipped bytimmasvit_base_patch14_dinov2.lvd142m. Weights are hosted publicly on Hugging Face undertimm/vit_base_patch14_dinov2.lvd142m— no gated access, no HF token.Implements clemsgrs/soma#232.
Why
soma's detection benchmark (Paper 1) needs a non-pathology ViT control to answer "does pathology-pretraining actually pay off for detection?" (and Paper 2 the "catches up late" curve). Almost every pathology tile encoder here (UNI, Virchow, GigaPath, H-optimus, Midnight, …) is a DINOv2-family ViT, so pairing them with a DINOv2 ViT trained on natural images holds architecture and the SSL objective fixed and varies only the pretraining domain — exactly the control the benchmark wants.
How
Structurally a plain
TimmTileEncoder, mirroringlunit/prost40m/uni: the dense (encode_tiles_dense) and attention (encode_tiles_attention) paths are inherited unchanged from the timm ViT base, so the control is dense-extraction- and attention-capable exactly like the pathology encoders.dynamic_img_size=Truelets the natively-518px backbone run at the 224px detection tile geometry via positional-embedding interpolation (a no-op at native size, verified in the shared dense-extraction suite).Metadata
leveltileoutput dimdefaultvariantinput_sizepatch_sizeprecisionsupported_spacing_umNonedefault_spacing_umsourcetimm/vit_base_patch14_dinov2.lvd142mSpacing model — split constraint from default
A natural-image model has no intrinsic micron-per-pixel spacing, so advertising
0.5as a supported/recommended spacing (the way the pathology encoders do) would be a lie: it would makevalidate_encoder_configreject any other spacing as "non-recommended" when in fact no spacing is more correct than another. This PR splits the previously overloadedsupported_spacing_uminto its two real roles:supported_spacing_um— the validation constraint: the spacing(s) a model was trained/validated for. Off-list requests raise unlessallow_non_recommended_settings=True. Set toNonehere to mark the encoder spacing-agnostic — the spacing check is skipped entirely.default_spacing_um— the single spacing a slide is tiled at when the encoder is selected by name with no explicitrequested_spacing_um. Set to 0.5 here so the control lands on the pathology encoders' tile geometry and drops in with zero config.Consequences:
dinov2-vitb14by name still tiles at 0.5 µm/px → matched geometry, zero config (acceptance criterion satisfied).default_spacing_umis optional and derived fromsupported_spacing_umwhen that is a single value, so the ~13 single-spacing pathology encoders need no edits. The multi-spacing guardrail is preserved:virchow2/musk/midnight(a list of spacings) still require an explicitrequested_spacing_umunless they opt in with their owndefault_spacing_um. Default resolution is kept lazy so tile-size validation never trips over a list/agnostic encoder.Other judgment calls
vit_base_patch14_dinov2over thereg4variant for the cleanest "vanilla ViT" control: a single CLS prefix token, which is the simplest dense/attention layout (likeuni/midnight/phikon). The reg4 variant is a trivial swap if you'd prefer to match the register-carrying pathology models (UNI2, H-optimus, Virchow2).dinov2/dinov2-base/dinov2-vitb→dinov2-vitb14.Tests
New
tests/test_dinov2_natimage.pymirrors the existing encoder suite:supported_spacing_um is None,default_spacing_um == 0.5) + alias resolution,validate_encoder_configaccepts any requested spacing (0.25 / 2.0) with no escape hatch,get_intermediate_layersoracle at the 224px detection geometry (offline,pretrained=False) — pins spatial registration, not just shape,The registry split (
supported_spacing_umacceptsNone; newdefault_spacing_um) lives inslide2vec/encoders/registry.py; the existing multi-spacing guard tests (virchow2/musk/midnightintest_regression_core/test_regression_models) are unchanged and still pass. Also updated the shared exact-membership lists (test_encoder_registry,test_patch_size_metadata,test_regression_core), docs (docs/models.rst), and the README count.Full suite (
-o addopts=""): 483 passed, 4 skipped (the 4 skips are the weight-dependent heavy tests, which run on the nightly heavy workflow). Separately verified end-to-end with the real downloaded weights on CPU:dense (1, 768, 16, 16)·attn (1, 12, 16, 16)·pooled (1, 768)at 224px.