BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation

CVF Open Access · Paper PDF · arXiv · Citation

Overview

Abstract: Biomedical Vision--Language Models (VLMs) have shown remarkable promise in few-shot medical diagnosis but face a critical bottleneck: fragility to prompt variations. Existing adaptation frameworks typically optimize visual and textual prompts as independent streams, relying on ideal “Golden Prompts.” In clinical reality, where descriptions are often noisy and heterogeneous, this modality isolation leads to unstable cross-modal alignment. To address this, we propose BiomedAP, a vision--informed dual-anchor framework with gated cross-modal fusion. BiomedAP enforces synergistic alignment through two mechanisms: (1) Gated Cross-Modal Fusion, which enables layer-wise interaction between modalities, acting as a dynamic noise regulator to suppress irrelevant textual cues; and (2) a Dual-Anchor Constraint that regularizes learnable prompts toward stable semantic centroids derived from both expert templates (High Anchors) and few-shot visual prototypes (Low Anchors). Extensive experiments across 11 benchmarks demonstrate that BiomedAP consistently surpasses baselines, achieving competitive few-shot accuracy and markedly enhanced robustness under prompt perturbations.

Method

Overall framework of BiomedAP with gated cross-modal fusion. The Dual-Anchor Constraint (DAC) constructs a High-Quality (Expert) anchor and a Low-Quality (Core) anchor, whose features ($f_{high}$, $f_{low}$) regularize the learnable text context to reduce prompt drift. Training uses a Confidence-Aware Adaptive Loss to balance anchor regularization with the task loss. A Unified Interaction Space applies bidirectional MHCA between text-context and visual-prompt tokens at selected layers, followed by class-wise aggregation at inference.

Supported Methods

Method	Paper	Configs	Training Scripts	Trainers
BiomedCoOp	CVPR 2025	link	link	link
CLIP	ICML 2021	link	link	link
CoOp	IJCV 2022	link	link	link
CoCoOp	CVPR 2022	link	link	link
KgCoOp	CVPR 2023	link	link	link
ProGrad	ICCV 2023	link	link	link
CLIP-Adapter	IJCV 2024	link	link	link
Tip-Adapter	ECCV 2022	link	link	link
LP	ICML 2021	link	link	link
LP++	CVPR 2024	link	link	link

---

Results

Results reported below show accuracy for few-shot scenarios as well as base and novel classes across 11 biomedical recognition datasets averaged over 3 seeds.

Few-shot Evaluation

Method	$K=1$	$K=2$	$K=4$	$K=8$	$K=16$
CLIP-Adapter	46.54	45.18	45.45	46.27	47.75
Tip-Adapter	49.11	54.06	58.93	63.46	67.18
Standard LP	51.80	54.54	60.49	67.56	68.70
LP++	53.57	53.55	57.26	64.89	68.70
CoOp	50.18	54.17	59.77	65.85	69.72
CoCoOp	48.53	51.28	54.69	61.09	65.10
KgCoOp	52.16	54.24	59.21	63.87	64.85
ProGrad	51.37	54.40	60.61	65.50	67.03
BiomedCoOp	56.69	58.58	64.45	68.84	72.13
Biomed-DPT	59.03	61.27	66.12	70.76	73.51
BiomedAP(Ours)	63.57	64.89	67.92	71.21	75.09

Base-to-Novel Generalization

Name	Base Acc.	Novel Acc.	HM
BiomedCLIP	47.84	65.42	53.81
CoOp	74.28	67.93	71.11
CoCoOp	72.95	70.02	71.48
KgCoOp	69.15	67.15	68.37
ProGrad	71.97	69.06	70.52
BiomedCoOp	76.16	73.09	74.62
Biomed-DPT	78.06	75.97	77.02
BiomedAP(Ours)	81.62	78.42	80.02

Installation

For installation and other package requirements, please follow the instructions detailed in INSTALL.md.

Data preparation

Please follow the instructions at DATASETS.md to prepare all datasets.

Training and Evaluation

Please refer to the RUN.md for detailed instructions on training, evaluating and reproducing the results using our pre-trained models.

---

Citation

If you find this repository useful for your research, please consider citing our CVPRW paper:

@InProceedings{Tong_2026_CVPR,
  author    = {Tong, Huanyang and Liu, Kai and Kuang, Fangjun and Chen, Huiling},
  title     = {BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  month     = {June},
  year      = {2026},
  pages     = {6969--6977}
}

For the arXiv version, you may cite:

@article{tong2026biomedap,
  title   = {BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation},
  author  = {Tong, Huanyang and Liu, Kai and Kuang, Fangjun and Chen, Huiling},
  journal = {arXiv preprint arXiv:2605.15736},
  year    = {2026},
  doi     = {10.48550/arXiv.2605.15736}
}

---

Acknowledgements

Our code builds upon the Biomed-DPT, BiomedCoOp, CoOp, MaPLe, and LP++ repositories. We are grateful to the authors for making their code publicly available. If you use our model or code, we kindly request that you also consider citing these foundational works.