ClipMD

ClipMD is a medical image-text matching model based on OpenAI's CLIP ViT32 model with a sliding window text encoder. ClipMD was fine-tuned on the ROCO dataset.

• ClipMD pretrained model on Hugging Face. Trained weights are also included in this repository.
• ClipMD paper on arxiv

Approach

Usage

First, download the ROCO dataset, clone this repository, and run:

pip install -r requirement.txt

To train the model on the ROCO dataset run:

python train.py PATH_TO_ROCO_ALLDATA

To test the model on the ROCO dataset run:

python recall_at_k.py PATH_TO_ROCO_ALLDATA