Our implementation of MedPLIB

🔧 Dependencies and Installation

• Python == 3.10.18
• PyTorch == 2.1.2+cu118
• transformers == 4.31.0
• xformers == 0.0.23.post1+cu118
• flash-attn == 2.5.2

Installation

1. Clone repo

git clone https://github.com/qin1122/MedPLIB-M.git
cd MedPLIB-M

2. Install dependent packages (use conda)

conda create --name medplib
conda activate fluxsr_env
conda install python==3.10.18
pip install -r requirements.txt

🗂️ Datasets

1. Please download MeCoVQA dataset from Google Drive.
2. Download the images from SA-Med2D-20M at Huggingface.
3. Data examples can be found in './data'

🏰 Pre-trained Checkpoints

Pre-trained MedPLIB checkpoint can be download here

⚡️ Quick Inference

TRANSFORMERS_OFFLINE=1 deepspeed --include=localhost:1 --master_port=64995 model/eval/vqa_infer.py \
    --version="/path/to/the/medplib_checkpoints" \
    --vision_tower='/path/to/the/clip-vit-large-patch14-336' \
    --answer_type='open' \
    --val_data_path='/path/to/the/pixel_grounding_json_file' \
    --image_folder='/path/to/the/image' \
    --vision_pretrained="/path/to/the/sam-med2d_b.pth" \
    --eval_seg \
    --moe_enable \
    --region_fea_adapter \
    --vis_mask \