Slot4ACir

Official implementation of the ICME 2025 paper "Slot Inversion for Asymmetric Composed Image Retrieval"Paper

Highlights

We propose a novel framework, Slot Inversion for Asymmetric Composed Image Retrieval (Slot4ACir), which adopts an asymmetric setting: lightweight models are employed on the query side, while large vision-language models are deployed on the gallery side. In this framework, we propose a lightweight inversion module based on slot attention, which maps an image into multiple textual tokens with distinct semantics. Additionally, the LLM sampler and the distillation alignment (DTA) loss are introduced to facilitate richer semantic interactions and the extraction of more informative representations, respectively.

Usage

Requirements

We use a single NVIDIA 4090/A100 GPU for training and evaluation.

conda create -n slot4acir -y python=3.9
conda activate slot4acir
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2
pip install -r requirements.txt

Prepare Datasets

We have uploaded all used datasets to Google Drive, which includes the original CIRR and FashionIQ dataset, as well as additional text data generated by the LLM sampler. Unzip and organize them in your dataset root folder as follows:

|-- YOUR_DATA_ROOT
   |-- CIRR
      |-- cirr
         |--captions
            |-- cap.rc2.llm.train.json
            |-- cap.rc2.test1.json
            |-- cap.rc2.train.json
            |-- cap.rc2.val.json
         ...
      |-- dev
      |-- test1
      |-- train
   |-- FashionIQ
      |-- captions
         |-- cap.dress.llm.train.json
         |-- cap.shirt.llm.train.json
         |-- cap.toptee.llm.train.json
         ...
      |-- image_splits
      |-- images

Finally, modify the base_path variable in the data_utils.py file to Path("YOUR_DATA_ROOT").

Training & Evaluation

Specify a dataset and start to train your lightweight CIR models.

CUDA_VISIBLE_DEVICES=0 \
python blip2_fine_tune.py \
--exp-name train_slot4acir_fiq \
--dataset fashioniq \
--blip-model-name blip2_light_cir \
--num-epochs 30 \
--num-workers 4 \
--learning-rate 2e-5 \
--batch-size 96 \
--llm-sampler \
--save-training \
--save-best \
--validation-frequency 1

CUDA_VISIBLE_DEVICES=0 \
python blip2_fine_tune.py \
--exp-name train_slot4acir_cirr \
--dataset cirr \
--blip-model-name blip2_light_cir \
--num-epochs 50 \
--num-workers 4 \
--learning-rate 1e-5 \
--batch-size 96 \
--llm-sampler \
--save-training \
--save-best \
--validation-frequency 1

# optional
# --light-model-name efficientnet-b0 by default (choose from ["efficientnet-b0", "efficientnet-b1", "efficientnet-b2", "efficientvit-m4", "efficientvit-m2", "mobilenetv3-s", "mobilenetv3-l"])
# --num-slots 8 by default can be 16, 32...
# --use-adapt (whether to use adaptive slot attention [ref: Adaptive Slot Attention: Object Discovery with Dynamic Slot Number, Fan et al.])
# --loss-setting itc dta by default (nargs='+', must contains itc)

After training, you can run the following script to generate json files, and upload them to the remote sever for evaluation on the CIRR test split. Note that the --config-path and --model-path need to changed to your own paths, which have been automatically saved in the log directory.

CUDA_VISIBLE_DEVICES=0 \
python cirr_test_submission.py \
--save-name slot_light \
--blip-model-name blip2_light_cir \
--config-path path to the .json config file \
--model-path path to the .pt checkpoint \

Acknowledgments

Some components of this code implementation are adapted from SPRC, slot-attention, mobilenetv3, EfficientViT and EfficientNet PyTorch. We sincerely appreciate for their contributions.

Citation

If you find our work useful for your research, please cite our paper.

@inproceedings{slot4acir,
   author = { Li, Haiwen and Chen, Zining and Liu, Ying and Su, Fei and Zhao, Zhicheng },
   booktitle = { 2025 IEEE International Conference on Multimedia and Expo (ICME) },
   title = { Slot Inversion for Asymmetric Composed Image Retrieval },
   year = {2025},
   pages = {1-6},
   doi = {10.1109/ICME59968.2025.11209393},
   url = {https://doi.ieeecomputersociety.org/10.1109/ICME59968.2025.11209393},
   publisher = {IEEE Computer Society},
   address = {Los Alamitos, CA, USA},
   month = Jul
}

Contact

If you have any question, please contact us. E-mail: lihaiwen@bupt.edu.cn.