Adaptive Image Transformer (AIT)

This repository provides the official implementation of the key modules introduced in "Adaptive Image Transformer for One-Shot Object Detection (CVPR 2021)."

TL;DR: AIT uses an encoder-decoder architecture for adaptive proposal-query alignment, along with selective channel attention (SCA) that amplifies or suppresses contributions across heads

tags: transformer, encoder-decoder, attention, object detection

---

Table of Contents 1. Stepup 1.1 Create Conda Environment 1.2 Installation 2. Prerequisites 3. Run Adaptive Image Transformer 4. Citation 5. Acknowledgements

🔥 Updates

• Accepted to CVPR 2021

🛠️ 1. Setup

The project is tested on Python 3.10 and NVIDIA CUDA GPUs

1.1 Create Conda Environment

conda create -n AIT python=3.10 -y
conda activate AIT

1.2 Installation

• Command line for installing PyTorch 2.6

pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124

2. Prerequisites

• Operating system
  • Ubuntu 22.04.5 LTS
• Graphics card
  • GPU: NVIDIA GeForce RTX 3090 or NVIDIA GeForce RTX 4090
• Framework and environment
  • pytorch: 2.6.0
  • cuda: 12.4
• Programming language
  • python: 3.10

Project structure

$ tree ait
ait/
└── transformer
    ├── Layers.py    # Encoder and Decoder layers
    ├── Models.py    # Adaptive Image Transformer (AIT) implementation
    ├── Modules.py   # Scaled dot product attention
    └── SubLayers.py # Selective Channel Attention (SCA) module

🚀 3. Run Adaptive Image Transformer

python adaptive_image_transformer.py

For example, 128 proposals from Faster R-CNN and a batch size of 4. The proposal input is shaped as batch_size × num_proposals along the first dimension, and the AIT module outputs refined proposal features. You'll see that the output shows:

--------------------------------------------------
Input shape:
        Proposal: 512 x 1024 x 7 x 7
          Query : 4 x 1024 x 8 x 8

Output shape:
        Reconstruted Proposal: 512 x 1024 x 8 x 8

🔖 4. Citation

If you find this work helpful for your research, please cite the following paper:

@inproceedings{ChenHL21,
  author       = {Ding-Jie Chen and He-Yen Hsieh and Tyng-Luh Liu},
  title        = {Adaptive Image Transformer for One-Shot Object Detection},
  booktitle    = {CVPR},
  pages        = {12247--12256},
  publisher    = {Computer Vision Foundation / {IEEE}},
  year         = {2021}
}

🙌 5. Acknowledgements

This repository is part of the official AIT project. We also thank the authors of faster-rcnn and CoAE for their excellent work and implementations!