CAFF-DETR

Repository for the paper "CAFF-DINO: Multi-spectral object detection transformers with cross-attention features fusion" [Helvig et al.] , accepted in the 20 ^th IEEE Workshop on Perception on Beyond the Visible Spectrum, taking part of the CVPR 2024 conference. The work explores the adaptation of DETRs architectures for backbone features fusion on IR-visible data, through cross-attention fusion.

Road-Map 🚧

• [11/11/2024] : CAFF-DINO original weights and logs added ! (longer training incoming 👨‍🍳). 🏋️‍♂️
• [11/10/2024] : CAFF-DINO released ! See this folder. 🦖
• [10/09/2024] : CAFF-Lite-DINO is released ! (weights from FLIR and LLVIP trainings are out too). See this folder. 🦎
• [10/2024] : progressive release of models and weights during the month (CAFF-DINO, CAFF-lite-DINO) 🦖 🦎
• [10/2024] : Annotation files for LLVIP and FLIR dataset are added, converted in COCO format 📔
• Several other adaptations of DEtection TRansformers (DETR, H-DETR) may be released later 🤖

Code cleaning : Parsers have been improved and the model is as easy to run as DETRs (heavy machines 😶‍🌫️).

Demo animations 🎥

Examples of multi-spectral object detections using our CAFF-DINO architecture, on FLIR then LLVIP. Labels are in red, while model's detections are in blue. Object Detection's confidence threshold is 50 %.

Core principles 💡

The core principles of the proposed fusion are described in the illustrations bellow. See the paper for more theoretical stuffs.

Illustration of the global detection transformer model : the proposed fusion approach inserts feature fusion operation (CAFF module) at each level of monospectral backbones, merging both modalities.

Illustration of the features fusion module. The main idea is to use a cross-attention operation to extract meaningful correlations between both spectra's features, as a complementary information, before a convolutional features fusion layer.

Annotations 📑

The annotations files are adapted from the standard MS-COCO format. For FLIR, a key "image_IR" is added, to load the correspondant thermal image. Using COCO annotations allows to use the pyCOCOtools API for metrics evaluation, which is "native" in DEtection TRansformers, while MUCH easier than the other techniques.

• LLVIP
  The original image collection can be downloaded here. These data need to be organized as following :

├── LLVIP/
│   ├── visible/
│   │   ├── train/
│   │   └── test/
│   ├── infrared/
│   │   ├── train/
│   │   └── test/
│   ├── coco_annotations/
│   │   ├── train.json
│   │   └── val.json

• FLIR-Aligned
  A source link to the data collection is added here ( ⚠️ various contradictory sources for this dataset). These data need to be organized as following :

├── FLIR_aligned_coco/
│   ├── train_RGB/
│   ├── val_RGB/
│   ├── train_thermal/
│   ├── val_thermal/
│   ├── annotations/
│   │   ├── train.json
│   │   └── val.json

Reference to prior work 🔖

• IR-Visible fusion model using attention operation for YOLO-v5 head: CFT-YOLO v5.

• DETR-DINO original architecture here.

Insights for future works/improvements 🚀

• Backbones frozen for each modality could be sub-optimal: finding ways to train backbones without parameters explosion (unsupervised mono-spectrum pre-training) ?
• Reducing the number of parameters of the fusion module (which is heavy): compression/interpolation, and compensating the decrease of performance with scaling (i.e. more attention layers in the fusion module) ?
• Fusion module empirically designed: grid/other to optimize fusion hyperparameters ?

Cite 📕

If the proposed fusion architecture is used for academic purpose, please consider citing our work:

@inproceedings{HelvigCAFFDINO2024,
  title={CAFF-DINO: Multi-spectral object detection transformers with cross-attention features fusion},
  author={Helvig, Kevin and Abeloos, Baptiste and Trouve-Peloux, Pauline},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  year={2024},
  organization={IEEE}
}