SCSFusion

Multi-scale Spatial Channel Joint Representation for General Multi-modality Image Fusion with Self-supervision

in IEEE Transactions on Multimedia (IEEE TMM) 🔥🔥🔥
by Jiawei Li, Jiansheng Chen, Jinyuan Liu, Hongwei Yu, Xinlong Ding and Huimin Ma

• [IEEE]

Pipeline of our proposed SCSFusion:

Fusion Network:

‼️Requirements

• python 3.10
• torch 1.13.0
• torchvision 0.14.0
• opencv 4.9
• numpy 1.26.4
• pillow 10.3.0

📑Dataset setting

The reorganized M2FD dataset can be download at [Here]

We give several test image pairs as examples in [M3FD] and [M2DF] datasets, respectively.

Moreover, you can set your own test datasets of different modalities under ./test_images/..., like:

test_images
├── ir
|   ├── 1.png
|   ├── 2.png
|   └── ...
├── vis
|   ├── 1.png
|   ├── 2.png
|   └── ...
├── flair
|   ├── 1.png
|   ├── 2.png
|   └── ...
├── t1
|   ├── 1.png
|   ├── 2.png
|   └── ...

🖥️Test

The pre-trained model has given in ./model/model_fusion_final.pth. Please run test.py to get fused results. You can check them in:

results
├── ir_vis
|   ├── 1.png
|   ├── 2.png
|   └── ...
├── flair_t1
|   ├── 1.png
|   ├── 2.png
|   └── ...

⌛Train

You can also utilize your own data to train a new fusion model with:

python train.py

The visual-focused maps can be obtained by VFM.

🌟Experimental results

we compared our proposed A2RNet with [TarDAL], [SeAFusion], [IGNet], [PAIF], [CoCoNet], [LRRNet] and [EMMA].

Fusion results:

IVIF

MIF

After retaining the fusion results of all methods on [YOLOv5] and [DeepLabV3+], we compare the corresponding detection and segmentation results with SCSFusion.

Segmentation results:

Please refer to the paper for more experimental results and details.

🗒️Citation

@article{li2026multi,
 title={Multi-Scale Spatial Channel Joint Representation for General Multi-Modality Image Fusion With Self-Supervision},
 author={Li, Jiawei and Chen, Jiansheng and Liu, Jinyuan and Yu, Hongwei and Ding, Xinlong and Ma, Huimin},
 journal={IEEE Transactions on Multimedia},
 year={2026},
 publisher={IEEE}
}

🧩Realted works

• Jiawei Li, Hongwei Yu, Jiansheng Chen, Xinlong Ding and et al. A2RNet: Adversarial Attack Resilient Network for Robust Infrared and Visible Image Fusion. Proceedings of the AAAI Conference on Artificial Intelligence, (AAAI), 2025, 39(5): 4770-4778. [Paper] [Code]
• Jiawei Li, Jiansheng Chen, Jinyuan Liu and Huimin Ma. Learning a Graph Neural Network with Cross Modality Interaction for Image Fusion. Proceedings of the 31st ACM International Conference on Multimedia (ACM MM), 2023: 4471-4479. [Paper] [Code]
• Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. GeSeNet: A General Semantic-guided Network with Couple Mask Ensemble for Medical Image Fusion. IEEE Transactions on Neural Networks and Learning Systems (IEEE TNNLS), 2024, 35(11): 16248-16261. [Paper] [Code]
• Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang and Nikola K. Kasabov. Learning a Coordinated Network for Detail-refinement Multi-exposure Image Fusion. IEEE Transactions on Circuits and Systems for Video Technology (IEEE TCSVT), 2023, 33(2): 713-727. [Paper]
• Jia Lei, Jiawei Li, Jinyuan Liu, Bin Wang, Shihua Zhou, Qiang Zhang, Xiaopeng Wei and Nikola K. Kasabov. MLFuse: Multi-scenario Feature Joint Learning for Multi-Modality Image Fusion. IEEE Transactions on Multimedia (IEEE TMM), 2025. [Paper] [Code]

🙇‍♂️Acknowledgement

We would like to express our gratitude to [Omni-SR] for inspiring our work! Please refer to their excellent work for more details.

📬Contact

If you have any questions, please create an issue or email to me (Jiawei Li).