Optical and SAR Cross-Modal Hallucination Collaborative Learning for Remote Sensing Missing-Modality Building Footprint Extraction

Abstract

Building footprint extraction using optical and synthetic aperture radar (SAR) images enables all-weather capability and significantly boosts performance. In practical scenarios, optical data may not be available, leading to the missing-modality challenge. To overcome this challenge, advanced methods employ mainstream knowledge distillation approaches with hallucination network schemes to improve performance. However, under complex SAR backgrounds, current hallucination-network-based methods suffer from cross-modal information transfer failure between optical and hallucination models. To solve this problem, this study introduces a cross-modal hallucination collaborative learning (CMH-CL) method, consisting of two components: modality-share information alignment learning (MSAL) and multimodal fusion information alignment learning (MFAL). The MSAL method facilitates cross-modal knowledge transfer between optical and hallucination encoders, thereby enabling the hallucination model to effectively mimic the missing optical modality. The MFAL method aligns semantic information between OPT-SAR and HAL-SAR fusion heads to strengthen their semantic consistency, thereby improving HAL-SAR fusion performance. By combining MSAL and MFAL, the CMH-CL method collaboratively alleviates cross-modal transfer failure problem between the optical and hallucination models, thereby improving performance in missing-modality building footprint extraction. Extensive experimental results obtained on a public dataset demonstrate the effectiveness of the proposed CMH-CL.

Overview

Pipeline

MSAL and MFAL

MSAL	MFAL

Getting Started

Training on MSAW Dataset

To train the model on the MSAW dataset, use the following command:

python3 engine/train_val.py --config cfgs/msaw/res50/msaw_bs8_deeplabv3p_s16_cmlcl_res50.yaml

Citation

If this work is helpful to your research, please cite it as follows:

@ARTICLE{wei2026optical,
  author={Wei, Tianyu and Chen, He and Liu, Wenchao and Chen, Liang and Gu, Panzhe and Wang, Jue},
  journal={IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing}, 
  title={Optical and SAR Cross-Modal Hallucination Collaborative Learning for Remote Sensing Missing-Modality Building Footprint Extraction}, 
  year={2026},
  volume={19},
  number={},
  pages={1183-1196},
  keywords={Optical imaging;Optical sensors;Adaptive optics;Synthetic aperture radar;Buildings;Semantics;Optical scattering;Radar polarimetry;Integrated optics;Knowledge transfer;Building footprint extraction;hallucination networks;modality-missing;modality-share information;synthetic aperture radar (SAR)},
  doi={10.1109/JSTARS.2025.3638382}}

If you have any question, please email to me (weity@bit.edu.cn).

Acknowledgment

This project is based on the following works:

• REF-Net (Github Code)
• DisOptNet (Github Code)
• Segmentation Models Pytorch (Github Code)

Thanks for their excellent works.