Multimodal Beamforming: Transformer-Based Sensing-Assisted Communication

Sensing-Assisted High Reliable Communication: A Transformer-Based Beamforming Approach
Yuanhao Cui, Jiali Nie, Xiaowen Cao, Tiankuo Yu, Jiaqi Zou, Junsheng Mu, Xiaojun Jing
IEEE Journal of Selected Topics in Signal Processing, 2024
[Paper] | [arXiv]

Official implementation of multimodal learning-based beamforming using Transformer architectures for sensing-assisted communication.

📜 Citation

If you find this work useful, please consider citing our paper:

@ARTICLE{10539181,
  author={Cui, Yuanhao and Nie, Jiali and Cao, Xiaowen and Yu, Tiankuo and Zou, Jiaqi and Mu, Junsheng and Jing, Xiaojun},
  journal={IEEE Journal of Selected Topics in Signal Processing},
  title={Sensing-Assisted High Reliable Communication: A Transformer-Based Beamforming Approach},
  year={2024},
  doi={10.1109/JSTSP.2024.3405859}
}

🚀 Installation

# Clone the repository
git clone https://github.com/yuanhao-cui/multimodal_beamforming.git
cd multimodal_beamforming

# Create a virtual environment (optional but recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

📂 Repository Structure

multimodal_beamforming/
├── main.py                 # Main training/testing script
├── model.py                # TransFuser model architecture
├── data.py                 # Data loading and preprocessing
├── config_seq.py           # Model and training configuration
├── scheduler.py            # Learning rate scheduler
├── Data_Augmentation/      # Data augmentation scripts
├── Data_Preprocessing/     # Raw dataset preprocessing
├── Dataset/                # Dataset directory (not included)
├── requirements.txt        # Python dependencies
└── README.md

📥 Dataset

The dataset required for training and evaluation can be downloaded from Google Drive:

🔗 Dataset Download

After downloading, extract and place the dataset inside the Dataset/ folder:

Dataset/
├── Multi_Modal/
├── Adaptation_dataset_multi_modal/
└── Multi_Modal_Test/

🏋️ Training

# Basic training
python main.py --id experiment_1 --epochs 150 --batch_size 64 --lr 5e-4

# Training with data augmentation
python main.py --id experiment_aug --augmentation 1 --flip 1

# Training with EMA (Exponential Moving Average)
python main.py --id experiment_ema --ema 1

Key Arguments

Argument	Default	Description
--id	test_cui	Experiment identifier
--epochs	150	Number of training epochs
--batch_size	64	Batch size
--lr	5e-4	Learning rate
--loss	focal	Loss function (ce or focal)
--scheduler	1	Use learning rate scheduler
--augmentation	1	Enable data augmentation
--ema	0	Enable exponential moving average

🧪 Testing

# Run testing
python main.py --id test_run --Test 1

Test results will be saved as beam_pred.csv and beam_pred_confidence_seq.csv.

📊 Results

Main Results (Distance-Based Accuracy Score)

Evaluated on the DeepSense 6G multimodal beam prediction challenge dataset.

Method	Overall	Scenario 31	Scenario 32	Scenario 33	Scenario 34
Images³⁴ + GPS (Flipping Aug.) ⭐	0.7844	0.7298	0.7852	0.8462	0.8433
Images³⁴ + GPS	0.7767	0.7253	0.8000	0.8038	0.8560
Images³⁴ + GPS (Image Aug.)	0.7127	0.5764	0.7654	0.8576	0.8483
Images³⁴ + Radar + LiDAR³⁴	0.7358	0.6649	0.7938	0.7919	0.8142
Raw Image³⁴ (Camera only)	0.7548	0.6982	0.7160	0.8024	0.8494
GPS (Angle calibration)	0.7425	0.6353	0.7704	0.8229	0.8906
Radar only	0.3563	0.2936	0.3160	0.4800	0.3842
LiDAR only	0.4422	0.3260	0.4272	0.6705	0.4707
Best leaderboard score	0.7162	0.6536	0.7074	0.8576	0.7120

Scenario descriptions: 31 = Unseen day location, 32 = Day location, 33/34 = Night locations
Data from Multimodal Transformers for Wireless Communications (IEEE JSTSP 2024)

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.