LFNet: Cross-Modal LiDAR-Fisheye Fusion Network for 3D Semantic Segmentation
(ICME 2025)

Weijian Zhang¹, Zhiwei Zhang², Tianfang Sun¹, Zhizhong Zhang¹, Xin Tan¹, Yuan Xie¹

¹ECNU, ²SJTU

Installation

This codebase is built upon Pointcept (commit 2b51931, 2025-11-30).

We recommend following the official instructions to set up the environment.

The instructions below describe the environment configuration used in our experiments.

Requirements

• Ubuntu: 20.04.6

1. Conda Environment

conda create -n lfnet python=3.10 -y
conda activate lfnet
conda install ninja -y
# Choose version you want here: https://pytorch.org/get-started/previous-versions/
pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url https://download.pytorch.org/whl/cu124
pip install h5py pyyaml
pip install sharedarray tensorboard yapf addict einops plyfile termcolor timm wandb scipy tensorboardX
pip install torch-scatter torch-sparse torch-cluster -f https://data.pyg.org/whl/torch-2.5.0+cu124.html
pip install torch-geometric

cd libs/pointops
python setup.py install

pip install open3d
pip install flash-attn==2.7.4.post1 --no-build-isolation
pip install peft
pip install opencv-python-headless

2. Data Preparation

Download dataset from KITTI-360. The following data are required:

1. Fisheye Images (355G)
2. Fisheye Calibration Images (11G)
3. Perspective Images for Train & Val (128G) (Use download_2d_perspective.sh)
4. Raw Velodyne Scans (119G)
5. Raw SICK Scans (0.4G) (We use the data_timestamps_sick.zip)
6. Accumulated Point Clouds for Train & Val (12G)
7. Calibrations (3K)
8. Vechicle Poses (8.9M)

After unzipping, the dataset should be organized as follows:

 LFNet
 │── data
 │	│── kitti360
 │	│	│── calibration
 │	│	│── data_2d_raw
 │	│	│── data_2d_semantics
 │	│	│── data_3d_bboxes
 │	│	│── data_3d_raw
 │	│	│── data_3d_semantics
 │	│	│── data_poses
 │	│	└── test

The KITTI-360 dataset provides 3D semantic annotations for accumulated point clouds. Run the script to generate frame-level annotations:

cd pointcept/datasets/preprocessing/kitti360/recoverKITTI360label
python recoverLabels_mp.py --dataset_root <path to LFNet>/data/kitti360 --output_root <path to LFNet>/data/kitti360_labels

Note1: This script can be slow (may take days) because it's CPU-heavy. We added multi-processing to speed it up, but haven't measured the exact improvement. It uses 4 workers by default. Try using --workers 12 to see if it runs faster.

Note2: If you modify the output_root, you need to modify the label_root in the training configuration as well.

Note3: The script is modified from recoverKITTI360label.

3. Training

1. Single-modal Baseline

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 sh scripts/train.sh -g 8 -d kitti360 -c semseg-pt-v3m1-0-base -n ptv3

2. Cross-modal LFNet

# Fisheye
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 sh scripts/train.sh -g 8 -d kitti360 -c semseg-pt-v3m1-0-base-dinov2-base-cam23-combineblock -n lfnet_base_cam23
# Pinhole
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 sh scripts/train.sh -g 8 -d kitti360 -c semseg-pt-v3m1-0-base-dinov2-base-cam01-combineblock -n lfnet_base_cam01

Acknowledgement

This repo is built upon Pointcept, PointTransformerV3, recoverKITTI360label.

Please cite our work if you find it useful.

@inproceedings{zhang2025lfnet,
  title={LFNet: Cross-Modal LiDAR-Fisheye Fusion Network for 3D Semantic Segmentation},
  author={Zhang, Weijian and Zhang, Zhiwei and Sun, Tianfang and Zhang, Zhizhong and Xin, Tan and Xie, Yuan},
  booktitle={2025 IEEE International Conference on Multimedia and Expo: Journey to the Center of Machine Imagination, ICME 2025-Conference Proceedings},
  year={2025},
  organization={IEEE Computer Society}
}