BEVal: A Cross-dataset Evaluation Study of BEV Segmentation Models for Autononomous Driving

Abstract

This paper addresses the limitation of current research in semantic bird's-eye view (BEV) segmentation for autonomous driving, which typically uses only one dataset for model optimization. We conduct a comprehensive cross-dataset evaluation of state-of-the-art BEV segmentation models to assess their performance across various datasets, setups, and semantic categories. They investigate how different sensors affect model generalization and explore multi-dataset training to improve performance. The study highlights the importance of enhancing model generalizability and adaptability for more robust BEV segmentation in autonomous driving applications.

Key points:

1. Current research often uses single datasets, leading to specialized models prone to domain shift.
2. The paper evaluates BEV segmentation models across multiple datasets and setups.
3. It investigates the impact of different sensors on model generalization.
4. Multi-dataset training experiments are conducted to improve performance.
5. The study emphasizes the need for more generalizable and adaptable models in autonomous driving.

Installation

Prerequisites

• Conda environment manager
• Having the trainval or mini split of the nuScenes dataset downloaded (Link to nuScenes website).
• Having the training or sample split of the Woven Planet dataset downloaded (Link to Woven Planet website).
• Download the subsampled point clouds for the Woven Planet dataset (Link to HuggingFace Dataset Repo):
  • Trainval split point clouds (Direct Download Link)
  • Mini split point clouds (Direct Download Link)

Setup

1. Clone this repository:

   git clone https://github.com/manueldiaz96/beval.git
   cd beval
   

2. Install the required dependencies following the commands under install.txt

   • If you have problems installing the environment, please open an issue.

3. Set up two shell variables pointing to the dataset repositories in your .bashrc:

   export NUSCENES=/path/to/nuscenes/dataset
   export LYFT=/path/to/woven/planet/dataset
   

   • If you plan to use the sample (a.k.a. mini) split for the Woven Planet Dataset, also set up the following environment variable:

     export LYFT_MINI=/path/to/woven/planet/dataset
     

4. Create a folder named subsampled_lidar in your Woven Planet Dataset Folder

   cd $LYFT
   mkdir subsampled_lidar
   cd subsampled_lidar
   

5. Extract the files from the downloaded zip file for the split you need into the subsampled_lidar folder

   unzip YOUR_SPLIT_subsampled_lidar_lyft.zip -d $LYFT/subsampled_lidar
   

After completing all these steps, you should have the environment working. You can test this by loading the beval environment and launching the following command:

conda activate beval
python train_lift_splat.py --cfg=configs/lss_lyft_vehicle.yaml

Training a model

To train a model, use the appropriate script with the desired configuration file:

python train.py --config config/your_config_file.yaml

Testing a model

The models trained for the paper results are now being hosted on HuggingFace - Link to model zoo.

To test any of our models please download manually the weights, or use the HuggingFace CLI to download the whole catalog:

hf download manutheeng/beval_model_zoo --local-dir ./model_zoo

Then, launch the test script for the model you require (lift_splat, LAPT or LAPT_PP) as:

python test_LAPT.py --cfg path/to/LAPT_config.yaml --weights path/to/LAPT_model.pt

Results

Cross-dataset evaluation

Cross-dataset training

Citation

This work is licensed under CC BY-NC. If our work has been useful in your research, please consider citing us:

Manuel Diaz-Zapata, Wenqian Liu, Robin Baruffa, and Christian Laugier. “BEVal: A Cross-dataset Evaluation Study of BEV Segmentation Models for Autononomous Driving”. In: 2024 18th International Conference on Control, Automation, Robotics and Vision (ICARCV). IEEE. 2024, pp. 704–709.

@article{beval,
  title={BEVal: A Cross-dataset Evaluation Study of BEV Segmentation Models for Autonomous Driving},
  author={Diaz-Zapata, Manuel and Liu, Wenqian and Baruffa, Robin and Laugier, Christian},
  booktitle={2024 18th International Conference on Control, Automation, Robotics and Vision (ICARCV)},
  pages={704--709},
  year={2024},
  organization={IEEE}
}