Official Repository for ScenarioControl

ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation
Lili Gao^1*, Yanbo Xu^2*, William Koch^2*, Samuele Ruffino¹, Luke Rowe³, Behdad Chalaki¹, Dmitriy Rivkin¹, Julian Ost^1,2, Roger Girgis^1,3, Mario Bijelic^1,2, Felix Heide^1,2
1 Torc Robotics, ² Princeton University, ³ Mila

European Conference on Computer Vision (ECCV), 2026

^* Equal contribution

We propose ScenarioControl, a vision-language controllable framework for learned driving scenario generation.

scenariocontrol.mp4

Table of Contents

1. Setup
2. Dataset Preparation
3. Pre-Trained Checkpoints
4. Training
5. Inference
6. Evaluation
7. Simulation
8. Citation
9. Acknowledgements

Setup

Start by cloning the repository

git clone https://github.com/princeton-computational-imaging/ScenarioControl.git
cd ScenarioControl

This repository assumes you have a "scratch" directory for larger files (datasets, checkpoints, etc.). If disk space is not an issue, you can keep everything in the repository directory:

export SCRATCH_ROOT=$(pwd) # prefer a separate drive? Point SCRATCH_ROOT there instead.

Define environment variables to let the code know where things live:

source $(pwd)/scripts/define_env_variables.sh

Conda Setup

# create conda environment
conda env create -f environment.yml
conda activate ScenarioControl

# login to wandb for experiment logging
export WANDB_API_KEY=<your_api_key>
wandb login

Dataset Preparation

Dataset for downloading is coming soon! You could also follow the steps below to extract and preprocess the data yourself.

NuPlan

We use the same extracted NuPlan data as SLEDGE, with minor modifications tailored for ScenarioControl. Our modified fork for extracting the Nuplan data is available here.

Step-by-Step Instructions

1. Install dependencies & download raw NuPlan data
   Follow the guide in the installation.md file of our forked repo.
   This will walk you through:

   • Downloading the NuPlan dataset
   • Setting up the correct environment variables
   • Installing the sledge-devkit

2. Extract NuPlan data
   Run the following in the forked Repo to preprocess the NuPlan data:

   cd $SLEDGE_DEVKIT_ROOT/scripts/autoencoder/rvae/
   bash feature_caching_rvae_temporal.sh
   bash feature_caching_rvae_temporal_test.sh
   python merge_meta.py
   

3. Extract train/val/test splits and preprocess data for training
   Run the following to extract train/val/test splits and create the preprocessed data for training.

   bash scripts/extract_nuplan_data_3dtemp_wimages.sh # create train/val/test splits and create eval set for computing metrics
   bash scripts/preprocess_nuplan_dataset_3dtemp_wimages.sh # preprocess data to facilitate efficient model training
   

4. Extract DINO patch features and depth maps for image conditioning
   Run the following to extract DINOv3 patch features and a monocular depth map for each camera frame.

   export NUPLAN_DATA_FOLDER=/path/to/nuplan-v1.1 # root of the raw nuPlan dataset (contains sensor_blobs/)
   bash scripts/extract_dino_depth_features.sh
   

5. Prepare captions for prompt conditioning
   Download our captions extracted with GPT-4.1-mini and place them under $SCRATCH_ROOT/captions_bev/gpt_bev

Waymo

Coming soon

Pre-Trained Checkpoints

Pre-trained checkpoints can be downloaded from Google Drive. Place the checkpoints directory into your scratch ($SCRATCH_ROOT) directory.

Checkpoints

• Autoencoder: put under $SCRATCH_ROOT/checkpoints/scenario_control_autoencoder3d_nuplan
• Unconditional Pretrained LDM: put under $SCRATCH_ROOT/checkpoints/scenario_control_ldm_base_nuplan
• LDM for image-conditioning: put under $SCRATCH_ROOT/checkpoints/scenario_control_ldm_img_cond_nuplan
• LDM for prompt-conditioning: put under $SCRATCH_ROOT/checkpoints/scenario_control_ldm_prompt_cond_nuplan

Autoencoder Latent Caching:

# Cached latents are saved to: ae.eval.cache_latents.latent_dir
python eval.py \
  dataset_name=nuplan \
  model_name=autoencoder3dtemp \
  ae.eval.run_name=scenario_control_autoencoder3d_nuplan \
  ae.dataset.load_images=True \
  ae.eval.cache_latents.enable_caching=True \
  ae.eval.cache_latents.split_name=[train|val|test] \
  --config-name=config3dtemp

Training

Pretrain Unconditional LDM

Trains the base unconditional LDM from scratch on all scene types (load_scene_type='012'). Image/prompt conditioning are later added on top of this checkpoint via ldm.train.finetune=True + ldm.train.pretrained_dir (see below).

CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py \
  dataset_name=nuplan \
  model_name=ldm_cond \
  ldm.model.autoencoder_run_name=scenario_control_autoencoder3d_nuplan \
  ldm.dataset.load_scene_type='012' \
  ldm.model.img_conditioning=False \
  ldm.train.run_name=scenario_control_ldm_base_nuplan \
  ldm.train.devices=4 \
  ldm.train.lr=5e-5 \
  ldm.train.track=True \
  ldm.train.save_top_k=-1 \
  ldm.train.check_val_every_n_epoch=5 \
  ldm.train.max_steps=500000 \
  ldm.train.num_samples_to_visualize=3 \
  ldm.datamodule.train_batch_size=64 \
  ldm.datamodule.val_batch_size=64 \
  --config-name=config3dtemp

Finetune LDM with Image Conditioning

Finetunes a pretrained unconditional LDM checkpoint to add single-image conditioning (DINO patch features + depth map, see Dataset Preparation). ldm.train.freeze_pretrained=True freezes every weight that loaded from ldm.train.pretrained_dir, training only the newly-added image-conditioning layers.

CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py \
  dataset_name=nuplan \
  model_name=ldm_cond \
  ldm.model.autoencoder_run_name=scenario_control_autoencoder3d_nuplan \
  ldm.dataset.load_single_img_cond=True \
  ldm.dataset.load_scene_type='12' \
  ldm.model.img_conditioning=True \
  ldm.model.decode_in_training=True \
  ldm.train.run_name=scenario_control_ldm_img_cond_nuplan \
  ldm.train.devices=4 \
  ldm.train.lr=5e-5 \
  ldm.train.track=True \
  ldm.train.finetune=True \
  ldm.train.pretrained_dir=$SCRATCH_ROOT/checkpoints/scenario_control_ldm_base_nuplan \
  ldm.train.freeze_pretrained=True \
  ldm.train.collision_weight=0.001 \
  ldm.datamodule.train_batch_size=64 \
  ldm.datamodule.val_batch_size=64 \
  --config-name=config3dtemp

Finetune LDM with Prompt Conditioning

Finetunes a pretrained unconditional LDM checkpoint to add text-prompt conditioning via a frozen UMT5 text encoder. Captions can be cached (coming soon!) and are encoded on the fly since ldm.dataset.use_cached_text_embeds=False; set it to True instead if you've precomputed a .pt text-embedding cache.

CUDA_VISIBLE_DEVICES=0,1,2,3 python train.py \
  dataset_name=nuplan \
  model_name=ldm_cond \
  ldm.model.autoencoder_run_name=scenario_control_autoencoder3d_nuplan \
  ldm.dataset.load_scene_type='01' \
  ldm.dataset.load_captions=True \
  ldm.model.text_conditioning=True \
  ldm.dataset.use_cached_text_embeds=False \
  ldm.model.decode_in_training=True \
  ldm.train.run_name=scenario_control_ldm_prompt_cond_nuplan \
  ldm.train.devices=4 \
  ldm.train.lr=5e-5 \
  ldm.train.track=True \
  ldm.train.finetune=True \
  ldm.train.pretrained_dir=$SCRATCH_ROOT/checkpoints/scenario_control_ldm_base_nuplan \
  ldm.train.freeze_pretrained=True \
  ldm.train.collision_weight=0.001 \
  ldm.datamodule.train_batch_size=64 \
  ldm.datamodule.val_batch_size=64 \
  --config-name=config3dtemp

Inference

Generate Initial Scenes with LDM with Conditioning

Initial Scene Generation (Image Conditioning):

python test.py \
  dataset_name=nuplan \
  model_name=ldm_cond \
  ckpt_path=$SCRATCH_ROOT/checkpoints/scenario_control_ldm_img_cond_nuplan/last.ckpt \
  ldm.model.autoencoder_run_name=scenario_control_autoencoder3d_nuplan \
  ldm.model.img_conditioning=True \
  ldm.model.decode_in_training=True \
  ldm.dataset.load_single_img_cond=True \
  ldm.dataset.load_scene_type='2' \
  ldm.eval.mode=initial_scene \
  ldm.eval.run_name=scenario_control_ldm_img_cond_nuplan_test \
  ldm.eval.num_samples=100 \
  ldm.eval.visualize=True \
  ldm.eval.visualize_gt=True \
  ldm.eval.cache_samples=True \
  ldm.datamodule.test_batch_size=64 \
  --config-name=config3dtemp

Initial Scene Generation (Prompt Conditioning):

python test.py \
  dataset_name=nuplan \
  model_name=ldm_cond \
  ckpt_path=$SCRATCH_ROOT/checkpoints/scenario_control_ldm_prompt_cond_nuplan/last.ckpt \
  ldm.model.autoencoder_run_name=scenario_control_autoencoder3d_nuplan \
  ldm.model.text_conditioning=True \
  ldm.model.decode_in_training=True \
  ldm.dataset.load_captions=True \
  ldm.dataset.use_cached_text_embeds=False \
  ldm.dataset.load_scene_type='0' \
  ldm.eval.mode=initial_scene \
  ldm.eval.run_name=scenario_control_ldm_prompt_cond_nuplan_test \
  ldm.eval.num_samples=100 \
  ldm.eval.visualize=True \
  ldm.eval.cache_samples=True \
  ldm.datamodule.test_batch_size=64 \
  --config-name=config3dtemp

Outpainting

Coming soon!

Evaluation

Coming soon!

Citation

If you find this work useful, please cite ScenarioControl:

@inproceedings{gao2026scenariocontrol,
  title     = {ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation},
  author    = {Gao, Lili and Xu, Yanbo and Koch, William and Ruffino, Samuele and Rowe, Luke and Chalaki, Behdad and Rivkin, Dmitriy and Ost, Julian and Girgis, Roger and Bijelic, Mario and Heide, Felix},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Acknowledgements

Special thanks to the authors of the following open-source repositories:

• Scenario Dreamer
• SLEDGE
• latent-diffusion
• ctrl-sim