[CVPR 2026 Highlight] No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency

Check Project Page for More Visuals

🧐Overview

In multi-modal learning from different sensors, such as RGB + thermal cameras, Near-Infared (NIR) cameras, or Synthetic Aperture Radar (SAR), most prior works assume paired data exist and only focus on designing the network for fusing the multi-modal features.

However, in real-world applications, especially in robotics and autonomous driving, we often encounter scenarios where perfectly aligned pairs do not exist.

Toward the goal, traditional pipelines require laborious calibration and depth estimation to establish cross-sensor correspondences, which can be costly and error-prone. In this work, we present the first scalable data processing framework that attempts to align the view from raw sensor sequsences.

🔧Environment

Before running any script, update the local paths for your machine. The repository uses /path/to/3D-RGBX as a placeholder; replace it with the absolute path to this cloned repository, or export RGBX_ROOT before running the pipeline:

export RGBX_ROOT=/path/to/3D-RGBX

Create the basic Python environment from the repository root. Python 3.10 and a CUDA-capable GPU are recommended for the default pipeline.

conda create -n 3d-rgbx python=3.10 -y
conda activate 3d-rgbx

# Install PyTorch for your CUDA version first. Example for CUDA 12.1:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

pip install -U pip setuptools wheel
pip install -r requirements.txt

# Install 3D Gaussian Splatting CUDA extensions from the repository root.
pip install -e ./dual-diff-gaussian-ray-splatter
pip install -e ./simple-knn

The densification scripts import NVIDIA Apex. If Apex is not already available in your environment, install it after PyTorch:

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation \
  git+https://github.com/NVIDIA/apex.git

Prepare the pretrained weights under checkpoints/ or override CHECKPOINT_ROOT when running the pipeline. The default pipeline expects, which can be downloaded here:

checkpoints/
  resnet34.pth
  densification_rgbt.pt
  densification_nir.pt
  densification_normal.pt
  minima_loftr.ckpt
  weights_xoftr_640.ckpt
  sam2.1_hiera_large.pt
  groundingdino_swint_ogc.pth

😺Data

RGBT-Scenes

We preprocess RGBT-Scenes with COLMAP. Download the data here. The folder contains RoadBlock, Parterre, LandScape, Dimsum, and Building. Put the dataset under:

RGBT-Scenes/

Each scene contains:

RGBT-Scenes/<scene_name>/
  rgb/
    train/
    test/
  thermal/                 # raw thermal images
  thermal_aug_1chan/        # augmented thermal images for matching
    train/
  sparse/0/                 # COLMAP results
  image_generation/         # StyleBooth baseline output

METU-VisTIR

Download the data here. Place the scenes under the repository root, or override SCENE_ROOT when running. The default layout is:

METU_VisTIR/
  scene_1/
    sparse/0/
      cameras.bin
      images.bin
      points3D.bin
    visible/
      images_1920_1080/
      images/
    thermal/
      images/
    normal_generation/
    image_generation/

image_generation/ and normal_generation/ are generated by data engine pipeline fro MINIMA. Generated normal maps will be used for subsequent normal map experiments.

NIR

For NIR sequences, place the data under (downloaded here):

NIR_colmap/
  <scene_name>/
    rgb_fil/
    nir_fil_aug/
    sparse/0/

The default example uses:

NIR_colmap/09-28-16-48-17-1/

Normal Maps

For normal-map experiments, use the METU-VisTIR scene layout and add generated normal maps:

METU_VisTIR/
  <scene_name>/
    sparse/0/
    visible/
      images_518_518/ or images_1920_1080/ or images/
    normal_generation/ or normal/images/

🚀Processing

Common Setup

Important: before running, you must update the repository path. Either edit RGBX_ROOT in scripts/pipeline_common.sh or export it in your shell:

export RGBX_ROOT=/path/to/3D-RGBX
export DENSIFICATION_ROOT=$RGBX_ROOT/densification
export CHECKPOINT_ROOT=$RGBX_ROOT/checkpoints

Replace /path/to/3D-RGBX with the absolute path to this repository. The placeholder path will not run as-is.

RGBT-Scenes

Edit execute_pipeline_rgbt.sh for the scene you want to process.

scene_name="Building"
SCENE_ROOT="./RGBT-Scenes/$scene_name"
RGB_DIR="$SCENE_ROOT/rgb/train"
TARGET_DIR="$SCENE_ROOT/thermal_aug_1chan/train"

Set the RGBT densification checkpoint if needed:

export RGBT_DENSIFICATION_CKPT=$CHECKPOINT_ROOT/densification_rgbt.pt

Run the full RGBT processing pipeline:

bash execute_pipeline_rgbt.sh

The script runs the following stages:

1. Generate semi-dense RGB-X matching maps with semidense_matching.py at three match thresholds.
2. Densify each matching result with densification/src/first_densi.py.
3. Average the three densified outputs with level_mean.py.
4. Filter the averaged maps with filtering.py.
5. Refine the filtered maps with densification/src/second_densi.py at three sample rates.
6. Average the refined outputs into the final result.

Outputs are written to:

demo/pipelines/<scene_name>/
  matching/
  dens/
  mean/
  filtered/
  refined/
  refined_mean/
  gs_model/              # trained 3DGS checkpoint and config
  gs_rendered/
    train/               # rendered train views
    test/                # rendered test views

After refined_mean/ is generated, execute_pipeline_rgbt.sh automatically trains and renders a 3D Gaussian Splatting model with:

RGB supervision:     ./RGBT-Scenes/<scene_name>/rgb/train
Thermal supervision: ./demo/pipelines/<scene_name>/refined_mean

Training uses the rgb/train split only. Rendering saves train and test views separately when rgb/test images are available. RGBT-Scenes are rendered without the METU width crop.

If your RGBT-Scenes folder is outside this repository, override the dataset root:

SCENE_ROOT=/path/to/RGBT-Scenes/Building \
  bash execute_pipeline_rgbt.sh

METU-VisTIR RGBT

Use the separate METU pipeline script:

RGBX_ROOT=/path/to/3D-RGBX \
scene_name=scene_1 \
  bash execute_pipeline_metu_rgbt.sh

Before launching the full processing job, you can validate paths and checkpoints without running matching, densification, or 3DGS:

RGBX_ROOT=/path/to/3D-RGBX \
scene_name=scene_1 \
DRY_RUN=1 \
  bash execute_pipeline_metu_rgbt.sh

By default, the METU script looks for:

METU_VisTIR/<scene_name>/
  sparse/0/
  visible/images_1920_1080/ or visible/images/
  thermal/images_1920_1080/, thermal/images/, infrared/images_1920_1080/, or infrared/images/

For the default layout, <scene_name> is directly under METU_VisTIR/, such as METU_VisTIR/scene_1. If your copy is nested by condition, set METU_CONDITION or override SCENE_ROOT.

If your METU folders use different names, override the paths explicitly:

SCENE_ROOT=/path/to/METU_VisTIR/scene_1 \
RGB_DIR=/path/to/METU_VisTIR/scene_1/visible/images_1920_1080 \
TARGET_DIR=/path/to/METU_VisTIR/scene_1/thermal/images \
  bash execute_pipeline_metu_rgbt.sh

To process another scene, only change scene_name:

RGBX_ROOT=/path/to/3D-RGBX \
scene_name=scene_2 \
  bash execute_pipeline_metu_rgbt.sh

METU outputs are written to:

demo/pipelines/METU_VisTIR/<scene_name>/
  matching/
  dens/
  mean/
  filtered/
  refined/
  refined_mean/
  gs_model/
  gs_rendered/
    train/
    test/

After refined_mean/ is generated, execute_pipeline_metu_rgbt.sh automatically trains and renders a 3DGS model. For METU-VisTIR scenes, render.py automatically applies the width crop and 518 resize.

NIR

Use the NIR pipeline:

RGBX_ROOT=/path/to/3D-RGBX \
scene_name=09-28-16-48-17-1 \
  bash execute_pipeline_nir.sh

NIR outputs are written to:

demo/pipelines/NIR_colmap/<scene_name>/
  matching/
  dens/
  mean/
  filtered/
  refined/
  refined_mean/
  gs_model/
  gs_rendered/
    train/
    test/

After refined_mean/ is generated, execute_pipeline_nir.sh automatically trains and renders a 3DGS model. The NIR target is passed through the existing --thermal argument used by the 3DGS code:

RGB supervision: ./NIR_colmap/<scene_name>/rgb_fil
NIR supervision: ./demo/pipelines/NIR_colmap/<scene_name>/refined_mean

Normal Maps

Use the normal-map pipeline:

RGBX_ROOT=/path/to/3D-RGBX \
scene_name=scene_1 \
  bash execute_pipeline_normal.sh

By default, the normal-map script looks for:

METU_VisTIR/<scene_name>/
  visible/images_518_518/ or visible/images_1920_1080/ or visible/images/
  normal_generation/ or normal/images/

Normal-map outputs are written to:

demo/pipelines/METU_VisTIR/<scene_name>/normal/
  matching/
  dens/
  mean/

The current code is fixed to 1-channel thermal or NIR. To run 3DGS for 3-channel normal maps, please make the following simple edits. First edit the 3DGS target-channel handling, change dual-diff-gaussian-ray-splatter/cuda_rasterizer/config.h:

#define NUM_CHANNELS_THERMAL 3

Then keep the target channel count consistent in the 3DGS Python code under gaussian-splatting-archive/:

• In scene/gaussian_model.py, initialize _temperature with 3 channels and load/save all temp_* attributes.
• In train.py, keep viewpoint_cam.original_temperature as 3 channels instead of expanding it to 1 channel.
• In train.py, build the normal-map validity mask per pixel, then apply the loss to all 3 channels at valid pixels.
• In render.py, and in render_video.py or render_novel_view.py if used, make sure the Gaussian model reloads the same 3-channel temperature field.

Rebuild the 3DGS rasterizer after editing the CUDA channel count:

cd /path/to/3D-RGBX
pip uninstall -y diff-gaussian-rasterization
pip install -e ./dual-diff-gaussian-ray-splatter --no-build-isolation

After execute_pipeline_normal.sh produces mean/, train and render normal-map 3DGS manually:

cd /path/to/3D-RGBX/gaussian-splatting-archive

python3 train.py \
  -s /path/to/3D-RGBX/METU_VisTIR/scene_1 \
  --thermal /path/to/3D-RGBX/demo/pipelines/METU_VisTIR/scene_1/normal/mean \
  --image /path/to/3D-RGBX/METU_VisTIR/scene_1/visible/images_518_518 \
  -m /path/to/3D-RGBX/demo/pipelines/METU_VisTIR/scene_1/normal/gs_model

python3 render.py \
  -m /path/to/3D-RGBX/demo/pipelines/METU_VisTIR/scene_1/normal/gs_model \
  --render_output /path/to/3D-RGBX/demo/pipelines/METU_VisTIR/scene_1/normal/gs_rendered

If you switch back to thermal or NIR 3DGS, restore NUM_CHANNELS_THERMAL to 1 and rebuild the rasterizer again.

📚Citation

If you find our work useful, please consider cite our paper:

@inproceedings{choyingwu3drgbx,
  title={No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency},
  author={Wu, Cho-Ying and Huang, Zixun and Huang, Xinyu and Ren, Liu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21836--21848},
  year={2026}
}