SYN3R: Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion

* The name SYN3R is an abbreviation for "Novel View Synthesis with Natural Video 3D-aware Regularization"

---

Clone the Code

git clone https://github.com/DecaYale/SYN3R.git

cd SYN3R 
git submodule update --init --recursive

Environmental Setups

We provide install method based on Conda package and environment management:

conda env create --file env.yml
conda activate SV2CGS 

Install 3DGS

cd FSGS
pip install submodules/diff-gaussian-rasterization-confidence
pip install submodules/simple-knn

# If errors like "fatal error: crypt.h: No such file or directory" are encountered, please install libxcrypt and expose its path to the environment. Then run the commands above again. 
conda install --channel=conda-forge libxcrypt
export CPATH=$PATH_TO_CONDA_ENV/include/ 

Install Diffuser

pip install -e ".[torch]"

Download the Thirdparty Pretrained Models

Before runing our code, you need to download the pretrained GMFlow models and Dust3R models. The downloaded model should be put in a folder called "pretrained" in the directory of this project, as we hardcode this part in our code:

#in train_v4.py
self.gmflow = GMFlowInference(model_name='pretrained/gmflow_sintel-0c07dcb3.pth') 
self.dust3r = Dust3RInference(weights_path='pretrained/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth', )

Data Preparation

In data preparation step, we download the official datasets and estimate the camera poses with SfM or transform the provided camera parameters to be compatible with 3D Gaussian Splatting. We may use all the frames for SfM but would train 3D Gaussian Splatting with sparsely sampled views. We do not directly use the reconstructed point clouds from SfM to initialize the 3D Gaussian Splatting.

-->

# Step 1: download processed datasets from https://huggingface.co/datasets/decayale/SV2CGS_DATA/tree/main 

# Step 2:  unzip the data to a target position we refer to as DATA_DIR in the scripts

unzip nerf_llff_data.zip 

unzip DTU.zip 

tar -xvf DL3DV.tar

Training

Train on LLFF dataset with 3 views

# modify the variable $PROJECT_DIR and $DATA_DIR according to the locations where your project and your data are before running the script
bash bash_scripts/batch_llff_train.sh output/llff/

Train on DTU dataset with 3 views

# modify the variable $PROJECT_DIR and $DATA_DIR according to the locations where your project and your data are before running the script
bash bash_scripts/batch_dtu_train.sh output/dtu/

Train on DL3DV dataset with 9 views

# modify the variable $PROJECT_DIR and $DATA_DIR according to the locations where your project and your data are before running the script
bash bash_scripts/batch_dl3dv_train.sh output/dl3dv/ 9 

Rendering and Evaluation

Run the following script to render and evaluate the images.

Evaluate on LLFF dataset

# modify the variable $PROJECT_DIR and $DATA_DIR according to the locations where your project and your data are before running the script
bash bash_scripts/batch_llff_eval.sh output/llff/

Evaluate on DTU dataset

# modify the variable $PROJECT_DIR and $DATA_DIR according to the locations where your project and your data are before running the script
bash bash_scripts/batch_dtu_eval.sh output/dtu/

Evaluate on DL3DV dataset

# modify the variable $PROJECT_DIR and $DATA_DIR according to the locations where your project and your data are before running the script
bash bash_scripts/batch_dl3dv_eval.sh output/dl3dv/

Acknowledgement

Special thanks to the following awesome projects!

• Gaussian-Splatting
• FSGS
• dust3r
• GMFlow

Citation

If you find our work useful for your project, please consider citing the following paper.

@inproceedings{xu2025novel,
        title={Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion},
        author={Xu, Yan and Wang, Yixing and Yu, Stella X},
        booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems}
        year={2025}
}