REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image

Xiaoxuan Ma  Jiashun Wang  Nicolás Ugrinovic  Yehonathan Litman  Kris Kitani

Carnegie Mellon University

   

⚡️ TL;DR: From a single casual image to a visually consistent and physically stable interactive 3D scene.

News

🚩 2026.06: Released the code.

2026.05: Released the arXiv paper and project page.

🛠️ Installation

Please follow INSTALL.md for detailed installation instructions.

🚀 Quick start

Follow the steps below to reconstruct a physically stable 3D scene from a single image and interactively inspect object stability in physics simulator.

Remember to set your API key before running (GEMINI_API_KEY — see INSTALL.md).

Step 1. Infer 3D scene from a single image

conda activate rest3d
bash 1_infer_scenecanon.sh. #Change `INPUT` to your image path (defaults to the demo image)

Outputs are saved to output/<image_name>/stage2/scene_canon/.

Step 2. Stabilize the scene

conda activate gym
bash 2_stable_scene.sh

Outputs are saved to output/<image_name>/stage3/:

• global_scene/ — physically stable scene
• global_scene_w_walls/ — (optional) additionally fits walls to the scene and adjusts wall-attached object positions accordingly

🤗 Visualize and interact with the physically stable scene

conda activate gym
bash 3_replay_in_simulator.sh

Follow the printed URL to open the scene in a browser and interactively inspect the physics simulation settling process.

Citation

If you find this work useful, please cite:

@article{ma2026rest3d,
  title     = {REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image},
  author    = {Ma, Xiaoxuan and Wang, Jiashun and Ugrinovic, Nicol\'{a}s and Litman, Yehonathan and Kitani, Kris},
  booktitle = {arXiv preprint arXiv:2605.30338},
  year      = {2026}
}

Acknowledgements

This repository builds upon the following excellent open-source projects: SAM 3, SAM 3D Objects, and Isaac Gym.

License

This project is released under the CC BY-NC 4.0. See LICENSE for details.

Non-commercial use only. For commercial licensing, please contact the author. Please note that it also relies on external libraries, which may be subject to their own licenses and terms of use.