DreamsatPose

6-DoF pose estimation for spacecraft from a single RGB image.

[Porject page] (https://dreamsatpose.github.io/) Link to the paper: arXiv | [ResearchGate](insert link)

---

1. Clone the repository

git clone https://github.com/ARCLab-MIT/dreamsat-pose.git
cd dreamsat-pose

---

2. Create the environment

conda create -n dreamsatpose python=3.10
conda activate dreamsatpose

---

3. Install dependencies

# Install PyTorch first — adjust cu118 to match your CUDA version
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

# Install everything else
pip install -r requirements.txt

---

4. Set up DINOv3

DreamsatPose uses the DINOv3 ViT-B/16 backbone (86M parameters).

4.1 Clone the DINOv3 repository

git clone https://github.com/facebookresearch/dinov3.git

Place it so the structure looks like this from inside dreamsat-pose/:

dreamsat-pose/
└── dinov3/
    ├── dinov3/
    │   └── hub/
    │       └── backbones.py
    └── ...

4.2 Download the weights

Download the ViT-B/16 distilled weights from HuggingFace: https://huggingface.co/facebook/dinov3-vits16-pretrain-lvd1689m

Then place the downloaded .pth file here:

dreamsat-pose/
└── dinov3/
    └── checkpoints/
        └── dinov3/
            └── dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth

4.3 Update config.py

Open config.py and set these two paths:

dino_repo_dir:     str = "./dinov3"
dino_weights_path: str = "./dinov3/checkpoints/dinov3/dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth"

---

5. Download the trained checkpoint

Download the trained checkpoint from HuggingFace: https://huggingface.co/JosianeUwumukiza/DreamsatPose/tree/main

Place it anywhere on your machine and update config.py:

checkpoint_path: str = "/path/to/GT2d3d_supervised_1000.pth"

---

6. Prepare your CAD model

Option A — You already have a CAD model

If you have a .obj file for your spacecraft, skip to Step 7.

Option B — Reconstruct from an image using Hunyuan3D-2

If you don't have a CAD model, you can reconstruct one from a single image.

Optional but strongly recommended — remove the background first using SAM. Hunyuan3D-2 produces significantly better meshes from foreground-only images. See Step 6.1 below before reconstructing.

Follow the installation and usage instructions here: https://github.com/Tencent-Hunyuan/Hunyuan3D-2

Use the masked image from Step 6.1 as input if you ran SAM. Once you have the .obj file, move to Step 7.

---

6.1 Background removal with SAM (optional but recommended)

Removing the background before reconstruction and inference improves both mesh quality and pose accuracy.

Follow the installation instructions here: https://github.com/facebookresearch/sam3

Place the downloaded weights in:

dreamsat-pose/
└── sam_weights/
    └── <your-sam-checkpoint>.pth

Then update config.py:

sam_checkpoint: str = "./sam_weights/<your-sam-checkpoint>.pth"
sam_model_type: str = "vit_h"

---

7. Run pose estimation

Update config.py with camera instrinsics file paths for inference, if you with to train or finetune check training/training.md

Single image with a precomputed mask

python run.py --image spacecraft.jpg --obj model.obj --mask mask.png

Single image, SAM runs automatically

python run.py --image spacecraft.jpg --obj model.obj --use_sam

Single image, no mask

python run.py --image spacecraft.jpg --obj model.obj

Batch of images

python run.py --image_dir ./images/ --mask_dir ./masks/ --obj model.obj --out results.json

Output

Each call returns:

Key	Description
R	(3, 3) rotation matrix
t	(3,) translation vector, same units as the .obj
q_wxyz	(4,) quaternion [w, x, y, z]
success	True if a pose was found
error	reason for failure if success=False

---

Acknowledgments

The research was sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.