PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery

 

 

Elkhan Ismayilzada¹    Yufei Zhang²    Zijun Cui¹

¹Michigan State University    ²Independent Researcher

---

News

• 2026-05-26: Demo code and pretrained checkpoint released.
• 2026-04-09: PAD-Hand selected as a Highlight at CVPR 2026!
• 2026-02-20: PAD-Hand accepted at CVPR 2026!

---

Setup

This project uses two separate conda environments:

Environment	Purpose
wilor	WiLoR hand detection and initial pose estimation
pad_hand	PAD-Hand diffusion refinement and rendering

1. WiLoR Environment (wilor)

Follow the installation instructions in the WiLoR repository.

Then initialize the submodule:

git submodule update --init --recursive

Place pretrained WiLoR weights under:

WiLoR/pretrained_models/
├── wilor_final.ckpt
├── detector.pt
├── model_config.yaml
└── dataset_config.yaml

2. PAD-Hand Environment (pad_hand)

conda create -n pad_hand python=3.7
conda activate pad_hand
pip install -r requirements.txt

Note: torch-scatter requires matching your CUDA and PyTorch versions. See the torch-scatter installation guide.

3. Assets

The MANO model files must be downloaded manually due to licensing. Please register and download from the MANO website and place the processed files under:

assets/
├── MANO_RIGHT.pkl
├── MANO_LEFT.pkl
├── MeshConv_template.ply
└── MeshConv_transform.pkl

Then run the preprocessing script to convert the official MANO files into the format expected by this codebase:

conda activate pad_hand
python mano_preprocessing.py

The mesh convolution module is adapted from MobRecon. The mesh data processing module can be installed from psbody-mesh. We thank them for generously sharing their outstanding work.

---

Demo

conda activate pad_hand
python demo.py \
  --video path/to/input.mp4 \
  --checkpoint path/to/pad_hand.pth \
  --output output.mp4

Example:

python demo.py --video demo_input.mp4 --checkpoint pad_hand.pt

Arguments

Argument	Required	Default	Description
--video	Yes	—	Input video path
--checkpoint	Yes	—	PAD-Hand model checkpoint (.pth) — download here
--output	No	demo_output.mp4	Output video path

The output is a side-by-side video: WiLoR prediction (left, blue) vs PAD-Hand refined (right, green).

WiLoR inference is automatically invoked as a subprocess inside demo.py using conda run -n wilor. No manual step needed.

---

Acknowledgements

This repository is based on

• DIP-Hand
• BayesDiff

Citation

@InProceedings{Ismayilzada_2026_CVPR,
    author    = {Ismayilzada, Elkhan and Zhang, Yufei and Cui, Zijun},
    title     = {PAD-Hand: Physics-Aware Diffusion for Hand Motion Recovery},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {28358-28368}
}