EgoExoMoCap: Distributed Ego-Exo Human Motion Capture

ECCV 2026 Oral (Spotlight)

Jiaxi Jiang^1,2*, Bharat Lal Bhatnagar¹, Nan Yang¹, Lingni Ma¹, Sebastian Starke¹, Robin Kips¹, Nadine Bertsch¹, Christian Holz², Federica Bogo¹

¹ Meta Reality Labs ² ETH Zürich

---

Human motion capture from head-mounted devices (HMDs) offers a scalable way to acquire real-world human motion and interaction data, which is crucial for applications in embodied AI and VR/AR. Existing approaches focus on either egocentric body tracking, estimating the motion of the subject wearing the device, or exocentric tracking, capturing the movements of people in the wearer’s surroundings. So far, these two paradigms have largely been explored in isolation. In this paper, we propose a novel distributed framework that jointly leverages ego- and exocentric multi-modal signals for human motion estimation from HMDs. Unlike traditional motion capture systems requiring bulky multi-camera setups or obtrusive mocap suits, our approach, EgoExoMoCap, is as simple as two (or more) people, each wearing a pair of smart glasses. The method leverages head (plus potentially wrist) tracking signals for accurate estimation of global motion in the 3D world and combines context-aware image features based on DINOv3 to achieve robustness in the presence of noise and occlusions. Extensive experiments on two in-the-wild datasets show that our approach can robustly reconstruct motion even in challenging scenarios.

Method

Given an egocentric and one or more exocentric streams from HMDs, we first roughly estimate 3D body poses from egocentric streams (EgoNet) to identify regions of interest in exocentric frames. From these, ViTPose-extracted 2D keypoints are unprojected into 3D rays and softly weighted by DINOv3-based confidence scores to form Exo Tokens. A Spatial Transformer fuses Ego and Exo tokens into View-Aggregated (VA) Tokens, followed by a Temporal Transformer for smoothness to output final full-body motions.

Installation

conda create -n egoexomocap python=3.10 -y
conda activate egoexomocap
# install PyTorch matching your CUDA version first (https://pytorch.org/)
pip install -r requirements.txt

Some dependencies are installed from source (see requirements.txt): dinov3, projectaria-tools, nymeria, human_body_prior.

SMPL-H body models are required but not distributed here (license). Download the "Extended SMPL+H model" from mano.is.tue.mpg.de (free registration) and place the neutral/male/female models at:

third_party/body_models/smplh/{male,female,neutral}/model.npz
third_party/body_models/smplh/SMPLH_NEUTRAL.npz

Repository layout

main_train.py            training entry point
main_test.py             evaluation entry point
models/                  model classes + base + selector + loss
                           model_egoexomocap.py  ego+exo fusion model (inherits the chain below)
                           model_egoposer_seq2seq.py, model_egoposer.py
networks/                egoexomocap (ego+exo net), fusion (observer-fusion module), egonet (coarse per-frame ego net)
data_loaders/            Nymeria dataset + selector
utils/                   options, transforms, metrics, logging helpers
options/                 the 8 config files for the reported results
model_zoo/               final checkpoints for the reported results (one model.pth per model)
splits/                  benchmark sequence splits + eval protocols
data_prep/
  shared/                DINO/ViT backbones + EgoNet-bbox extraction (dataset-agnostic)
  nymeria/               Nymeria download + SMPL-H + observer video/features
  egohumans/             EgoHumans -> Nymeria-schema pair extraction (4 stages)

Configs

Configs are named by the tracker input type (the head/wrist signal driving the model), which is the real axis of variation:

• nymeria_real_tracker — real headset device signals (real Aria head + wristband wrists). Trained and evaluated on Nymeria.
• egohumans_syn_tracker — synthetic FK-derived head/wrist (EgoHumans has no wrist trackers, so inputs come from GT SMPL by FK). Trained on Nymeria-FK, evaluated zero-shot on EgoHumans.

config (train / test)	eval dataset	inputs
train_nymeria_real_tracker_3pt.yaml / test_nymeria_real_tracker_3pt.yaml	Nymeria	3-point (head + wrists)
train_nymeria_real_tracker_1pt.yaml / test_nymeria_real_tracker_1pt.yaml	Nymeria	1-point (head only)
train_nymeria_syn_tracker_3pt.yaml / test_egohumans_syn_tracker_3pt.yaml	EgoHumans (zero-shot)	3-point
train_nymeria_syn_tracker_1pt.yaml / test_egohumans_syn_tracker_1pt.yaml	EgoHumans (zero-shot)	1-point

Body-pose ground truth comes from NymeriaPlus official SMPL (smplh_nymeriaplus.npz). Training uses GT observer bboxes (the default bbox_source); evaluation uses EgoNet-derived observer bboxes (bbox_source: egonet in each config's test set) — i.e. the trained model's own EgoNet proposes the crop at test time.

Data preparation

The raw datasets are obtained separately: Nymeria (VRS + MPS, plus NymeriaPlus SMPL-H GT) by requesting access and exporting its per-sequence signed-URL manifests, and EgoHumans from the EgoHumans repo (OAuth download). See data_prep/README.md for the full guide — dataset access, the ViTPose / DINOv3 checkpoints, and the path / env-var configuration; the condensed pipeline below assumes those are in place.

Run all data_prep/ scripts from the repository root. Put the repo root and data_prep/shared on the path so the shared DINO/ViT backbones resolve: PYTHONPATH=.:data_prep/shared. VRS-reading steps need the pymeria env (projectaria_tools / nymeria SDK); the GPU feature steps need torch + transformers.

export PYTHONPATH=.:data_prep/shared

Nymeria

# 1. download sequences (see splits/ for the benchmark split) + NymeriaPlus SMPL (GT)
python data_prep/nymeria/download_test_split_parallel.py
python data_prep/nymeria/download_train_split_parallel.py
python data_prep/nymeria/download_nymeriaplus_smpl.py

# 2. SMPL-H GT @ 30 fps from NymeriaPlus (also establishes the meta.json /
#    timestamps.csv grid that every downstream step aligns to)
python data_prep/nymeria/extract_smplh_30fps_from_nymeriaplus.py   # smplh_nymeriaplus.npz
python data_prep/nymeria/extract_wearer_smplh_fk.py \
    --input_smplh smplh_nymeriaplus.npz --output_fk wearer_smplh_fk_nymeriaplus.npz

# 3. observer (exo) video + head/wrist + bbox, then DINO/ViT features
python data_prep/nymeria/extract_observer_video.py      # observer_full.mp4, wearer_ego.npz, observer.npz, bbox.npz
python data_prep/nymeria/extract_features_from_video.py # vitpose.npz, dino_features.npz

# 4. EgoNet-derived bboxes + features recomputed on them (for eval, bbox_source: egonet).
#    Uses the trained model's own EgoNet, so run this after training — or download the
#    released checkpoint first (see Evaluation) and point --opt at its test config.
python data_prep/shared/extract_egonet_bboxes.py --opt options/test_nymeria_real_tracker_3pt.yaml
python data_prep/shared/rerun_vitpose_dino_egonet_bbox.py   # vitpose_egonet.npz, dino_features_egonet.npz

EgoHumans (zero-shot)

Produces Nymeria-schema sequences under dataset/egohumans_extracted/, then reuse the shared EgoNet-bbox step above with an EgoHumans config.

python data_prep/egohumans/extract_egohumans_pairs_stage1.py
python data_prep/egohumans/extract_egohumans_pairs_stage2_bbox.py
python data_prep/egohumans/extract_egohumans_pairs_stage2_video.py
python data_prep/egohumans/extract_egohumans_pairs_stage3_features.py

Training

Train with any of the four train_* configs (see Configs):

# Nymeria, real device tracker (3-point: head + wrists)
python main_train.py -opt options/train_nymeria_real_tracker_3pt.yaml

# Nymeria-FK, synthetic tracker (for zero-shot EgoHumans evaluation)
python main_train.py -opt options/train_nymeria_syn_tracker_3pt.yaml

Swap _3pt for _1pt for the head-only variants. Training uses GT observer bboxes (data-prep step 3). Checkpoints and logs are written under results/<task>/ (models/<iter>.pth, train.log); the checkpoint / validation / print intervals and the loss weights live in each config's train: block (checkpoint_save, checkpoint_test, checkpoint_print, *_weight), and gpu_ids selects the device(s).

After training, generate the EgoNet-derived bboxes and features from your trained checkpoint (data-prep step 4) before running the evaluation below.

Evaluation

Download the final checkpoints from Google Drive and place the four folders under model_zoo/, so each model.pth sits at:

model_zoo/nymeria_real_tracker_3pt/model.pth
model_zoo/nymeria_real_tracker_1pt/model.pth
model_zoo/egohumans_syn_tracker_3pt/model.pth
model_zoo/egohumans_syn_tracker_1pt/model.pth

Every test config's pretrained_model: already points at the matching one.

Nymeria (real tracker):

python main_test.py -opt options/test_nymeria_real_tracker_3pt.yaml

EgoHumans (zero-shot, synthetic tracker):

python main_test.py -opt options/test_egohumans_syn_tracker_3pt.yaml

Fuse multiple observer views at test time with --multi_observer (--protocol restricts to a scene subset):

python main_test.py -opt options/test_egohumans_syn_tracker_3pt.yaml \
    --multi_observer --protocol splits/egohumans_multiobs_protocol.json

Citation

@article{jiang2026egoexomocap,
  title={EgoExoMoCap: Distributed Ego-Exo Human Motion Capture},
  author={Jiaxi Jiang and Bharat Lal Bhatnagar and Nan Yang and Lingni Ma and Sebastian Starke and Robin Kips and Nadine Bertsch and Christian Holz and Federica Bogo},
  journal={arXiv preprint arXiv:2607.15868},
  year={2026}
}

Acknowledgements

This code builds on EgoPoser, which is itself adapted from AvatarPoser. We train and evaluate on the Nymeria and EgoHumans datasets, and use the SMPL-H body model for the human representation. The observer (exo) branch relies on ViTPose for 2D keypoint detection and DINOv3 for visibility-gating features. We thank the authors of these projects for releasing their code, models, and data.