3dhands

Pipeline for egocentric 3D hand tracking from iPhone video.

Input:

• RGB video (video.mp4)
• ARKit camera poses (poses.csv)
• Camera intrinsics (intrinsics.json)

Output:

• Per-frame 3D world-space hand joints (21 joints per hand) with orientation and confidence.

What This Repo Does

1. Extracts frames and timestamps from video.
2. Builds Pi3X multimodal conditions (condition.npz) from ARKit poses + intrinsics.
3. Runs MediaPipe Hands for 2D joint pixels.
4. Runs Pi3X for dense 3D geometry.
5. Fuses 2D joints with Pi3X 3D and writes output.json.
6. Provides validation tools:

• 2D reprojection overlay video
• quantitative eval report
• 3D exports (simple and "pretty")

Repository Layout

• src/handtrack/cli.py main CLI entrypoint
• src/handtrack/conditions.py ARKit -> Pi3X condition builder
• src/handtrack/pi3x_runner.py Pi3X inference wrapper
• src/handtrack/fuse.py 2D + 3D fusion
• src/handtrack/visualize.py overlay rendering
• src/handtrack/analysis.py eval + 3D export tools
• docs/pipeline.md detailed command reference

Setup

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
git clone https://github.com/yyfz/Pi3.git vendor/Pi3
.venv/bin/pip install -e vendor/Pi3
.venv/bin/pip install -e .

Notes:

• Pi3X model weights are downloaded automatically from Hugging Face at first run.
• MediaPipe task model is downloaded automatically if not already present.

Data Format

Expected files:

• data/<run_id>/video.mp4
• data/<run_id>/poses.csv with columns:
  • timestamp,pos_x,pos_y,pos_z,quat_w,quat_x,quat_y,quat_z
• data/<run_id>/intrinsics.json with:
  • fx, fy, cx, cy

Coordinate conversion:

• ARKit is Right-Up-Backward.
• Pi3X expects OpenCV Right-Down-Forward.
• Conversion is handled in conditions.py.

End-to-End Run

PYTHONPATH=src .venv/bin/python -m handtrack.cli run \
  --video data/1770767105/video.mp4 \
  --poses data/1770767105/poses.csv \
  --intrinsics data/1770767105/intrinsics.json \
  --work-dir data/1770767105/work \
  --output data/1770767105/output.json \
  --max-frames 30 \
  --pi3-width 448 \
  --pi3-height 252 \
  --pi3-chunk-size 4

Validate Results

Reprojection overlay (Pi3 3D -> image):

PYTHONPATH=src .venv/bin/python -m handtrack.cli overlay \
  --frames-index data/1770767105/work/frames.csv \
  --output-json data/1770767105/output.json \
  --conditions data/1770767105/work/condition.npz \
  --pi3x data/1770767105/work/pi3x.npz \
  --mediapipe data/1770767105/work/mediapipe.json \
  --source reproject \
  --show-error \
  --out-video data/1770767105/overlay.mp4

Quantitative metrics:

PYTHONPATH=src .venv/bin/python -m handtrack.cli eval \
  --frames-index data/1770767105/work/frames.csv \
  --output-json data/1770767105/output.json \
  --mediapipe data/1770767105/work/mediapipe.json \
  --conditions data/1770767105/work/condition.npz \
  --report-json data/1770767105/eval_report.json \
  --per-frame-csv data/1770767105/eval_per_frame.csv

Pretty 3D sequence export:

PYTHONPATH=src .venv/bin/python -m handtrack.cli export-3d-pretty-seq \
  --output-json data/1770767105/output.json \
  --conditions data/1770767105/work/condition.npz \
  --out-obj data/1770767105/hand_pretty_seq_30.obj \
  --frame-start 0 --frame-end 29 --frame-stride 1 --hand-id 0

Interpreting Metrics

From eval_report.json:

• reprojection.median_px: typical pixel error (lower is better).
• reprojection.p90_px: 90th percentile error (tail quality).
• bone_length_consistency.*.cv: temporal stability of each bone length (std/mean).

Rules of thumb:

• Median reprojection below ~10 px is typically good.
• Bone CV below ~0.2 is usually stable for this task.

Share-Safe Defaults

Local/private assets are ignored by git:

• /data/
• /vendor/
• /models/
• .venv/

So you can share the repo code without internal dataset files.