SAM3D Video Pose

3D pose estimation from video using SAM3 segmentation and SAM-3D-Body mesh reconstruction. Works on humans, animals, and other subjects — point it at any video with a text prompt and get 3D COCO keypoints, skeleton overlays, and animated visualizations.

Demo

Adult walking	Toddler walking	Infant in crib	Macaque walking

Getting Started

The fastest way to see the pipeline in action is with the built-in demos. Each downloads a short video from Wikimedia Commons, processes it end-to-end, and produces a skeleton overlay video + animated GIF.

# Pick a demo
python scripts/demo.py --demo-adult      # adult walking
python scripts/demo.py --demo-infant     # baby in crib
python scripts/demo.py --demo-nhp        # crab-eating macaque
python scripts/demo.py --demo-toddler    # 18-month-old walking

Each demo runs the full pipeline:

1. Download video from Wikimedia Commons
2. Segment subject with SAM3 (text-prompted)
3. Estimate 3D mesh + COCO keypoints with SAM-3D-Body
4. Bundle-adjust bone lengths
5. Generate 3D skeleton GIF
6. Overlay skeleton on original video with OpenPose-style colors

Output lands in output/demo_<name>/ with the overlay video, skeleton GIF, and CSV keypoint files.

Notes on Skeletal Geometry

This pipeline applies the MHR adult pose prior to all extractions. Age/species-specific adjustments are in development.

Process Your Own Video

# Local file
python scripts/process_video.py path/to/video.mp4 \
    --text-prompt "a person" \
    --export-coco-csv \
    --max-frames 300

# URL (auto-downloads and converts)
python scripts/process_video.py "https://example.com/video.webm" \
    --text-prompt "a dancer" \
    --export-coco-csv

Visualize Results

# Animated 3D skeleton GIF
python scripts/visualize_3d_keypoints.py output/video_name/video_name_3D_smoothed_adjusted.csv \
    --mode animation --output skeleton.gif

# Overlay skeleton on original video
python scripts/overlay_skeleton_on_video.py \
    data/video.mp4 \
    output/video_name/video_name_3D_smoothed_adjusted.csv \
    output/video_name/video_name_meshes

Tracking a Specific Individual Across Multiple People

When your video contains multiple people, two extra steps after process_video.py let you identify and lock on to one specific individual.

Step 1 — Process the video with a broad prompt

Use a prompt that segments everyone in the scene. Tune --chunk-size to fit your GPU memory — larger chunks reduce track-switching at boundaries but cost more VRAM.

python scripts/process_video.py session.mp4 \
    --text-prompt "a person" \
    --export-coco-csv \
    --chunk-size 100        # frames per SAM3 processing chunk; default 50

The same physical person may receive a different obj_* ID across chunk boundaries. The next step resolves that.

Step 2 — Merge fragmented tracks with merge_tracks.py

Computes DINOv2 appearance embeddings for every tracked object and merges any two that look like the same person (cosine similarity ≥ --threshold). Outputs track_merge_map.json.

python scripts/merge_tracks.py \
    --run-dir output/session/session \
    --video   session.mp4 \
    --threshold 0.65        # lower = more aggressive merging
    --auto                  # apply merges without interactive prompts

track_merge_map.json maps every fragmented obj_id to a canonical identity:

{ "4": 1, "7": 1, "9": 3 }   // obj_4 and obj_7 are the same person as obj_1

Step 3 — Select your target with identify_target.py

Scores each canonical track against a CLIP text prompt describing that specific person, and writes semantic_target.json. Make the prompt as distinctive as possible relative to others in the scene.

python scripts/identify_target.py \
    --run-dir output/session/session \
    --video   session.mp4 \
    --prompt  "the young child sitting in front"
    # other examples:
    # "the person in the red shirt on the left"
    # "the infant in the highchair"
    # "the experimenter standing behind the table"

Output: semantic_target.json

{ "global_target_id": 1, "prompt": "the young child sitting in front", "score": 0.31 }

All downstream analysis scripts read semantic_target.json automatically — no --target flag needed once it exists.

Full workflow at a glance

process_video.py   --text-prompt "a person" --chunk-size 100
        ↓
merge_tracks.py    (DINOv2 appearance → track_merge_map.json)
        ↓
identify_target.py --prompt "the child in the red shirt"
        ↓            (CLIP text match → semantic_target.json)
downstream scripts read semantic_target.json automatically

Tips

• Prompt breadth vs. specificity: use a broad prompt for process_video.py so every person is segmented. Save the specific description for identify_target.py.
• Chunk size vs. memory: if you hit OOM, halve --chunk-size. If tracks fragment excessively at boundaries, increase it.
• Tuning merge threshold: if the same person splits into many short tracks, lower --threshold (try 0.55). If unrelated people merge together, raise it (try 0.75).
• Verifying the merge: inspect track_merge_map.json before running identify_target.py — a bad merge map will confuse CLIP scoring.
• Multiple targets: run identify_target.py multiple times with different --prompt values to track several individuals independently (each call overwrites semantic_target.json, so save the outputs under different names).

---

Installation

Prerequisites

• Python 3.10
• CUDA-capable GPU with 16GB+ VRAM
• Git with submodule support

Setup

git clone --recursive https://github.com/quietscientist/sam3d-video-pose.git
cd sam3d-video-pose

With uv (recommended):

uv venv --python 3.10
source .venv/bin/activate
uv pip install -e .

With pip:

python3.10 -m venv .venv
source .venv/bin/activate
pip install -e .

HuggingFace token (required for model downloads):

echo "HF_TOKEN=your_token_here" > .env

Get your token from https://huggingface.co/settings/tokens.

Configuration

YAML Config Files

Demo configs live in configs/sam3d/. Use them as templates for your own:

experiment_name: my_analysis
text_prompt: "a person"
output_dir: "output"

processing:
  max_frames: 300
  export_coco_csv: true
  bundle_adjustment: true
  constrain_torso: false        # generally keep false
  temporal_smooth_window: 11
  temporal_smooth_polyorder: 3
  smoothing_sigma: 2.0

tracking:
  max_num_objects: 20            # enough candidates to include every visible person
  new_det_thresh: 0.9            # high bar for spawning new tracks
  global_identity_tracking: true # assign stable global IDs across frames/chunks
  global_similarity_threshold: 0.5

quality:
  enable_filter: false

visualization:
  flip_z: true
  fps: 10

Tracking Parameters

SAM3's tracking behavior is tunable via the tracking: section in config or --tracking-params on the CLI:

Parameter	Default	Effect
max_num_objects	10000	Max tracked objects/candidates; keep this high enough for every visible person
new_det_thresh	0.7	Score needed to spawn a new track (higher = stickier)
score_threshold_detection	0.5	Min detection confidence
assoc_iou_thresh	0.1	Det-to-track matching threshold (lower = stickier)
min_trk_keep_alive	-1	How long tracks survive without matches
max_trk_keep_alive	30	Budget for matched frames
hotstart_delay	15	Frames before output starts
recondition_every_nth_frame	16	Re-init from detections frequency
global_identity_tracking	true	Wrapper-level all-person global-ID association across frames/chunks
global_iou_threshold	0.1	Mask IoU threshold for short-gap global-ID matching
global_similarity_threshold	0.5	Min DINOv2 crop-embedding cosine similarity for long-gap matching
global_center_threshold	0.2	Normalized center distance threshold for very short-gap motion matching
global_max_age	-1	Max frames to keep a missing global track alive (-1 = forever)
global_max_num_objects	10000	Replacement SAM3 candidate cap when global mode sees max_num_objects <= 1
global_use_appearance	true	Enable DINOv2 appearance embeddings for global re-identification
appearance_model_id	facebook/dinov2-small	Hugging Face image model used for global appearance embeddings
target_lock	false in global mode	Legacy single-target filter; ignored when global_identity_tracking is true
target_lock_candidate_count	5	SAM3 candidate count used internally when target lock is active
target_max_center_jump	0.15	Max normalized frame-to-frame center jump before rejecting a switch
target_min_iou	0.02	Min mask IoU with the previous target for continuity
target_reacquire_after	12	Missed frames before long-gap re-identification switches to appearance matching
target_similarity_threshold	0.5	Min DINOv2 crop-embedding cosine similarity needed to reacquire a target

CLI override example:

python scripts/process_video.py video.mp4 \
    --tracking-params '{"global_identity_tracking": true, "max_num_objects": 20, "global_similarity_threshold": 0.5}'

Text Prompts

SAM3 uses text prompts for segmentation. More specific prompts help with tracking:

• "a person" -- general person tracking
• "a baby" -- general infant
• "a toddler walking in the center" -- spatial + action hint
• "a monkey" -- non-human primate

Output Structure

output/video_name/
    video_name_3D_raw.csv                    # Raw COCO keypoints before smoothing
    video_name_3D_smoothed.csv               # Temporally smoothed keypoints
    video_name_3D_smoothed_adjusted.csv      # Smoothed + bundle-adjusted (main output)
    video_name_skeleton.gif                  # 3D skeleton animation
    video_name_skeleton_overlay.mp4          # Skeleton overlaid on video
    video_name_meshes/                       # Per-frame MHR parameters
        frame_NNNN_obj_<global_id>/
            mhr_parameters.npz              # Camera params + 70-point skeleton
            keypoints.json                  # COCO keypoint subset
    video_name_all_keypoints.json           # All keypoints (JSON)
    video_name_mesh_results.json            # Mesh metadata
    video_name_global_tracks.json           # Global person ID spans and counts

All CSV files use long format: one row per keypoint per frame.

frame,x,y,z,part_idx
0,0.123,0.456,0.789,0
0,0.234,0.567,0.890,1
...

part_idx follows the COCO-17 ordering (0=nose, 1=left_eye, ..., 16=right_ankle).

Pipeline Architecture

Video + Text Prompt
    |
    v
SAM3 Segmentation (text-prompted tracking across frames)
    |
    v
SAM-3D-Body (per-frame 3D mesh + 70-point MHR keypoints)
    |
    v
COCO 17-point extraction + temporal smoothing
    |
    v
Bundle adjustment (fixed bone lengths)
    |
    v
Visualization (3D GIF + video overlay)

GPU Memory

The pipeline uses two GPUs when available:

• GPU 0: SAM3 segmentation (~15GB)
• GPU 1: SAM-3D-Body mesh estimation (~8GB)

For single GPU, reduce --max-frames or use --skip-mesh-saving.

Troubleshooting

CUDA Out of Memory: Reduce --max-frames or use --skip-mesh-saving.

Dark video after conversion: The pipeline validates brightness after webm-to-mp4 conversion. If you see a warning, the retry with explicit color range normalization should fix it.

Identity switches after absences: Keep global_identity_tracking: true, raise max_num_objects high enough to include every visible person, and tune global_similarity_threshold (higher is stricter).

Module import errors: Run pip install -e . and git submodule update --init --recursive.

Acknowledgments

• SAM3 for video segmentation
• SAM-3D-Body for 3D body reconstruction