SAM 3.1 on Apple Silicon (MPS)

Run Meta's SAM 3.1 Object Multiplex video tracking natively on the Apple GPU (Metal / MPS). Fork of facebookresearch/sam3, which is CUDA only and aborts on Mac in ~20 places.

Why this exists

• Image concept segmentation already works on Mac out of the box via HuggingFace transformers (Sam3Model / Sam3Processor on MPS). If you only need image masks, use that. No fork needed, and it is faster.
• SAM 3.1 video tracking (Object Multiplex) is the part that is not in transformers and is CUDA only. This repo makes that run on MPS. That is the point.

What works

• Video (the headline): SAM 3.1 Object Multiplex tracking, text prompted detect + track, end to end on MPS, correct masks.
• Image: the native image path runs here too, but for image only work prefer transformers (above). facebook/sam3.1 ships only the video multiplex checkpoint, so for images 3.1 == 3.

Requirements

• Apple Silicon Mac (M1/M2/M3/M4).
• macOS 14+, Python 3.12, PyTorch 2.12+.
• Hugging Face account; accept the facebook/sam3 and facebook/sam3.1 licenses; huggingface-cli login.
• RAM: image runs on 16 GB. Video is memory heavy, see below.

Quickstart

git clone https://github.com/JordiSpranger/sam3.1-mps.git
cd sam3.1-mps
/opt/homebrew/bin/python3.12 -m venv .venv
.venv/bin/pip install torch torchvision numpy timm ftfy regex iopath \
    huggingface-hub pillow tqdm typing_extensions einops pycocotools psutil \
    scipy scikit-image opencv-python-headless
.venv/bin/pip install --no-deps .
.venv/bin/huggingface-cli login

# Video (the point): text prompted multiplex tracking on a folder of jpgs
# (frames named 00000.jpg, 00001.jpg, ...)
PYTORCH_ENABLE_MPS_FALLBACK=1 .venv/bin/python run_video_mps.py --video <jpeg_dir> --text "box"

# Image (optional; for image only use transformers instead): concept panel, saves overlay to out/
PYTORCH_ENABLE_MPS_FALLBACK=1 .venv/bin/python run_mps.py sam3.1 --save

• PYTORCH_ENABLE_MPS_FALLBACK=1 is required (one op, _assert_async, has no MPS kernel).

Performance to expect (M4 Max, fp16 default)

• No torch.compile and no FlashAttention (both CUDA only), so slower than an NVIDIA GPU but functional.
• Image concept segmentation: ~0.8 s image encode + ~0.2 s per concept. 20 concept panel ~0.18 fps. Single concept ~1 s.
• Video multiplex tracking: ~0.3 fps (about 3 to 5 s per frame).

RAM to expect (read this)

• ru_maxrss and most tools under report MPS memory. Use torch.mps.driver_allocated_memory(). run_video_mps.py prints both.
• Image: ~4 GB.
• Video, 12 frame clip, measured:
  • fp16 (default): ~19 GB in use, ~77 GB driver peak (reserved).
  • fp32: ~35 GB in use, ~110 GB driver peak.
• The driver peak is what shows up in Activity Monitor. It is the Metal allocator's reserved working set, not steady state usage.
• Driver peak scales with input resolution (fixed at 1008) squared and with max_num_objects.

Adjust to your Mac hardware

• Keep fp16 on (default). SAM3_MPS_FP16=0 forces fp32 (more memory, not faster).
• 64 to 128 GB: video runs at default settings.
• 16 to 36 GB: image is fine; for video use short clips and fewer objects:
  • Fewer frames per session (process the video in chunks).
  • Lower max_num_objects and multiplex_count in build_sam3_predictor (default 16 each).
• These do not help on Apple Silicon (verified):
  • offload_video_to_cpu / offload_state_to_cpu: identical memory, 2x slower. Unified memory means CPU and GPU RAM are the same pool.
  • torch.mps.empty_cache(): does not reduce the in frame peak.
  • Watermark ratio caps (PYTORCH_MPS_HIGH/LOW_WATERMARK_RATIO): OOM if set below the working set.

What changed vs upstream

• Full list in FORK_NOTES.md. Summary:
  • triton import stub (image path imports a CUDA only kernel that is never called).
  • Device agnostic builders and predictor (upstream hardcodes .cuda() and device="cuda" widely).
  • Real valued RoPE instead of complex64 (no complex matmul on MPS).
  • Unfused fp32 addmm_act (upstream forces a bf16 fused kernel with no MPS support).
  • Root multiplex bug: MPS advanced indexing (t[index_tensor], read and write) silently returns garbage indices even for a valid index like [0]. Fixed with index_select + CPU routed writes + zero init of empty allocations.
  • fp16 autocast on MPS (bf16 has no Metal kernel).
• All changes are guarded so the CUDA path is unchanged; only non CUDA devices take the new branch.

License and attribution

• This is a derivative work of facebookresearch/sam3, distributed under the SAM License (see LICENSE). Model weights are gated on Hugging Face under the same license.
• Built with SAM 3 by Meta AI. Original README preserved as UPSTREAM_README.md.