Sam 3.1 Upgrade

dive-dsa-slicer/example-docker-containers/.env.example

@@ -1,6 +1,14 @@
 # Copy to .env and set your Hugging Face token (read access).
-# Request access: https://huggingface.co/facebook/sam3
+# Request access: https://huggingface.co/facebook/sam3.1
 HF_TOKEN=hf_your_token_here
 
 # Optional image tag
 # TAG=latest
+
+# Optional SAM3Demo GPU memory tuning (see README.md)
+# SAM3_MAX_NUM_OBJECTS=2
+# SAM3_GROUNDING_BATCH_SIZE=1
+# SAM3_POSTPROCESS_BATCH_SIZE=2
+# SAM3_PROPAGATION_CHUNK_SIZE=2
+# SAM3_OFFLOAD_VIDEO_TO_CPU=true
+# SAM3_USE_FA3=true

dive-dsa-slicer/example-docker-containers/Dockerfile

@@ -51,7 +51,7 @@ RUN \
     # install step \
     pip install --no-cache-dir opencv-python-headless && \
     # SAM dependencies
-    pip install --no-cache-dir sam2 sam3 huggingface_hub psutil pycocotools torch pillow && \
+    pip install --no-cache-dir sam2 'sam3 @ git+https://github.com/facebookresearch/sam3.git' huggingface_hub psutil pycocotools torch pillow einops && \
     # clean up \
     rm -rf /root/.cache/pip/*
 

dive-dsa-slicer/example-docker-containers/README.md

@@ -38,28 +38,63 @@ Uses SAM 2 video propagation from an existing track bounding box or mask. Requir
 
 ## SAM3Demo
 
-Uses [SAM 3](https://github.com/facebookresearch/sam3) open-vocabulary text prompts to segment all matching instances on the current DIVE frame and ingest them as new tracks with masks.
+Uses [SAM 3.1](https://github.com/facebookresearch/sam3) (Object Multiplex) open-vocabulary text prompts to segment all matching instances on the current DIVE frame and ingest them as new tracks with masks.
 
 ### Parameters
 
 - **TextPrompt** (required): phrase describing what to detect (e.g. `person`, `vehicle`).
 - **DIVEFrameId**: frame index (auto-filled from the DIVE UI when launched from the app).
 - **DIVETrackType**: classification label for new tracks.
 - **ConfidenceThreshold**: minimum detection score (default `0.5`).
-- **TrackingFrames**: number of frames to track forward from the prompt frame (`1` = single frame only; values greater than `1` use SAM3 video propagation, similar to SAM2Demo).
+- **TrackingFrames**: number of frames to track forward from the prompt frame (`1` = single frame only; values greater than `1` propagate masks with SAM 3.1 video tracking, similar to SAM2Demo).
 
 ### Device selection
 
-Single-frame inference (`TrackingFrames` = 1) uses CUDA when available, otherwise CPU. Video propagation (`TrackingFrames` > 1) requires CUDA because the SAM3 video predictor loads the model on GPU. Apple MPS is detected but SAM3 runs on CPU for single-frame mode only.
-
-If you see `RuntimeError: The NVIDIA driver on your system is too old` while the log says `Using inference device: cpu`, PyTorch cannot use the GPU (driver/CUDA mismatch). Single-frame mode should still work on CPU after rebuilding the image (the container patches SAM3’s hardcoded CUDA precompute paths). For GPU inference, update the host NVIDIA driver to match the PyTorch CUDA version in the image, or rebuild with a CPU-only PyTorch wheel.
+SAM 3.1 multiplex inference requires CUDA. If you see `RuntimeError: The NVIDIA driver on your system is too old`, update the host NVIDIA driver to match the PyTorch CUDA version in the image.
 
 ### Hugging Face checkpoints
 
-SAM 3 weights are gated on Hugging Face. Request access to [facebook/sam3](https://huggingface.co/facebook/sam3), then either:
+SAM 3.1 weights are gated on Hugging Face. Request access to [facebook/sam3.1](https://huggingface.co/facebook/sam3.1), then either:
+
+1. Copy `.env.example` to `.env`, set `HF_TOKEN`, and run `docker compose build --no-cache` to prefetch `sam3.1_multiplex.pt` into `/opt/SAM3/models`, or
+2. Set `HF_TOKEN` (or run `hf auth login`) at runtime so `build_sam3_predictor` can download on first use, or
+3. Set `SAM3_CHECKPOINT` to a local `sam3.1_multiplex.pt` path inside the container.
+
+The image installs `sam3` from GitHub (SAM 3.1 APIs are not yet on PyPI) and downloads the text tokenizer vocab (`bpe_simple_vocab_16e6.txt.gz`) into `/opt/SAM3/assets`. Override with `SAM3_BPE_PATH` if needed.
+
+### Reducing GPU memory
+
+SAM 3.1 multiplex loads a large detector + tracker stack on GPU. The checkpoint size is similar to SAM 3, but peak VRAM is usually higher because every job uses the full multiplex path (including single-frame jobs).
+
+**Job parameters (no code changes):**
+
+- Set **TrackingFrames** to `1` when you only need one frame (still uses multiplex, but avoids propagation memory).
+- Use a higher **ConfidenceThreshold** to drop weak detections (fewer masks/objects held in memory).
+
+**Container environment variables** (on GPUs ≤20 GiB, conservative defaults apply automatically):
+
+| Variable | Typical default (≤20 GiB) | Effect |
+|----------|---------------------------|--------|
+| `SAM3_MAX_NUM_OBJECTS` | `4` | Cap parallel object slots (was 16 in upstream sam3). |
+| `SAM3_GROUNDING_BATCH_SIZE` | `2` | Detector grounding chunk size (upstream default `16` often OOMs on 16 GiB). |
+| `SAM3_POSTPROCESS_BATCH_SIZE` | `2` | Postprocess batch size. |
+| `SAM3_PROPAGATION_CHUNK_SIZE` | `3` | Propagate a few frames per stream, then `empty_cache` (set `0` for one shot). |
+| `SAM3_OFFLOAD_VIDEO_TO_CPU` | `true` | Keep decoded frames on CPU. |
+| `SAM3_USE_FA3` | `true` on Ampere+ only | Flash Attention 3 when supported. |
+
+Example if propagation still OOMs on a 16 GiB GPU:
+
+```bash
+SAM3_MAX_NUM_OBJECTS=2 \
+SAM3_GROUNDING_BATCH_SIZE=1 \
+SAM3_PROPAGATION_CHUNK_SIZE=2 \
+SAM3_OFFLOAD_VIDEO_TO_CPU=true
+```
+
+Also set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` (enabled by default in `SAM3Demo.py`).
+
+### Pre-Ampere GPUs (Turing / Volta / Pascal)
 
-1. Copy `.env.example` to `.env`, set `HF_TOKEN`, and run `docker compose build --no-cache` to prefetch weights into `/opt/SAM3/models`, or
-2. Set `HF_TOKEN` (or run `hf auth login`) at runtime so `build_sam3_image_model` can download on first use, or
-3. Set `SAM3_CHECKPOINT` to a local `sam3.pt` path inside the container.
+If you see `RuntimeError: No available kernel` during propagation, the GPU is below CUDA capability 8.0 (no Flash Attention). `SAM3Demo.py` patches sam3 to use math SDPA and runs inference in fp32 on those GPUs. Ensure you are on the latest `SAM3Demo.py` from this repo.
 
-The image build also downloads the text tokenizer vocab (`bpe_simple_vocab_16e6.txt.gz`) into `/opt/SAM3/assets` (the PyPI `sam3` package does not include it). Override with `SAM3_BPE_PATH` if needed.
+**Hard limits:** Model weights still occupy several GB on GPU; there is no CPU fallback for SAM 3.1 multiplex in this demo. For the lowest VRAM on single-frame text prompts only, the older SAM 3 image-model path (`sam3.pt`) used less memory but is not SAM 3.1.