Carve

ML-assisted segmentation and matting for VFX, comp, and roto work. Browser-based. Designed for stills and frame sequences.

Status: deployed and in production use — running daily against live film and OTT VFX work.

---

What it does

• Auto multi-ID — SAM 2 generates every distinct mask in a plate and exports them as an ObjectID EXR or per-layer PNGs.
• Prompt — click positive / shift-click negative points; SAM 2 returns up to 3 candidate masks per click set.
• Search — text-prompted segmentation via SAM 3 ("the man in the red shirt", "all the cars", "the dog").
• Matte — refine any source mask into a soft alpha with one of three engines:
  • Fast (pymatting, closed-form, MIT) — CPU, ~80–90 % of ViTMatte quality.
  • Hero (ViTMatte) — best on hair/fur. Research-licensed weights — internal use only.
  • Auto (BiRefNet, MIT) — trimap-free; predicts the subject directly from the plate.
• Paint brush — refine the SAM source mask by hand before matting. Paint / erase / undo.
• Sequences — drop numbered frames (shot_0001.exr); they're auto-grouped. Then:
  • Run any op on every frame in one click.
  • Propagate a single keyframe's mask through the sequence using SAM 2's video predictor.
  • Refine the propagated masks with pymatting or ViTMatte (per-frame soft alphas with full Alpha / Trimap / SAM / Comp views).
  • Export padded EXR / PNG sequences with DCC-standard naming (shot.####.exr).
• Comp view — every soft-matte result has a checker-BG composite preview so you can judge edge quality at a glance.

---

Stack

Layer	Stack
Frontend	Next.js 16 · React 19 · Tailwind v4 · PixiJS v8 · Zustand · TanStack Query · Sonner · Radix
Backend	FastAPI · Python 3.12 · stdlib sqlite3 (WAL) for the index
Models	SAM 2 / SAM 3 (Meta) · ViTMatte (HuggingFace) · BiRefNet · pymatting
Runtime	uvicorn + systemd on a single H100 GPU host
Frontend host	Vercel
Tunnel	Cloudflare named tunnel

Frontend → tunnel → FastAPI → CUDA H100. SQLite index sits next to the JSON sidecars and rebuilds from disk on schema bump or accidental delete, so it can never get permanently out of sync.

---

Quick start (dev)

Backend

# from repo root
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e .                        # carve + deps
bash scripts/install_sam3.sh            # SAM 3 isn't on PyPI — patches HF cache
huggingface-cli login                   # SAM 3 weights are gated
uvicorn carve.api.main:app --port 8787 --reload

Storage roots default to /data/carve/storage — override with CARVE_STORAGE_ROOT=/tmp/carve (or set CARVE_* env vars; see backend/carve/config.py).

Frontend

cd frontend
npm install
echo 'NEXT_PUBLIC_CARVE_API=http://localhost:8787' > .env.local
npm run dev          # http://localhost:3000

Tests

# backend (index + storage)
pytest tests/test_storage_index.py

# end-to-end (browser-driven; needs the backend reachable)
cd frontend
E2E_TUNNEL_URL=http://localhost:8787 \
E2E_BASE_URL=http://localhost:3000 \
  npx playwright test

---

Architecture in one paragraph

A single FastAPI process owns the in-process JobRunner with a global asyncio.Lock over the GPU. Every job (auto / prompt / search / matte / propagate) is a coroutine that awaits the lock, does its work, persists progress to disk (throttled 2 Hz) + the SQLite index, and releases. Result payloads stay as JSON sidecars + per-job output directories. The index is a thin SQLite layer over those JSONs — when a process restart finds an empty or schema-mismatched index, it rebuilds from disk. Frontend polls per-job via TanStack Query; sequence pages use a single bulk POST /plates/jobs for N-plate hydration. Frame sequences are recognised from filename patterns (name.0042.ext / name_0042.ext / name-0042.ext) at upload time. Multi-plate jobs (propagation) are surfaced through a job_plate_members association table so they appear in every member plate's history.

---

Key endpoints

Method	Path	Purpose
POST	/upload	Plate upload with SHA-256 dedup
GET	/plates	List all plates (indexed)
GET	/plates/{id}	Stream the plate image
DELETE	/plates/{id}	Delete plate + storage
GET	/plates/{id}/jobs	History for one plate
POST	/plates/jobs	Bulk latest-by-kind for N plates
GET	/sequences	Group plates by detected sequence
DELETE	/sequences/{id}	Delete every frame in a sequence
POST	/sequences/{id}/apply	Run a blueprint (chain of ops) across the sequence, optionally clamped to a frame range
POST	/sequences/{id}/refine	Per-frame pymatting/ViTMatte using a finished propagate as source
GET	/sequences/{id}/export	Stream a padded EXR/PNG zip (shot.####.ext)
POST	/jobs/auto /jobs/prompt /jobs/search /jobs/matte /jobs/propagate	Submit a single job
GET	/jobs/{id}	Job status + result
GET	/jobs/{id}/files/{name:path}	Stream a job's output file (path-traversal blocked)
GET	/healthz	Liveness + model status + GPU stats

---

Layout

carve/
├─ backend/carve/
│  ├─ api/           # FastAPI routes
│  ├─ jobs/          # Job runner + handlers + registry
│  ├─ models/        # SAM 2, SAM 3, ViTMatte, BiRefNet, pymatting wrappers
│  ├─ storage/       # LocalStorage + SQLite index
│  ├─ utils/         # image / exr helpers
│  └─ config.py
├─ frontend/src/
│  ├─ app/           # Next.js routes: /, /plates, /sequences/[id], /p/[plateId]
│  ├─ components/    # PlateViewer, MaskPainter, panels, shell, UI primitives
│  └─ lib/           # api client, zustand store, hooks
├─ tests/            # pytest
└─ scripts/          # SAM 3 installer, sample uploads, demo flow

---

Production status — honest read

What's solid

• Single-user / small-team internal use on a private link.
• 14/14 e2e (Playwright) + 15/15 backend index tests pass against the live deployment.
• SQLite index, atomic JSON writes, content-hash dedup, FIFO GPU mutex.
• Pixi viewer reuses the Application across plate swaps; black-canvas race is gone.

What's not production for external users yet

• No auth. Anyone with the URL can upload, run, delete.
• No backups. All data lives on one VM disk.
• ViTMatte weights are research-only license — internal use only.
• Tunnel URL rotates if cloudflared restarts; the named-tunnel migration is on the next-up list.
• No monitoring / alerting.

See the conversation history (or git log) for the audit waves that brought it here. The next-up items above are documented as TODO but not yet shipped.

---

License

Carve is proprietary source, published here as a portfolio reference. Ask before redistributing or reusing it in a product.

Third-party model licenses:

• SAM 2 — Meta SAM License (commercial OK with restrictions).
• SAM 3 — Meta SAM License (commercial OK except military/weapons).
• BiRefNet — MIT.
• pymatting — MIT (Apache-2.0 components).
• ViTMatte (hustvl/vitmatte-small-composition-1k) — research/non-commercial only. Do not ship outputs to paid clients.