Support <=12GB GPUs: lazy depth model, frame resize, lower tracker caps - #26
Open
Jpalmer95 wants to merge 1 commit into
Open
Support <=12GB GPUs: lazy depth model, frame resize, lower tracker caps#26Jpalmer95 wants to merge 1 commit into
Jpalmer95 wants to merge 1 commit into
Conversation
Three small fixes that let the video-guided animation pipeline run on consumer 12GB cards (tested RTX 4070 Ti) without changing results on larger GPUs: 1. optimization.py: skip loading Video-Depth-Anything (~1.5GB VRAM) when depth/depth_gt_raw.pt already exists. Companion script precompute_depth.py (new, repo root) runs VDA standalone and frees it before the optimization loop. 2. optimization.py: resize loaded frames to --img_size (upstream assumed extracted frames already matched, crashing with tensor size mismatch whenever the source video's resolution differed). 3. save_utils.py: lower CoTracker full-mode caps 15000/4000 -> 6000/2000. Dense meshes (200k+ verts from modern image-to-3D models) OOM'd the corr-volume einsum even at 512px. Verified end-to-end: rigged 31-joint insect mesh (68k->31k verts after decimation), 40-frame 384px driving video, 300 optimization iterations at ~11.6GB peak, output motion visually matches the reference video.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small, opt-in-compatible fixes that let the video-guided animation pipeline run on consumer 12GB cards (tested: RTX 4070 Ti, torch 2.1.1/cu118) without changing behavior on larger GPUs.
Changes
animation/optimization.py — lazy DepthModule. Skip loading Video-Depth-Anything (~1.5GB VRAM) when
depth/depth_gt_raw.ptalready exists. New companion script precompute_depth.py (repo root) runs VDA standalone so the model is freed before the optimization loop starts. This alone was the difference between OOM and a comfortable run at 512px.animation/optimization.py — resize frames to --img_size on load. The loader previously assumed extracted frames already matched --img_size, producing
RuntimeError: The size of tensor a (512) must match the size of tensor b (960)whenever the source video's native resolution differed. Now resizes with LANCZOS when needed.animation/utils/save_utils.py — lower tracker caps 15000/4000 → 6000/2000. Dense meshes from modern image-to-3D models (200k+ verts, e.g. TRELLIS.2 outputs) OOM'd CoTracker3's corr-volume einsum even at 512px in 'full' tracking mode. Sampled mode with 2000 points still converges well.
Verification
End-to-end on a 31-joint rigged insect mesh (31k verts after decimation), 40-frame 384px driving video: 300 optimization iterations at ~11.6GB peak VRAM, output motion visually matches the reference video. The stock deer example also still runs (200 iters, 512px, ~8GB peak).
Happy to gate any of these behind CLI flags instead if you'd prefer the defaults unchanged.