ComfyUI-SAM3DBody-BetaSwap

A drop-in extra node for PozzettiAndrea/ComfyUI-SAM3DBody that replaces plain skeleton-pose rendering with a body-shape- and proportion-aware swap for Kijai's WanAnimate pipeline: the driver's motion stays 100% driver, but the pose skeleton fed to Wan is reshaped to the reference character's proportions before rendering — height, build, limb length, headwear clearance, even clothing volume — so the generated body actually matches the reference instead of just wearing the driver's skeleton.

What it does (verified against the node's own tooltips/source, not guessed)

Standard WanAnimate pose rendering draws the driver's skeleton as-is — a tall reference character ends up with the driver's short arms, a head-and-shoulders reference photo gives no idea what the body build should be, hats clip into hair, faces don't take on the reference's proportions. SAM3DBodyBetaSwapPoseRender runs Meta's SAM 3D Body on both the driver and the reference and works out, per frame, how to bend the driver's motion onto the reference's body:

• Height — measures the swap/driver skeletal height ratio (pose-invariant bone-chain sums, EMA'd over frames) and shifts the figure in depth so the projected size matches, anchoring the pelvis pixel to the driver's so footing doesn't jump.
• Build / limb thickness — measures how much bulkier or thinner the reference body is than the driver's and scales stick width and mask silhouette accordingly (auto by default, tunable).
• Clothing volume — measures how much wider a clothed reference silhouette is than its bare MHR mesh (per zone: torso / thigh / shin, via GrabCut or a provided mask) and re-adds that volume to the mask so Wan has room to draw the actual outfit instead of being capped by the driver's segmentation.
• Multi-reference body/face split — an optional second reference image (reference_body_image) supplies body form (shape + scale) while the main reference supplies identity (face) — for when your best face reference is a head crop with no body to measure from.
• Headwear clearance — measures how much room whatever is worn on the reference's head needs above the bare skull and opens the mask by that much every frame, so hats/hair aren't clipped.
• Face shape — scales the driver's dlib68 face landmarks about the nose tip by the reference head-size ratio, so face geometry follows the reference's proportions while expression stays 100% driver.
• Driver hands, always — hand pose is always taken from the driver (POSEDATA), independent of body retargeting.
• Temporal smoothing / jitter filter — both auto by default: an EMA on the swap-minus-driver offset field (not the points), so driver motion passes through with no lag and only per-frame SAM3D recon noise gets smoothed; the jitter filter measures the driver recon's own noise floor per clip and only smooths as far as needed to match it.
• Diagnostics — optional, measurement-only: writes a per-frame CSV (jitter before/after, scale factors, mask block flips) and a reference-mesh report next to ComfyUI's output folder. Changes no output pixel.

Full parameter-by-parameter behavior is documented in the node's own tooltips (INPUT_TYPES in nodes/beta_swap_pose_render.py) — that's the source of truth, not this README.

Base workflow credit

The surrounding WanAnimate pipeline (pose/face detection → segmentation → LoRA-blended WanVideo animate) started from MDMZ's Wan 2.2 Animate: Swap Characters & Lip-Sync workflow — credited there as built in collaboration with MDMZ, using Kijai's ComfyUI-WanAnimatePreprocess and ComfyUI-WanVideoWrapper. This repo's copy has junk nodes removed, settings changed, auto-resolution added, and multi-reference + the BetaSwapPoseRender node stitched in — enough changes that it's a different workflow, but the base structure and idea are MDMZ's.

Requirements

• ComfyUI-SAM3DBody installed and working (provides LoadSAM3DBodyModel, .process helpers this node imports)
• Kijai's ComfyUI-WanAnimatePreprocess (or -V2) installed next to ComfyUI-SAM3DBody in custom_nodes/ — the node locates it by walking up from its own folder and looking for that directory name, and dynamically loads pose_utils/pose2d_utils.py, pose_utils/human_visualization.py, and utils.py from it. It will not import without this.
• ComfyUI-WanVideoWrapper for the animate/sampling side of the workflow
• Tested on an RTX 4090 24GB / 64GB system RAM. A 24-frame 720×1280 test clip: SAM3DBody recon + full BetaSwap retargeting + WanAnimate sampling (4 steps) completed in ~130 seconds end to end.

Install

nodes/beta_swap_pose_render.py is not a standalone package — it imports .process from ComfyUI-SAM3DBody itself, so it has to live inside that package's own nodes/ folder:

copy nodes/beta_swap_pose_render.py  →  ComfyUI/custom_nodes/ComfyUI-SAM3DBody/nodes/beta_swap_pose_render.py

Unverified — check this yourself: whether ComfyUI-SAM3DBody auto-scans everything in its nodes/ folder or needs an explicit import added to its own nodes/__init__.py. I haven't opened that file, so I'm not claiming either way. If the node "SAM 3D Body: beta-Swap Pose Render (Wan Animate)" doesn't show up in ComfyUI after restart, that import line is what's missing.

Workflow

workflows/1-Beta_multiref.json — the full pipeline: driver video → pose/face detection → SAM2 segmentation → SAM3DBodyBetaSwapPoseRender (multi-reference: separate body-form and face-identity images) → WanAnimate sampling with the relight + lightx2v LoRAs. Load your own driving video into the VHS_LoadVideo node and your own reference images into the LoadImage nodes, point LoadSAM3DBodyModel at your local ComfyUI/models/sam3dbody folder.

Author

Node code (everything past the base WanAnimate pipeline) by stark622.

License

Not yet specified. Note: ComfyUI-SAM3DBody itself is dual-licensed (MIT wrapper / SAM License for the vendored model code) — see its own repo for terms if you're redistributing alongside it.