feat(cosmos3-edge): add Thor AV denoise engine and TeaCache#158
Draft
LiangSu8899 wants to merge 3 commits into
Draft
feat(cosmos3-edge): add Thor AV denoise engine and TeaCache#158LiangSu8899 wants to merge 3 commits into
LiangSu8899 wants to merge 3 commits into
Conversation
…le-graph engine Add config="cosmos3_edge" for Cosmos3-Edge (4B, two-tower MoT, 28L/2048, GQA 16/8, non-gated relu2 FFN) AV inverse dynamics on Thor SM110: - Official Cosmos Framework baseline runner plus action-only, live-handoff, boundary/prepare dump-replay, and VAE probe paths behind one frontend. - Torch/FVK bring-up references, boundary dumps, static und-K/V engine, and a native one-launch UniPC step for the fixed 30-step AV schedule. - models/cosmos3_edge/pipeline_thor.py: the optimized denoise engine. All six gen-tower projections in per-tensor FP8 E4M3 with calibrated static activation scales; fused quant chain (residual_add_rms_norm_fp8, quantize_fp8_static, relu2->fp8); warp-per-head fused qk-norm+RoPE; FA4 attention over per-layer joint und+gen K/V; M=60 slim last layer; the whole 30-step denoise replayed as a single CUDA graph. - Optional NVFP4 W4A4 FFN via flash_rt_fp4 with fused bf16 res+rms->fp4 and fp16 relu2->fp4 quantizers. - benchmarks/cosmos3_edge_thor_denoise.py with latency and final-action gates. Thor, 15 back-to-back iterations, official eager denoise 33.14s, gates cos>=0.999 / rel_l2<3% (both engines also pass on a second-seed dump): --engine bf16-eager 11.998s (2.76x) cos 0.9999959 rel_l2 0.29% --engine fp8 5.792s (5.72x) cos 0.9999834 rel_l2 0.59% --engine fp8 --ffn-fp4 5.456s (6.07x) cos 0.9997067 rel_l2 2.42%
… table Fixed compute-step schedules baked into the whole-denoise CUDA graph: skipped steps reuse the last computed velocity while the native UniPC still advances all 30 steps. --teacache-computes K picks K evenly spaced computed steps (always step 0 and the final step); --teacache-steps takes an explicit list. For AV inverse dynamics the action velocity field is dominated by the static clean-video conditioning, so every swept schedule (15/10/6/4/3/2 computes) stays within quantization-level error of the official action; the 3- and 2-compute tiers also pass on a second-seed dump. Thor, 15 back-to-back iterations, official eager denoise 33.14s, gates cos>=0.999 / rel_l2<3%: fp8, no cache 5.792s (5.72x) rel_l2 0.59% fp8 + TeaCache 15 2.877s (11.52x) rel_l2 0.56% fp8 + TeaCache 6 1.153s (28.74x) rel_l2 0.57% fp8 + TeaCache 3 0.576s (57.56x) rel_l2 0.54% fp8 + TeaCache 2 0.384s (86.36x) rel_l2 0.59% fp8 + fp4-ffn + TC2 0.362s (91.57x) rel_l2 2.34% The 2-compute tiers are an aggressive benchmark operating point, not yet a dataset-level task-success claim. Full table in docs/cosmos3_edge_thor.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Jetson AGX Thor implementation of the Cosmos3-Edge AV
inverse-dynamics denoise boundary, together with the official reference
runner, reproducible benchmark harness, correctness gates, and user
documentation.
The no-cache FlashRT path executes all 30 velocity evaluations and delivers
the production optimization result. TeaCache is an explicit optional
approximation layered on top.
Performance
All rows measure the same
av_inverse_030-step denoise boundary with 15back-to-back hot iterations. Accuracy is the final action versus the official
eager output. The gates are cosine >= 0.999 and relative L2 < 3%. Both
no-cache engines and the 3- and 2-compute TeaCache schedules were rechecked
with a second seed.
The production claim is the no-cache 5.72x FP8 / 6.07x FP8+NVFP4 result.
Those rows preserve all 30 velocity evaluations and separate the FlashRT
engine gain from step caching.
Pipeline optimization
cosmos3_edgePyTorch/Thor frontend and keepsexisting model routes unchanged.
prefix.
activation scales.
work.
slim 60-row final action layer.
paths for boundary validation.
TeaCache
TeaCache bakes a fixed compute-step set into the 30-step graph. Skipped steps
replay native UniPC while retaining the most recent velocity buffer, with no
new kernel or runtime branch.
These TeaCache numbers are workload-specific benchmark observations, not a
dataset-level task-success claim. Camera setup, robot dynamics, task
distribution, conditioning, and fine-tuned checkpoints can materially change
the valid schedule. Every deployment must qualify TeaCache case by case
against its own checkpoint and task data. The 2-compute rows are aggressive
benchmark operating points; 3 computes is the suggested starting point for
this measured workload.
User and developer impact
flash_rt.load_model(..., config="cosmos3_edge", framework="torch", hardware="thor").benchmark commands.
fails during model resolution.
Validation
Build validation used Jetson AGX Thor (
sm_110), CUDA 13.0.88, PyTorch2.9.0a0, and NGC PyTorch 25.09.
GPU_ARCH=110build offlash_rt_kernels: passedGPU_ARCH=110build offlash_rt_fp4: passedcheckpoint fixtures were not supplied
git diff --check: passedScope
The table is denoise P50 in a hot process, not raw-request end-to-end latency.
Official tokenization, video preprocessing, and Wan2.2 VAE encode remain
outside this optimized boundary. VAE encode is the next major end-to-end
target.