Skip to content

feat(cosmos3-edge): add Thor AV denoise engine and TeaCache#158

Draft
LiangSu8899 wants to merge 3 commits into
mainfrom
feat/cosmos3-edge-thor
Draft

feat(cosmos3-edge): add Thor AV denoise engine and TeaCache#158
LiangSu8899 wants to merge 3 commits into
mainfrom
feat/cosmos3-edge-thor

Conversation

@LiangSu8899

Copy link
Copy Markdown
Member

Summary

Adds a Jetson AGX Thor implementation of the Cosmos3-Edge AV
inverse-dynamics denoise boundary, together with the official reference
runner, reproducible benchmark harness, correctness gates, and user
documentation.

The no-cache FlashRT path executes all 30 velocity evaluations and delivers
the production optimization result. TeaCache is an explicit optional
approximation layered on top.

Performance

All rows measure the same av_inverse_0 30-step denoise boundary with 15
back-to-back hot iterations. Accuracy is the final action versus the official
eager output. The gates are cosine >= 0.999 and relative L2 < 3%. Both
no-cache engines and the 3- and 2-compute TeaCache schedules were rechecked
with a second seed.

Configuration Denoise P50 Speedup vs Official Cosine Rel-L2 Peak memory (GiB)
Official Cosmos3-Edge (eager) 33.14 s 1.00x reference reference not recorded
FlashRT FP8, no step cache 5.792 s 5.72x 0.999983 0.59% 7.85
FlashRT FP8 + NVFP4 FFN, no step cache 5.456 s 6.07x 0.999707 2.42% 7.54
FlashRT FP8 + TeaCache, 15 computes 2.877 s 11.52x 0.999985 0.56% 7.85
FlashRT FP8 + TeaCache, 6 computes 1.153 s 28.74x 0.999984 0.57% 7.85
FlashRT FP8 + TeaCache, 3 computes 0.576 s 57.56x 0.999986 0.54% 7.85
FlashRT FP8 + TeaCache, 2 computes 0.384 s 86.36x 0.999983 0.59% 7.85
FlashRT FP8 + NVFP4 FFN + TeaCache, 2 computes 0.362 s 91.57x 0.999727 2.34% 7.54

The production claim is the no-cache 5.72x FP8 / 6.07x FP8+NVFP4 result.
Those rows preserve all 30 velocity evaluations and separate the FlashRT
engine gain from step caching.

Pipeline optimization

  • Registers a dedicated cosmos3_edge PyTorch/Thor frontend and keeps
    existing model routes unchanged.
  • Uses static AV buffers and pre-installs the invariant und-conditioned K/V
    prefix.
  • Quantizes generated-stream projections to FP8 E4M3 with calibrated static
    activation scales.
  • Adds optional NVFP4 W4A4 FFN up/down projections.
  • Fuses residual, RMSNorm, quantization, ReLU2, Q/K normalization, and RoPE
    work.
  • Uses FA4 generated-stream attention, a native UniPC scheduler step, and a
    slim 60-row final action layer.
  • Captures the complete 30-step denoise loop in one CUDA Graph.
  • Includes the official action-only reference runner and fixture-backed replay
    paths for boundary validation.

TeaCache

TeaCache bakes a fixed compute-step set into the 30-step graph. Skipped steps
replay native UniPC while retaining the most recent velocity buffer, with no
new kernel or runtime branch.

These TeaCache numbers are workload-specific benchmark observations, not a
dataset-level task-success claim. Camera setup, robot dynamics, task
distribution, conditioning, and fine-tuned checkpoints can materially change
the valid schedule. Every deployment must qualify TeaCache case by case
against its own checkpoint and task data. The 2-compute rows are aggressive
benchmark operating points; 3 computes is the suggested starting point for
this measured workload.

User and developer impact

  • Adds flash_rt.load_model(..., config="cosmos3_edge", framework="torch", hardware="thor").
  • Adds a documented official-baseline-first workflow and explicit denoise
    benchmark commands.
  • Makes all checkpoint and fixture paths explicit; no host layout is assumed.
  • Keeps TeaCache off unless a compute schedule is explicitly selected.
  • Keeps the integration isolated to the Thor route; unsupported hardware
    fails during model resolution.

Validation

Build validation used Jetson AGX Thor (sm_110), CUDA 13.0.88, PyTorch
2.9.0a0, and NGC PyTorch 25.09.

  • GPU_ARCH=110 build of flash_rt_kernels: passed
  • GPU_ARCH=110 build of flash_rt_fp4: passed
  • Core and FP4 import plus Cosmos binding inventory: passed
  • Cosmos CUDA kernel canaries: 17 passed
  • Cosmos routing/reference unit tests: 69 passed, 20 skipped because external
    checkpoint fixtures were not supplied
  • Repository install/API/calibration smoke: 86 passed
  • Benchmark and baseline CLI help checks: passed
  • git diff --check: passed
  • Private-path, credential, and local-artifact scan: passed

Scope

The table is denoise P50 in a hot process, not raw-request end-to-end latency.
Official tokenization, video preprocessing, and Wan2.2 VAE encode remain
outside this optimized boundary. VAE encode is the next major end-to-end
target.

…le-graph engine

Add config="cosmos3_edge" for Cosmos3-Edge (4B, two-tower MoT, 28L/2048,
GQA 16/8, non-gated relu2 FFN) AV inverse dynamics on Thor SM110:

- Official Cosmos Framework baseline runner plus action-only, live-handoff,
  boundary/prepare dump-replay, and VAE probe paths behind one frontend.
- Torch/FVK bring-up references, boundary dumps, static und-K/V engine, and
  a native one-launch UniPC step for the fixed 30-step AV schedule.
- models/cosmos3_edge/pipeline_thor.py: the optimized denoise engine. All six
  gen-tower projections in per-tensor FP8 E4M3 with calibrated static
  activation scales; fused quant chain (residual_add_rms_norm_fp8,
  quantize_fp8_static, relu2->fp8); warp-per-head fused qk-norm+RoPE; FA4
  attention over per-layer joint und+gen K/V; M=60 slim last layer; the whole
  30-step denoise replayed as a single CUDA graph.
- Optional NVFP4 W4A4 FFN via flash_rt_fp4 with fused bf16 res+rms->fp4 and
  fp16 relu2->fp4 quantizers.
- benchmarks/cosmos3_edge_thor_denoise.py with latency and final-action gates.

Thor, 15 back-to-back iterations, official eager denoise 33.14s, gates
cos>=0.999 / rel_l2<3% (both engines also pass on a second-seed dump):

  --engine bf16-eager   11.998s  (2.76x)  cos 0.9999959  rel_l2 0.29%
  --engine fp8           5.792s  (5.72x)  cos 0.9999834  rel_l2 0.59%
  --engine fp8 --ffn-fp4 5.456s  (6.07x)  cos 0.9997067  rel_l2 2.42%
… table

Fixed compute-step schedules baked into the whole-denoise CUDA graph: skipped
steps reuse the last computed velocity while the native UniPC still advances
all 30 steps. --teacache-computes K picks K evenly spaced computed steps
(always step 0 and the final step); --teacache-steps takes an explicit list.

For AV inverse dynamics the action velocity field is dominated by the static
clean-video conditioning, so every swept schedule (15/10/6/4/3/2 computes)
stays within quantization-level error of the official action; the 3- and
2-compute tiers also pass on a second-seed dump.

Thor, 15 back-to-back iterations, official eager denoise 33.14s, gates
cos>=0.999 / rel_l2<3%:

  fp8, no cache        5.792s   (5.72x)  rel_l2 0.59%
  fp8 + TeaCache 15    2.877s  (11.52x)  rel_l2 0.56%
  fp8 + TeaCache 6     1.153s  (28.74x)  rel_l2 0.57%
  fp8 + TeaCache 3     0.576s  (57.56x)  rel_l2 0.54%
  fp8 + TeaCache 2     0.384s  (86.36x)  rel_l2 0.59%
  fp8 + fp4-ffn + TC2  0.362s  (91.57x)  rel_l2 2.34%

The 2-compute tiers are an aggressive benchmark operating point, not yet a
dataset-level task-success claim. Full table in docs/cosmos3_edge_thor.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant