cosmos3: re-pin to cosmos-framework HEAD (5eee9ed) + unblock torch.compile on H100/H200

[bluecrayon52]

What

Re-pin the Cosmos 3 sample (3.test_cases/pytorch/cosmos3) from cosmos-framework 90cd348 to HEAD 5eee9ed, adapt to the framework's breaking changes, drop a now-redundant monkeypatch, and unblock torch.compile-enabled training on H100/H200 (p5/p5en).

Changes

Framework adaptation (breaking changes carried from the pin bump):

• Update framework import paths data.vfm → data.generator (B1/B2) across the action-policy src + tests.
• Rename action mode "policy" → "wam" (B3), matching the framework's renamed vocabulary.
• Bump the cosmos-framework pin 90cd348 → 5eee9ed (Dockerfile + build script + env_vars.example).
• Drop the NormMonitor monkeypatch (norm_monitor_guard.py + sitecustomize.py) — fixed upstream; one runtime patch (the wandb→OTLP bridge) remains.
• Re-pin FFmpeg to a 7.1.x gpl-shared build (the previous asset 404'd; torchcodec 0.10 requires FFmpeg ≤ 7).

torch.compile fix (H100/H200 shared-memory OOM):

• Set TORCHINDUCTOR_MIX_ORDER_REDUCTION=0 as a Dockerfile ENV, with a README note under "Software versions".

On PyTorch 2.10 (new in 2.10; the same code compiles on 2.9), Inductor's mix-order-reduction fusion fuses the Qwen3-VL RMSNorm backward reductions together with the RoPE token-mask into a single persistent Triton reduction kernel over the full hidden axis. That kernel needs ~272 KB of shared memory, over the ~227 KB-per-SM limit on H100/H200, so the first loss.backward() fails to compile with:

torch._inductor.exc.InductorError: RuntimeError: No valid triton configs.
OutOfMemoryError: out of resource: shared memory, Required: 278688, Hardware limit: 232448

Disabling just that fusion makes Inductor emit the reductions as separate looped (non-persistent) kernels that fit shared memory — torch.compile stays enabled with no measured throughput regression. This is an upstream PyTorch bug with a fix in flight; tracked at pytorch/pytorch#175250. The env var can be removed once the pinned PyTorch version includes the fix.

Validation

Validated end-to-end on the re-pinned image (clean HEAD framework + the Dockerfile ENV, no source patch), single- and multi-node:

• Post-train (2× p5en, action-policy, real warm-start): EFA/NCCL up (NET/OFI … provider efa, efa-direct, 16 nics, 16 ranks), torch.compile succeeds, 0 OOM, 100 iterations, Done with training, DCP checkpoint + latest_checkpoint.txt written.
• Generate (SDG, vLLM-Omni Super V2V): server ready; POST /v1/videos/sync → HTTP 200, valid MP4 (h264, 1280×704, 121 frames).
• Eval-serve (LIBERO policy server): GET /info → HTTP 200 with a valid model-info payload.
• HyperPod-EKS manifests: parity confirmed; the compile fix rides in the shared Dockerfile ENV, so no per-manifest change is needed.

Notes

• The TORCHINDUCTOR_MIX_ORDER_REDUCTION=0 env var is a temporary workaround for the pinned torch 2.10; remove it once torch carries the #175250 fix.