[bug] Qwen3-VL vision tower diverges from transformers 5.15.0: visual-token embeddings mismatch (image/video conditioning parity broken on main)

[SolitaryThinker]

Summary

FastVideo's MiniMax-H3 Qwen3-VL conditioner diverges from official transformers (5.15.0) in the vision tower: on identical pixel_values, the visual-token embeddings differ before any decoder layer runs. Text-only conditioning is bit-exact; image/video-conditioned paths (FL2VA keyframes, Ref2VA) drift from the official reference.

Found while GPU-validating #1711 (encoder truncation) — and explicitly not caused by that PR: it reproduces byte-for-byte at that PR's base commit 8208536cd with the full 64-layer stack.

Evidence (GB200, driver 580.82.07, torch 2.12.0+cu130, transformers 5.15.0, real MiniMax-H3 checkpoint)

Running the in-tree parity gate on main-equivalent code:

MINIMAX_H3_RUN_ENCODER_PARITY=1 MINIMAX_H3_MODEL_ROOT=<model dir> \
  python -m pytest tests/local_tests/encoders/test_minimax_h3_qwen3_vl_parity.py -v -s

• text case: all 65 hidden states bit-exact (atol=0), final normed state included.
• image case: fails at hidden_states[0] — the embeddings entry, below every decoder layer: Mismatched elements: 338336 / 445440 (76.0%); greatest absolute difference: 1.125 at index (0, 65, 1939).
• Position probe: of the case's 87 tokens, exactly the 70 <|image_pad|> positions differ; all 17 text/marker positions are bit-identical. First diverging index is 0.
• video case: same signature — fails at layer 0, 58536/158720 (36.9%) mismatched = the visual-token fraction.
• Identical failure at base 8208536cd and at [perf]: MiniMax H3 - build the Qwen3-VL encoder only as far as it is read (-13.7 GB) #1711's head a02634fc7 → orthogonal to truncation.

Impact

• Downstream, the layer-50 tap drifts on visually-conditioned inputs (measured image-case tap drift: max_abs 2314, mean_abs 0.161 — large at individual positions).
• The in-tree recorded evidence for this gate reportedly showed all-states exactness when it was authored, so this is either version skew (the vision tower changed in a newer transformers) or a latent divergence in the ported tower (e.g. patch merger / rotary / interpolation details).

Suggested next steps

1. Bisect against transformers versions to separate "we drifted" from "they moved".
2. Compare tower submodule outputs (patch embed → blocks → merger) at the first diverging visual token.
3. Until fixed, treat FL2VA/Ref2VA conditioning parity claims vs official as unverified on main.

Repro artifacts (job logs, probe script) available on request.

🤖 Generated with Claude Code

[github-actions]

Welcome to FastVideo! Thanks for opening your first issue.

To help us investigate, please include:

• FastVideo version: pip show fastvideo
• GPU: nvidia-smi output (GPU model, driver, CUDA version)
• Python version: python --version
• OS: e.g., Ubuntu 22.04

If this is a bug, a minimal reproduction script helps us fix it faster.

Useful links:

• Documentation
• Contributing Guide
• Slack

[primorLee]

I traced the divergence to Qwen3-VL position-table interpolation. FastVideo still quantizes the bilinear weights to the position embedding dtype and builds coordinates with the older linspace path, while Transformers 5.15 keeps closed-form weights and accumulation in float32, then casts only when adding them to the vision hidden states. On a deterministic bf16 CPU fixture, the current boundary already drifts by max_abs=0.015625 before any vision block. A native implementation of the 5.15 semantics matches the official helper exactly (atol=0, rtol=0) for single-image, multi-frame video, and packed mixed grids on both Windows and Linux. I am preparing a focused PR with always-on CPU regression coverage plus the existing real-checkpoint parity gate.