C2: SOTA board — dense NVFP4 fused MoE is the SM120 MoE decode SOTA, not quadbit sparse

[vitali87]

Summary

C2 builds the final SM120 MoE SOTA board: a direct, controlled, same-harness comparison of the C1
sparse route-slot D2 policy against the strongest dense/NVFP4 baseline that actually runs on this hardware.
It answers the question C1 could not: is quadbit sparse MoE a decode SOTA? It is not. The board is doing
exactly its job — catching an overclaim before it reaches the headline.

What the board measures

Every row runs through the same harness (graph_gate4 / glm_graph_gate), the same PPL passage
(mito80), the same decode-only formula (63/(wall64-wall1)), the same graph mode (captured), and
the same memory accounting (vLLM per-worker logs). The only change between baseline and quadbit rows is
the MoE selector. The dense baseline is vLLM's native FlashInfer-CUTLASS fused NVFP4 MoE (QB_MOE=off),
with the plugin's attention/DSA unblock so the model loads at all (vanilla vLLM cannot init it on SM120).

New controlled harness knob (no kernel work): baseline=dense_nvfp4 → QB_MOE=off.

Board (captured, same harness)

model	GPUs	dense NVFP4 fused (baseline)	quadbit D2 native captured	dense advantage
DeepSeek-V4-Flash	4	48.248 tok/s, 40.83 GiB/GPU, 0.18 pool, PPL 4.1222	5.972 tok/s, 51.7 GiB/GPU, 2.08 pool, PPL 4.0943	8.1× decode, less memory
GLM-5.2	8	33.810 tok/s, 54.62 GiB/GPU, 0.10 pool, PPL 3.9572	5.367 tok/s, 68.98 GiB/GPU, 0.80 pool, PPL 4.0674	6.3× decode, less memory

DSA native on all four rows (sparse_mla_sm120_decode_dsv4 / FLASHINFER_MLA_SPARSE_SM120); all coherent,
drop=0. C1's 11.3× was measured against the crippled dequant loop (0.514 tok/s), not this dense path.

Verdicts

• V1 DeepSeek decode: NO — dense 48.248 vs D2 5.972 = 8.1× faster.
• V2 GLM decode: NO — dense 33.810 vs D2 5.367 = 6.3× faster.
• V3 memory: sparse INCREASES memory — dual residency costs +27% (DeepSeek) / +26% (GLM) weight
  memory, larger graph pool, and −62% KV on GLM (236,672 vs 629,760 tokens).
• V4 quality: matched-to-slightly-worse — mito80 PPL is 80-token noise (excluded from ranking);
  downstream MC smoke is the metric: DeepSeek D2 −0.79 pt, GLM D2 −0.95 pt vs dense.
• V5 label: "graph-enabled transfer result, not speed SOTA." The dense NVFP4 fused MoE is the SM120
  MoE decode SOTA.

What quadbit sparse MoE genuinely is (not overclaimed)

The only deployed 2:4-sparse FP4 MoE on SM120; training-free capability-preserving structural sparsity that
transfers across architectures (DeepSeek → GLM) and graph-captures with native DSA (downstream within
~1 pt); and a prefill/large-M kernel Pareto point (paper §5 — a microbench, not re-measured in serving
here). Not a decode-speed or decode-memory win.

Next bottleneck (precise, not softened)

At decode (M=1–2) the sparse serving path (fixed-cap device routing + per-expert 2:4 sparse_moe_mm_2lvl

• dual-residency group_gemm_nvfp4 anchor, under Python-level per-expert loops) is 6–8× slower than a
  single autotuned fused NVFP4 grouped GEMM; the 2:4 advantage is a large-M bandwidth effect absent at
  decode. The only missing piece is a kernel: a fused sparse grouped decode GEMM at tiny M, plus dropping
  dual residency. That is genuine new CUDA — identified, not started (per the C2 guardrail).

Baselines that do not run on SM120 (reported, not hidden)

• Vanilla vLLM (no plugin) on DeepSeek-V4-Flash NVFP4: fails to init (ue8m0 FP8 attention asserts). The
  plugin unblock is what lets the dense baseline run at all.
• SGLang NVFP4 MoE for these models: unavailable (not in serve image; no SM120 DSA path).

Deliverables

docs/c2/{sota_board,deepseek_sota,glm_sota,verdict}.md; raw logs docs/audit/logs/c2_*.log;
claims_checklist.md §11; command_manifest.md C2 block; README + paper §10.1/§12/abstract synced to the
honest C2 finding (rebuilt paper.tex/paper.pdf, zero net new overfull boxes).

Guardrails

No "production-wide SOTA" (the board shows the opposite for decode). No "beats dense FP4 generally". Failed
baselines reported. CUTLASS 80b sparse prior art preserved. No mismatched-PPL quality claim. No custom CUDA
started.

Merge checklist

• Greptile 5/5, zero unresolved greptile/gemini comments
• All bot outputs (CodeQL, CI, coverage) addressed
• Merge with a true merge commit (gh pr merge --merge), never squash

[gemini-code-assist]

Code Review

This pull request establishes the C2 SOTA board, comparing the quadbit sparse D2 policy against the dense NVFP4 fused MoE baseline on SM120 hardware. The results, documented across several new markdown files and logs, demonstrate that the dense baseline is 6-8x faster at decode and more memory-efficient, framing the sparse policy's value around quality preservation and cross-architecture graph transfer rather than decode speed. Code changes in harness/serve_dsv4.py add support for running this dense baseline. The review feedback is highly consistent and correct, pointing out several instances in the new documentation files where references to other markdown files should be formatted as relative links, and glob patterns should be wrapped in backticks.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[greptile-apps]

Greptile Summary

This PR adds the C2 same-harness SOTA board for dense NVFP4 fused MoE versus quadbit sparse D2. The main changes are:

• Adds the baseline=dense_nvfp4 harness path that selects QB_MOE=off while keeping the graph-gate protocol unchanged.
• Adds raw C2 DeepSeek and GLM dense/sparse run logs.
• Adds C2 board, model-specific detail pages, and verdict docs.
• Updates the README, claims checklist, command manifest, and paper artifacts with the C2 result.

Confidence Score: 5/5

Safe to merge with minimal risk.

The code change is a narrow harness selector, and the documentation values match the committed C2 logs. No actionable bugs or security issues were identified.

No files require special attention.

T-Rex Logs

What T-Rex did

• The C2 consistency check executable script was generated to validate the consistency checks.
• The generated script was run to perform the C2 consistency checks and produce results.
• The command transcript shows an exit code of 0 and per-row parsed versus expected comparisons.
• The transcript notes that a full Modal/SM120 rerun is unavailable locally and intentionally not performed.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
harness/serve_dsv4.py	Adds the baseline=dense_nvfp4 selector, which maps to QB_MOE=off while preserving the graph gate path.
docs/c2/sota_board.md	Adds the consolidated C2 board with dense and sparse metrics aligned to the raw logs.
docs/c2/verdict.md	Adds the final C2 verdict that dense NVFP4 fused MoE wins decode and memory.
docs/c2/deepseek_sota.md	Adds DeepSeek-specific C2 comparison details, including the corrected 1.10 GiB D2 graph pool.
docs/c2/glm_sota.md	Adds GLM-specific C2 comparison details with metrics aligned to the committed logs.
docs/claims_checklist.md	Extends the claims checklist with C2 negative and guardrail claims.
docs/audit/command_manifest.md	Records the C2 commands and measured outcomes.
README.md	Adds the C2 dense-vs-sparse SOTA board headline and links to the verdict.
docs/paper.md	Syncs the paper narrative with the C2 finding.
docs/paper.tex	Syncs the LaTeX source with the updated C2 board and claims.
docs/paper.pdf	Regenerates the PDF to reflect the updated paper text.
docs/audit/logs/c2_ds_dense_baseline_C.log	Adds raw DeepSeek dense NVFP4 baseline run evidence.
docs/audit/logs/c2_ds_d2_native_C.log	Adds raw DeepSeek D2 captured run evidence.
docs/audit/logs/c2_glm_dense_baseline_C.log	Adds raw GLM dense NVFP4 baseline run evidence.
docs/audit/logs/c2_glm_d2_native_C.log	Adds raw GLM D2 captured run evidence.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant User as Modal CLI
participant Harness as serve_dsv4.py graph_gate4/glm_graph_gate
participant Env as Environment selectors
participant Plugin as qb_sm120 plugin
participant VLLM as vLLM engine
participant MoE as FlashInfer-CUTLASS NVFP4 MoE

User->>Harness: "run graph gate with baseline=dense_nvfp4"
Harness->>Env: "set QB_MOE=off, QB_GRAPH=1, QB_DENSE=nvfp4"
Harness->>VLLM: initialize captured LLM with same PPL/decode protocol
VLLM->>Plugin: load SM120 attention/DSA unblock
Plugin-->>VLLM: "leave MoE unpatched when QB_MOE=off"
VLLM->>MoE: execute native fused dense NVFP4 MoE
VLLM-->>Harness: emit PPL, decode tok/s, KV, graph pool metrics
Harness-->>User: record C2 dense baseline log and board row

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant User as Modal CLI
participant Harness as serve_dsv4.py graph_gate4/glm_graph_gate
participant Env as Environment selectors
participant Plugin as qb_sm120 plugin
participant VLLM as vLLM engine
participant MoE as FlashInfer-CUTLASS NVFP4 MoE

User->>Harness: "run graph gate with baseline=dense_nvfp4"
Harness->>Env: "set QB_MOE=off, QB_GRAPH=1, QB_DENSE=nvfp4"
Harness->>VLLM: initialize captured LLM with same PPL/decode protocol
VLLM->>Plugin: load SM120 attention/DSA unblock
Plugin-->>VLLM: "leave MoE unpatched when QB_MOE=off"
VLLM->>MoE: execute native fused dense NVFP4 MoE
VLLM-->>Harness: emit PPL, decode tok/s, KV, graph pool metrics
Harness-->>User: record C2 dense baseline log and board row

Loading

_{Reviews (2): Last reviewed commit: "fix(c2): correct DeepSeek D2 graph pool ..." | Re-trigger Greptile}

[vitali87]

@greptile review