Skip to content

Parity-suite coverage gaps + DSV4 CP validation (successor to #1333, reframed for the single-oracle era) #1586

Description

@tianhaox

Successor to #1333, which is closed as a graveyard: of its five sections, §1/§2/§4.5 were already struck RESOLVED, §3's premise has since become false (the calib data shipped), and §5's first item was done in code but never checked off. This issue carries forward only what is still real, re-verified against main on 2026-08-23 and reframed — #1333's wording assumed a Python reference engine, which the #1357 ladder deleted.

Framing change that applies throughout: there is no second engine. "Both engines agree" is now "the compiled engine agrees with its frozen goldens" (parity_tests/goldens/, refreshed via pin_goldens.py). See .claude/rules/rust-core/parity.md Rule 1.


A. Parity-suite coverage gaps

Against aic-core/rust/aiconfigurator-core/parity_tests/test_engine_step_parity.py. Each item states what I verified.

  • Dense CP coverage is still zero. Verified: 4 CP cases exist (up from Follow-ups from the perf_interp v2 redesign (PR #1303): rust CP model, rust shared-layer, DSV4 CP calib data, minor gaps #1333's 3), all b200_sxm + sglang, all MoE — Qwen3-235B-A22B (cp8, cp4), Kimi-K2.5 (cp8), DeepSeek-V4-Flash (cp8, isl=8192). Add a dense long-context case (e.g. Llama-3.1-70B or Qwen3-32B, tp1/cp4, isl 16k–32k). Note cli_estimate has no cp_size param, so CP is mixed-step-only today; consider threading cp through the static/agg surfaces.
  • Sparse-CP numeric parity — now unblocked and triggered. Follow-ups from the perf_interp v2 redesign (PR #1303): rust CP model, rust shared-layer, DSV4 CP calib data, minor gaps #1333 asked for a symmetric-failure assertion that would "flip to a numeric-parity assertion the day the calib tables ship". They shipped: dsv4_csa_topk_calib_perf.parquet is present on h100_sxm, h200_sxm, b200_sxm, b300_sxm and gb200 under sparse_attention/sglang/0.5.14/. So this should go straight in as a numeric case, not a failure-symmetry one.
  • pp_size > 1 coverage is zero. Verified: no pp_size=2+ anywhere in the suite. Add one dense pp=2 case.
  • attention_dp is backend-skewed. Verified: sglang ×6, vllm ×1, trtllm ×0. Add a trtllm adp8 case.
  • Beyond-range / shared-layer hold is unexercised. No case exercises extrapolation where a shared-layer sibling point moves the anchor (e.g. Kimi-K2.5 or DSV3 on h200 sglang at isl ≥ 32k). This is precisely where feat(rust-core): add context-parallelism and catch up with latest perf DB refactors; make Spica tests non-blocking #1313's loaders changed answers.
  • Tolerance — partially addressed, needs a decision not a change. Verified: PARITY_RTOL = 0.01 is still the general tolerance, but HYBRID_PARITY_RTOL = 1e-4 already exists for the shared-table class with a comment explaining why. Decide whether the static surfaces should join the tighter tier rather than blanket-tightening.

Dropped from #1333 §5: the anti-vacuous guard, which is donetest_engine_step_parity.py:2025 ("Anti-vacuous guard: prove the golden comparison actually bites") plus a sibling in test_compile_engine_parity.py. It was never checked off.

B. DSV4 CP validation track

Different in kind from section A — this is data/calibration quality, not test coverage. Carried from #1333 §3; I did not re-verify these two against current data, so triage before starting.

  • All-gather calibration. The CP comm terms are formula-derived. Instrument a real sglang DSV4 CP timeline (cp_utils-style, as GLM-5 did) to pin the full-isl latent gather (kv_lora+rope per token, pre-cache-write), the kv_score gather, and the replicated per-rank compressor compute (currently unmodeled — consider an analytic term first, standalone bench later). Version-annotate the comm model and test with the sglang commit.
  • Unsaturated-anchor guard. step=0 boundary utils are still climbing (+25.6% median over-prediction if held). Add a util-slope threshold before holding, or borrow plateau util from larger-step slices. Land with its own LOO validation.

Not carried forward

  • Follow-ups from the perf_interp v2 redesign (PR #1303): rust CP model, rust shared-layer, DSV4 CP calib data, minor gaps #1333 §1, §2, §4.5 — struck RESOLVED in that issue already.
  • §3's "csa_topk_calib is collected nowhere" premise — false as of the sglang 0.5.14 drop.
  • §4's "caller-less wideep compute variants hold on the proxy until a caller appears" — dormant by design; it re-opens when a caller appears, not on a schedule.
  • §4's "pre-existing cargo test failures on main (parse_b200_sxm, 4× py::tests)" — not verified here (would need a full Rust build). If still failing, that deserves its own issue rather than a bullet in a coverage list.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions