Skip to content

Topology-aware source metadata for MoE and P/D scaleout #471

Description

@athreesh

Summary

ModelExpress source identity is currently mostly model + backend + TP/PP/EP + dtype/quantization.

That is enough for simple P2P cold-start cases, but it isn't enough for mapping topologies that users now care about: WideEP, TEP/DEP, P/D disagg, shadow/failover engines, and EPLB-style expert layout.

We should explore an engine-agnostic topology metadata contract so MX can tell when two workers are actually compatible sources for each other.

Motivation

Large MoE deployments have new requirements, especially when served with EP. Oftentimes, the EPLB info is essential for ensuring the transfer starts a new worker that matches the source worker.

A worker's compatibility can depend on factors like:

  • whether it's a P/D worker, agg worker, or shadow engine,
  • sharding layout,
  • WideEP rank layout,
  • expert placement,
  • whether EPLB /redundant experts are enabled,
  • whether expert traffic expects NVL/MNNVL, but KV or weight movement can use RDMA,
  • engine runtime (SGL, vLLM, TRT-LLM, etc.)

Today MX has no first-class place to represent this. We need to explore what fields affect source compatibility, which fields are per-worker runtime state, and how different engines want to describe the same topology. Especially for MoE, (wide)EP, P/D deployments.

Proposal

Add a topology descriptor to MX metadata that is common across engines. The first version should focus on source compatibility and discovery, not on making MX own runtime scheduling.

Some possible fields:

  • serving worker role: prefill, decode, aggregated, shadow, etc.
  • parallelism shape: TP, PP, DP, EP, CP/PCP where available,
  • rank identity: global rank, local rank, DP rank, EP rank, engine group id,
  • topology domain: node, rack, NVL domain, RDMA fabric, compute domain,
  • expert metadata: logical experts, physical slots, redundant experts, placement strategy,
  • EPLB metadata: enabled/disabled plus a stable config/layout digest,
  • transport expectations: RDMA, NVL/MNNVL, local-only, or mixed paths,
  • framework-specific escape hatch for fields that do not normalize cleanly yet.

This ideally should stay backend-agnostic. vLLM DP+EP, SGLang WideEP, and TRT-LLM disagg/WideEP should all be able to populate the same shape, but map to engine specific implementations.

Acceptance

  • MX can distinguish incompatible MoE/P/D workers that currently look identical.
  • MX can track info such as EPLB and use that to select the right worker/replicate for target new workers.
  • The schema isn't engine-specific.
  • Exisiting TP/PP/EP deployments continue to work unchanged.
  • A new doc that details a WideEP or P/D scaleout example

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions