Validate P2P transfers across different GPU SKUs

[zhengluo-nv]

Problem

P2P workers with the same SourceIdentity are assigned the same mx_source_id and can discover one another as peers. GPU SKU is not currently part of SourceIdentity, and cross-SKU P2P transfer has not been validated.

If an inference engine uses a different in-memory weight representation for different GPU SKUs, workers that are not transfer-compatible may currently be grouped together. We do not yet know the compatibility rules for combinations such as B200 → B300 or H200 → B200.

Goal

Define and validate the GPU compatibility requirements for P2P weight transfer, then make peer discovery reject incompatible workers if needed.

Scope

• Run a P2P transfer with the same model, rank, parallelism, dtype, and framework version across different source and destination GPU SKUs.
• Test representative GPU pairs in both directions where direction may affect compatibility.
• Verify both transfer completion and model correctness after the transfer.
• Determine whether compatibility should be keyed by GPU SKU or by a more precise engine/hardware property.
• If a GPU-dependent compatibility field is required, include it in SourceIdentity/mx_source_id or otherwise reject incompatible peers before transfer.
• Document the resulting compatibility rule and any unsupported combinations.

Acceptance criteria

• A reproducible heterogeneous-GPU test covers at least one representative cross-SKU pair.
• The test verifies transferred weights are usable and produce correct model behavior, not only that the transfer API succeeds.
• The compatibility behavior for representative combinations such as B200/B300 and H200/B200 is documented when hardware is available.
• Incompatible workers cannot be silently treated as transferable peers.
• Regression coverage protects the chosen compatibility behavior.