Skip to content

[Distributed][HCCL] Harden all_to_all_single split validation and add 2/4/8-card tests#2757

Merged
lvyufeng merged 7 commits into
candle-org:masterfrom
lvyufeng:feat/distributed-split-group-parent-seq
Mar 8, 2026
Merged

[Distributed][HCCL] Harden all_to_all_single split validation and add 2/4/8-card tests#2757
lvyufeng merged 7 commits into
candle-org:masterfrom
lvyufeng:feat/distributed-split-group-parent-seq

Conversation

@lvyufeng

@lvyufeng lvyufeng commented Mar 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Harden HCCL all_to_all_single split validation to use store-synchronized call sequencing and full matrix consistency checks before dispatch.
  • Ensure invalid split pairing fails at Python validation stage on all ranks, avoiding partial-rank fallthrough into HCCL recv/send errors.
  • Expand HCCL all_to_all_single multicard coverage to 2/4/8 for both async unequal semantics and invalid split validation, with transient init retry handling.

Key Changes

  • src/mindtorch_v2/distributed/__init__.py
  • tests/mindtorch_v2/test_hccl_all_to_all_single_async_unequal_multicard.py
  • tests/mindtorch_v2/test_hccl_all_to_all_single_invalid_splits_multicard.py

Test Plan

  • PYTHONPATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:src pytest -q tests/mindtorch_v2/test_hccl_all_to_all_single_invalid_splits_multicard.py
  • PYTHONPATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:src pytest -q tests/mindtorch_v2/test_hccl_all_to_all_single_async_unequal_multicard.py
  • PYTHONPATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:src python tests/mindtorch_v2/test_hccl_all_to_all_multicard.py
  • PYTHONPATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:src pytest -q tests/mindtorch_v2/test_hccl_mvp_multicard.py
  • PYTHONPATH=src pytest -q tests/mindtorch_v2/contract/test_schema_registration_order.py tests/mindtorch_v2/contract/test_schema_coverage.py

Notes

  • CI environments without Ascend devices may not execute the HCCL multicard paths; these checks were validated on local Ascend hardware.

@lvyufeng
lvyufeng force-pushed the feat/distributed-split-group-parent-seq branch from 56a14c5 to 9b24d6e Compare March 8, 2026 07:05
@lvyufeng
lvyufeng merged commit d29ee8e into candle-org:master Mar 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant