feat: Phase 44 — multi-node distributed training

[aarambh-darshan]

Summary

Extends v2 §27's single-node NCCL data parallelism to multiple nodes —
still data-parallel only, not model/pipeline-parallel — so training can scale
past whatever a single machine's GPU count offers. The gradient all-reduce
math is unchanged from v2; only the topology it runs over grows, and the
rendezvous that shares the NCCL unique id now supports a TCP transport so
nodes without a shared filesystem can join the world.

Bumps the workspace version to 4.0.0-alpha.4.

Motivation

v2 §27 proved data-parallel training across the GPUs of one machine, but
its ceiling is the GPU count of a single box. Phase 44 lifts only that ceiling:
the world is now N nodes × M GPUs instead of 1 node × M GPUs. The
gradient all-reduce math is byte-for-byte unchanged — what changes is the
topology it runs over and the rendezvous that bootstraps it.

Per the roadmap's honesty note, Kaggle notebooks do not provide genuine
multi-node access, so this phase is validated on CPU via a distributed
unit-test suite that exercises every multi-node code path (topology math,
TCP rendezvous over loopback, retry policy, rank-zero decision, device-count
fix) without needing CUDA hardware.

What changed

crates/aarambh-studio-train/src/distributed.rs (extended)

• MultiNodeTopology — combines num_nodes, gpus_per_node,
  node_rank, and local_rank into the global rank and world size that
  NCCL and the data loader see. Invariants:
  • world_size = num_nodes * gpus_per_node
  • rank = node_rank * gpus_per_node + local_rank
  • The global rank zero — the only rank that logs and checkpoints — is
    exactly the first node's first GPU, never every node's local rank zero.
• RendezvousTransport enum (File default | Tcp { endpoint }) —
  File reproduces v2 single-node behaviour byte-for-byte; Tcp (Phase 44)
  lets genuinely separate nodes exchange the 128-byte NCCL unique id over
  the network.
• Rendezvous trait + FileRendezvous + TcpRendezvous — pure
  standard-library I/O that exchanges a Vec<u8> blob, so the entire
  rendezvous layer compiles and is unit-tested on CPU without the cuda
  feature. The actual NCCL Id only enters at the call site, behind
  #[cfg(feature = "cuda")].
• RetryPolicy — exactly one retry on a transient (timeout /
  connection-refused) error, then fail loudly. Full elastic training is
  explicitly out of scope.
• Device-count fix — v2 required device_count >= world_size on every
  worker, which is wrong for multi-node (a 2-node × 2-GPU world has
  world_size = 4 but each node only has 2 GPUs). A multi-node worker now
  needs only gpus_per_node devices locally; single-node runs keep v2's
  >= world_size check byte-identical.
• DistributedConfig gains five fields — num_nodes, node_rank,
  gpus_per_node, rendezvous, retry_attempts — all defaulting to the
  single-node v2 behaviour. Only num_nodes >= 2 activates multi-node mode.
• New env overrides: AARAMBH_STUDIO_NUM_NODES,
  AARAMBH_STUDIO_NODE_RANK, AARAMBH_STUDIO_GPUS_PER_NODE,
  AARAMBH_STUDIO_DIST_RENDEZVOUS_ENDPOINT, AARAMBH_STUDIO_DIST_RETRIES.

crates/aarambh-studio-train/src/config.rs

• Extended the rank-0 log line to include num_nodes, gpus_per_node,
  node_rank, and the rendezvous transport when running multi-node.

crates/aarambh-studio-train/src/lib.rs

• Re-exports the new public types (MultiNodeTopology,
  RendezvousTransport, Rendezvous, FileRendezvous, TcpRendezvous,
  RetryPolicy, NCCL_ID_BYTES, build_rendezvous).

New files

• configs/multinode_smoke.toml — CPU smoke config with num_nodes = 2,
  gpus_per_node = 1, TCP rendezvous on loopback, retry_attempts = 1.
• scripts/phase44_smoke.sh — runs the distributed unit tests, a CPU
  fallback training smoke, and writes a scorecard to
  artifacts/phase44_multi_node_smoke.json.
• docs/phase44_multi_node.md — Phase 44 design, mechanism, CPU/CUDA
  honesty policy, fault-tolerance scope, and test catalogue.

Docs / version bump

• Cargo.toml workspace version → 4.0.0-alpha.4; Cargo.lock regenerated.
• ROADMAP_V4.md — Phase 44 task checklist marked [x].
• CHANGELOG.md — [4.0.0-alpha.4] entry added.
• ARCHITECTURE_V4.md §58 — "Implementation (Phase 44)" subsection added.
• README.md — version, "Current Boundaries" (multi-node is data-parallel
  only), phase-doc link, and citation version updated.

Backward compatibility

Every existing single-node config (e.g. configs/wikitext103_small_2gpu.toml)
deserialises to byte-identical v2 behaviour: num_nodes defaults to 1, so
world_size and rank are taken as explicitly configured and the topology
is inactive. Only num_nodes >= 2 activates multi-node mode. The gradient
all-reduce, bucketing, checkpoint format, and optimiser are all unchanged.

Tests

15 new/updated CPU unit tests in aarambh-studio-train (no cuda feature
required), including the 4 roadmap-named acceptance tests:

Test	Gate
world_size_one_node_reproduces_v2_single_node_behaviour_exactly	backward compat
gradient_all_reduce_correctness_across_simulated_multi_node_topology	all-reduce math across 2-node × 2-GPU (4 ranks)
rank_zero_checkpoint_writes_from_exactly_one_process_globally	only global rank 0 checkpoints
transient_nccl_timeout_triggers_single_retry_then_fails_loudly	single-retry fault policy
multi_node_topology_derives_global_rank_and_world_size	topology math
invalid_multi_node_topology_rejected	topology validation
multi_node_config_requires_gpus_per_node_devices_not_world_size	device-count fix
file_rendezvous_round_trips_id_bytes	file transport
file_rendezvous_receive_times_out_when_rank0_never_publishes	file timeout
tcp_rendezvous_broadcasts_id_bytes_across_loopback	TCP transport (4 ranks, loopback)
sharded_data_loader_partitions_across_global_world_size_not_local_gpus	global world_size drives the shard count
multi_node_topology_validate_requires_tcp_endpoint_when_configured	TCP endpoint validation
+ 3 inherited v2 tests	regression coverage

CI verification

All quality gates pass locally:

• cargo fmt --all --check ✅
• cargo check --workspace --all-targets --locked ✅
• cargo clippy --workspace --all-targets --locked -- -D warnings -D clippy::undocumented_unsafe_blocks ✅
• cargo test --workspace --no-fail-fast --locked ✅ (20 crates)
• RUSTDOCFLAGS="-D warnings -D missing_docs" cargo doc --workspace --no-deps ✅
• scripts/phase28_release_audit.sh (v4.0.0-alpha.4) ✅
• CLI smoke (--version + 29 subcommand --help) ✅
• scripts/phase44_smoke.sh ✅ (checkpoint saved, 20 tensors)

Out of scope (documented, not implied)

• Full elastic training (nodes joining/leaving mid-run, checkpoint-and-resume
  on node failure) — only the single-retry behaviour ships; the rest is
  flagged as future work.
• Model/pipeline parallelism — this phase is data-parallel only.
• Real-hardware multi-node throughput benchmarks — reported only where
  genuinely available, labelled with the validation path (external multi-VM
  tunnel vs. single-machine loopback simulation).

Milestone

Multi-node data-parallel training runs correctly on the documented
validation path, with gradient correctness verified against the
single-node v2 baseline on identical data. Real-hardware multi-node
throughput numbers are reported only where genuinely available and are
clearly labelled as such — never implied from the simulation path.