feat: Phase 44 — multi-node distributed training

ARCHITECTURE_V4.md

@@ -494,6 +494,39 @@ checkpoint-and-resume on node failure) is explicitly out of scope — a
 genuinely large feature in its own right that this project does not
 attempt to half-implement.
 
+### Implementation (Phase 44, v4.0.0-alpha.4)
+
+The multi-node surface lives in `aarambh-studio-train/src/distributed.rs`:
+
+- `MultiNodeTopology { num_nodes, gpus_per_node, node_rank, local_rank }`
+  derives `global_world_size = num_nodes * gpus_per_node` and
+  `global_rank = node_rank * gpus_per_node + local_rank`; `is_global_rank0()`
+  is true only for the first node's first GPU, so exactly one process
+  globally logs and checkpoints.
+- `RendezvousTransport` enum (`File` default | `Tcp { endpoint }`):
+  `File` reproduces v2 single-node behaviour byte-for-byte; `Tcp` (Phase 44)
+  lets genuinely separate nodes exchange the 128-byte NCCL unique id over
+  the network.
+- `Rendezvous` trait + `FileRendezvous` + `TcpRendezvous`: pure
+  standard-library I/O that exchanges a `Vec<u8>` blob, so the entire
+  rendezvous layer compiles and is unit-tested on CPU without the `cuda`
+  feature. The actual NCCL `Id` only enters at the call site, behind
+  `#[cfg(feature = "cuda")]`.
+- `RetryPolicy`: one retry on a transient (timeout / connection-refused)
+  error, then fail loudly.
+- Device-count fix: a multi-node worker needs only `gpus_per_node` devices
+  locally (not the full global `world_size`); single-node runs still need
+  `world_size`.
+- `DistributedConfig` gains `num_nodes`, `node_rank`, `gpus_per_node`,
+  `rendezvous`, `retry_attempts`, all defaulting to the single-node v2
+  behaviour. Only `num_nodes >= 2` activates multi-node mode, deriving
+  `world_size` and `rank` from the topology. Every existing single-node
+  config deserialises to byte-identical v2 behaviour.
+
+The gradient all-reduce math (`all_reduce_gradients`, `sync_bucket`,
+`all_reduce_flat`) is unchanged from v2 §27 — only the topology it runs
+over and the rendezvous that bootstraps it change.
+
 ---
 
 ## 59. Test-Time Compute Scaling

CHANGELOG.md

@@ -2,6 +2,91 @@
 
 > From first principles. From zero. From Rust.
 
+## [4.0.0-alpha.4] - 2026-08-16
+
+### Added
+
+- **Phase 44 — Multi-Node Distributed Training:** Extends v2 §27's
+  single-node NCCL data parallelism to multiple nodes — still
+  data-parallel only, not model/pipeline-parallel — so training can scale
+  past whatever a single machine's GPU count offers. The gradient
+  all-reduce math is unchanged from v2; only the topology it runs over
+  grows, and the rendezvous that shares the NCCL unique id now supports a
+  TCP transport so nodes without a shared filesystem can join the world.
+  - New `MultiNodeTopology` (`aarambh-studio-train`): combines
+    `num_nodes`, `gpus_per_node`, `node_rank`, and `local_rank` into the
+    global rank and world size that NCCL and the data loader see. The
+    invariant `world_size = num_nodes * gpus_per_node` and
+    `rank = node_rank * gpus_per_node + local_rank` holds by construction,
+    so the global rank zero — the only rank that logs and checkpoints — is
+    exactly the first node's first GPU, never every node's local rank zero.
+  - New `RendezvousTransport` enum (`File` default | `Tcp { endpoint }`):
+    `File` reproduces v2 single-node behaviour byte-for-byte (a
+    shared-filesystem rendezvous directory); `Tcp` (Phase 44) lets
+    genuinely separate nodes exchange the 128-byte NCCL unique id over the
+    network — rank 0 binds `endpoint`, every other rank connects to
+    receive the id. Required for multi-node runs whose nodes do not share
+    a filesystem.
+  - New `Rendezvous` trait + `FileRendezvous` + `TcpRendezvous`
+    implementations: pure standard-library I/O that exchange a `Vec<u8>`
+    blob, so the entire rendezvous layer compiles and is unit-tested on
+    CPU without the `cuda` feature. The actual NCCL `Id` type only enters
+    at the call site, behind `#[cfg(feature = "cuda")]` — the same
+    structure v2 used for its own distributed code.
+  - New `RetryPolicy`: implements the roadmap's "exactly one retry on a
+    transient NCCL rendezvous timeout, then fail loudly" behaviour,
+    without attempting full elastic training (explicitly out of scope). A
+    transient error (timeout or connection-refused during the brief window
+    before rank 0 is listening) is retried once; non-transient errors
+    (shape mismatch, unsupported build, invalid config) propagate
+    immediately.
+  - Device-count fix: v2 required `device_count >= world_size` on every
+    worker, which is wrong for multi-node (a 2-node × 2-GPU world has
+    `world_size = 4` but each node only has 2 GPUs). Phase 44 changes the
+    check to require `device_count >= gpus_per_node` (multi-node) and keep
+    `>= world_size` (single-node) — so a multi-node worker only needs the
+    GPUs it actually hosts, not the whole global world.
+  - `DistributedConfig` gains five fields — `num_nodes`, `node_rank`,
+    `gpus_per_node`, `rendezvous`, `retry_attempts` — all defaulting to
+    the single-node v2 behaviour (`num_nodes = 1`, `rendezvous = File`,
+    `retry_attempts = 1`). Every existing single-node config deserialises
+    to byte-identical v2 behaviour; only `num_nodes >= 2` activates
+    multi-node mode, deriving `world_size` and `rank` from the topology.
+  - New env overrides: `AARAMBH_STUDIO_NUM_NODES`,
+    `AARAMBH_STUDIO_NODE_RANK`, `AARAMBH_STUDIO_GPUS_PER_NODE`,
+    `AARAMBH_STUDIO_DIST_RENDEZVOUS_ENDPOINT`, `AARAMBH_STUDIO_DIST_RETRIES`.
+  - New config: `configs/multinode_smoke.toml` (CPU smoke with
+    `num_nodes = 2`, `gpus_per_node = 1`, TCP rendezvous on loopback,
+    `retry_attempts = 1`); new script `scripts/phase44_smoke.sh`; new doc
+    `docs/phase44_multi_node.md`.
+  - Tests (CPU, no cuda, 15 total): `world_size_one_node_reproduces_v2_single_node_behaviour_exactly`,
+    `gradient_all_reduce_correctness_across_simulated_multi_node_topology`,
+    `rank_zero_checkpoint_writes_from_exactly_one_process_globally`,
+    `transient_nccl_timeout_triggers_single_retry_then_fails_loudly`,
+    `multi_node_topology_derives_global_rank_and_world_size`,
+    `invalid_multi_node_topology_rejected`,
+    `multi_node_config_requires_gpus_per_node_devices_not_world_size`,
+    `file_rendezvous_round_trips_id_bytes`,
+    `file_rendezvous_receive_times_out_when_rank0_never_publishes`,
+    `tcp_rendezvous_broadcasts_id_bytes_across_loopback`,
+    `sharded_data_loader_partitions_across_global_world_size_not_local_gpus`,
+    `multi_node_topology_validate_requires_tcp_endpoint_when_configured`,
+    plus the three inherited v2 tests (`config_env_overrides_world_rank_and_local_rank`,
+    `gradient_average_matches_two_rank_mean`, `invalid_rank_is_rejected`).
+    The TCP rendezvous test binds an ephemeral loopback port and runs four
+    threads as the four ranks of a 2-node × 2-GPU world.
+
+### Honesty note on hardware
+
+Kaggle notebooks do not provide genuine multi-node access. This phase is
+validated using (a) the `distributed` unit-test suite exercising the real
+multi-node code paths (topology, TCP rendezvous over loopback, retry
+policy, rank-zero decision, device-count fix) on CPU, and (b) a documented
+single-machine loopback simulation or external multi-VM tunnel for the
+real NCCL path. Real-hardware multi-node throughput numbers are reported
+only where genuinely available and are clearly labelled as such — never
+implied from the simulation path.
+
 ## [4.0.0-alpha.3] - 2026-08-15
 
 ### Added

Cargo.toml

@@ -24,7 +24,7 @@ members = [
 resolver = "2"
 
 [workspace.package]
-version = "4.0.0-alpha.3"
+version = "4.0.0-alpha.4"
 edition = "2024"
 rust-version = "1.89"
 description = "From first principles. From zero. From Rust."

README.md

@@ -18,14 +18,17 @@ with hybrid Gated DeltaNet, DeepSeek Sparse Attention,
 fine-grained MoE with shared experts, Multi-Token Prediction (MTP), on-policy
 distillation, native quantization-aware training, native video/document input,
 bounded long-horizon tool-use chains, persistent forgetting diagnostics, and
-Max thinking mode (16,384-token budget). **v4.0.0-alpha.3** continues the v4 arc
+Max thinking mode (16,384-token budget). **v4.0.0-alpha.4** continues the v4 arc
 with Multi-Head Latent Attention (Phase 41), a native Audio modality
-(Phase 42), and sparse/grouped MoE dispatch (Phase 43) — a frozen audio
+(Phase 42), sparse/grouped MoE dispatch (Phase 43), and multi-node
+distributed training (Phase 44) — a frozen audio
 spectrogram transformer plus trainable projector that lets the model hear and
 reason about audio clips (the same frozen-encoder-plus-projector recipe vision,
-video, and documents use), and real sparse expert dispatch where each token
+video, and documents use), real sparse expert dispatch where each token
 computes only its assigned top-k experts rather than every expert on every
-token then masked (numerically equivalent to the dense path, faster on CUDA).
+token then masked (numerically equivalent to the dense path, faster on CUDA),
+and data-parallel training extended across multiple nodes over a TCP
+rendezvous so the world can scale past a single machine's GPU count.
 
 > [!IMPORTANT]
 > This is a source and engineering project. It does not publish crates to
@@ -255,13 +258,16 @@ CUDA checks require a CUDA-capable environment and are intentionally opt-in.
 | [docs/phase38_forgetting.md](docs/phase38_forgetting.md) | Capability curves, routing drift, training/self-learning hooks, and Manas JSONL |
 | [docs/phase41_mla.md](docs/phase41_mla.md) | MLA configuration, retrofit, and KV-cache report |
 | [docs/phase42_audio.md](docs/phase42_audio.md) | Audio encoder, mel-spectrogram, fusion, tuning, inference, and audio-QA evaluation |
+| [docs/phase43_sparse_moe.md](docs/phase43_sparse_moe.md) | Sparse/grouped dispatch design, CPU/CUDA honesty, and equivalence proof |
+| [docs/phase44_multi_node.md](docs/phase44_multi_node.md) | Multi-node topology, TCP rendezvous, single-retry fault policy, and validation paths |
 | [RELEASE.md](RELEASE.md) | Source-release process and artifact policy |
 | [CHANGELOG.md](CHANGELOG.md) | Versioned implementation history |
 
 ## Current Boundaries
 
 - No pretrained model, GGUF, adapter, or binary ships — you train your own.
-- MoE uses dense masked dispatch (not sparse grouped). Multi-GPU is single-node.
+- MoE uses dense masked dispatch on CPU (sparse dispatch is CUDA-only, Phase 43).
+  Multi-node training is data-parallel only (Phase 44), not model/pipeline-parallel.
 - Tool chains are generated and orchestrated but never executed by the runtime.
 - Video is visual-only H.264 MP4; audio is WAV PCM only (no MP3/FLAC/Ogg).
 - Documents are pixel-based (no OCR/table parser).
@@ -284,7 +290,7 @@ reproducible bugs and scoped feature requests. Report vulnerabilities through
   author  = {Aarambh Dev Hub},
   year    = {2026},
   url     = {https://github.com/AarambhDevHub/aarambh-studio},
-  version = {3.0.0},
+  version = {4.0.0-alpha.4},
   license = {Apache-2.0}
 }
 ```

ROADMAP_V4.md

@@ -531,7 +531,7 @@ because that claim would not be honest.
 
 **`aarambh-studio-train`:**
 ```
-[ ] src/distributed.rs (extended)
+[x] src/distributed.rs (extended)
       Node rank vs local rank distinction (world_size = nodes ×
       gpus_per_node)
       Multi-node NCCL initialisation over TCP rendezvous
@@ -542,7 +542,7 @@ because that claim would not be honest.
       Minimal fault handling: a single retry on a transient NCCL
       timeout before failing loudly — this phase does not attempt full
       elastic/fault-tolerant training, which is explicitly out of scope
-[ ] Rank-zero logging/checkpointing extended to first-node-rank-zero
+[x] Rank-zero logging/checkpointing extended to first-node-rank-zero
     specifically, so multi-node runs do not produce duplicate
     checkpoints from every node's local rank zero
 ```