Pipeline-parallel inference across an attested Apple Silicon cluster (+ Gemma 4)

[crypt0fairy]

Summary

Adds multi-device clustering so a co-located, individually-attested Apple
Silicon cluster serves as one logical provider, splitting a model layer-wise
across nodes over an encrypted MLX ring — without breaking Darkbloom's
operator-blind guarantee. This unlocks models too large for any single Mac
(e.g. Gemma-4-26B, GPT-OSS-120B) by pooling the cluster's combined unified memory.

Two commits:

1. Pipeline-parallel cluster — the full clustering stack (trust, transport, decode, batching).
2. Gemma 4 support — wires the Gemma 4 shard into the cluster.

Depends on the fork submodules: Layr-Labs/mlx#2 (ring backend), Layr-Labs/mlx-swift#8
(enable ring), Layr-Labs/mlx-swift-lm#53 (GPT-OSS/Gemma-4 pipeline shards).

Architecture

consumer ─(E2E)→ COORDINATOR ─(WS, head only)→ HEAD ─(encrypted ring)→ PEERS
                  sees ONE provider               serves + streams      own layer slice

• Only the head connects to the coordinator; it registers as a single provider
  (the coordinator's Provider == one socket model is preserved). Peers join only
  the ring and loop in lockstep via a per-request control-round all_gather.
• Layer split is memory-weighted (each node ∝ usable RAM), largest-remainder.
• Activations are encrypted per hop: X25519 key agreement over the ring +
  per-token ChaCha20-Poly1305, AAD-bound to cluster/request/layer/seq. A tap on
  the wire sees only ciphertext, with forward secrecy.

Core modules (provider-swift/Sources/ProviderCore/Cluster/): ClusterRoster,
ClusterHandshake, ClusterLinkCrypto, LayerPartition, ClusterPlan,
MLXDistributed, ActivationCodec, ClusterPipeline, ClusterServer,
DistributedInferenceEngine, PipelineModelShard + GPT-OSS/Gemma-4 adapters,
and the ClusterBatch* continuous-batching path.

Trust & Attestation

The privacy guarantee rests on decrypting only inside attested hardware. The cluster
extends single-node attestation to every node:

• Each member's Secure Enclave signs its attestation blob; the head relays them;
  the coordinator verifies each and sets cluster trust = min(member trust).
• The coordinator signs a ClusterRoster; nodes verify it before forming pairwise
  ClusterHandshake links (so they only ring with coordinator-authorized peers).

Honest gaps (not yet closed — flagged for review):

• Per-member attestation gather is opt-in (DARKBLOOM_CLUSTER_ATTEST=1), off by
  default; with it off the coordinator attests the head and peers are trusted
  transitively.
• Relayed peers cap at self_signed (no per-peer MDM/MDA over their own link).
• No per-member challenge/nonce freshness yet (member-blob replay window).
• A dedicated coordinator-adjacent Cluster Attestation Service (per-node enroll +
  roster issuance) is the proposed path to close these without disturbing the
  one-provider-per-socket serving model.

Performance (measured, batch=1, Thunderbolt)

Clustering's value is fitting models that don't fit one node, not speeding up
ones that do (pipeline runs GPUs sequentially at batch=1 — see
docs/cluster-benchmark.md). Continuous batching recovers ~2.5× aggregate
throughput. TP/EP analysis + the TP comms floor are documented in
docs/architecture/cluster-tensor-expert-parallel.md.

Validation

• Crypto/planning: ClusterCryptoTests (roster, handshake, link cipher, split).
• Sharding correctness: monolithic-vs-shard logit oracles at 0.0000 diff for
  GPT-OSS and Gemma 4.
• Real hardware: GPT-OSS-20B and Gemma-4-26B-A4B-it-qat-4bit each sharded
  across two Macs over Thunderbolt (encrypted ring, coherent generation, ~28 tok/s).

Robustness

An orphaned peer used to busy-spin its control loop on a dead ring (pegging a CPU
core indefinitely); it now backs off exponentially and exits.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

[vercel]

@crypt0fairy is attempting to deploy a commit to the EigenLabs Team on Vercel.

A member of the Team first needs to authorize it.