Scalability: qualify AICR runtime configurations at 2,304 nodes

[mchmarny]

Summary

Establish an explicit, reproducible scalability contract for AICR runtime configurations through 2,304 nodes. Convert scale-test findings into public recipes, qualified component settings, preflight validation, and repeatable evidence.

Problem

AICR can carry Helm values through recipe configuration and overrides, but it does not currently define a scale-selection contract:

• --nodes is advisory and only affects registry-declared paths.
• --set, --set-json, and --set-file transport user-selected values; they do not derive scale-safe settings.
• --dynamic exposes selected values at install time; it does not compute them and is not supported by every deployer.
• No public recipe currently covers GB300 + EKS + Ubuntu + training + Slurm.
• Current AICR component versions and deployment topology must be qualified directly rather than inheriting settings from a different stack.

Without an umbrella effort, large-scale support depends on undocumented overrides and one-off operational knowledge.

Scope

• Add a GB300 EKS Ubuntu training Slurm recipe.
• Define an explicit scale-qualified recipe configuration contract.
• Requalify current AICR component defaults at scale.
• Add preflight validation for scale-sensitive runtime prerequisites.
• Add a repeatable 36-to-2,304-node qualification lane and evidence format.

Non-goals

AICR will not provision or manage Kubernetes infrastructure. This epic does not add ownership for:

• managed control-plane or etcd sizing
• node-pool provisioning
• kubelet reservations
• host sysctls
• cluster DNS architecture
• registry or image-cache infrastructure
• CNI or conntrack infrastructure

AICR may detect and report relevant prerequisites, but infrastructure mutation remains with the cluster provider or provisioner.

This epic also does not copy historical pins or tuning values without qualifying them against AICR's current public component versions and architecture.

Success criteria

• All five child issues are complete.
• Identical scale inputs resolve to identical versioned recipe and bundle outputs.
• Scale-specific behavior is explicit; node count alone never silently selects a specialized configuration.
• Qualification uses current AICR component pins and records the exact resolved artifacts.
• Preflight failures distinguish AICR-owned configuration defects from external infrastructure prerequisites.
• The 2,304-node claim is backed by published pass/fail criteria and repeatable evidence.
• User and contributor documentation explains the supported scale contract and its boundaries.

Operational considerations

The largest qualification lane will be release-gated or manually triggered rather than required for every pull request. Lower-cost checks should run continuously where practical.