Requalify scale defaults for AICR runtime components

[mchmarny]

Summary

Requalify scale-sensitive settings for AICR's current NFD, NVSentinel, Topograph, Slurm, and monitoring components before adopting them as scale defaults.

Problem

Large-cluster tuning results cannot be copied directly when component versions or deployment topology differ. In particular, AICR deploys standalone NFD while some prior configurations tuned GPU Operator's embedded NFD. Current public pins must be measured directly.

Work

At representative qualification bands, measure and evaluate:

• CPU and memory requests and limits
• replica count, concurrency, worker, and queue settings
• Kubernetes client QPS and burst
• rollout and reconciliation duration
• scheduler and topology convergence
• monitoring discovery, scrape coverage, and cardinality
• failure behavior under partial rollout and control-plane throttling

For each component, record the exact AICR version, rendered values, cluster size, workload, and observed resource envelope.

Success criteria

• A qualification matrix covers NFD, NVSentinel, Topograph, Slurm, and monitoring at the agreed scale bands.
• Results use current AICR public pins and resolved artifacts.
• Embedded-NFD results are not treated as proof for standalone NFD without direct validation.
• Every adopted setting has a measured rationale and supported range.
• Default changes remain conservative; scale-only values live in the explicit scale contract.
• Values are represented through registry-declared paths or versioned component configuration.
• Regression tests and documentation accompany adopted settings.
• Results identify external infrastructure limits separately from AICR-owned component limits.

Non-goal

This task does not tune managed control planes, etcd, kubelets, host sysctls, DNS, registries, or CNI infrastructure.