feat(platform): k8s manifests for QuanvNN MVP (Sprint 2C)#41
Merged
Conversation
Align with platform-bot mapping (PR #7, ffd16f5). The bot exposes __QUANVNN_IRSA_ROLE_ARN__ resolved from terraform output quanvnn_irsa_role_arn (k8s-platform PR #39). Without this rename, the bot would not substitute the placeholder and the rendered ServiceAccount annotation would carry a literal __IRSA_ROLE_ARN__ string — IRSA would silently fail with assumeRoleWithWebIdentity errors at first run. Identity chain: SA quanvnn-sa in ml namespace, derived from nameOverride=quanvnn + releaseName=quanvnn + suffix '-sa'.
Resolves AUDIT_QUANVNN_MVP.md HIGH finding — the NVIDIA k8s-device- plugin DaemonSet was not installed by any ApplicationSet. Without it, the kubelet does not advertise nvidia.com/gpu, so GPU Jobs stay Pending forever even after Cluster Autoscaler provisions a g4dn node. - New ApplicationSet at sync-wave 2 (after istio-base/istiod/CNI). - Targets kube-system, project: platform, releaseName: nvidia-device-plugin. - Restricts the DaemonSet to nodes labelled node-group=gpu (matches terraform.tfvars node_groups.gpu.labels) — CPU nodes unaffected. - Tolerates the GPU taint nvidia.com/gpu=true:NoSchedule. - Pinned via __NVIDIA_DEVICE_PLUGIN_VERSION__ in platform.yaml (0.19.1). Bot follow-up (Sprint 2D): platform-bot must learn to map versions.nvidia_device_plugin → __NVIDIA_DEVICE_PLUGIN_VERSION__ in internal/gitops/placeholders.go before this ApplicationSet can sync.
Platform-owned PriorityClasses for GPU workloads. They are cluster- scoped resources, so they cannot live in the chart-owned ml AppProject (whose clusterResourceWhitelist is empty). The platform AppProject scope owns them, deployed via a dedicated ApplicationSet at sync-wave -2 (parallels istio-base CRDs — no inter-dependency). - gpu-high-priority (value: 1000) — inference, time-sensitive. - gpu-batch-priority (value: 500) — QuanvNN training (default for ML). Workloads opt in via spec.priorityClassName in their pod template. QuanvNN ml-batch-job chart will reference gpu-batch-priority in Sprint 3 (chart values + template wiring).
Sprint 3 will introduce a dataSync initContainer using public.ecr.aws/aws-cli to s3 sync the QuanvNN dataset from S3 to a PVC before the main training container starts. Without this allowlist entry, Kyverno would deny QuanvNN pods at admission time when policy mode flips to Enforce. The allowlist addition is conservative: the prefix public.ecr.aws/aws-cli covers exactly the AWS CLI image and nothing else under public.ecr.aws. No change to validationFailureAction — Audit remains until the policy hardening sprint.
Resolves AUDIT_QUANVNN_MVP.md MEDIUM finding — the chart referenced imagePullSecret.secretName 'ghcr-pull-secret' in job-cpu.yaml and job-gpu.yaml but no template existed to provision it from AWS Secrets Manager. Operators had to pre-create the Secret manually. - New template templates/external-secret.yaml, conditional behind .Values.imagePullSecret.enabled (chart default flipped from true to false — opt-in is safer for existing consumers). - Added .Values.imagePullSecret.secretStoreRef (name + kind) — defaults to the canonical ClusterSecretStore 'aws-secrets-manager'. - API version external-secrets.io/v1beta1 — matches platform's pinned ESO 0.10.7 and the existing grafana-admin-externalsecret pattern. - Target Secret type kubernetes.io/dockerconfigjson — matches the imagePullSecrets consumer in job templates. - Schema extended to lock the new secretStoreRef shape. Existing chart defaults render zero ExternalSecret (verified via helm template). quanvnn-dev.yaml already opts in (enabled: true).
Resolves PR_41_REVIEW.md F1 + F2 (BLOCKERS). Without this patch, the two new ApplicationSets shipped by PR #41 (nvidia-device-plugin + priority-classes) would be denied at sync time by argocd-application-controller: F1: 'application repo https://nvidia.github.io/k8s-device-plugin is not permitted in project platform' — fixed by appending the chart repo to spec.sourceRepos. F2: 'cluster-scoped resource PriorityClass.scheduling.k8s.io is not permitted in project platform' — fixed by appending an entry to spec.clusterResourceWhitelist. These were silent at chart-render time (helm lint, helm template, yaml.safe_load all pass). They surface only at ArgoCD sync. A CI gate diffing every new AS repoURL against the referenced project's sourceRepos, and every new cluster-scoped GVK against the clusterResourceWhitelist, would catch this pre-merge — tracked as a Sprint 3 follow-up in the review report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sprint 2C from
roadmap.md— Kubernetes manifests + Helm chart only.Five distinct commits, each addressing one Sprint 2C item.
Pre-flight: integrity review GO_WITH_WARNINGS, 0 blockers.
Commits
7345510fix(values): rename IRSA placeholder for QuanvNN94697b7feat(argocd): NVIDIA device plugin ApplicationSet0d02210feat(scheduling): GPU PriorityClasses + ApplicationSet0ec75cffeat(kyverno): allow public.ecr.aws/aws-cli for initContainersa4d5980feat(chart): conditional ExternalSecret for ghcr-pull-secretAudit findings addressed
added at sync-wave 2.
ExternalSecret template → conditional template added (opt-in).
__IRSA_ROLE_ARN__→__QUANVNN_IRSA_ROLE_ARN__,aligning with platform-bot (PR [FEATURE] GPU node group — Terraform + NVIDIA device plugin + MIG time-slicing #7, ffd16f5).
public.ecr.aws/aws-cli(Sprint 3 dataSync initContainer).
Identity chain (audit §6)
quanvnn-sainml. Locked by chart helpers (verified viahelm templatein pre-2C integrity review).__QUANVNN_IRSA_ROLE_ARN__(this PR).__QUANVNN_DATASETS_BUCKET_NAME__(NOT consumedin this PR — Sprint 3 will use it). Note: canonical name uses NAME
suffix and plural "datasets".
Validation
helm lintpasses withquanvnn-dev.yaml(after C1 and after C5).helm templaterenders zeroExternalSecretwith chart defaults,exactly one when
imagePullSecret.enabled=true.helm templateescape verification: rendered.dockerconfigjsonbody contains literal
{{ .dockerconfig | toString }}for ESO toevaluate at sync time.
YAML.load_fileon all newApplicationSets and PriorityClass manifests.
kubectl applyagainst a live cluster — Sprint 2 orchestrationhandles that via the bot.
Deviations from prompt — justified
argocd/applicationsets/; the repoconvention is
argocd/platform/<domain>/<name>.yaml. Usedargocd/platform/gpu/for both new ASes (NVIDIA + PriorityClasses).imagePullSecretschema: prompt suggested new field names(
create,name,secretStoreRef,remoteSecretKey); existingchart already had
enabled,secretName,awsSecretPath. Reusedthe existing names, flipped default
enabled: true → false(saferopt-in), added
secretStoreRefonly.Follow-ups (separate sessions)
internal/gitops/placeholders.gomust learn
versions.nvidia_device_plugin→__NVIDIA_DEVICE_PLUGIN_VERSION__. Until shipped, the NVIDIA AScarries a literal placeholder and won't sync — flagged in commit
message of
94697b7.bootstrap eks --env dev,seed the secret at
platform/dev/ghcr-pull-token(orplatform/ghcr-pull-tokenper current quanvnn-dev value) with aJSON object containing a
dockerconfigkey.Out of scope
eks → re-hydrate (roadmap §2.12-2.17).
and the new Kyverno allowlist entry).
Review-driven follow-up commits
e402435fix(argocd): extend platform project for Sprint 2C ASes(resolves F1 + F2 from PR_41_REVIEW.md — adds
https://nvidia.github.io/k8s-device-plugintosourceReposandscheduling.k8s.io/PriorityClasstoclusterResourceWhiteliston theplatformAppProject).