feat(ml): ml-batch-job chart + quanvnn application — audit-driven fixes folded

[JackMaarek]

Summary

Folds the two audit-driven fixes from AUDIT_QUANVNN_MVP.md (2026-04-26) into
the existing feature branch, ready to merge to main.

Audit findings addressed

• BLOCKER-1 (HIGH, §2.4): chart args ↔ mount paths ↔ image contract misalignment.
  Resolved via Fix A — Helm-only, no QuanvNN code change.
• MEDIUM (§2.5): quanvnn.yaml pinned project: platform while
  argocd/projects/ml.yaml was dead code. Repointed to project: ml and
  added ignoreDifferences for batch/Job /status and Pod controller-uid.

Changes

• kubernetes/helm/values/quanvnn-dev.yaml — paths, args, gpu.enabled
• argocd/applications/ml/quanvnn.yaml — project, ignoreDifferences

Validation

• helm lint passes.
• helm template rendered output shows the corrected paths
  (/data/dubai_data/Dubai256, /data/quanv_ckpt, /app/output/fusion).
• kubectl apply --dry-run=client passes on the ArgoCD Application.

Out of scope (Sprint 2)

• NVIDIA device plugin, PriorityClasses, S3+IRSA, ExternalSecret template,
  Kyverno allowlist, AWS SM seed.

Identity chain (audit §6)

ServiceAccount: quanvnn-sa in ml. OIDC subject:
system:serviceaccount:ml:quanvnn-sa. Do NOT rename nameOverride,
releaseName, or serviceAccount.name — silently breaks IRSA in Sprint 2.

Follow-up commits (review-driven, 2026-04-26)

Doc-grounded review (PR_38_REVIEW.md, Context7 cross-checks) flagged two
items requiring action; PodMonitor namespace selector was already correct
(prometheus-values.yaml uses match-all {}, CASE B — no commit).

• 5c559dc docs(ml-batch-job): correct istio-injection state for ml namespace
  Stale README claimed namespace had injection enabled with per-pod opt-out.
  Actual: kubernetes/namespaces/ml-namespace.yaml has no istio-injection
  label (= disabled). README aligned; obsolete sidecar.istio.io/inject=false
  block removed.
• 04697ad fix(argocd/ml): cover prefixed controller-uid label in ignoreDifferences
  Per Kubernetes labels-annotations-taints reference (Context7 / k8s docs):
  K8s 1.27+ Job controller sets BOTH controller-uid AND
  batch.kubernetes.io/controller-uid simultaneously on owned Pods. K8s 1.33
  still sets both (verified via researchMode query). Added the prefixed
  jsonPointer /metadata/labels/batch.kubernetes.io~1controller-uid (RFC 6901
  encoding) alongside the legacy entry to prevent ArgoCD drift on the new key.

[JackMaarek]

Review — ml-batch-job chart + QuanvNN Application

Reviewed against Helm best practices (Context7 / helm.sh), Prometheus Operator PodMonitor patterns, and platform conventions.

Overall shape is solid — the chart is reusable, values are well-structured, the ArgoCD Application uses multi-source correctly, and the conformity audit fixes are all applied. Three issues must be resolved before merge; four are non-blocking suggestions.

---

🔴 BLOCKING — must fix before merge

1. Prometheus has no RBAC to scrape the ml namespace

pod-monitor.yaml creates a PodMonitor in the ml namespace, but kube-prometheus-stack only provisions RBAC (Role + RoleBinding) for Prometheus in the namespaces it controls by default (monitoring, kube-system, default). Without a Role granting get/list/watch on pods and a matching RoleBinding for the Prometheus ServiceAccount in the ml namespace, the PodMonitor will be silently ignored at runtime — no scrape targets, no errors, nothing in Prometheus UI.

Fix: add a Role + RoleBinding in kubernetes/manifests/monitoring/ (or as chart templates guarded by metrics.enabled):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-ml-scraper
  namespace: ml
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus-ml-scraper
  namespace: ml
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus-ml-scraper
subjects:
  - kind: ServiceAccount
    name: kube-prometheus-stack-prometheus
    namespace: monitoring

Also verify that prometheus.prometheusSpec.podMonitorNamespaceSelector in prometheus-values.yaml is set to match-all (matchLabels: {}) or explicitly includes the ml namespace — otherwise the Prometheus Operator itself won't watch for PodMonitors outside its own namespace.

---

2. Empty image.cpu.repository renders silently as :cpu-latest

job-cpu.yaml:

image: "{{ .Values.image.cpu.repository }}:{{ .Values.image.cpu.tag }}"

If image.cpu.repository is not overridden (chart default is ""), this renders as :cpu-latest — syntactically accepted by the Kubernetes API but fails with InvalidImageName at scheduling time. The error is hard to trace back to a missing value.

Use Helm's required function to catch this at helm template / helm install time:

image: "{{ required "image.cpu.repository must be set" .Values.image.cpu.repository }}:{{ .Values.image.cpu.tag }}"

Same fix for image.gpu.repository inside the {{- if .Values.gpu.enabled }} block in job-gpu.yaml.

---

3. README.md namespace table is stale after Istio injection fix

The table in README.md reads:

| `ml` | ML batch workloads | enabled (Jobs opt-out via annotation) | restricted |

But ml-namespace.yaml no longer carries istio-injection: enabled — it was removed during the conformity audit. The table should show disabled to reflect the actual state.

---

🟡 SUGGESTIONS — non-blocking, recommended for follow-up

4. Missing values.schema.json

Helm recommends a values.schema.json (JSON Schema draft-07) alongside values.yaml to enforce types, enums, and required fields at helm install / helm lint time — before any template rendering runs. This is the correct layer to constrain workload.kind to ["Job"], pullPolicy to the three valid strings, and namespace.name to a non-empty string.

Minimal schema for the critical fields:

{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["namespace", "image"],
  "properties": {
    "workload": {
      "type": "object",
      "properties": {
        "kind": { "type": "string", "enum": ["Job"] }
      }
    },
    "namespace": {
      "type": "object",
      "required": ["name"],
      "properties": { "name": { "type": "string", "minLength": 1 } }
    },
    "image": {
      "type": "object",
      "properties": {
        "cpu": {
          "type": "object",
          "properties": {
            "pullPolicy": { "type": "string", "enum": ["Always", "IfNotPresent", "Never"] }
          }
        },
        "gpu": {
          "type": "object",
          "properties": {
            "pullPolicy": { "type": "string", "enum": ["Always", "IfNotPresent", "Never"] }
          }
        }
      }
    }
  }
}

---

5. No ttlSecondsAfterFinished on Job specs

Without a TTL, completed and failed Jobs accumulate indefinitely. The quanvnn Application intentionally retains them for log inspection — valid, but a long TTL makes the retention policy explicit rather than implicit:

spec:
  ttlSecondsAfterFinished: 604800  # 7 days — retain for log inspection, then auto-clean
  backoffLimit: {{ .Values.cpu.backoffLimit }}

Expose this as a configurable value in values.yaml (default 604800, set to 0 to disable retention).

---

6. workload.kind has no template guard

values.yaml documents workload.kind: "Job" with a note that only Job is currently implemented. But no template enforces this — a user setting workload.kind: Deployment silently gets a Job. Add a fail guard at the top of job-cpu.yaml:

{{- if ne .Values.workload.kind "Job" }}
{{- fail (printf "workload.kind '%s' is not supported. Only 'Job' is currently implemented." .Values.workload.kind) }}
{{- end }}

---

7. completions and parallelism not explicit on Job specs

Kubernetes defaults both to 1 when omitted, but making them explicit documents intent and avoids surprises if the defaults ever change in a future API version:

spec:
  completions: 1
  parallelism: 1
  backoffLimit: {{ .Values.cpu.backoffLimit }}

---

✅ What's correct

• Multi-source ArgoCD Application with $values alias — correct pattern for separating chart from per-env values ref
• sync-wave: "9" — correctly placed after monitoring stack (waves 4–8)
• CreateNamespace=false in syncOptions — namespace is platform-managed, not chart-managed
• prune: true + selfHeal: true — right for a commit-triggered Job model
• ignoreDifferences on /status for batch/Job — prevents ArgoCD from flagging Completed Jobs as OutOfSync
• PodMonitor instead of ServiceMonitor — correct for ephemeral Jobs with no Service endpoint
• Grafana dashboard ConfigMap in monitoring namespace with grafana_dashboard: "1" — correct sidecar discovery pattern
• podSecurityContext.seccompProfile: RuntimeDefault — satisfies PSS restricted profile
• readOnly: true on data and checkpoint PVCs in quanvnn-dev.yaml — correct least-privilege for read-only inputs
• Named templates prefixed with ml-batch-job.* — correct Helm convention, prevents subchart name conflicts
• __ENV__ / __TARGET_REVISION__ / __REPO_URL__ placeholders throughout — correct for platform-bot hydration model