Readiness gate Job uses ArgoCD Replace=true, which cannot succeed on upgrade (immutable Job fields)

[atif1996]

Summary

The readiness gate Job generated for the ArgoCD deployers is annotated with
argocd.argoproj.io/sync-options: Replace=true. That option cannot succeed against a Job that
already exists, so the first time an upgrade changes the rendered gate manifest the sync fails
permanently and the Application is left OutOfSync.

Why it fails

spec.selector and spec.template.metadata.labels are generated by the Job controller at
creation time (the batch.kubernetes.io/controller-uid labels) and are immutable. The rendered
manifest correctly omits them. Replace=true makes Argo CD send the whole object, so the API
server sees those immutable fields going from their generated values to null and rejects it:

one or more objects failed to apply, reason: error when replacing "/dev/shm/2799122012":
Job.batch "gpu-operator-readiness-gate" is invalid:
  spec.selector: Required value
  spec.template.metadata.labels: Invalid value: null: `selector` does not match template `labels`
  spec.selector: Invalid value: null: field is immutable
  spec.template: Invalid value: {...}: field is immutable
(retried 5 times)

After the retries Argo CD backs off, and the Application stays OutOfSync / Healthy
indefinitely.

When it triggers

Not on every sync. While the rendered Job manifest is unchanged Argo CD sees no diff and never
attempts a sync, so a freshly created gate is fine.

It triggers on upgrade: the gate image is ghcr.io/nvidia/aicr-gate:v<version>, so bumping
the AICR version changes the Job spec, Argo CD must sync, and the replace fails. Reproduced
going to 0.19.0 against a gate Job created by an earlier version.

Root cause

pkg/bundler/gatemanifest/manifest.go, jobMetadataAnnotations. The Helm branch already uses
the correct delete-then-recreate pattern; only the Argo CD branch is affected:

func jobMetadataAnnotations(deployer config.DeployerType) string {
	switch deployer {
	case config.DeployerHelm:
		return `  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation`     // correct
	case config.DeployerArgoCD, config.DeployerArgoCDHelm:
		return `  annotations:
    argocd.argoproj.io/sync-options: Replace=true`        // cannot work on an existing Job

Introduced in #1110 and unchanged since.

Suggested fix

Use the Argo CD equivalent of the Helm pattern, so the Job is deleted and recreated rather than
replaced in place:

argocd.argoproj.io/hook: Sync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation

Adding Force=true alongside Replace=true would also work, since that makes Argo CD delete
and recreate, but the hook form mirrors the existing Helm behaviour and states the run-once
intent directly.

These currently assert the Replace=true string and would need updating:

• pkg/bundler/gatemanifest/manifest_test.go:42
• pkg/bundler/readiness_test.go:144
• goldens under pkg/bundler/deployer/argocd/testdata/ and pkg/bundler/deployer/argocdhelm/testdata/

Impact

Any cluster deploying AICR bundles through Argo CD ends up with a permanently OutOfSync
readiness Application after its first AICR upgrade. Health stays Healthy, so it is easy to
miss, but it fails any policy or test that requires all Argo CD applications to be Synced.

Workaround

Delete the Job and force a sync. A refresh alone is not enough, because auto-sync has already
backed off after its retries:

kubectl delete job gpu-operator-readiness-gate -n gpu-operator
kubectl patch application gpu-operator-readiness -n argocd --type merge \
  -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"syncStrategy":{"hook":{}}}}}'

Verified: the gate Job is recreated, completes 1/1, and the Application returns to
Synced / Healthy.

[mchmarny]

Triaged: Priority P2 → P1. Linked PR #2408 fixes an Argo CD upgrade path that otherwise leaves the readiness Application permanently OutOfSync.