fix(dev): move prd root Application to bootstrap for reproducibility

[asad-miah]

Purpose

prd-base (the prd root app-of-apps) currently depends on two fixes that exist only on the cluster and cannot be reproduced from git: a Prune=false annotation and a removed tracking-id. Both were applied by hand with kubectl during the v0.13.0 release because the only file that could carry them (kubernetes/clusters/prd/base.yaml) is deliberately excluded from its own kustomization. If prd-base is ever recreated, both fixes would have to be reapplied from memory or prd's whole app tree gets pruned again. This moves the manifest to kubernetes/bootstrap/prd-base.yaml, outside every synced tree, so the arrangement is self-describing and survives a rebuild.

What Changed

• Moved kubernetes/clusters/prd/base.yaml to kubernetes/bootstrap/prd-base.yaml, and added the argocd.argoproj.io/sync-options: Prune=false annotation directly to the committed manifest.
• Updated kubernetes/clusters/prd/kustomization.yaml's comment to explain the new location instead of describing an in-directory exclusion.
• Updated .github/workflows/publish-release.yml: the FILES array now points at the new path, and a guarded argocd app get/app create step runs before pinning prd-base, so a missing root Application self-heals during release instead of failing as a PermissionDenied auth error.
• Fixed the stale path comment in kubernetes/clusters/prd/rhesis/rhesis-application.yaml.
• Updated kubernetes/README.md: bootstrap command, directory tree, promotion-gate paragraphs, and a new "Why prd's root Application lives in bootstrap/" section covering both the self-reference problem and the prune/tracking-id problem.

Additional Context

Closes #2544. The exclusion of base.yaml/prd-base.yaml from its own kustomization is intentional and unchanged — publish-release.yml overwrites targetRevision imperatively via argocd app set, and a commit can't embed its own hash, so a self-managed manifest would revert the pin on every sync. Only the file's location changes here. Per the issue, this must land on main between releases (not ride a release branch) — promote-prd-config's drift check diffs kubernetes/clusters/prd against main and only tolerates targetRevision: differences.

Testing

• kubectl kustomize kubernetes/clusters/prd still renders 17 Applications and does not contain prd-base.
• grep -rn "clusters/prd/base.yaml" .github kubernetes returns nothing.
• kubectl apply -f kubernetes/bootstrap/prd-base.yaml --dry-run=client validates the manifest.
• Simulated the publish-release.yml write-back logic against a scratch copy of the repo: every file in the updated FILES array (including the new kubernetes/bootstrap/prd-base.yaml path) has exactly one targetRevision: line, and the sed rewrite only touches that line, leaving annotations/comments intact.
• Not yet tested against a live cluster: reconciling the live prd-base object, and exercising the new argocd app create guarded path by deleting prd-base on dev/stg first (flagged in the issue as a manual follow-up).

[peqy]

Overall this makes the prd root Application location/self-healing behavior much more reproducible and the README updates are clear.

Improvement: tighten the “prd-base missing” guard in publish-release.yml so we only app create when we’re sure it’s actually missing (not just any CLI failure).

Found 1 issue (1 improvement).

[peqy]

Improvement: This argocd app get check treats any failure as “missing app”, so transient connectivity issues / TLS errors / RBAC misconfig could incorrectly trigger app create and then fail the job even if prd-base actually exists.

Fix: capture and inspect the error output and only create on the specific “missing app” signature you’re expecting, otherwise fail with the original error. Alternatively: attempt argocd app set first; if it fails with the “missing” error, then app create and retry app set/sync.

[asad-miah]

the argocd app get check is a bare one-shot call with no retry — every other ArgoCD call in this same job goes through retry_argocd, which already exists precisely to absorb transient network blips (per its own comment). A single connectivity hiccup on get immediately falls into the create branch today, which is inconsistent with the rest of the file's pattern.

[peqy]

Yep — agree. The app get should also go through retry_argocd to avoid a one-off blip dropping into the create path.

I’d still avoid treating any post-retry failure as “missing”: wrap retry_argocd argocd app get ... while capturing stderr, and only app create if the output matches the specific “missing app” signature you’re seeing; otherwise print the error and fail the job.

[peqy]

Yep, agreed — routing app get through retry_argocd is the right fix for consistency and to avoid a transient blip falling into the create branch. The updated approach (retry + only create on the specific PermissionDenied signature, otherwise fail with the original error) addresses my original concern.

[asad-miah]

Fixes pushed in a346ea8 — added kubernetes/bootstrap/prd-base.yaml to the drift-check diff paths per Harry's review, and reworded the retry_argocd comment/log message for accuracy (it retries network-level failures, not the PermissionDenied/missing-app case, by design).

[peqy]

Thanks — yep, this addresses my original concern.

• Adding kubernetes/bootstrap/prd-base.yaml to the drift-check path closes the “unvalidated change slips into prd” gap.
• The retry_argocd-wrapped app get + “only attempt create on PermissionDenied, otherwise fail with the original error” is the right behavior, and the clarified comment/logging makes the intent much clearer.

No further changes needed from my side on this thread.

[peqy]

Looks good — moving prd-base into kubernetes/bootstrap/ makes the bootstrap state reproducible and avoids the self-reference/prune footguns described.

One small suggestion: in publish-release.yml, the branch that keys off PermissionDenied logs prd-base not found — creating..., but your own comment notes this signature can also be a real RBAC issue. Might be worth wording that message as “PermissionDenied from app get; attempting create in case it’s the missing-app case” to avoid misleading diagnosis.

[harry-rhesis]

Thanks for this, Asad. The README rewrite is really thorough, and making the bootstrap state reproducible from git is a big operational win. A couple of things worth looking at before merge.

Drift check no longer covers prd-base.yaml

The "Verify main hasn't drifted" step (line 66) diffs only kubernetes/clusters/prd. Now that prd-base.yaml lives in kubernetes/bootstrap/, a non-targetRevision change to it (e.g. path, repoURL, syncPolicy) would be promoted silently without stg validation. Same class of risk the check was built to prevent.

Fix is small:

UNEXPECTED=$(git diff "$GITHUB_SHA" HEAD -- kubernetes/clusters/prd \
  kubernetes/bootstrap/prd-base.yaml \
  | grep -E '^[+-][^+-]' \
  | grep -v 'targetRevision:' || true)

retry_argocd fast-exits on "permission denied", so the retry is a no-op for missing apps

retry_argocd (line 161) treats permission denied as non-retryable and returns on the first attempt. Since ArgoCD reports a non-existent app as "PermissionDenied", the retry_argocd argocd app get prd-base call will never actually retry the missing-app case. It fails immediately and falls through to the create branch.

The commit message says "route the get through retry_argocd so a transient blip gets retried," but that only holds for transient errors that don't contain "permission denied." The specific failure mode this guard was built for (missing app = PermissionDenied) still gets no retry.

The end behavior is still correct (the create path fires), so this is more about the comment being honest than the logic being wrong. Either update the comment to note the retry only covers network-level failures, or use a separate retry loop for this call that doesn't fast-exit on "permission denied."

Minor: log message and comment path

• The "prd-base not found — creating..." message runs on any "permission denied" match, including real RBAC failures. The peqy reviewer suggested something like "PermissionDenied on app get; attempting create (may be a missing app or a real RBAC issue)", which would make diagnosis clearer.
• rhesis-application.yaml comment says (bootstrap/prd-base.yaml), which is relative to kubernetes/. From four levels deep in kubernetes/clusters/prd/rhesis/, that reads ambiguously. kubernetes/bootstrap/prd-base.yaml (full repo-relative path) would be clearer.

[peqy]

Looks good. Moving prd-base into kubernetes/bootstrap/ + committing Prune=false makes the prd root reproducible, and the updated publish-release.yml now retries app get and only attempts create on the expected PermissionDenied signature (otherwise fails with the real error). Ship it.

[asad-miah]

@harry-rhesis thanks for the thorough review — all three points fixed and pushed in a346ea8:

• Drift check now diffs kubernetes/bootstrap/prd-base.yaml alongside kubernetes/clusters/prd, so a non-targetRevision change to it will fail the promotion gate like any other prd Application manifest.
• Reworded the retry_argocd comment to be accurate: it only helps network-level failures — the missing-app (PermissionDenied) case still fast-exits on attempt 1 by design, since retrying it wouldn't change the outcome either way. Went with the comment fix rather than a separate retry loop, since retrying a real permission error repeatedly wouldn't accomplish anything.
• Log message updated to "PermissionDenied on app get; attempting create (may be a missing app or a real RBAC issue)", and the rhesis-application.yaml comment now uses the full repo-relative path (kubernetes/bootstrap/prd-base.yaml) instead of the ambiguous bootstrap/prd-base.yaml.