ci: consolidate the duplicated cluster secret setup into one implementation

[leiicamundi]

Description

The logic that provisions test-namespace secrets before a chart install exists twice:

• .github/actions/cluster-setup-secrets/action.yaml — ~124 lines of bash
• scripts/camunda-core/pkg/kube/platforms.go — applySecretsForEKS and applyExternalSecretsForGKERosa

Only the bash runs in CI. test-integration-runner.yaml passes --namespace-override unconditionally (lines 767 and 1205), and runner_execute.go gates the Go path on ExternalSecrets: opts.NamespaceOverride == "", so applySecretsFor* is skipped on every CI run. The Go implementation is reached only by manual deploys that omit the override.

The cost of this showed up while fixing #7212. One bug fix — deleting the target Secret before applying the replicate-from stub, because the replicator's replicated-from-version annotation survives kubectl apply and makes it skip the refill — had to be written twice, in two languages. The two polling timeouts are both 300s by coincidence: {1..30} x sleep 10 on one side, replicatedSecretTimeout on the other, with nothing tying them together.

It also breaches the repository's own review rule in .github/instructions/code-review.instructions.md:

If the workflow logic calls external APIs, parses JSON, or contains branching/orchestration, is that logic implemented in Go under scripts/<feature>/ (or existing Go tooling)?
If Bash remains, is it only thin glue (roughly <=20 lines), with business logic moved to Go?

The bash block orchestrates branches and parses JSON with jq across roughly 124 lines.

Motivation / Use Case

Two implementations of the same behaviour drift silently. The one covered by unit tests is not the one that runs in CI, and the one that runs in CI has no tests at all. Every future fix has to be applied twice, or it is only half applied.

What makes this non-trivial

The two implementations are not behaviourally equivalent today, so the Go path cannot simply be switched on. Four divergences found while comparing them:

1. Vault store selection. The action hardcodes the -vault credentials manifest on EKS. The Go path derives the -vault suffix from externalSecretsStore == "vault-backend", and CI resolves vaultBackedSecrets=false on EKS, so it would apply the non-vault manifest and break EKS credentials.
2. Apply retries. The action retries each apply 8 times with backoff, explicitly because the External Secrets validating webhook may not be ready yet. The Go path applies once.
3. Stale ExternalSecret cleanup for integration-test-credentials. The action does it on both branches; the Go path only does it for the TLS secret on EKS.
4. Secret materialisation check. The action waits for the Ready condition and then verifies that the expected keys actually landed on the Secret. The Go path stops at the Ready condition.

Each of these breaks gke, rosa or eks silently if the switch is made without porting it.

Acceptance Criteria

• One implementation owns test-namespace secret provisioning for gke, rosa and eks.
• The four divergences above are resolved rather than dropped.
• The surviving implementation has test coverage for both the EKS replicate-from path and the GKE/ROSA ExternalSecret path.
• .github/actions/cluster-setup-secrets/action.yaml either disappears or is reduced to thin glue, in line with the repository rule quoted above.
• Validated on all three platforms, including an EKS cell. Note that EKS cells only run in the merge queue, since documentstore is tier 2 and test-chart-version.yaml gates the matrix with tier: ${{ github.event_name == 'pull_request' && '1' || '' }}.

Options considered

• (a) Add a CLI entrypoint to scripts/camunda-core (which has no cmd/ today) and have the action call it. Adds surface to build and maintain inside the job.
• (b) Let the runner own secrets: remove the pre-apply from the action and drop the NamespaceOverride == "" gate. Deletes the bash entirely and activates code that already has unit tests, once the four divergences above are closed. Preferred.

[distro-ci]

🧭 Triage Assessment

Field	Value
Work type	TASK
Kind	refactor
Severity	⚠️ Mid
Likelihood	🔶 Mid
Confidence	0.93

📝 Rationale

Within SM Platform Distribution scope because the team boundaries assign ownership of Helm Charts, camunda/camunda-platform-helm, and automated deployment/testing tooling to Distribution. This is a TASK rather than a BUG: the requested outcome is restructuring duplicated CI and Go secret-provisioning implementations, with behavior preserved across GKE, ROSA, and EKS. It is not a feature request or customer-facing capability. The best kind is refactor because the primary goal is code restructuring, testability, and removal of duplicated orchestration without intended behavior change. The impact is mid: the duplication has already caused fixes to be implemented twice and can break CI or manual deployments, but the problem is limited mainly to internal deployment and validation tooling, with no direct production outage or customer data impact demonstrated. The likelihood is mid: drift can recur whenever secret provisioning changes, and related work has already identified the same CI/Go path divergence, but the failure is not shown to occur on every run. The issue is sufficiently specified because it identifies the two implementations, four behavioral divergences, the preferred remediation, platform coverage, tests, and validation requirements. The closest repository issues are broader Go-over-Bash migration and CI-maintenance efforts, plus a version-specific ExternalSecret setup bug, but none is a direct duplicate.

🔍 Similar Issues

#4705, #5383, #6262

---

🤖 This issue was automatically triaged by an AI-based system.
✋ If you believe this assessment is incorrect, update the labels or add the triage:manual label for human review.