fix(velero): make the R2 credentials stub create-only so Flux stops clobbering real creds#1979
Merged
Merged
Conversation
…lobbering real creds The velero-r2-credentials Secret in git is a bootstrap stub (PLACEHOLDER values) that exists only to break the chart's existingSecret deadlock on fresh installs. Flux re-applied it on every infrastructure-controllers reconcile, overwriting the real R2 credentials written by the ExternalSecret. ESO normally overwrites it back within a minute, but whenever the ExternalSecret cannot sync (as during the 2026-06-10 OpenBao data loss) the placeholder sticks: the BackupStorageLocation goes Unavailable and every kopia maintenance job fails with "Credential access key has length 11, should be 32" (= len(PLACEHOLDER)). kustomize.toolkit.fluxcd.io/ssa: IfNotPresent keeps the bootstrap behaviour (create when missing) and stops Flux from ever patching the live Secret again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jun 10, 2026
Contributor
Author
ℹ️ The 🧪 System Test failure here is pre-existing on main, not caused by this PR: every CI run since ~18:30 UTC today fails identically (including the unrelated Renovate PRs #1971/#1972/#1974). The test cluster's |
Contributor
|
🎉 This PR is included in version 1.47.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
velero-r2-credentials.yamlis a bootstrap stub (aws_access_key_id=PLACEHOLDER) that breaks the Velero chart'sexistingSecretdeadlock on fresh installs. Because it sits in theinfrastructure-controllersKustomization without any SSA hint, Flux re-applies the PLACEHOLDER values over the ESO-written real credentials on every reconcile. ESO normally wins the ping-pong back within its refresh interval — but whenever the ExternalSecret is wedged (today: the 2026-06-10 OpenBao data loss left it inSecretSyncedError), the placeholder sticks.Observed on prod right now:
BackupStorageLocation default→Unavailable: ... Credential access key has length 11, should be 32(11 =len("PLACEHOLDER"))*-default-kopia-maintain-jobpod failing every ~5 minutes with the same errorFix
Add
kustomize.toolkit.fluxcd.io/ssa: IfNotPresentto the stub: Flux still creates it when missing (bootstrap deadlock stays solved), but never patches the live Secret again, so the ExternalSecret's real values persist even while ESO is degraded.Validation
kubectl kustomize k8s/clusters/local/✅kubectl kustomize k8s/clusters/prod/✅Part of the 2026-06-10 OpenBao incident remediation (see the companion vault-seed / vault-config / vault-snapshot PRs).
🤖 Generated with Claude Code