What happened
A CI deploy to kind failed after roughly 20 minutes:
argocd argocd-redis-6c6f5486cb-nnwp5 0/1 ImagePullBackOff 0 30m
Failed to pull image "ecr-public.aws.com/docker/library/redis:8.2.3-alpine":
429 Too Many Requests
##[error]Applications did not converge within 1200s
Run: https://github.com/nebari-dev/nebari-infrastructure-core/actions/runs/33446494722
The same commit deployed and passed on the identical kind shape twenty minutes earlier in deployment-tests.yml, and a re-run of the failed job went green, so this is transient rate limiting rather than a broken image reference.
Where the image comes from
Not from us. Nothing in this tree references ecr-public.aws.com. It is the default in the upstream argo/argo-cd chart, which we pin at 9.7.1 in pkg/argocd/config.go. Upstream moved to ECR Public to get out from under Docker Hub's rate limits; unauthenticated ECR Public has its own limit, and GitHub-hosted runners share egress IPs with every other tenant pulling the same way.
Frequency
Observed once. I do not know whether this is weekly or a one-off, and that is worth measuring before choosing an expensive fix.
Why it matters more now
#642 wires the user journey suite into all five provider jobs in deployment-tests.yml, hard-failing. A transient pull failure now costs a full deploy cycle (up to 90 minutes on AWS and Azure) and produces a red that reads like a platform fault. On a release run, which fans out to all five providers, it would fail the lot.
Options, roughly in order of appeal
Pre-pull and kind load for the kind-based jobs. ci.yml's deploy job and deployment-tests.yml's local job both run kind on the runner, so the image can be pulled once on the host (with a retry) and side-loaded into the cluster. Removes the in-cluster pull for the job that runs most often. Does nothing for AWS, Azure, or Hetzner, where nodes pull directly.
Override redis.image.repository. There is a seam: DefaultConfig() in pkg/argocd/config.go already passes a Values map containing a redis key. I would be cautious here. Docker Hub's anonymous limits are stricter on shared CI IPs, public.ecr.aws is the same backend under a different hostname, and changing the registry that every Nebari operator pulls from in order to fix our CI flakiness is a platform decision rather than a CI fix.
Authenticated pulls. ECR Public grants higher limits with credentials. The aws job already has OIDC; the kind-based jobs do not, and this adds a credential to the deploy path for every operator.
Retry the deploy step. Crude and general. A 429 is transient, but a retry costs another full deploy cycle and does not help if the whole window is rate-limited.
Precedent in this repo
The Hetzner job already seeds ~/.hetzner-k3s/k3s-releases.yaml using authenticated gh api calls, specifically so it stops sharing an unauthenticated budget with every other tenant of the runner's IP. Same class of problem, solved by removing the unauthenticated dependency rather than retrying it.
Related
The diagnosis cost of this failure is tracked separately: the deploy waited the full 1200 seconds on a condition that was terminal within seconds, and then reported a symptom rather than the cause. See the companion issue on fail-fast convergence detection.
What happened
A CI deploy to kind failed after roughly 20 minutes:
Run: https://github.com/nebari-dev/nebari-infrastructure-core/actions/runs/33446494722
The same commit deployed and passed on the identical kind shape twenty minutes earlier in
deployment-tests.yml, and a re-run of the failed job went green, so this is transient rate limiting rather than a broken image reference.Where the image comes from
Not from us. Nothing in this tree references
ecr-public.aws.com. It is the default in the upstreamargo/argo-cdchart, which we pin at 9.7.1 inpkg/argocd/config.go. Upstream moved to ECR Public to get out from under Docker Hub's rate limits; unauthenticated ECR Public has its own limit, and GitHub-hosted runners share egress IPs with every other tenant pulling the same way.Frequency
Observed once. I do not know whether this is weekly or a one-off, and that is worth measuring before choosing an expensive fix.
Why it matters more now
#642 wires the user journey suite into all five provider jobs in
deployment-tests.yml, hard-failing. A transient pull failure now costs a full deploy cycle (up to 90 minutes on AWS and Azure) and produces a red that reads like a platform fault. On a release run, which fans out to all five providers, it would fail the lot.Options, roughly in order of appeal
Pre-pull and
kind loadfor the kind-based jobs.ci.yml's deploy job anddeployment-tests.yml'slocaljob both run kind on the runner, so the image can be pulled once on the host (with a retry) and side-loaded into the cluster. Removes the in-cluster pull for the job that runs most often. Does nothing for AWS, Azure, or Hetzner, where nodes pull directly.Override
redis.image.repository. There is a seam:DefaultConfig()inpkg/argocd/config.goalready passes aValuesmap containing arediskey. I would be cautious here. Docker Hub's anonymous limits are stricter on shared CI IPs,public.ecr.awsis the same backend under a different hostname, and changing the registry that every Nebari operator pulls from in order to fix our CI flakiness is a platform decision rather than a CI fix.Authenticated pulls. ECR Public grants higher limits with credentials. The
awsjob already has OIDC; the kind-based jobs do not, and this adds a credential to the deploy path for every operator.Retry the deploy step. Crude and general. A 429 is transient, but a retry costs another full deploy cycle and does not help if the whole window is rate-limited.
Precedent in this repo
The Hetzner job already seeds
~/.hetzner-k3s/k3s-releases.yamlusing authenticatedgh apicalls, specifically so it stops sharing an unauthenticated budget with every other tenant of the runner's IP. Same class of problem, solved by removing the unauthenticated dependency rather than retrying it.Related
The diagnosis cost of this failure is tracked separately: the deploy waited the full 1200 seconds on a condition that was terminal within seconds, and then reported a symptom rather than the cause. See the companion issue on fail-fast convergence detection.