OCPBUGS-98462: add jitter to AWS endpoint service requeue delay#8996
OCPBUGS-98462: add jitter to AWS endpoint service requeue delay#8996jparrill wants to merge 1 commit into
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe AWS endpoint controller replaces the fixed 20-second retry interval used when the network load balancer is inactive with a randomized delay. New base and jitter constants define the delay calculation, and retry logs now report the computed requeue duration. Suggested reviewers: 🚥 Pre-merge checks | ✅ 11✅ Passed checks (11 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
jparrill
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@jparrill: This pull request references Jira Issue OCPBUGS-98462, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jparrill The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@jparrill: This pull request references Jira Issue OCPBUGS-98462, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
Bot
added
area/hypershift-operator
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
hypershift-operator/controllers/platform/aws/controller.go (1)
359-361: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd a bounded retry-jitter test.
Please add or extend the controller tests to exercise this error path and assert that
RequeueAfterremains within the documented 16–25 second range, without asserting one exact random value.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hypershift-operator/controllers/platform/aws/controller.go` around lines 359 - 361, Add a controller test covering the reconciliation error path that logs “reconciliation failed, retrying with jitter” and returns RequeueAfter. Execute the path and assert the result stays within the documented 16–25 second inclusive range, checking bounds rather than an exact random value; reuse existing test fixtures and symbols for the AWS controller reconciliation flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@hypershift-operator/controllers/platform/aws/controller.go`:
- Around line 359-361: Add a controller test covering the reconciliation error
path that logs “reconciliation failed, retrying with jitter” and returns
RequeueAfter. Execute the path and assert the result stays within the documented
16–25 second inclusive range, checking bounds rather than an exact random value;
reuse existing test fixtures and symbols for the AWS controller reconciliation
flow.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: c7817516-3d1d-4792-b165-a4ef1d77f5cf
📒 Files selected for processing (1)
hypershift-operator/controllers/platform/aws/controller.go
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8996 +/- ##
==========================================
- Coverage 43.80% 43.80% -0.01%
==========================================
Files 772 772
Lines 96100 96101 +1
==========================================
Hits 42097 42097
- Misses 51084 51085 +1
Partials 2919 2919
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
When multiple HostedClusters are created in parallel, their AWS endpoint service controllers all hit Route53 ChangeResourceRecordSets simultaneously. On failure (e.g. NLB not yet active), all controllers requeued after a fixed 20s, retrying at the same instant and potentially sustaining API throttling. Replace the fixed 20s requeue with 15s base + random 1-10s jitter so concurrent controllers desynchronize their retries. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com>
|
/label acknowledge-critical-fixes-only |
openshift-ci
Bot
added
the
acknowledge-critical-fixes-only
|
@jparrill: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Nice, thanks /lgtm |
|
Scheduling tests matching the |
Test Failure Analysis CompleteJob Information
Test Failure AnalysisErrorSummaryThe codecov/patch check failed because none of the 3 new executable lines introduced by the PR are exercised by unit tests. The PR adds jitter to the AWS endpoint service requeue delay (replacing a fixed 20-second constant with a 15s base + random 1–10s jitter) to prevent thundering-herd retries. All 3 modified lines reside inside an error-handling branch ( Root CauseThe codecov/patch failure is caused by zero test coverage on the modified lines in the error-requeue path of
These lines are inside an This is a false positive / acceptable coverage gap. The change is a 3-line operational improvement with minimal risk. Writing a unit test would require mocking the entire AWS reconciliation stack to trigger the error branch, which is disproportionate effort for a simple jitter addition. The repo's overall coverage is already low (43.80% target), and this is a well-understood error-requeue pattern. Recommendations
Evidence
|
3 participants
Summary
Fixes
Root Cause
When 14+ HostedClusters are created in parallel (as in
e2e-aws-ovn), their AWS endpoint service controllers all call Route53ChangeResourceRecordSetssimultaneously. On failure (typically NLB not yet active), all controllers requeued after a fixed 20s — retrying at the exact same instant and sustaining Route53 API throttling (Throttling: Rate exceeded).Frequency
Searched 40 builds of
periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn(29 failures total):AWSEndpointAvailable=Falseand 9 test failuresExisting throttle handling
The CPO-side
awsprivatelinkcontroller already handles throttling with a 2min backoff (throttleRequeueDelay, line 61 ofcontrol-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go). This fix targets the HO-side endpoint service controller where the fixed 20s delay caused synchronized retries across all HCs.Changes
lbNotActiveRequeueDuration = 20s(fixed, all controllers retry simultaneously)lbNotActiveRequeueBase = 15s+rand(1-10s)jitter (16-25s range, desynchronized)log.Info("retrying in 20s")log.Info("retrying with jitter", "requeueAfter", requeueAfter)Test plan
go buildpassese2e-aws-ovnperiodic — Route53 throttling should no longer cascade across HCs🤖 Generated with Claude Code
Summary by CodeRabbit