Repository navigation
YARN-12022. Fix CapacityOverTimePolicy rejecting reservations and updates within the quota - #8815
Open
joseluisll wants to merge 1 commit into
Open
joseluisll wants to merge 1 commit into
joseluisll wants to merge 1 commit into
Conversation
…ns within the quota. CapacityOverTimePolicy#validate rejects reservations that are far within the average and instantaneous limits when the user has an allocation in progress at the end of the checked window (the reservation end plus the validity window), or a periodic allocation crossing the period boundary. The allocation is then open-ended up to Long.MAX_VALUE in the merged view, which overflows the integral computation. These tests reproduce the issue and fail until it is fixed. Do not merge the CapacityOverTimePolicy integral view up to Long.MAX_VALUE. CapacityOverTimePolicy#validate merged the user consumption and the reservation over [Long.MIN_VALUE, Long.MAX_VALUE]. The consumption is taken with getRangeOverlapping(checkStart, checkEnd), which keeps the start of an allocation in progress at checkEnd but not its end, so that allocation extended to Long.MAX_VALUE. Computing the integral up to there overflowed, and reservations far within the quota were rejected. End the merge at checkEnd + validWindow instead. The integral is only checked over [checkStart, checkEnd], and it only depends on points up to the time it is evaluated at, so the check is unchanged in that range except for the overflow. Ending the merge at checkEnd instead would add a point there, where the integral approximation overestimates the window and could reject reservations within the quota. Explain why CapacityOverTimePolicy ends the merge after checkEnd. Ending the merge of the integral view at checkEnd would add a point inside the checked range, where the approximated integral overestimates the window. Say so next to the code rather than only in the commit history. Add a test for CapacityOverTimePolicy rejecting updates within the quota. InMemoryPlan#updateReservation validates the new version of a reservation while the old version is still in the plan, so the user's consumption already includes it. CapacityOverTimePolicy#validate then adds the old version again instead of subtracting it, and updates far within the average limit are rejected. The test fails until this is fixed. It also checks that an update over the limit is still rejected. Subtract the old version of an updated reservation in CapacityOverTimePolicy. InMemoryPlan#updateReservation validates the new version of a reservation while the old version is still in the plan, so the user's consumption already includes it. CapacityOverTimePolicy#validate merged the old version into that consumption with RLEOperator.add, counting it twice, so updates far within the average limit were rejected. Subtract it instead, as the comment above says and as availableResources() already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joseluisll
force-pushed
the
YARN-12022-upstream
branch
from
October 12, 2026 11:52
4a7d4ef to
f99af95
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
CapacityOverTimePolicy#validaterejects reservations that are well within the user's average and instantaneous limits. There are two causes:Integral overflow when an allocation is open at the end of the window. The user's consumption is read with
getRangeOverlapping(checkStart, checkEnd). This keeps the start of an allocation that is still in progress atcheckEnd, but not its end. The merged view went up toLong.MAX_VALUE, so that allocation stayed open untilLong.MAX_VALUEand the integral computation overflowed. This happens when the user has an allocation in progress at the end of the checked window (reservation end + validity window), or a periodic allocation that crosses the period boundary.Fix: end the merge at
checkEnd + validWindowinstead ofLong.MAX_VALUE. Only the integral over[checkStart, checkEnd]is checked, and it depends only on points up to the time it is evaluated at. So the check is the same as before in that range, without the overflow. Ending the merge atcheckEndwould not work: it adds a point inside the checked range, where the approximated integral overestimates the window and could still reject reservations within the quota. A code comment explains this.Old version counted twice on update.
InMemoryPlan#updateReservationvalidates the new version while the old one is still in the plan, so the user's consumption already includes it.validatethen merged the old version in again withRLEOperator.addinstead of removing it, and updates well within the average limit were rejected. Fix: useRLEOperator.subtract. This matches the existing comment above that code and whatavailableResources()already does.How was this patch tested?
Three new tests in
TestCapacityOverTimePolicy, each of which fails without the fix:testUserAllocationInProgressAtEndOfCheckedWindow: a reservation within the quota is accepted while the user has an allocation in progress past the checked window.testPeriodicAllocationCrossingPeriodBoundary: same, with a periodic allocation that crosses the period boundary.testUpdateReservationWithinQuota: an update within the quota is accepted, and an update over the limit is still rejected withPlanningQuotaException.The existing
TestCapacityOverTimePolicyand reservation tests pass.For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles? (N/A)AI Tooling
Contains content generated by Claude Code.