Fix stranded snapshot markers on publish failure

[Sukriti0717]

Description

Adds retryOrFailOnClusterManagerFailOver helper that retries a failed cluster-state publish (marker removal) with bounded exponential backoff (1s/2s/4s, max 3 retries) when the node is still the elected cluster-manager. Without this, a publish failure on a stable cluster-manager strands the in-progress snapshot marker forever, blocking index deletion and close.

Also fixes a latent stale-read bug in stateWithoutSnapshotV2: the inner execute() was reading from the captured outer state parameter instead of the live currentState, which could clobber concurrent
cluster-state changes. The method is refactored into a task factory (createStateWithoutSnapshotV2Task) that makes this bug structurally impossible to reintroduce.

The retry behavior is gated behind the SNAPSHOT_RESILIENCE feature flag (default off). The stale-read fix is unconditional.

Depends on #22516 (PR-C). Will rebase once that merges.

Testing

• RetryOrFailOnClusterManagerFailOverTests: 10 unit tests covering helper decision logic, stale-state bug fix, and retried-task behavior.
• All tests pass with -Dtests.iters=20 for seed stability.

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit c3a19e9)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Missing listener notification on retry path In handleFinalizationFailure, when the SNAPSHOT_RESILIENCE flag is on and a FailedToCommitClusterStateException occurs, the code calls removeFailedSnapshotFromClusterState(snapshot, e, repositoryData, null) but skips the previous failSnapshotCompletionListeners + failAllListenersOnMasterFailOver calls. If the retry inside removeFailedSnapshotFromClusterState eventually succeeds, clusterStateProcessed will call failSnapshotCompletionListeners(snapshot, failure) (good), but if retries are exhausted the fallback fails listeners with a generic "Failed to remove snapshot from cluster state" wrapper rather than the original finalization failure. Additionally, failAllListenersOnMasterFailOver was previously called unconditionally on NotClusterManagerException too; now with the flag on, only the FailedToCommit branch is handled and NotClusterManagerException falls through with no listener notification at all. Verify the NotClusterManager path still fails listeners. if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING) && ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) != null) { removeFailedSnapshotFromClusterState(snapshot, e, repositoryData, null); } else { failSnapshotCompletionListeners( snapshot, new SnapshotException(snapshot, "Failed to update cluster state during snapshot finalization", e) ); failAllListenersOnMasterFailOver(e); } Missing cluster-manager check before retry retryOrFailOnClusterManagerFailOver retries on any FailedToCommitClusterStateException as long as NotClusterManagerException is not present in the cause chain. However, a publish can fail with FailedToCommitClusterStateException while the node has subsequently stepped down as cluster-manager (before receiving the NotClusterManager signal). The helper does not consult clusterService.state().nodes().isLocalNodeElectedClusterManager() before scheduling retries, contradicting the PR description's stated behavior ("retries ... when the node is still the elected cluster-manager"). Consider explicitly checking cluster-manager status before retrying to avoid submitting doomed tasks. if (ExceptionsHelper.unwrap(e, NotClusterManagerException.class) != null) { failoverFallback.run(); return; } if (ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) == null) { logger.error("Unexpected failure during cluster state update", e); failoverFallback.run(); assert false : new AssertionError("Unexpected failure during cluster state update", e); return; } if (attempt >= maxRetries) { logger.warn("Exhausted {} retries for [{}], falling back to failover handling", maxRetries, source); failoverFallback.run(); return; } final int nextAttempt = attempt + 1; final TimeValue delay = computeBackoff(retryBackoff, attempt); logger.info("Publish failed for [{}] (attempt {}), scheduling retry in [{}]", source, nextAttempt, delay); try { threadPool.schedule(() -> clusterService.submitStateUpdateTask(source, taskFactory.get()), delay, ThreadPool.Names.GENERIC); } catch (OpenSearchRejectedExecutionException ex) { logger.warn("Retry scheduling rejected for [{}], falling back to failover handling", source); failoverFallback.run(); } } Assertion after fallback may mask bugs in tests In retryOrFailOnClusterManagerFailOver, when an unexpected exception type is encountered, failoverFallback.run() is invoked and then assert false is thrown. In assertions-enabled test/dev environments this leaves partial state (fallback already executed) before the AssertionError propagates to the caller. Since this is called from onFailure inside a cluster state task, the AssertionError propagates into the cluster state applier thread and may cause hard-to-diagnose failures. Consider logging and returning without asserting, or asserting before invoking the fallback. if (ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) == null) { logger.error("Unexpected failure during cluster state update", e); failoverFallback.run(); assert false : new AssertionError("Unexpected failure during cluster state update", e); return; }

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to c3a19e9

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Prevent overflow in exponential backoff calc The multiplication base.millis() * (1L << Math.min(attempt, 30)) can still overflow to a negative value for large base values (e.g., base of ~10 seconds with attempt=30 overflows Long). The subsequent Math.min would then pick the negative overflowed value instead of the day-cap. Compute the shift safely (e.g., use Math.multiplyExact in a try/catch or check for overflow) before clamping. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [3378-3381] - static TimeValue computeBackoff(TimeValue base, int attempt) { - final long delayMillis = Math.max(0L, Math.min(base.millis() * (1L << Math.min(attempt, 30)), TimeValue.timeValueDays(1).millis())); - return TimeValue.timeValueMillis(delayMillis); +static TimeValue computeBackoff(TimeValue base, int attempt) { + final long shift = 1L << Math.min(attempt, 30); + long product; + try { + product = Math.multiplyExact(base.millis(), shift); + } catch (ArithmeticException e) { + product = Long.MAX_VALUE; } + final long delayMillis = Math.max(0L, Math.min(product, TimeValue.timeValueDays(1).millis())); + return TimeValue.timeValueMillis(delayMillis); +} Suggestion importance[1-10]: 6 __ Why: Valid concern: with a large base.millis() and attempt near 30, base.millis() * (1L << 30) can overflow to a negative value, bypassing the day cap. Using Math.multiplyExact addresses this edge case, though in practice base is bounded by configuration.	Low
General	Unblock repository before deleting index The test blocks the cluster-manager on the index file and never explicitly unblocks the repository before asserting the index deletion succeeds. If a new cluster-manager is elected and inherits a mock repository that is still blocked, the delete may hang indefinitely. Consider calling unblockNode("test-repo", blockedNode) (or unblock on all nodes) before asserting the delete acknowledges, to prevent flaky test hangs. server/src/internalClusterTest/java/org/opensearch/snapshots/SnapshotCleanupRetryIT.java [79-81] logger.info("--> verify index deletion is unblocked"); + unblockAllDataNodes("test-repo"); assertAcked(client().admin().indices().prepareDelete("test-idx").get()); } Suggestion importance[1-10]: 6 __ Why: Valid concern to reduce test flakiness. Without unblocking the repository, the index deletion could hang if the new cluster-manager attempts repository operations. Adding an unblock call is a reasonable defensive measure for an integration test.	Low
	Fail listeners when v2 cleanup gives up When the feature flag is disabled, onFailure for the v2 cleanup task only logs a warning and does not perform any fallback, matching prior behavior. However, when the flag is enabled, if retryOrFailOnClusterManagerFailOver decides retry is not applicable (e.g., NotClusterManagerException), the fallback here only logs and doesn't invoke failAllListenersOnMasterFailOver. This may leave listeners waiting; ensure the fallback matches semantics of the pre-existing failover handling. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2533-2543] if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING)) { retryOrFailOnClusterManagerFailOver( e, attempt, source, () -> createStateWithoutSnapshotV2Task(source, attempt + 1), () -> { logger.error("Giving up on removing v2 snapshot state after {} attempts", attempt + 1); + failAllListenersOnMasterFailOver(e); } ); } Suggestion importance[1-10]: 5 __ Why: Reasonable point that the giveup path only logs and may leave listeners waiting, but it's unclear whether v2 cleanup has associated snapshot completion listeners that need to be failed here. The suggestion improves symmetry with pre-existing failover handling.	Low
	Wrap failure with SnapshotException for listeners When FailedToCommitClusterStateException is encountered during finalization, the code calls removeFailedSnapshotFromClusterState which itself will retry on FailedToCommitClusterStateException. However, listeners are never notified of the finalization failure in the success path of retry — the removal task calls failSnapshotCompletionListeners(snapshot, failure) with the passed failure argument, which is e. Verify that failSnapshotCompletionListeners is actually invoked; the current flow may leave callers of the create-snapshot API hanging if the removal task's clusterStateProcessed doesn't call it. Consider explicitly failing listeners here before delegating cleanup. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2326-2335] if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING) && ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) != null) { - removeFailedSnapshotFromClusterState(snapshot, e, repositoryData, null); + removeFailedSnapshotFromClusterState( + snapshot, + new SnapshotException(snapshot, "Failed to update cluster state during snapshot finalization", e), + repositoryData, + null + ); } else { failSnapshotCompletionListeners( snapshot, new SnapshotException(snapshot, "Failed to update cluster state during snapshot finalization", e) ); failAllListenersOnMasterFailOver(e); } Suggestion importance[1-10]: 3 __ Why: The suggestion mainly asks to verify listener notification behavior and proposes wrapping the exception. The impact is marginal and speculative without confirming actual listener leaks; the downstream removeFailedSnapshotFromClusterState does invoke failSnapshotCompletionListeners.	Low

---

Previous suggestions

Suggestions up to commit de51453

Category	Suggestion	Impact
Possible issue	Avoid retry when no longer cluster-manager When the failure is a FailedToCommitClusterStateException wrapped inside a NotClusterManagerException (or vice versa), the current condition still routes to removeFailedSnapshotFromClusterState, which will likely fail again because this node is no longer cluster-manager. Check for NotClusterManagerException first and only retry via cleanup when the sole cause is a commit failure. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2326-2328] private void handleFinalizationFailure(Exception e, SnapshotsInProgress.Entry entry, RepositoryData repositoryData) { Snapshot snapshot = entry.snapshot(); if (ExceptionsHelper.unwrap(e, NotClusterManagerException.class, FailedToCommitClusterStateException.class) != null) { logger.debug(() -> new ParameterizedMessage("[{}] failed to update cluster state during snapshot finalization", snapshot), e); if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING) + && ExceptionsHelper.unwrap(e, NotClusterManagerException.class) == null && ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) != null) { removeFailedSnapshotFromClusterState(snapshot, e, repositoryData, null); Suggestion importance[1-10]: 7 __ Why: Valid concern: if the exception is a NotClusterManagerException (potentially wrapping FailedToCommitClusterStateException), attempting removeFailedSnapshotFromClusterState may fail again. Ordering the check to prioritize NotClusterManagerException improves correctness.	Medium
	Prevent overflow before clamping backoff The multiplication base.millis() * (1L << 30) can still overflow to a negative value when base.millis() is large (e.g., a few seconds), because Math.min is evaluated after the multiplication. The Math.max(0L, ...) masks the overflow rather than preventing it, potentially yielding an incorrect clamp. Perform the min on the shift result using division or use Math.multiplyHigh/explicit overflow check before comparing to the day cap. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [3378-3381] static TimeValue computeBackoff(TimeValue base, int attempt) { - final long delayMillis = Math.max(0L, Math.min(base.millis() * (1L << Math.min(attempt, 30)), TimeValue.timeValueDays(1).millis())); - return TimeValue.timeValueMillis(delayMillis); + final long cap = TimeValue.timeValueDays(1).millis(); + final long baseMillis = Math.max(0L, base.millis()); + final int shift = Math.min(attempt, 30); + final long shifted = (baseMillis == 0L || shift > 62 || baseMillis > (Long.MAX_VALUE >> shift)) + ? cap + : baseMillis << shift; + return TimeValue.timeValueMillis(Math.min(shifted, cap)); } Suggestion importance[1-10]: 6 __ Why: Legitimate overflow risk: base.millis() * (1L << 30) can overflow to negative for larger base values, and Math.max(0L, ...) doesn't fully prevent incorrect clamping. Fix is reasonable though attempt is capped at 30 which limits practical impact.	Low
General	Include exception in fallback error log The retryOrFailOnClusterManagerFailOver helper asserts (fails in tests) on unexpected exceptions and runs the fallback. Here the fallback only logs, so unexpected exceptions in production will be swallowed silently after the assert is disabled. Additionally, when the feature flag is disabled, no logging beyond the initial warn happens — consider ensuring at minimum that unexpected exception paths are surfaced. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2528-2544] @Override public void onFailure(String src, Exception e) { logger.warn( () -> new ParameterizedMessage("failed to remove in progress snapshot v2 state after cluster manager switch {}", e), e ); if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING)) { retryOrFailOnClusterManagerFailOver( e, attempt, source, () -> createStateWithoutSnapshotV2Task(source, attempt + 1), - () -> { - logger.error("Giving up on removing v2 snapshot state after {} attempts", attempt + 1); - } + () -> logger.error( + () -> new ParameterizedMessage("Giving up on removing v2 snapshot state after {} attempts", attempt + 1), + e + ) ); } } Suggestion importance[1-10]: 3 __ Why: Minor logging improvement to include the exception in the fallback error message; helpful for debugging but not a functional issue.	Low

Suggestions up to commit 4cc3b75

Category	Suggestion	Impact
Possible issue	Prevent long overflow in backoff computation base.millis() * (1L << Math.min(attempt, 30)) can still overflow to a negative value for a large base (e.g., base > ~8.5 seconds with attempt=30 exceeds Long.MAX_VALUE). Since Math.min compares signed longs, overflow would make the result negative and defeat the cap. Compute the shift first and use Math.multiplyExact or check for overflow before applying the day cap. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [3379-3380] -final long delayMillis = Math.max(0L, Math.min(base.millis() * (1L << Math.min(attempt, 30)), TimeValue.timeValueDays(1).millis())); +final int shift = Math.min(attempt, 30); +long product; +try { + product = Math.multiplyExact(base.millis(), 1L << shift); +} catch (ArithmeticException ex) { + product = TimeValue.timeValueDays(1).millis(); +} +final long delayMillis = Math.max(0L, Math.min(product, TimeValue.timeValueDays(1).millis())); return TimeValue.timeValueMillis(delayMillis); Suggestion importance[1-10]: 7 __ Why: Correct observation: base.millis() * (1L << 30) can overflow to a negative value for larger base values, and Math.min on a negative long would return the negative product, breaking the intended cap. Using Math.multiplyExact addresses this edge case.	Medium
	Ensure listeners get terminal notification on retry path When handleFinalizationFailure is invoked because the finalize step failed to commit cluster state, calling removeFailedSnapshotFromClusterState immediately will submit another cluster state update — but the snapshot completion listeners are never notified of the original finalization failure. The subsequent success path in removeFailedSnapshotFromClusterState.clusterStateProcessed calls failSnapshotCompletionListeners(snapshot, failure) with the passed-in failure, so this works only if e is propagated as the failure; verify listeners actually receive a terminal notification and that runNextQueuedOperation is invoked to prevent stuck queued snapshots. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2326-2335] if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING) && ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) != null) { - removeFailedSnapshotFromClusterState(snapshot, e, repositoryData, null); + removeFailedSnapshotFromClusterState( + snapshot, + new SnapshotException(snapshot, "Failed to update cluster state during snapshot finalization", e), + repositoryData, + null + ); } else { failSnapshotCompletionListeners( snapshot, new SnapshotException(snapshot, "Failed to update cluster state during snapshot finalization", e) ); failAllListenersOnMasterFailOver(e); } Suggestion importance[1-10]: 6 __ Why: The suggestion raises a valid concern about listener notification semantics on the retry path, wrapping the exception in a SnapshotException for clarity. However, removeFailedSnapshotFromClusterState's clusterStateProcessed does call failSnapshotCompletionListeners, so the impact is moderate and mainly cosmetic/clarifying.	Low
General	Clarify fallback behavior for exhausted v2 retries When the feature flag is disabled, onFailure here becomes a no-op after the warn log — this changes prior behavior which only logged, so this is fine, but when the flag is enabled and retries are exhausted, the fallback merely logs and does nothing to clear stranded V2 snapshot markers. Consider whether the fallback should still attempt some cleanup or explicitly document this trade-off. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2533-2543] if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING)) { retryOrFailOnClusterManagerFailOver( e, attempt, source, () -> createStateWithoutSnapshotV2Task(source, attempt + 1), - () -> { - logger.error("Giving up on removing v2 snapshot state after {} attempts", attempt + 1); - } + () -> logger.error( + "Giving up on removing v2 snapshot state after {} attempts; markers may remain until next failover", + attempt + 1 + ) ); } Suggestion importance[1-10]: 2 __ Why: The suggestion only proposes a minor log message clarification and raises a general observation without providing a concrete fix for the potential stranded markers issue.	Low

Suggestions up to commit 70de2e4

Category	Suggestion	Impact
Possible issue	Guard backoff computation against overflow The multiplication base.millis() * (1L << 30) can still overflow to a negative value for large base (e.g. any base > ~8 seconds combined with attempt near 30 overflows a signed long). The Math.min then picks the negative overflow value rather than the daily cap. Compute the shift first and detect overflow explicitly, or use Math.multiplyExact with a try/catch to clamp on overflow. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [3378-3381] static TimeValue computeBackoff(TimeValue base, int attempt) { - final long delayMillis = Math.max(0L, Math.min(base.millis() * (1L << Math.min(attempt, 30)), TimeValue.timeValueDays(1).millis())); + final long cap = TimeValue.timeValueDays(1).millis(); + final int shift = Math.min(attempt, 30); + long delayMillis; + try { + delayMillis = Math.multiplyExact(base.millis(), 1L << shift); + } catch (ArithmeticException ex) { + delayMillis = cap; + } + delayMillis = Math.max(0L, Math.min(delayMillis, cap)); return TimeValue.timeValueMillis(delayMillis); } Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies a real overflow bug: base.millis() * (1L << 30) can overflow to negative for base values > ~8 seconds, and Math.min would then pick the negative value. Using Math.multiplyExact properly clamps to the cap.	Medium
	Avoid recursive retry on publish failure When finalization fails with a FailedToCommitClusterStateException, calling removeFailedSnapshotFromClusterState submits another cluster-state update which will itself likely fail with the same publish error, leading to recursive retry cycles. Consider guarding this path so the retry logic in removeFailedSnapshotFromClusterState is the sole retry point, or ensure the entry-point is only used when the node is still confirmed cluster-manager to avoid duplicated retry/backoff sequences. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2326-2335] if (ExceptionsHelper.unwrap(e, NotClusterManagerException.class, FailedToCommitClusterStateException.class) != null) { logger.debug(() -> new ParameterizedMessage("[{}] failed to update cluster state during snapshot finalization", snapshot), e); if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING) - && ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) != null) { + && ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) != null + && ExceptionsHelper.unwrap(e, NotClusterManagerException.class) == null) { removeFailedSnapshotFromClusterState(snapshot, e, repositoryData, null); } else { failSnapshotCompletionListeners( snapshot, new SnapshotException(snapshot, "Failed to update cluster state during snapshot finalization", e) ); failAllListenersOnMasterFailOver(e); } } else { Suggestion importance[1-10]: 5 __ Why: The suggestion raises a valid concern about potentially recursive retry paths, but the improved code adds a check that is somewhat redundant since FailedToCommitClusterStateException and NotClusterManagerException are typically mutually exclusive outcomes. Moderate value.	Low
General	Skip retry if no longer cluster-manager Scheduled retries will keep re-submitting even if this node has since lost cluster-manager role, causing repeated NotClusterManagerException failures until retries exhaust. Before submitting the retry task, verify the local node is still the cluster-manager (e.g. clusterService.state().nodes().isLocalNodeElectedClusterManager()) and invoke failoverFallback immediately otherwise to shorten recovery time. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [3358-3371] if (attempt >= maxRetries) { logger.warn("Exhausted {} retries for [{}], falling back to failover handling", maxRetries, source); failoverFallback.run(); return; } final int nextAttempt = attempt + 1; final TimeValue delay = computeBackoff(retryBackoff, attempt); logger.info("Publish failed for [{}] (attempt {}), scheduling retry in [{}]", source, nextAttempt, delay); try { - threadPool.schedule(() -> clusterService.submitStateUpdateTask(source, taskFactory.get()), delay, ThreadPool.Names.GENERIC); + threadPool.schedule(() -> { + if (clusterService.state().nodes().isLocalNodeElectedClusterManager() == false) { + failoverFallback.run(); + return; + } + clusterService.submitStateUpdateTask(source, taskFactory.get()); + }, delay, ThreadPool.Names.GENERIC); } catch (OpenSearchRejectedExecutionException ex) { Suggestion importance[1-10]: 5 __ Why: Adding a pre-check for cluster-manager status before submitting the retry could shorten recovery time in failover scenarios, but the existing logic handles this via NotClusterManagerException fallback anyway, so this is a minor optimization.	Low
	Handle failure when feature flag disabled When the feature flag is disabled the previous comment noted that execute never fails so onFailure should not fire; however, with dynamic-node cluster-manager loss onFailure can indeed be invoked (e.g. NotClusterManagerException). The current code silently swallows the failure in that case. Consider always logging at warn (already done) but also ensuring behavior is not silently degraded when the flag is off — either always attempt cleanup or explicitly document the no-op. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2528-2544] @Override public void onFailure(String src, Exception e) { logger.warn( () -> new ParameterizedMessage("failed to remove in progress snapshot v2 state after cluster manager switch {}", e), e ); if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING)) { retryOrFailOnClusterManagerFailOver( e, attempt, source, () -> createStateWithoutSnapshotV2Task(source, attempt + 1), - () -> { - logger.error("Giving up on removing v2 snapshot state after {} attempts", attempt + 1); - } + () -> logger.error("Giving up on removing v2 snapshot state after {} attempts", attempt + 1) ); + } else { + logger.error("failed to remove in progress snapshot v2 state after cluster manager switch (retries disabled)", e); } } Suggestion importance[1-10]: 3 __ Why: Adding an else branch to log at error level when the feature flag is disabled is a minor observability improvement, not a critical fix.	Low

Suggestions up to commit 503a6b3

Category	Suggestion	Impact
Possible issue	Prevent overflow in backoff computation The expression base.millis() * (1L << Math.min(attempt, 30)) can still overflow long if base.millis() is large (e.g., minutes/hours) because the multiplication happens before the Math.min clamp. Compute the shifted multiplier first and use Math.multiplyExact or check for overflow before clamping, otherwise a negative delayMillis is possible before the Math.max(0L, ...) guard—which is fine, but the clamp to 1 day is bypassed on overflow, producing zero delay. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [3378-3381] static TimeValue computeBackoff(TimeValue base, int attempt) { - final long delayMillis = Math.max(0L, Math.min(base.millis() * (1L << Math.min(attempt, 30)), TimeValue.timeValueDays(1).millis())); + final long maxMillis = TimeValue.timeValueDays(1).millis(); + final int shift = Math.min(attempt, 30); + long delayMillis; + try { + delayMillis = Math.multiplyExact(base.millis(), 1L << shift); + } catch (ArithmeticException e) { + delayMillis = maxMillis; + } + delayMillis = Math.max(0L, Math.min(delayMillis, maxMillis)); return TimeValue.timeValueMillis(delayMillis); } Suggestion importance[1-10]: 6 __ Why: Valid concern: with a large base.millis() (e.g., hours), the multiplication before clamping could overflow, producing an incorrect zero or negative delay that bypasses the 1-day clamp. Using Math.multiplyExact provides safer overflow handling.	Low
General	Handle flag-disabled path explicitly When the feature flag is disabled, onFailure only logs and does nothing—leaving the v2 snapshot marker stranded, which contradicts the PR's goal. Additionally, the pre-existing comment stated "execute never fails, so we should never hit this," but now that we're handling publish failures, this branch is reachable. Ensure the non-flag path still logs at warn level (currently does) and consider that the flag-disabled path should retain prior behavior explicitly. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2528-2544] @Override public void onFailure(String src, Exception e) { logger.warn( () -> new ParameterizedMessage("failed to remove in progress snapshot v2 state after cluster manager switch {}", e), e ); if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING)) { retryOrFailOnClusterManagerFailOver( e, attempt, source, () -> createStateWithoutSnapshotV2Task(source, attempt + 1), - () -> { - logger.error("Giving up on removing v2 snapshot state after {} attempts", attempt + 1); - } + () -> logger.error("Giving up on removing v2 snapshot state after {} attempts", attempt + 1) ); + } else { + logger.warn("Feature flag disabled; v2 snapshot cleanup will rely on next cluster-manager to retry"); } } Suggestion importance[1-10]: 4 __ Why: The suggestion mainly adds an extra log statement for the flag-disabled path. The prior behavior is retained (just log warn), so the impact is minor and mostly cosmetic.	Low
	Clarify branch selection for failure type When the feature flag is enabled but the failure is a NotClusterManagerException (not a commit failure), the code falls through to the else branch and calls failAllListenersOnMasterFailOver, which is correct. However, verify the condition ordering: currently only FailedToCommitClusterStateException triggers the new retry-based cleanup path. Consider explicitly checking NotClusterManagerException first so the intent is clearer and to avoid future regressions if the outer unwrap list changes. server/src/main/java/org/opensearch/snapshots/SnapshotsService.java [2324-2335] if (ExceptionsHelper.unwrap(e, NotClusterManagerException.class, FailedToCommitClusterStateException.class) != null) { logger.debug(() -> new ParameterizedMessage("[{}] failed to update cluster state during snapshot finalization", snapshot), e); if (FeatureFlags.isEnabled(FeatureFlags.SNAPSHOT_RESILIENCE_SETTING) + && ExceptionsHelper.unwrap(e, NotClusterManagerException.class) == null && ExceptionsHelper.unwrap(e, FailedToCommitClusterStateException.class) != null) { removeFailedSnapshotFromClusterState(snapshot, e, repositoryData, null); } else { Suggestion importance[1-10]: 3 __ Why: The existing code already achieves the correct behavior since FailedToCommitClusterStateException and NotClusterManagerException are distinct types. The suggested change is a minor readability improvement with limited impact.	Low

Suggestions up to commit 5987e9d

[github-actions]

✅ Gradle check result for 99f35b1: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 52.56410% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.40%. Comparing base (03a4de3) to head (c3a19e9).

Files with missing lines	Patch %	Lines
...ava/org/opensearch/snapshots/SnapshotsService.java	52.56%	37 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #22518      +/-   ##
============================================
- Coverage     71.42%   71.40%   -0.03%     
  Complexity    76786    76786              
============================================
  Files          6148     6148              
  Lines        357980   358031      +51     
  Branches      52177    52183       +6     
============================================
- Hits         255689   255638      -51     
- Misses        81937    82094     +157     
+ Partials      20354    20299      -55     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shourya035]

Please add some integration tests to ensure that the feature works end to end with induced cluster manager failures

[github-actions]

Persistent review updated to latest commit 1cb6abd

[github-actions]

❌ Gradle check result for 1cb6abd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit a9a5f13

[github-actions]

✅ Gradle check result for a9a5f13: SUCCESS

[github-actions]

Persistent review updated to latest commit 104557c

[github-actions]

Persistent review updated to latest commit a0f69b9

[github-actions]

Persistent review updated to latest commit 5577c6f

[github-actions]

❌ Gradle check result for 5577c6f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[shourya035]

Please address the resolved comments. Looks like the comments were marked resolved but the code changes were not made

[github-actions]

❌ Gradle check result for b0edc93: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit 328edbe

[github-actions]

Persistent review updated to latest commit 74f33fb

[github-actions]

❌ Gradle check result for 74f33fb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit 91090a9

[github-actions]

❌ Gradle check result for 91090a9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit 5987e9d

[github-actions]

❌ Gradle check result for 5987e9d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit 503a6b3

[github-actions]

✅ Gradle check result for 503a6b3: SUCCESS

[github-actions]

Persistent review updated to latest commit 70de2e4

[github-actions]

Persistent review updated to latest commit 4cc3b75

[github-actions]

❌ Gradle check result for 4cc3b75: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit de51453

[github-actions]

❌ Gradle check result for de51453: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit c3a19e9

[github-actions]

✅ Gradle check result for c3a19e9: SUCCESS