perf(envelope): jittered backoff between OCC retries; bounded Retry-After (#72) - #86
Merged
Conversation
…fter (#72) execute() retried OccConflict up to MAX_OCC_RETRIES with zero delay and the RetryExhausted 503 advertised Retry-After: 0. Server-broken deadlocks (40P01) from the pick/cancel-vs-reaper ABBA lock ordering (ADR-0019) are translated to OccConflict and absorbed by this same loop, so a loser re-ran instantly against the same hot rows -- re-forming the deadlock or re-losing the CAS -- and could burn all five attempts in microseconds, returning 503 where a few ms of spacing would have let it commit. Retry-After: 0 then invited the herd to rejoin on the same beat. Add full-jitter exponential backoff (base 10ms, cap 200ms) between OCC retries, injected via sleep/rand seams so the unit suite stays deterministic; no pause after the final attempt and the idempotency-claim (EXISTS) replay branch stays backoff-free. Advertise a bounded, non-zero Retry-After (1-3s, jittered) on the 503 instead of 0. Correctness is unaffected -- the CAS and guards still arbitrate, and the 503 terminal (ADR-0020) is unchanged; only the spacing and the Retry-After value changed. The ABBA lock-order inversion is deliberately left in place; backoff mitigates it rather than resolving it at source. Documented in ADR-0028. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # docs/adr/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #72.
Problem
execute()retriedOccConflictup toMAX_OCC_RETRIES(5) with zero delay between attempts, and theRetryExhausted503 advertisedRetry-After: 0. Server-broken deadlocks (40P01) from the pick/cancel-vs-reaper ABBA lock ordering (ADR-0019) are translated toOccConflict(ADR-0020) and absorbed by this same loop. With no spacing, a loser re-ran instantly against the same hot rows — re-forming the deadlock or re-losing the same CAS — and could burn all five attempts in microseconds, returning 503 where a few ms of spacing would have let it commit.Retry-After: 0compounded it by inviting the herd to rejoin on the same beat.Correctness was never at risk (CAS/guards arbitrate), but the load harness — built to exercise exactly this contention — would read an inflated 503 rate.
Fix
rand() * min(cap, base · 2^attempt)withbase10ms,cap200ms. Injected viasleep/randseams so the unit suite stays deterministic. No pause after the final attempt; the idempotency-claim (EXISTS) replay branch stays backoff-free.Retry-After(1–3s, jittered) on theRetryExhausted503 instead of0.The 503 status itself (ADR-0020) is unchanged; only the spacing and the header value changed.
executegains keyword-onlysleep/randseams (defaulted), so the 13 call sites are untouched.Scope / deliberate non-goal
The ABBA lock-order inversion (ADR-0019) is left in place: backoff mitigates the deadlock-driven 503s rather than resolving them at source. Aligning the reaper and pick/cancel lock orders is a larger, separately-reasoned change. Recorded as the accepted V1 posture in ADR-0028.
Tests
test_envelope.py: pure_occ_backoff_delay(full-jitter exponential, capped, scales with jitter);executebacks off between each attempt pair with the right schedule and not after the last; the replay branch never backs off; existing retry tests inject a no-op sleep to stay instant.test_api_errors.py:RetryExhausted503 advertises a bounded non-zeroRetry-After.test_api_allocate.py: updated the exhaustion test from== "0"to1 ≤ Retry-After ≤ 3.make verifygreen locally: 511 passed, 99.15% coverage.🤖 Generated with Claude Code