Skip to content

perf(envelope): jittered backoff between OCC retries; bounded Retry-After (#72) - #86

Merged
JumpTechCode merged 2 commits into
mainfrom
perf/occ-backoff-jitter
Jun 22, 2026
Merged

perf(envelope): jittered backoff between OCC retries; bounded Retry-After (#72)#86
JumpTechCode merged 2 commits into
mainfrom
perf/occ-backoff-jitter

Conversation

@JumpTechCode

Copy link
Copy Markdown
Collaborator

Closes #72.

Problem

execute() retried OccConflict up to MAX_OCC_RETRIES (5) with zero delay between attempts, and the RetryExhausted 503 advertised Retry-After: 0. Server-broken deadlocks (40P01) from the pick/cancel-vs-reaper ABBA lock ordering (ADR-0019) are translated to OccConflict (ADR-0020) and absorbed by this same loop. With no spacing, a loser re-ran instantly against the same hot rows — re-forming the deadlock or re-losing the same CAS — and could burn all five attempts in microseconds, returning 503 where a few ms of spacing would have let it commit. Retry-After: 0 compounded it by inviting the herd to rejoin on the same beat.

Correctness was never at risk (CAS/guards arbitrate), but the load harness — built to exercise exactly this contention — would read an inflated 503 rate.

Fix

  • Full-jitter exponential backoff between OCC retries: rand() * min(cap, base · 2^attempt) with base 10ms, cap 200ms. Injected via sleep/rand seams so the unit suite stays deterministic. No pause after the final attempt; the idempotency-claim (EXISTS) replay branch stays backoff-free.
  • Bounded, non-zero Retry-After (1–3s, jittered) on the RetryExhausted 503 instead of 0.

The 503 status itself (ADR-0020) is unchanged; only the spacing and the header value changed. execute gains keyword-only sleep/rand seams (defaulted), so the 13 call sites are untouched.

Scope / deliberate non-goal

The ABBA lock-order inversion (ADR-0019) is left in place: backoff mitigates the deadlock-driven 503s rather than resolving them at source. Aligning the reaper and pick/cancel lock orders is a larger, separately-reasoned change. Recorded as the accepted V1 posture in ADR-0028.

Tests

  • test_envelope.py: pure _occ_backoff_delay (full-jitter exponential, capped, scales with jitter); execute backs off between each attempt pair with the right schedule and not after the last; the replay branch never backs off; existing retry tests inject a no-op sleep to stay instant.
  • test_api_errors.py: RetryExhausted 503 advertises a bounded non-zero Retry-After.
  • test_api_allocate.py: updated the exhaustion test from == "0" to 1 ≤ Retry-After ≤ 3.

make verify green locally: 511 passed, 99.15% coverage.

🤖 Generated with Claude Code

JumpTechCode and others added 2 commits June 21, 2026 20:29
…fter (#72)

execute() retried OccConflict up to MAX_OCC_RETRIES with zero delay and
the RetryExhausted 503 advertised Retry-After: 0. Server-broken deadlocks
(40P01) from the pick/cancel-vs-reaper ABBA lock ordering (ADR-0019) are
translated to OccConflict and absorbed by this same loop, so a loser
re-ran instantly against the same hot rows -- re-forming the deadlock or
re-losing the CAS -- and could burn all five attempts in microseconds,
returning 503 where a few ms of spacing would have let it commit.
Retry-After: 0 then invited the herd to rejoin on the same beat.

Add full-jitter exponential backoff (base 10ms, cap 200ms) between OCC
retries, injected via sleep/rand seams so the unit suite stays
deterministic; no pause after the final attempt and the idempotency-claim
(EXISTS) replay branch stays backoff-free. Advertise a bounded, non-zero
Retry-After (1-3s, jittered) on the 503 instead of 0.

Correctness is unaffected -- the CAS and guards still arbitrate, and the
503 terminal (ADR-0020) is unchanged; only the spacing and the
Retry-After value changed. The ABBA lock-order inversion is deliberately
left in place; backoff mitigates it rather than resolving it at source.
Documented in ADR-0028.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@JumpTechCode
JumpTechCode merged commit dfdc697 into main Jun 22, 2026
7 checks passed
@JumpTechCode
JumpTechCode deleted the perf/occ-backoff-jitter branch June 22, 2026 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OCC retry loop has no jittered backoff and advertises Retry-After: 0, raising 503 rate under contention

1 participant