PR #649 makes the fetch rate ceiling global (one shared cursor in shared_cache, reserved atomically). It leaves the backoff multiplier and the clean-response streak per-process.
That is safe but incomplete:
- A per-process backoff reaches the shared cursor as an already-widened interval, so it can only ever slow the aggregate — it cannot overshoot the ceiling. No correctness bug today.
- But it is uncoordinated: a 429 seen by one django-q2 worker does not slow the others. The destination is telling the whole deployment to back off and only one process hears it.
- And decay is N times too fast:
_CLEAN_RESPONSES_BEFORE_DECAY is 100 consecutive clean responses, but N processes each count their own streak, so the deployment as a whole crosses that threshold N times over and recovers faster than the single agreed schedule intends. At the production dispatch cap that is roughly 2x; it scales with the cap.
The reasoning was worked out in the now-closed PR #648 and is correct — see its description. Its framing: "a shared next_allowed_at advanced by per-process intervals is incoherent".
What would close this
Move backoff_multiplier and the clean-response counter into the same shared row the pace cursor already occupies, preserving PR #644's asymmetric schedule unchanged (one 429/503 doubles; 100 consecutive clean halves; never past the configured ceiling). Storage moves; semantics do not.
Note the open question this reopens: #644's decay threshold of 100 consecutive clean responses was a judgement call, not a measurement — at 7/s a full recovery from x16 to x1 costs 400 clean responses, roughly a minute of sustained good behaviour. Making the streak global changes what that number means (it becomes deployment-wide rather than per-worker), so it is worth confirming the value at the same time rather than porting it blindly.
Related: #649 (the global ceiling), #644 (throttle-not-halt), #652 (the recurring per-process trap this is an instance of).
PR #649 makes the fetch rate ceiling global (one shared cursor in
shared_cache, reserved atomically). It leaves the backoff multiplier and the clean-response streak per-process.That is safe but incomplete:
_CLEAN_RESPONSES_BEFORE_DECAYis 100 consecutive clean responses, but N processes each count their own streak, so the deployment as a whole crosses that threshold N times over and recovers faster than the single agreed schedule intends. At the production dispatch cap that is roughly 2x; it scales with the cap.The reasoning was worked out in the now-closed PR #648 and is correct — see its description. Its framing: "a shared
next_allowed_atadvanced by per-process intervals is incoherent".What would close this
Move
backoff_multiplierand the clean-response counter into the same shared row the pace cursor already occupies, preserving PR #644's asymmetric schedule unchanged (one 429/503 doubles; 100 consecutive clean halves; never past the configured ceiling). Storage moves; semantics do not.Note the open question this reopens: #644's decay threshold of 100 consecutive clean responses was a judgement call, not a measurement — at 7/s a full recovery from x16 to x1 costs 400 clean responses, roughly a minute of sustained good behaviour. Making the streak global changes what that number means (it becomes deployment-wide rather than per-worker), so it is worth confirming the value at the same time rather than porting it blindly.
Related: #649 (the global ceiling), #644 (throttle-not-halt), #652 (the recurring per-process trap this is an instance of).