Commit faab517
committed
fix(run-engine): drop drained ck variants from the fair order on ack, TTL expiry and dead-letter
Only the vtime dequeue removed a variant from :ckVtime. Ack, TTL expiry and the
dead-letter path all ZREM a drained variant from ckIndex and left its ckVtime
entry behind, with a tag that had stopped advancing, until some later scan
happened to visit and collect it. Ack is much the most common of the three: it is
what a cancellation of a still-queued run runs through.
The limitations note called this bounded and self-healing, and it is, but on two
weaker grounds than it claimed. The 24h state TTL is refreshed by every enqueue,
nack and serving dequeue, so on a queue that is never quiet for a full day it
never fires, leaving floor advance as the only collection route. And that route
can be held still by a workload that keeps minting concurrency keys, because each
fresh variant registers at the floor and its first serve records the floor as the
minimum servable tag. It is also not fairness-neutral: registration is NX, so a
REUSED key inherits the stale high tag and is deprioritised, which means a
cancel-drain remembers history that a serve-drain forgets.
Adds vtime variants of the three commands rather than editing them, so the
flag-off scripts stay byte-identical by construction (this is a pure addition to
the file). Ack and dead-letter take the key as one more KEYS slot from the call
site; the TTL sweep derives it in Lua, as it already derives ckIndexKey, because
it discovers the queues it touches inside the script.
Tests cover all three paths and fail without the fix. The old stranded-entry test
is kept rather than deleted, retargeted at the pre-fix command directly, since
GC-on-scan is still load-bearing for an older instance during a rolling deploy.
Reported by Devin on #4367.1 parent 44ce4ef commit faab517
2 files changed
Lines changed: 579 additions & 39 deletions
0 commit comments