Skip to content

Commit 444c221

Browse files
authored
fix(run-engine): stop requeued runs with a lapsed ttl being orphaned in the queue (#4669)
## Summary A run triggered with a `ttl` could get permanently stuck showing as queued. If the run started executing and was then requeued after a failure (a stalled heartbeat, a worker dying mid-run) once its TTL had already elapsed, the next dequeue pass silently dropped it from every queue structure. The run stayed QUEUED in the database forever, and nothing (dequeue, the TTL consumer, queue repair) could ever see it again. ## Root cause Enqueue registers a TTL entry for the TTL consumer, and the first dequeue removes it ("the run is executing, not expired"). A nack rewrote the message preserving the original `ttlExpiresAt` without re-registering that entry. The next dequeue pass then took the expired-TTL branch: remove the run from the queue sorted sets and leave the message for the TTL consumer to finalize. But the consumer's entry was gone, so nothing ever finalized the run. The fix has two halves: - `nackMessage` drops `ttlExpiresAt` from the rewritten message. TTL only applies to runs that have never been dequeued (the same contract as `includeTtl` on re-enqueues), so a requeued run stays dequeuable and is never expired by its original deadline. - The dequeue expired-TTL branches now (re-)register the TTL entry instead of assuming it exists, so any message still carrying a lapsed `ttlExpiresAt` with no TTL entry (including ones written before this fix) finalizes as EXPIRED instead of orphaning. ## Verification New engine test suite `ttlNackRequeue.test.ts` (testcontainers, real Redis and Postgres). All four tests fail before the fix and pass after: - a heartbeat-stalled EXECUTING run with a lapsed TTL is requeued and dequeued again instead of orphaned (the full production failure chain) - requeue-after-failure strips `ttlExpiresAt` so later dequeues do not treat the run as expired - a lapsed-TTL message whose TTL entry is missing is re-registered by dequeue and finalized as EXPIRED, for both plain and concurrency-key queues Also ran the existing ttl, heartbeats, dequeuing and attemptFailures engine suites plus the full run-queue suite (149 tests) against the change.
1 parent 2496a8a commit 444c221

3 files changed

Lines changed: 559 additions & 3 deletions

File tree

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: fix
4+
---
5+
6+
Runs triggered with a `ttl` could get permanently stuck in the queued state if they started executing and were then requeued after a failure (for example a worker dying mid-run) once the TTL had already elapsed. Requeued runs now dequeue normally: a run's TTL only applies while it is waiting to start for the first time.

0 commit comments

Comments
 (0)