Add highly available background task delivery

[zzstoatzz]

closes #21218

this PR replaces process-local deferred-task delivery with Docket-backed delivery so TaskWorkers can receive background tasks from any Prefect server replica.

Implementation

• Each deferred task state is represented by an idempotently keyed Docket execution.
• Prefect uses task-key-specific Redis queues in Docket's namespace for TaskWorker routing.
• Offer and claim transitions are atomic; a process death cannot leave a false-ready marker or remove an unclaimed delivery.
• Scheduled and retry deliveries have independent durable capacity reservations using the existing 1,000/100 defaults.
• Queues are FIFO within each class and retry work is selected before scheduled work.
• Claims have renewable visibility. Disconnects requeue immediately; abrupt process death is redelivered after visibility expires.
• A database reconciliation loop restores missing Docket executions for deferred SCHEDULED task runs. This covers Redis restoration, the database/Docket transaction boundary, pre-upgrade runs, and unknown-task drops during mixed-version rollout.
• Acceptance is idempotent and leaves a bounded tombstone so reconciliation cannot race a TaskWorker's transition out of SCHEDULED.

HA model

The Prefect database remains the durable source of truth for task-run state. Docket and shared Redis provide scheduling, leases, and cross-replica delivery. Any API replica can publish and any API replica serving a matching TaskWorker subscription can claim.

The implementation was exercised against real Redis by killing the API process holding an unacknowledged delivery. A second replica received the same task-run/state delivery and acknowledged it; a replica subscribed to another task key never received it.

Mixed-version workers that do not recognize deliver_task_run can drop an execution. Reconciliation recreates it while the database task run remains deferred and SCHEDULED, so the run becomes deliverable when an upgraded Docket worker is available.

Results

Docket results are not Prefect task results. Docket tracks completion of the delivery operation. PrefectDistributedFuture still observes the Prefect task run and loads its return value through the existing ResultStore; parameter and return-value persistence are unchanged.

Compatibility

The TaskWorker subscription protocol, Task.delay, Task.map(..., deferred=True), PrefectDistributedFuture, task-run REST/state schemas, and result storage are unchanged.

The existing queue-size settings and prefect.server.task_queue imports remain available. server.tasks.scheduling.delivery_visibility_timeout is additive. A separate review discussion can decide whether the compatibility-only process-local queue implementation should be deprecated or removed in a minor release.

Docket is raised to >=0.23.1. Self-hosted HA requires every server replica to share the configured Docket Redis backend, as described by the existing server HA documentation.

Performance

OSS end-to-end benchmark, PostgreSQL + Redis, one API server, one TaskWorker, 2,000 mapped no-op deferred tasks:

• 2,000 completed, 0 failed
• 33.05 submissions/s
• 32.97 end-to-end completions/s
• queue latency p50 47 ms / p95 60 ms / p99 81 ms
• sampled results at indices 0, 1,000, and 1,999 were correct

This is slower than the earlier equivalent main baseline (~39 tasks/s); the durable Redis reservation and atomic delivery work is visible overhead. The change prioritizes HA correctness, and the benchmark is included so that tradeoff is explicit.

Validation

• 1,865 settings tests passed (3 skipped)
• 557 affected orchestration, subscription, background-task, and delivery tests passed (1 skipped)
• 16 focused delivery tests pass, including FIFO, retry priority, bounded backpressure, idempotency, worker restart, disconnect redelivery, and visibility renewal
• real-Redis cross-process API failover passed
• 2,000-task OSS end-to-end run completed with correct stored results
• Ruff and Pyright pass for the delivery implementation

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
prefect-oss	🟢 Ready	View Preview	Jul 23, 2026, 7:52 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[codspeed-hq]

Merging this PR will not alter performance

✅ 8 untouched benchmarks
⏩ 1 skipped benchmark¹

---

_{Comparing codex/docket-background-task-ha (906b582) with main (d2d873b)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩