`ExponentialRetry` + `Cron`/`Perpetual`: exhausted retries cause endless rescheduling

[amenck]

Bug

When a Cron or Perpetual task uses ExponentialRetry with a finite attempts count and the task fails persistently, the retries exhaust as expected — but then the task is rescheduled for a fresh cycle with attempt reset to 1. The entire retry sequence repeats every cycle, forever. There is no built-in way to say "retries are exhausted, stop scheduling this task."

A secondary issue makes subclassing ExponentialRetry to work around this harder than it should be: ExponentialRetry.__aenter__ constructs ExponentialRetry(...) rather than type(self)(...), so subclass method overrides are silently dropped at runtime unless __aenter__ is also overridden.

Repro

import asyncio
from datetime import timedelta

from docket import Docket
from docket.dependencies import ExponentialRetry, Perpetual
from docket.worker import Worker

fail_count = 0


async def main() -> None:
    global fail_count

    async with Docket(name="repro", url="memory://") as docket:

        async def always_failing(
            _perpetual: Perpetual = Perpetual(every=timedelta(0)),
            _retry: ExponentialRetry = ExponentialRetry(
                attempts=3,
                minimum_delay=timedelta(0),
                maximum_delay=timedelta(0),
            ),
        ) -> None:
            global fail_count
            fail_count += 1
            raise ValueError(f"attempt {fail_count}")

        docket.register(always_failing, names=["always-failing"])
        await docket.add(always_failing)()

        async with Worker(docket, schedule_automatic_tasks=False) as worker:
            try:
                await asyncio.wait_for(worker.run_until_finished(), timeout=2)
            except asyncio.TimeoutError:
                pass

    # Expected: 3 (one per attempt)
    # Actual: hundreds
    print(f"Total executions: {fail_count}")
    assert fail_count == 3, f"Expected 3, got {fail_count}"


asyncio.run(main())

Environment

• pydocket 0.21.1 (behavior unchanged on 0.22.0)
• Python 3.11

Happy to contribute a fix here, if we all agree that this is not the intended behavior.

[chrisguidry]

Thanks for the report @amenck, but I intended retries to operate on a smaller scale than Perpetual. Let's say you have a task that runs every hour, but you want to make it resilient to brief hiccups. That's when you'd use Perpetual and Retry together, so that the Perpetual is ensuring that it's always happening hourly, and the Retry gives it a certain number of attempts within that. The demo you've shown here is kind of smooshing those timescales together in a way that wouldn't be that realistic. It's definitely the case that each instance of a Perpetual should be considered a first attempt.

Definitely would love the fix you mentioned on ExponentialRetry, though, and perhaps if you do that you can try out a subclass of either Retry, ExponentialRetry, or Perpetual that worked the way you were thinking, where the dependency keeps track of a total number of attempts over all scheduling?

[Mr-Squirrel]

This is becoming a real pain for me as well. docket will not stop spamming

Getting redeliveries
Getting new deliveries
Scheduling due tasks

Over and over and over and over when just starting up a vanilla FastMCP with background tasks enabled

[chrisguidry]

@Mr-Squirrel Those aren't related to this issue, those are the debug level logs from docket. Do you have your log level set to DEBUG?

[Mr-Squirrel]

Yes I have log level debug. debugging the issue where my application never boots due to docket stuck in an infinite loop. What I figured out so far is that docket fails to clean up on interrupt/terminate leaving zombie processes. When the server tries to boot up again it goes into this infinite loop. Once I kill att other processes its works again. Notably the logs above stop looping