Skip to content

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

retry-after-clock

Turn a Retry-After header into a wait that is still correct on a client whose clock is wrong, and that does not land in the same millisecond as everybody else's.

import { planRetry, waitFor } from 'retry-after-clock';

const response = await fetch(url);
if (response.status === 429 || response.status === 503) {
  const plan = planRetry(response.headers, {
    attempt,
    jitter: { key: instanceId },   // stable position in the window, per client
  });

  if (plan === null) return myOwnBackoff(attempt);   // server gave no instruction

  await waitFor(plan, { signal });
  return retry();
}

plan carries the numbers the decision was made from:

{
  waitMs: 33_912,        // what to actually wait, from the moment the response arrived
  floorMs: 30_000,       // the server instructed minimum, never retry before this
  jitterMs: 3_912,       // added on top, never subtracted
  spreadMs: 7_500,       // width of the window the jitter was drawn from
  source: 'http-date',
  instructedMs: 30_000,  // before the age of the response was taken off
  ageMs: 0,
  skewMs: 41_204,        // measured, reported, and used by nothing
  clampedToZero: false,
  deadline: { monotonicAt: 918_244.3, /* ... */ }
}

The obvious version computes the wait in the wrong frame

Retry-After has two forms. The integer form is fine:

Retry-After: 120

The other form is an absolute instant, and this is where the bug lives:

Retry-After: Tue, 04 Aug 2026 12:00:30 GMT

The four line implementation is Date.parse(header) - Date.now(), floored at zero. That subtracts a client instant from a server instant. The difference between those two clocks is not zero, and it is not small: a survey of any large fleet finds machines minutes out, VMs that resumed from a snapshot hours out, and containers whose NTP client never came up at all.

So on a client running forty seconds fast, a thirty second backoff computes as -10000, floors to zero, and the client returns immediately into the limiter that just told it to stop. On a client running forty seconds slow the same header produces a seventy second wait, and the request that eventually goes out looks to the caller like a timeout.

Neither client has any way to notice. The header is well formed, Date.parse succeeds, the number is plausible, and there is no error anywhere. It also passes its own tests, because in a test the client clock and the server clock are the same clock.

What this does instead

Responses carry the server's own reading of the current time, in the Date header. Both instants are then from the same clock:

floor = parse(Retry-After) - parse(Date)

The client clock does not appear in that expression, so no amount of skew changes the answer. A test with a client forty seconds fast, forty seconds slow, and a full day out in either direction produces the same 30000 ms floor from the same headers.

The skew is still measured, and reported as skewMs so it can be logged or alerted on. Nothing in the computed wait reads it. That is the point: the number is available for observability and structurally cannot influence the result.

When Retry-After is an HTTP date and the response has no Date header, there is no server frame to work in, and the module throws rather than quietly substituting the client clock:

Retry-After was sent as an HTTP-date ("Tue, 04 Aug 2026 12:00:30 GMT") but the response
carried no Date header, so there is no record of what the server thought the time was.
Subtracting the client clock instead makes the wait wrong by the full amount of any skew:
a client running forty seconds fast computes a wait of zero and retries straight back into
the limiter. Ask the origin to send a Date header, which RFC 9110 requires on responses
like this one, or pass onMissingDate: "trust-client-clock" to accept the error explicitly.

The escape hatch exists because some origins really do omit Date, and an exception is not always the useful answer. It has to be asked for by name.

Waiting is a second clock problem

Removing skew from the arithmetic is not enough if the wait itself is held as a wall clock instant. Date.now() + 30000 is a deadline that an NTP correction landing fifteen seconds later can move by a minute in either direction, and a laptop resuming from sleep can move by hours.

Deadlines here live on the monotonic clock (performance.now()), which counts elapsed time and never steps. plan.deadline.monotonicAt is the real deadline. estimatedWallAt exists so a log line can say roughly when the retry is due, and is labelled an estimate because comparing against it would put the wall clock back in the critical path.

waitFor re-reads the monotonic clock in a loop rather than trusting one timer. Timers under-deliver in ways this module does not control: a background browser tab has them throttled, a busy event loop fires them late, setTimeout silently overflows past 2^31 milliseconds and fires immediately, and some runtimes round a delay down and fire a millisecond early. Re-arming against the deadline turns every one of those into an extra iteration instead of an early retry.

The response was not new when it arrived

Retry-After is relative to the moment the response was generated, and the client sees it later. A response that spent twenty seconds in a shared cache and two hundred milliseconds on the wire has already consumed part of its own interval.

RFC 9111 defines an apparent_age for exactly this, as the client's receive time minus the server's Date. That formula is unusable here for the reason the whole module exists: it subtracts a server instant from a client instant, so on a skewed client the age is wrong by the skew, and a client forty seconds fast would compute a forty second age on a response that was never cached at all.

What is used instead is skew free by construction:

age = Age header + (responseReceivedAt - requestSentAt)

The Age header is a duration the caches computed among themselves, and the round trip is measured with two readings of the client's own monotonic clock, so the client's offset from real time cancels. Pass requestSentAt and responseReceivedAt to get the second term. Passing only one of them throws, because a half measured round trip produces a wait that is too long by an unknown amount.

Honouring the value exactly is the other half of the bug

Five thousand clients get a 429 with Retry-After: 30. An implementation that waits exactly thirty seconds sends all five thousand back in the same millisecond, and scores itself as perfectly compliant while rebuilding the spike the header was sent to prevent. The next response is another 429 with another Retry-After: 30, and the herd stays locked in step indefinitely.

The usual fix is full jitter: pick uniformly between zero and the backoff. That is correct for a backoff a client chose for itself and wrong for one a server handed down. Retry-After is a floor, not a target, and half the population arriving early is the same stampede, only sooner.

So the window opens at the floor and extends upward:

wait = floor + random(0, spread)
spread = clamp(floor * 0.25, 250 ms, 30_000 ms)

wait >= floor is an invariant, asserted across the range of draws, and the random source is validated before it is used. A source returning NaN throws rather than producing a NaN deadline that resolves to an immediate retry somewhere far away from here.

Random dispersion has a weakness across rounds: every client redraws, so clients that collided once can collide again, and a client that keeps failing keeps rerolling. Passing jitter.key derives the position in the window from the client's identity and the attempt number instead. The population stays evenly spread on every round, a given client holds a stable position within a round, and two clients whose keys happen to hash close together on attempt one do not stay adjacent on attempt two.

The parser refuses instead of guessing

Every value that reaches the arithmetic above has to be exactly what the server meant, so the parsing is strict and Date.parse is not used anywhere.

Bare digits are checked first, and that ordering is load bearing. Date.parse('2000') returns the first of January 2000 in every major engine. A parser that tries the date branch first turns Retry-After: 2000, a thirty three minute backoff, into an instant twenty six years in the past, which floors to a wait of zero.

All three HTTP date forms are parsed by hand. IMF-fixdate, the obsolete RFC 850 form with its two digit year, and asctime with its space padded day. Date.parse is specified to fall back to implementation defined heuristics beyond a narrow set of formats, and those heuristics are what cause the damage: Date.parse('Feb 30 2024 00:00:00 GMT') returns the first of March rather than failing, and an abbreviation like EST is honoured, moving the instant five hours.

Refused, each with a message that says what to do about it:

  • a zone that is not literally GMT, including numeric offsets and including UTC
  • a day that does not exist in that month, and 29 February in a common year
  • a time of day outside 23:59:60
  • a day name that disagrees with the calendar date, since the two halves are redundant and a disagreement means one of them is corrupt
  • a lowercased or translated month or day name
  • a negative, signed, fractional or exponent delay
  • two values joined into one field, which is a singleton header that some hop appended to
  • an empty field, which cannot be read as zero because zero is a real instruction a server would have written as 0
  • a Retry-After instant more than a second before the response's own Date, which would otherwise become a hot loop against a service that is already refusing requests

maxWaitMs refuses too, rather than clamping. Clamping a two hour backoff down to a five minute limit means retrying an hour and fifty five minutes earlier than the server said was safe, which is the failure this module exists to prevent. The honest response to a wait that is longer than you are willing to take is to stop retrying.

Known limitations

UTC is refused even though it names the same offset as GMT. The grammar says GMT, and a parser that accepts one near miss is a parser that grows a list of accepted near misses, which is how EST gets in. This will reject a small number of real, harmless responses. Catch NON_GMT_ZONE if you would rather accept them.

The Date header is written when the response is generated, not when it is sent. An origin that buffers for a long time before flushing produces an interval that is slightly wide. The error is on the safe side, and it is bounded by the origin's own latency.

Age is only as good as the caches on the path. A cache that does not emit Age makes the response look fresher than it is, so the wait comes out long rather than short. Pass ageMs if you have a better estimate.

Jitter is statistical, not a scheduler. It reduces the chance of collisions across a population; it does not enforce an arrival rate. If you need a guaranteed rate, this is the wrong layer, and the answer is a shared limiter rather than a client side one.

Math.random is the default source. It is per process and not cryptographic. It is adequate for breaking up a herd and it is injectable if your environment gives you something better.

Nothing here performs a request. There is no transport, no retry loop, and no policy about which status codes deserve a retry. It reads headers and returns a number.

A leap second is accepted and rolled into the following minute. 23:59:60 is a real instant a correctly configured server can emit, and rejecting it would mean rejecting a header that is more precise than the parser.

Test

npm install
npm test   # 99 tests: clock skew, response age, herd dispersion, malformed headers, monotonic deadlines

Several tests carry the naive implementation alongside the real one and assert that it fails: that a client forty seconds fast computes a wait of zero from the same headers, that Date.parse('2000') lands in the past, and that five thousand clients honouring Retry-After: 30 exactly all return in a single millisecond.

License

MIT

About

Client side Retry-After handling that survives clock skew and synchronized retry herds. Zero dependencies.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages