Dynamic-batcher throughput progressively degrades over time when `default_queue_policy.default_timeout_microseconds > 0` (waiting_consumer_count_ drift on payload merges)

[jimmystique]

Description

With dynamic batching and rate limiting off (the default), a model configured with a non-zero dynamic_batching.default_queue_policy.default_timeout_microseconds progressively loses throughput over minutes of steady traffic, even though the GPU and the model instances are largely idle. Throughput drops well below the offered load and does not recover on its own. In our case it stayed degraded (non-zero) under continuous load, but the mechanism can drive it all the way to zero (a full per-model stall) in the limit; see the root-cause note below. Latency (nv_inference_queue_duration_us) climbs, the pending-request queue grows, execs/s decays, and per-instance activity becomes uneven (some instances receive little or no work). A model reload restores full throughput, but pausing traffic does not, so the bad state is persistent per-model and not tied to in-flight requests.

Root cause: dispatch readiness is tracked by InstanceQueue::waiting_consumer_count_, meant to
equal the number of instances currently parked in RateLimiter::DequeuePayload. It is moved by
exactly two sites on the per-model queue:

• -1 per enqueued payload: RateLimiter::EnqueuePayload (src/rate_limiter.cc, unconditional DecrementConsumerCount()).
• +1 per dequeue call :RateLimiter::DequeuePayload (src/rate_limiter.cc, IncrementConsumerCount()).

But InstanceQueue::Dequeue (src/instance_queue.cc) merges multiple queued payloads into a
single dequeue when the batch is unsaturated and max_queue_delay_ns_ > 0. A dequeue that merges k extra payloads therefore sees k+1 decrements but only 1 increment, a permanent -(k-1) leak that is never returned. Because almost all requests are handled 1:1 (their +1/-1 cancel), the counter is the difference of two large, nearly-equal flows, so only the merged-away surplus leaks through.

When support_prefetching_ is false (i.e. any non-zero default_timeout_microseconds; see src/scheduler_utils.cc), both dispatch gates read this counter via RateLimiter::PayloadSlotAvailable (WaitingConsumerCount() > 0):

1. DynamicBatchScheduler::Enqueue only wakes the batcher if the gate passes.
2. DynamicBatchScheduler::WaitForPayloadSlotAvailable loops on the same predicate with a 500 ms
   timeout.

Once waiting_consumer_count_ <= 0, the batcher's dispatch gate keeps failing: new requests no longer wake it, and WaitForPayloadSlotAvailable only proceeds once WaitingConsumerCount() > 0 (its 500 ms timeout just retries, it never dispatches while the count is non-positive). Throughput degrades; under continuous load a trickle still gets through, but the drift is unbounded and can reach a full per-model stall (0). The state is persistent: pausing traffic does not recover it, only a model reload does (it rebuilds InstanceQueue with the counter reset).

Triton Information

Triton server 2.70.0 (release r26.06). The relevant scheduler / rate-limiter code (src/rate_limiter.cc, src/instance_queue.cc, src/scheduler_utils.cc) is unchanged on main as of this writing, so it affects current releases as well.

To Reproduce

This bug is model-agnostic since it depends only on the scheduler configuration, not on the model framework, inputs, or outputs. Any model with max_batch_size > 1, instance_count > 1, and the dynamic-batching config below reproduces it. (In our case it showed up on a CPU model pool with instance_count: 12, but the framework is irrelevant.)

Model config that triggers it:

# config.pbtxt (any backend)
max_batch_size: 8
instance_group [ { count: 12 } ]   # any count > 1
dynamic_batching {
  max_queue_delay_microseconds: 2000
  default_queue_policy {
    default_timeout_microseconds: 150000   # any non-zero value disables prefetching
  }
}

Expected behavior

waiting_consumer_count_ should reflect the true number of waiting consumers, and dispatch should not decay over time. Throughput should stay stable for the lifetime of the loaded model, regardless of default_timeout_microseconds.

Workaround (not a fix): set default_queue_policy.default_timeout_microseconds: 0 (or remove default_queue_policy) to enable prefetching, which decides dispatch on live queue size instead of the counter. This also disables the per-request queue timeout, so it is a mitigation only.

I have a fix ready (credit back one consumer count per merged payload in InstanceQueue::Dequeue) with a regression test, and will open a PR against triton-inference-server/core shortly and link it here.

[jimmystique]

PR opened against triton-inference-server/core here: triton-inference-server/core#509.