llm-priority-broker

A single-permit, priority-aware broker that sits in front of a rate-limited LLM API. Standard library only. No dependencies.

Point your clients at this instead of at the provider. Exactly one request is in flight at any moment, and when the gate frees up, the highest-priority waiter goes next.

client A ──┐
client B ──┤                  ┌──────────────┐
client C ──┼──> broker :18790 │ 1 in flight  │──> https://api.provider.com
client D ──┤                  │ priority heap│
client E ──┘                  └──────────────┘

The problem this solves

Some hosted models bill concurrency, not just tokens. If one request consumes your plan's entire concurrency allowance, then "just retry on 429" is exactly the wrong answer. Every caller retries, they collide, and the provider spends its capacity rejecting you instead of serving you.

That failure mode is not theoretical. This component exists because of an incident where back-to-back requests collided repeatedly and the retry storm made it worse.

Try it

python3 examples/demo_priority.py

Six callers arrive while one request is already running. Two are high priority.

  t+0.00s  normal  in-flight    -> running (arrived first)
  t+0.00s  normal  batch-job-a  queued
  t+0.05s  normal  batch-job-b  queued
  t+0.11s  HIGH    paying-user  queued
  t+0.16s  normal  batch-job-c  queued
  t+0.21s  HIGH    support-req  queued
  t+0.27s  normal  batch-job-d  queued
  t+0.58s  HIGH    paying-user  -> running
  t+0.83s  HIGH    support-req  -> running
  t+1.09s  normal  batch-job-a  -> running
  t+1.34s  normal  batch-job-b  -> running
  t+1.59s  normal  batch-job-c  -> running
  t+1.85s  normal  batch-job-d  -> running

Both high-priority callers jumped the queue. Neither interrupted the request that was already running.

Usage

python3 llm_broker.py --upstream api.example.com
python3 llm_broker.py --upstream api.example.com --port 9001 --host 127.0.0.1

Then send requests to the broker instead of the provider:

# Normal priority
curl http://127.0.0.1:18790/v1/chat/completions -d @body.json

# High priority: same path with a /prio prefix. Jumps the queue.
curl http://127.0.0.1:18790/prio/v1/chat/completions -d @body.json

# Liveness, including current queue depth
curl http://127.0.0.1:18790/healthz
# {"ok":true,"waiting":3}

Headers and bodies pass through untouched, minus hop-by-hop headers. Your API key travels on the request as usual; the broker never stores or inspects it.

Design decisions, and why

One request in flight, enforced by a gate rather than by convention. Clients cannot be trusted to self-limit, and a misbehaving client during an incident is exactly the client you cannot reach.

Priority is non-preemptive. A high-priority arrival goes next. It never interrupts work already running. Preemption would mean discarding a call you have already paid for.

Retries happen while holding the gate. This is the counterintuitive one. On a 429 the broker keeps the gate and retries behind it, so nothing else can pile on while the provider recovers. Releasing the gate and letting callers retry is strictly worse: it turns one stalled request into a stampede.

A settle delay after success. Providers can lag releasing concurrency units between back-to-back requests. Pausing briefly before handing off prevents the exact collision this component exists to avoid. That value was tuned in response to a real outage, not chosen on a whiteboard.

There is deliberately no model and no clever scheduling in this path. Serialization and backpressure are invariants, not judgment calls. Anything adaptive here would be slower, non-deterministic, and impossible to reason about at three in the morning. All the judgment lives in the callers.

It fails fast rather than hanging. A broker that hangs takes every caller down with it. On internal error it returns 502 immediately, and a bounded upstream timeout stops one stuck call from holding the gate forever.

Tuning

All in llm_broker.py:

Constant	Default	What it controls
UPSTREAM_TIMEOUT	300s	Bounds a hung upstream so the gate cannot lock forever
ERR_RETRIES	3	Retries on transient connection and socket errors
RL_MAX_WAIT	60s	Total wall-clock budget for retrying 429s
RL_BACKOFF	2,3,5,5,8,8,10	Seconds between 429 retries, steady at the last value
SETTLE	0.7s	Pause after success before releasing the gate

Keep RL_MAX_WAIT under your callers' own timeout, so a late success still gets delivered instead of the caller giving up first.

Tests

python3 -m unittest discover -s tests -v

Covers mutual exclusion under contention, priority ordering, FIFO within a priority level, and the guarantee that priority never preempts work in flight.

Scope

This is deliberately small. It does one job.

It does not do load balancing across providers, caching, token accounting, request shaping, or failover. Those are real problems and they belong in the callers, where the context to decide them actually exists.

License

MIT. See LICENSE.