fleet-router

One OpenAI-compatible endpoint in front of a heterogeneous local inference fleet, routing each request to the node that already has the model warm in VRAM (residency-aware), with capability filtering, health failover, and per-node load caps.

Status: built and verified: 76 tests (unit plus integration through the real LiteLLM proxy). Residency routing, SSE streaming, health failover, concurrency fan-out, saturation, and in_flight load-cap accounting are all proven. The residency adapters are confirmed against real llama-swap v228, real LM Studio, and real Ollama v0.31.2 (reference/ollama_smoke.py drives a cold-load miss, then a warm residency hit, through the proxy; 2026-07).

Approach: COMPOSE (stand on giants, build only what's ours)

No tool ships fleet-wide VRAM-residency routing, but it's a thin plugin on top of mature pieces:

Layer	Decision	What
per-node model load + idle unload	REUSE llama-swap	on-demand swap; reuses your existing GGUFs
single OpenAI endpoint (auth, spend, Postgres, /v1/models, health, SSE, metrics)	REUSE LiteLLM proxy	the gateway shell
fleet-wide residency routing decision	BUILD	ResidencyRoutingStrategy (LiteLLM CustomRoutingStrategy) fed by a background poller, the thin glue nobody ships

Validated by spikes before committing: the strategy routes by a background-polled snapshot at ~1µs hot-path cost, and the LiteLLM proxy server loads it via callbacks config and routes live. See reference/ and ../autoeverything/research/.

Differentiator & position

The one thing no off-the-shelf tool ships: fleet-wide, VRAM-residency-aware routing across heterogeneous nodes, through one endpoint, preferring the node that already has the model warm. Everything else (auth, spend, SSE, health, per-node swap, metrics) is reused, so this is mostly glue: about 1k lines of routing decision on top of LiteLLM and llama-swap.

The nearest tools each miss residency a different way (landscape: a 106-agent verified run, every candidate scored N on this axis):

• generic load balancers never read VRAM state (LiteLLM, Kong, Portkey, paddler)
• placement is chosen at deploy time, not per request (GPUStack)
• the residency mechanism is right but single-host, or it picks which replica of one already-loaded model rather than which node has it loaded (llama-swap, vLLM production-stack)

LiteLLM even ships the plugin hook (CustomRoutingStrategy); it just doesn't ship the residency signal. That signal is the only thing fleet-router adds.

First-mover, not a moat. A narrow feature on a fast-moving base: llama-swap and paddler ship often, and a fleet-residency feature could land upstream any time. The gap is current, not durable.

What it is not: not an inference engine (llama.cpp/llama-swap run the models); not a general load balancer (LiteLLM already is one); not a from-scratch gateway; not a Kubernetes/datacenter system (it targets a Pi + tablet + consumer-GPU fleet); not a model-by-query-type router (that's semantic-router, orthogonal).

Run it

python3.13 -m venv .venv && . .venv/bin/activate    # Python 3.11-3.13, NOT 3.14
pip install -e '.[proxy,dev]'                        # '.[dev]' alone for the light units
pytest -q -p no:libtmux                              # -> 76 passed

FLEET_CONFIG=fleet.yaml python -m fleet_router.app   # launch the proxy on :8080
# then point any OpenAI client at http://localhost:8080 with the master key

Run with --num_workers 1 (the live in_flight counter + metrics are per-worker). /metrics exposes fleet_requests_total, fleet_request_latency_seconds, fleet_residency_hits_total, fleet_inflight, fleet_healthy_nodes.

fleet.yaml

models: is the capability source of truth per node (a Pi lists only small GGUF). A node is lmstudio (residency from /api/v0/models), llamaswap (residency from /running), or ollama (residency from /api/ps; model names use Ollama's repo:tag form).

server:
  listen: "0.0.0.0:8080"
  master_key: "sk-fleet"

nodes:
  # Reuse your EXISTING LM Studio as a node - non-disruptive, no llama-swap needed.
  - name: rig-4090
    base_url: "http://localhost:1234"
    kind: lmstudio
    vram_mb: 16000
    max_concurrency: 1
    roles: [chat, completion]
    models: ["qwen/qwen3-coder-30b", "zai-org/glm-4.7-flash"]

  # A second box via llama-swap, reusing LM Studio's downloaded GGUFs (no re-download):
  #   cmd: llama-server --model ~/.cache/lm-studio/models/<pub>/<repo>/<file>.gguf --port ${PORT}
  - name: box-2
    base_url: "http://10.0.0.20:8080"
    kind: llamaswap
    vram_mb: 24000
    max_concurrency: 1
    roles: [chat]
    models: ["qwen/qwen3-coder-30b"]

  # Or reuse an EXISTING Ollama the same way; it does its own load/unload (keep_alive):
  - name: box-3
    base_url: "http://10.0.0.30:11434/v1"     # Ollama OpenAI endpoint
    residency_url: "http://10.0.0.30:11434"   # server root; /api/ps gives loaded state
    kind: ollama
    vram_mb: 24000
    max_concurrency: 1
    roles: [chat]
    models: ["ornith-1.0-9b:latest"]

aliases:
  - { from: "smart", to: "qwen/qwen3-coder-30b" }

Layout

fleet_router/
  config.py         # fleet.yaml schema + alias/deployment helpers   (U0 contracts)
  snapshot.py       # residency snapshot (node -> NodeState)          (U0)
  capability.py     # can_serve(node, model, role)                   (U0)
  strategy.py       # ResidencyRoutingStrategy - the routing brain   (U1)
  poller.py         # background residency poller                    (U2)
  adapters/         # lmstudio/llamaswap/ollama residency adapters   (U2)
  litellm_config.py # fleet.yaml -> LiteLLM proxy config             (U3)
  metrics.py        # Prometheus metric contract                     (U5)
  mocknode.py       # mock node for the integration harness          (U4)
  app.py            # proxy registrar + entrypoint (wires it all)    (U7)
deploy/             # llama-swap node provisioning                   (U6)
tests/              # unit + integration; fixtures
reference/          # the proven spikes (port these, don't redesign)