DEP: Runtime-Free Router and Gateway On-Ramp

[PeaBrane]

Area

router, gateway

Summary

Migrate Dynamo routing toward a runtime-free router core and service API that can serve both Gateway/EPP sidecar-style integration and HTTP push-router/frontend use cases, while preserving the existing Dynamo-runtime-backed frontend/router path during migration. V1 should be a practical runtime-free on-ramp, not a full parity pass with every default-router feature.

Motivation

The Dynamo router is one of the easiest Dynamo capabilities for users to want, but today the cleanest production paths often make router adoption feel like Dynamo runtime adoption. Existing vLLM/SGLang/TRT-LLM users frequently want a low-risk first proof: keep their current workers and request plane, add smarter KV/load-aware routing, measure TTFT/throughput, and roll back cleanly.

The current Gateway API Inference Extension (GAIE) integration is pragmatic: the GAIE EPP framework is Go, while Dynamo's mature router lives in Rust under dynamo-llm and still assumes Dynamo runtime discovery, model config, preprocessing, worker state, and router bookkeeping. The result is Go EPP plugin glue plus CGO/C bindings into the existing Dynamo-native router. This preserves parity, but it is not the cleanest long-term on-ramp for runtime-free gateway or raw-engine deployments.

Proposal

Define and implement a runtime-free router core and service API below the Dynamo runtime/frontend layer. The core should support the routing semantics needed for a first useful KV/load-aware path: token-aware prefix/cache scoring, active-load accounting, worker eligibility, disaggregated prefill/decode decisions where enabled, and reservation lifecycle bookkeeping.

Expose that core through at least two delivery surfaces:

1. Sidecar / Gateway decision mode

   • POST /route: query-only endpoint selection; no active-load mutation.
   • POST /route_and_reserve: atomic endpoint selection plus active-load reservation.
   • POST /reservations/{id}/prefill_complete: mark prompt-side work complete.
   • DELETE /reservations/{id}: release active-load state on completion or cancellation.
   • Intended for Gateway/EPP, Envoy-like request planes, and customers that want routing decisions while keeping their own forwarding layer.

2. HTTP push-router mode

   • OpenAI-compatible HTTP endpoints such as /v1/chat/completions and/or /v1/responses.
   • The router owns worker selection, reservation, forwarding to raw HTTP workers, response relay, prefill-complete detection, and cleanup.
   • Intended for users who do not have or do not want a Kubernetes Gateway, and who want an SMG-like front door.

Deployments should normally choose one operating mode at a time: query-only sidecar, route-and-reserve sidecar, or push router. Mixing these modes in one deployment should be treated as an advanced configuration because ownership of active-load state, retries, cleanup, and rejection behavior changes by mode.

Mode ownership should be explicit:

• In pure POST /route sidecar mode, the router is decision-only: no reservation, no router-owned queueing, no router-owned admission/rejection policy, and no priority scheduling. The caller's gateway/request plane owns dispatch, request admission, retries, and the response path.
• In POST /route_and_reserve sidecar mode, the router owns active-load mutation and reservation lifecycle. It may return basic routing/reservation failures such as no eligible worker, but broader queueing, priority, and admission policy remain outside V1 unless explicitly promoted.
• In HTTP push-router mode, the router owns dispatch and response lifecycle, so queueing, admission control, and priority are natural future router features, but still not required for the V1 on-ramp.

V1 Scope

V1 should optimize for the runtime-free on-ramp, not full feature parity with the default Dynamo router. It should include the minimum pieces needed to prove routing against existing workers: worker registration, token-aware device-local prefix scoring, basic active-load accounting, reservation lifecycle APIs, sidecar query mode, route-and-reserve mode, and optionally the HTTP push-router relay.

V1 should preserve HA/recovery behavior that already exists below the router service boundary. In particular, indexer microservice recovery via peer-to-peer state sharing and gap recovery from worker replay should remain part of the runtime-free path rather than being treated as excluded parity work.

V1 should likely exclude policy/features that are still evolving in the default/main router:

• router-owned queueing and priority scheduling
• admission control / rejection policy beyond simple no eligible worker or reservation failure
• lower-tier, shared-cache, or external-cache indexing/scoring
• output-block tracking and other advanced response-progress policies

Those features can remain explicit follow-up phases once the default router behavior has stabilized and the runtime-free API boundary is validated.

For Kubernetes Gateway integration, keep the GAIE/EPP adapter thin. The Go EPP plugin should translate GAIE request/endpoints into the runtime-free router API, then set gateway routing hints through headers/body mutations. The routing implementation should live in Rust/router service code, not in Go.

The existing Dynamo-native frontend/router and current GAIE CGO bridge should remain supported during migration. This DEP proposes a migration path and shared-core target, not an immediate removal of the current integration.

Requirements

MUST

• The router core MUST be usable without dynamo-runtime or Dynamo worker registration.
• The router core MUST preserve the V1 routing semantics it exposes: token-aware block hashing, basic device-local prefix scoring, active-load accounting/reservation lifecycle, worker eligibility, and aggregated/disaggregated worker selection where enabled.
• V1 MUST preserve existing indexer microservice HA/recovery behavior, including peer-to-peer recovery and gap recovery from worker replay.
• The service API MUST support query-only routing and route-plus-reservation routing as separate operations.
• Reservation lifecycle APIs MUST make sidecar/request-plane-owned dispatch safe for active-load tracking.
• The migration MUST preserve the existing Dynamo-runtime-backed router/frontend path until parity, performance, and operational behavior are validated.

SHOULD

• Lower-tier/shared-cache scoring SHOULD be treated as follow-up scope unless explicitly promoted into V1.
• Push-router mode SHOULD use standard HTTP reverse-proxy streaming patterns and clean up reservations on normal completion, upstream failure, and client disconnect.
• Gateway/EPP integration SHOULD use a thin adapter that calls the runtime-free router API and writes routing hints; Go should remain adapter glue, not the routing implementation.
• The design SHOULD benchmark in-process CGO versus same-pod HTTP/gRPC/UDS router-service calls, including tokenization and KV lookup costs.
• V1 SHOULD explicitly exclude router-owned queueing, priority scheduling, full admission-control/rejection policy, and lower-tier/shared-cache indexing/scoring unless a narrower behavior is required for correctness.

Open Questions

• Should the first production target be Gateway/EPP sidecar mode, HTTP push-router mode, or both behind one service binary?
• Should the router service expose HTTP, gRPC, Unix-domain sockets, or multiple transports?
• Which, if any, minimal reservation failure/cleanup semantics are required in V1 beyond no eligible worker and stale-reservation cleanup?
• What is the minimum native worker signal surface required for vLLM/SGLang/TRT-LLM workers without Dynamo runtime wrappers?
• How should worker registration and reservation cleanup work with multiple router replicas while preserving the indexer microservice recovery model?
• Which current dynamo-llm::KvRouter logic should move down into dynamo-kv-router or another runtime-free crate?

References

• Dynamo router guide: https://github.com/ai-dynamo/dynamo/blob/main/docs/components/router/router-guide.md
• Dynamo GAIE guide: https://github.com/ai-dynamo/dynamo/blob/main/docs/kubernetes/inference-gateway.md
• Current GAIE EPP adapter: https://github.com/ai-dynamo/dynamo/tree/main/deploy/inference-gateway/epp
• Current C bindings for GAIE/EPP router calls: https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/c/src/lib.rs
• Runtime-free router primitives already exist in dynamo-kv-router: https://github.com/ai-dynamo/dynamo/tree/main/lib/kv-router

[PeaBrane]

@athreesh @itay @nnshah1 @atchernych @hutm