Problem
For cost efficient but low-latency routing, using staged_router as a deterministic step for weak/strong is useful, but having the strong target as the llm classifer would allow a greater granularity of cost control and potentially of PII management. If the deterministic step indicates esclation to the llm classifer, then upstream classification can be configured to control cloud costs.
Proposed solution
This would be simply that the "strong" target can route to the llm classifier natively, without the need for an intermediate step and would give native visibility of routing decisions in both the deterministic and probabilstic methods, giving users a single pane of a glass to configure costs and cost savings.
Additionally, a mechanism to optimise prompts being sent to the llm_classifer could be of value. Going further, having the llm classifer given a recipie for adaptive training to override the staged_router heuristics and potentially return back to the staged router weak target/local, with an explicit instruction to process locally. Thbis could be used in the case where the llm classifer determines there could be PII in the prompt, as a secondary control for users who have DLP/PII deterministic classifiers between the staged_router and the llm classifer, for additional control against sensitive information leakage upsteam to cloud.
Alternatives considered
I looked at using staged router to use litellm as the capable/strong target, but visibility of routing decisions is fragmented.
Scope notes
- Which surface owns the change: server API or configuration, routing algorithm,
protocol type, translation codec, upstream client, skill-distillation contract,
Python binding, or launcher?
- Does it change a public Rust, PyO3, Python, CLI, or deployment-TOML interface?
- Backward-compatibility concerns?
Additional context
Links to related issues, prior art in other routers, references.
Problem
For cost efficient but low-latency routing, using staged_router as a deterministic step for weak/strong is useful, but having the strong target as the llm classifer would allow a greater granularity of cost control and potentially of PII management. If the deterministic step indicates esclation to the llm classifer, then upstream classification can be configured to control cloud costs.
Proposed solution
This would be simply that the "strong" target can route to the llm classifier natively, without the need for an intermediate step and would give native visibility of routing decisions in both the deterministic and probabilstic methods, giving users a single pane of a glass to configure costs and cost savings.
Additionally, a mechanism to optimise prompts being sent to the llm_classifer could be of value. Going further, having the llm classifer given a recipie for adaptive training to override the staged_router heuristics and potentially return back to the staged router weak target/local, with an explicit instruction to process locally. Thbis could be used in the case where the llm classifer determines there could be PII in the prompt, as a secondary control for users who have DLP/PII deterministic classifiers between the staged_router and the llm classifer, for additional control against sensitive information leakage upsteam to cloud.
Alternatives considered
I looked at using staged router to use litellm as the capable/strong target, but visibility of routing decisions is fragmented.
Scope notes
protocol type, translation codec, upstream client, skill-distillation contract,
Python binding, or launcher?
Additional context
Links to related issues, prior art in other routers, references.