Followup to #974 (B2 — initial SGLang runtime backend). Once we have
real workload data on which SGLang flags operators reach for, expand
SGLangConfig to cover the remaining SGLang CLI surface. Candidates:
- LoRA hot-loading (
--lora-paths runtime updates) — needs a separate
CRD API for dynamic adapter registration
- Speculative decoding perf tuning
(--speculative-accept-threshold-single,
--speculative-accept-threshold-acc)
- Data-parallel rendezvous / multi-replica orchestration
PodMonitor for SGLang's /metrics endpoint (scraped today but not
wired into HPA defaults)
- Remaining minor flags: model override (
--model), reasoning-content
(--reasoning-content), return-logprob (--return-logprob), log
level (--log-level), trust-remote-code, skip-tokenizer-init, etc.
- A typed
LoRAConfig (vs the current []string for loraModules)
with named adapters
Filed by the same operator running LLMKube's Foreman on a self-hosted
fleet. Will coordinate with the maintainers on which subset is highest
value for the foreman workload.
Followup to #974 (B2 — initial SGLang runtime backend). Once we have
real workload data on which SGLang flags operators reach for, expand
SGLangConfigto cover the remaining SGLang CLI surface. Candidates:--lora-pathsruntime updates) — needs a separateCRD API for dynamic adapter registration
(
--speculative-accept-threshold-single,--speculative-accept-threshold-acc)PodMonitorfor SGLang's/metricsendpoint (scraped today but notwired into HPA defaults)
--model), reasoning-content(
--reasoning-content), return-logprob (--return-logprob), loglevel (
--log-level), trust-remote-code, skip-tokenizer-init, etc.LoRAConfig(vs the current[]stringforloraModules)with named adapters
Filed by the same operator running LLMKube's Foreman on a self-hosted
fleet. Will coordinate with the maintainers on which subset is highest
value for the foreman workload.