Skip to content

feat(runtime): expand SGLangConfig to kitchen-sink (B3 followup to #974) #1060

Description

@itsmiso-ai

Followup to #974 (B2 — initial SGLang runtime backend). Once we have
real workload data on which SGLang flags operators reach for, expand
SGLangConfig to cover the remaining SGLang CLI surface. Candidates:

  • LoRA hot-loading (--lora-paths runtime updates) — needs a separate
    CRD API for dynamic adapter registration
  • Speculative decoding perf tuning
    (--speculative-accept-threshold-single,
    --speculative-accept-threshold-acc)
  • Data-parallel rendezvous / multi-replica orchestration
  • PodMonitor for SGLang's /metrics endpoint (scraped today but not
    wired into HPA defaults)
  • Remaining minor flags: model override (--model), reasoning-content
    (--reasoning-content), return-logprob (--return-logprob), log
    level (--log-level), trust-remote-code, skip-tokenizer-init, etc.
  • A typed LoRAConfig (vs the current []string for loraModules)
    with named adapters

Filed by the same operator running LLMKube's Foreman on a self-hosted
fleet. Will coordinate with the maintainers on which subset is highest
value for the foreman workload.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions