feat(observability): per-InferenceService SLO declaration via Pyrra integration

[Defilan]

Motivation

LLMKube currently exposes inference latency, error rate, and queue wait metrics (per #409 / PR #410) but has no first-class concept of an SLO. Operators who want to commit to "TTFT p95 ≤ 2s, availability ≥ 99.5% over 28 days" today have to hand-roll Prometheus recording rules and alert thresholds and own the error-budget arithmetic themselves.

For LLMKube to be a credible Tier-1 platform, declarative SLOs and error budgets need to be a built-in capability. This issue scopes a v0.1 of that capability that ships before LLMKube is asked to commit to enterprise SLOs externally.

Proposed approach: integrate Pyrra rather than build a new CRD

Pyrra is a CNCF-incubating project that takes a declarative SLO spec via its own ServiceLevelObjective CRD and generates Prometheus recording rules + alert rules from it. Mature, multi-vendor, and integrates cleanly with kube-prometheus-stack which is already what LLMKube ships.

Why Pyrra over a new LLMKube SLO CRD:

• The error-budget math is non-trivial and Pyrra has been getting it right for years
• We do not want to maintain another CRD when an upstream solves the same problem
• Operators already familiar with Pyrra get zero learning curve
• We can layer LLMKube-specific UX on top (e.g., a spec.slo shorthand on InferenceService) without owning the underlying engine

Why not a custom LLMKube CRD:

• Reinvents the wheel on burn-rate alerts and multi-window multi-burn-rate logic
• Larger surface area for us to maintain forever
• No interoperability story with non-LLMKube workloads in the same cluster

Why not just document "use Pyrra alongside LLMKube":

• Ergonomic gap: an operator declaring an InferenceService should not have to also write a separate Pyrra CRD that duplicates service.namespace references
• We can do better than that with a thin shorthand

Scope

v0.1 (this issue)

1. Add spec.slo shorthand to InferenceService with two minimal fields:
   • name: SLO name (e.g., tinyllama-availability)
   • objective: target as decimal (e.g., 0.995 for 99.5% availability)
   • Optionally a window defaulting to 28d and an indicator enum (availability | latency_p95 | error_rate)
2. Controller renders a Pyrra ServiceLevelObjective resource on apply, in the same namespace as the InferenceService, owner-referenced for garbage collection
3. Helm chart values to enable / disable the Pyrra integration (off by default; opt-in via pyrra.enabled=true)
4. Documentation at docs/observability/slo.md covering: when to enable, supported indicators, how to view error budgets in Grafana, what happens when budgets burn down
5. Reference Grafana dashboard showing error budget per SLO + burn rate

Explicitly out of scope for v0.1 (defer to follow-up issues)

• Auto-remediation on SLO breach (that's #10)
• Custom indicators beyond availability / latency / error rate
• Multi-cluster SLO aggregation
• A LLMKube-native SLO CRD that supplants Pyrra (we genuinely do not need this)

Implementation sketch

// api/v1alpha1/inferenceservice_types.go
type InferenceServiceSpec struct {
    // ... existing fields ...

    // SLO declares a service-level objective for this inference service.
    // When set, the controller creates a Pyrra ServiceLevelObjective
    // resource in the same namespace; Pyrra generates the recording +
    // alert rules. Requires the cluster to have Pyrra installed.
    // +optional
    SLO *SLOSpec `json:"slo,omitempty"`
}

type SLOSpec struct {
    // Name is the SLO identifier; appears in Pyrra dashboards.
    // +kubebuilder:validation:Required
    Name string `json:"name"`

    // Objective is the target ratio (e.g., 0.995 for 99.5%).
    // +kubebuilder:validation:Minimum=0.5
    // +kubebuilder:validation:Maximum=0.99999
    Objective float64 `json:"objective"`

    // Window is the rolling window the objective is measured over.
    // +kubebuilder:default="28d"
    Window string `json:"window,omitempty"`

    // Indicator selects which signal the objective is measured against.
    // +kubebuilder:validation:Enum=availability;latency_p95;error_rate
    // +kubebuilder:default=availability
    Indicator string `json:"indicator,omitempty"`
}

The controller's reconcile path adds a small Pyrra resource render step after the existing Deployment + Service apply. PromQL templates per indicator type are bundled in internal/controller/slo/templates.go so behavior stays predictable.

Testing

Unit

• Render path: given a sample InferenceService with spec.slo set, controller produces an exactly-equal Pyrra resource with the right PromQL template
• Validation: objective outside [0.5, 0.99999] is rejected at admission
• Owner reference: deleting the InferenceService garbage-collects the Pyrra resource

Integration / e2e

• Deploy a TinyLlama with spec.slo.objective=0.995 against the e2e Kind suite (Pyrra installed via Helm dep), confirm the Pyrra resource exists, confirm Pyrra generates Prometheus rules from it, hit the endpoint with induced failures, confirm error budget burn registers

Manual

• Deploy on a real cluster, force an SLO breach, screenshot the Pyrra dashboard for the docs

Acceptance criteria

• InferenceServiceSpec.SLO field exists with the documented validation
• Controller reconcile creates / updates / deletes Pyrra ServiceLevelObjective resources following the InferenceService lifecycle
• Helm chart adds an opt-in Pyrra dependency or values toggle
• docs/observability/slo.md covers all four supported indicators with examples
• Reference Grafana dashboard at docs/grafana/llmkube-slo.json imports cleanly
• Unit + e2e tests pass in CI
• CRD-Sync check passes (chart-bundled CRD updated to match)

References

• Pyrra: https://github.com/pyrra-dev/pyrra
• Multi-window multi-burn-rate alerts (Google SRE workbook): https://sre.google/workbook/alerting-on-slos/
• Existing recording rules: charts/llmkube/templates/prometheusrule.yaml (added in PR #410)
• Related: #10 (advanced SLO auto-remediation; depends on this landing first)