feat(api): GPUQuota CRD + admission webhook + RBAC bundle for multi-tenant deployments

[Defilan]

Motivation

LLMKube's InferenceService currently has no concept of "this team owns this much GPU." A cluster shared across multiple teams (R&D, app teams, factory ops, etc.) has no built-in way to:

• Cap how many GPUs team A can use at any time
• Reject deployments that would exceed a team's quota
• Bind cost attribution (already in InferCost) to a quota structure that maps to chargeback business units

Operators today have to roll this themselves with ResourceQuota (which works for nvidia.com/gpu count but says nothing LLMKube-specific) plus custom RBAC. We can do meaningfully better.

This issue scopes a v0.1 multi-tenancy capability: a GPUQuota CRD enforced via admission webhook, plus an opinionated RBAC bundle in the Helm chart.

Proposed approach

A new GPUQuota CRD (cluster or namespace-scoped)

type GPUQuotaSpec struct {
    // Selector picks which namespaces this quota applies to. Mutually exclusive
    // with NamespaceRef (which restricts to a single namespace).
    Selector *metav1.LabelSelector `json:"selector,omitempty"`
    NamespaceRef string `json:"namespaceRef,omitempty"`

    // Hard caps on GPU count and total VRAM bytes summed across all
    // InferenceServices in the matched namespace(s).
    GPUCount int32 `json:"gpuCount"`
    VRAMBytes int64 `json:"vramBytes,omitempty"`

    // Optional priority floor: only InferenceServices at or above this
    // priority can consume from this quota. Maps to existing priority
    // class enum (critical/high/normal/low/batch).
    MinPriority string `json:"minPriority,omitempty"`

    // Optional CostBudget reference: if set, the InferCost TokenBudget
    // matching this name in the same namespace is treated as the spend
    // ceiling for this quota. Hooks the cost half (already shipping in
    // InferCost) into the resource quota.
    CostBudgetRef string `json:"costBudgetRef,omitempty"`
}

type GPUQuotaStatus struct {
    UsedGPUCount int32   `json:"usedGPUCount"`
    UsedVRAMBytes int64  `json:"usedVRAMBytes"`
    AdmissionDenials int64 `json:"admissionDenials"`
    LastDenial *metav1.Time `json:"lastDenial,omitempty"`
}

Admission webhook

A validating webhook at inferenceservices.inference.llmkube.dev admission:

1. Resolves applicable GPUQuota for the InferenceService's namespace (selector or direct ref)
2. Computes incremental usage if this InferenceService were admitted
3. Rejects if it would exceed gpuCount, vramBytes, or violate minPriority
4. Records the denial in the GPUQuota status counter

The webhook is opt-in via Helm values: multitenancy.enabled=true. With it off, behavior is identical to today.

RBAC bundle

Two new ClusterRoles ship with the chart, gated by the same opt-in:

• llmkube-tenant: read own-namespace Model, InferenceService, UsageReport (InferCost), TokenBudget (InferCost). Create/update Model and InferenceService only.
• llmkube-tenant-admin: superset; adds the ability to manage GPUQuota for own namespace and bind TokenBudget.

These are namespace-scoped via RoleBinding, not ClusterRoleBinding. Operators bind them per team-namespace.

Hook into InferCost

GPUQuota.spec.costBudgetRef does NOT modify the InferCost CRDs; it just resolves at admission time and consults InferCost's existing TokenBudget status to make a decision. InferCost remains the source of truth for cost; LLMKube remains the source of truth for resource. Clean separation.

Scope

v0.1 (this issue)

1. GPUQuota CRD + reconciler that maintains live Status.UsedGPUCount / UsedVRAMBytes from InferenceServices in scope
2. validating admission webhook for InferenceService that consults applicable GPUQuotas
3. Helm chart multitenancy.enabled toggle that installs the webhook + the two RBAC ClusterRoles
4. Documentation at docs/operations/multi-tenancy.md covering: when to use, RBAC binding examples, cost budget integration, what happens at quota breach
5. Reference Grafana dashboard panel showing per-quota utilization

Explicitly out of scope for v0.1 (defer to follow-up issues)

• Hierarchical quotas (org → BU → team)
• Quota borrowing / preemption between namespaces
• Network bandwidth or storage quotas (this is GPU-only)
• Auto-scaling quotas based on time-of-day
• Federation across clusters (depends on multi-site work)

Implementation sketch

api/v1alpha1/gpuquota_types.go              // new CRD types
internal/controller/gpuquota_controller.go  // status reconciler
internal/webhook/inferenceservice_admit.go  // validating webhook
charts/llmkube/templates/webhook.yaml       // chart-installed when toggle on
charts/llmkube/templates/rbac/tenant-role.yaml
charts/llmkube/templates/rbac/tenant-admin-role.yaml
docs/operations/multi-tenancy.md
docs/grafana/llmkube-quota.json

Testing

Unit

• GPUQuota status reconciliation: given InferenceServices in scope, status correctly aggregates GPU count + VRAM
• Admission decision matrix: combinations of (in-quota / over-quota), (priority floor satisfied / not), (cost budget breached / not) for both allow and deny outcomes

Integration / e2e

• Two namespaces, two GPUQuotas; submit InferenceServices that fit + ones that exceed; assert admission decisions match
• GPUQuota deletion cleans up admission state correctly
• Webhook unavailable: failure policy is Fail (deny) on admission errors so quota cannot be bypassed by webhook outage

Manual

• Walkthrough on shadowstack: two pretend tenant namespaces, two GPUQuotas, deploy hits and exceeds budget, screenshots for docs

Acceptance criteria

• GPUQuota CRD lands with documented validation
• Validating webhook rejects over-quota InferenceServices and surfaces clear error to user
• Status counter on GPUQuota updates within one reconcile cycle of InferenceService changes
• Helm multitenancy.enabled toggle off is the default and a no-op
• RBAC bundle ships in the chart and is documented
• Reference dashboard imports cleanly
• Unit + e2e tests pass in CI
• CRD-Sync check passes

References

• Existing InferenceService priority enum: api/v1alpha1/inferenceservice_types.go
• InferCost TokenBudget for cost-side budgets: https://github.com/defilantech/InferCost (CRD ships in v0.2.x)
• Kubernetes built-in ResourceQuota (nvidia.com/gpu count works but is not GPU-aware beyond count)
• Related: #10 (SLO auto-remediation may interact with quota when degrading), #415 (SLO framework, sibling capability)