Skip to content

feat: add kserve #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
henrichter-sap wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat: add kserve #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
26ff33e
feat: add kserve
henrichter-sap Jul 7, 2026
e02e7c5
review: remove stale epp configuration
henrichter-sap Jul 7, 2026
9492ecf
review: truncate llmisvc name
henrichter-sap Jul 7, 2026
c9240b6
downgrade epp tag
henrichter-sap Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 4 additions & 15 deletions .github/workflows/e2e.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,25 +56,13 @@ jobs:
--state-values-file ${{ github.workspace }}/helm/example.values.cpu.yaml \
--state-values-file ${{ github.workspace }}/test/e2e/values.yaml

- name: Wait for gateway
- name: Wait for LLMInferenceService
run: |
kubectl wait gateway/inference-gateway \
--namespace thalamus \
--for=condition=Programmed=True \
--timeout=120s

- name: Wait for model pod
run: |
kubectl rollout status deployment/vllm-tiny-llm \
kubectl wait LLMInferenceService/arnir0-tiny-llm \
--namespace thalamus \
--for=condition=Ready \
--timeout=600s

- name: Wait for EPP
run: |
kubectl rollout status deployment/vllm-tiny-llm-epp \
--namespace thalamus \
--timeout=120s

- name: Port-forward gateway
run: |
kubectl port-forward svc/inference-gateway \
Expand Down Expand Up @@ -104,3 +92,4 @@ jobs:
-l app.kubernetes.io/name=agentgateway || true
echo "=== Gateway / HTTPRoutes / InferencePools ==="
kubectl get gateway,httproutes,inferencepools -n thalamus -o yaml
kubectl get llminferenceservices -n thalamus -o yaml
2 changes: 1 addition & 1 deletion helm/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# renovate: datasource=github-releases depName=kubernetes-sigs/gateway-api
GATEWAY_API_VERSION ?= v1.5.1
# renovate: datasource=github-releases depName=kubernetes-sigs/gateway-api-inference-extension
GATEWAY_API_INFERENCE_EXTENSION_VERSION ?= v1.5.0
GATEWAY_API_INFERENCE_EXTENSION_VERSION ?= v1.4.0

.PHONY: vendor-crds
vendor-crds: ## Re-download vendored upstream CRDs from their pinned release tags.
Expand Down
16 changes: 3 additions & 13 deletions helm/example.values.cpu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,10 @@ gpu-operator:

thalamus:
models:
- slug: tiny-llm
model: arnir0/Tiny-LLM
accelerator: cpu
- name: arnir0/Tiny-LLM
config: cpu
replicas: 1
extraArgs:
args:
Comment on lines +11 to +14

@coderabbitai coderabbitai Bot Jul 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Finish removing the legacy EPP override from this example.

This example now uses the KServe model schema, but the same file still contains thalamus.epp.resources below. Since this PR removes the legacy EPP values/templates, that block is dead config and should be deleted or migrated to the new config model.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@helm/example.values.cpu.yaml` around lines 11 - 14, The example still
contains legacy thalamus.epp.resources configuration even though it now uses the
KServe model schema. Remove that dead EPP override from the example values file,
or migrate the resource settings into the new config model if they are still
needed, and keep the example aligned with the updated schema used by the
arnir0/Tiny-LLM entry.

- "--max-model-len=512"
- "--dtype=float32"
- "--gpu-memory-utilization=0.50"
Expand All @@ -23,12 +22,3 @@ thalamus:
limits:
cpu: "4"
memory: "6Gi"

epp:
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "200m"
memory: "512Mi"
8 changes: 4 additions & 4 deletions helm/example.values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@

thalamus:
models:
- slug: qwen3-6-27b
model: Qwen/Qwen3.6-27B
accelerator: nvidia
extraArgs:
- name: Qwen/Qwen3.6-27B
config: nvidia
replicas: 1
args:
- "--tensor-parallel-size=2"
- "--reasoning-parser=qwen3"
resources:
Expand Down
2 changes: 1 addition & 1 deletion helm/gateway-api-inference-extension/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ apiVersion: v2
name: gateway-api-inference-extension
description: Vendored Gateway API Inference Extension CRDs.
type: application
version: 1.5.0
version: 1.4.0
Loading