test(e2e): add scheduler policy E2E suite (#2793)

* test(e2e): add scheduler policy E2E suite

Add a Ginkgo E2E suite that asserts deterministic placement for HAMi's
node/GPU binpack and spread policies, comma-separated policy chains, and
per-Pod device scoring weight overrides. Adds a `make e2e-policy-test`
target and `hack/e2e-policy-test.sh` runner.

The suite waits on the scheduler's in-memory cache (via /metrics) so
placement assertions never observe stale utilization: pods are confirmed
released from every scheduler replica's cache after deletion and namespace
cleanup. Metrics are scraped from all running scheduler-extender replicas
rather than a guessed leader, so the suite is correct under HA and across
leader changes. GPU allocation decoding tolerates the trailing-separator
quirk of EncodePodSingleDevice instead of hard-coding the decoded length.

🤖 Assisted-by: Claude Code

Signed-off-by: Tim <tim.wang03@sap.com>

* address latest comment

Signed-off-by: Tim <tim.wang03@sap.com>

* fix(e2e): wait for scheduler GPU registration after device-plugin ready

After labeling a node and waiting for the device-plugin to become Ready,
the scheduler still needs time to process the node update and register
the node's GPU capacity. Without this wait, pods may fail to schedule
with '1 node unregistered' errors.

Add schedulerCanScheduleGPU check in ensureGPUNodeLabeled to verify the
node has GPU resources in its Capacity/Allocatable before proceeding.

Signed-off-by: Tim <tim.wang03@sap.com>

* fix(e2e): abort immediately on terminal pod phases in policy tests

waitForAllocatedPod previously polled for the full 5-minute timeout even
when a pod reached the terminal Failed phase (which can never transition
to Running). This caused the first policy spec to block for 300 seconds
before reporting failure, and the Ordered/Serial suite then skipped all
remaining specs.

Use gomega.StopTrying to abort the Eventually loop as soon as the pod
enters Failed or Succeeded, and surface container exit codes and
termination reasons in the failure message so the root cause is
immediately visible in CI logs.

Signed-off-by: Tim <tim.wang03@sap.com>

* fix(e2e): harden policy suite against device-plugin transitions

The policy E2E suite fails when the previous pod suite's cleanup removes
the gpu=on label and the policy suite re-adds it within ~133ms. The
DaemonSet controller may terminate the device-plugin pod after our
readiness checks pass, causing container startup failures.

Three layered fixes:

1. Add hasTerminatingDevicePlugin check to ensureGPUNodeLabeled and
   registeredGPUNodes — blocks test start while a device-plugin pod
   is still being torn down.

2. Switch RestartPolicy from Never to OnFailure — kubelet retries
   container startup through transient device-plugin unavailability
   instead of parking in permanent Failed phase.

3. Add StopTrying + podTerminalReason in waitForAllocatedPod — if a
   pod still reaches terminal Failed, abort in ~2s with exit codes
   and termination reasons instead of burning the full 5min timeout.

Signed-off-by: Tim <tim.wang03@sap.com>

* fix(e2e): wait for device-plugin healthy devices before creating GPU pods

The Pod E2E suite creates GPU pods immediately after adding the gpu=on
label, without waiting for the device-plugin to register healthy devices
with kubelet. This causes intermittent 'no healthy devices present'
admission errors when the device-plugin hasn't finished initializing.

Add WaitForDevicePluginReady to test/utils that polls until:
- At least one device-plugin pod is Running and Ready on the node
- No device-plugin pods are terminating (DaemonSet revision transition)
- The node reports GPU resources in Allocatable (healthy device registration)

Call it from the Pod E2E BeforeAll after labeling the node.

Signed-off-by: Tim <tim.wang03@sap.com>

---------

Signed-off-by: Tim <tim.wang03@sap.com>

Author: haitwang-cloud

Makefile

@@ -122,3 +122,7 @@ e2e-mig-test:
 .PHONY: e2e-mig-smoke-test
 e2e-mig-smoke-test:
 	./hack/e2e-mig-smoke-test.sh "${KUBE_CONF}"
+
+.PHONY: e2e-policy-test
+e2e-policy-test:
+	HAMI_E2E_REQUIRE_POLICY_TOPOLOGY=true bash ./hack/e2e-policy-test.sh "${KUBE_CONF}"

hack/e2e-policy-test.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# Copyright 2026 The HAMi Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+KUBE_CONF=${1:-"${HOME}/.kube/config"}
+export KUBE_CONF
+
+REPO_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
+cd "${REPO_ROOT}"
+
+if [ ! -f "${KUBE_CONF}" ]; then
+  echo "Error: kubeconfig file not found at ${KUBE_CONF}" >&2
+  exit 1
+fi
+
+if ! kubectl --kubeconfig "${KUBE_CONF}" get nodes >/dev/null; then
+  echo "Error: cannot reach Kubernetes API server via ${KUBE_CONF}" >&2
+  exit 1
+fi
+
+GINKGO_VERSION=$(go list -m -f '{{.Version}}' github.com/onsi/ginkgo/v2)
+if [ -z "${GINKGO_VERSION}" ]; then
+  echo "Error: could not determine Ginkgo version from go.mod" >&2
+  exit 1
+fi
+
+go run "github.com/onsi/ginkgo/v2/ginkgo@${GINKGO_VERSION}" \
+  run -v --procs=1 ./test/e2e/policy/ -- --kubeconfig="${KUBE_CONF}"

test/e2e/pod/test_pod.go

@@ -57,6 +57,10 @@ var _ = ginkgo.Describe("Pod E2E Tests", ginkgo.Ordered, func() {
 		ginkgo.By("Adding node labeling")
 		_, err = utils.AddNodeLabel(clientSet, nodeName, NodeLabelKey, NodeLabelValue)
 		gomega.Expect(err).NotTo(gomega.HaveOccurred())
+
+		ginkgo.By("Waiting for device-plugin to register healthy GPU devices")
+		err = utils.WaitForDevicePluginReady(clientSet, nodeName, 5*time.Minute, 2*time.Second)
+		gomega.Expect(err).NotTo(gomega.HaveOccurred(), "device-plugin did not become ready on node %s", nodeName)
 	})
 
 	ginkgo.AfterEach(func() {