gang-scheduling check aborts on a throttled pod read instead of retrying

[yuanchen8911]

Problem

waitForGangTestPods treats every error from its pod Get as a hard, non-retryable failure:

// validators/conformance/gang_scheduling_check.go
pod, err := clientset.CoreV1().Pods(run.namespace).Get(ctx, run.pods[i], metav1.GetOptions{})
if err != nil {
    return false, errors.Wrap(errors.ErrCodeInternal,
        fmt.Sprintf("failed to get gang test pod %s", run.pods[i]), err)
}

Returning a non-nil error from a wait.PollUntilContextCancel condition aborts the poll immediately. So a single throttled read ends the check, even though the very next poll interval would have succeeded.

On a loaded cluster that read gets throttled by client-go's own rate limiter, and the check fails with:

[TIMEOUT] gang test pods did not complete in time: [INTERNAL] failed to get gang test pod gang-worker-<id>-0:
client rate limiter Wait returned an error: context deadline exceeded

This is client-side throttling inside the validator, not a cluster fault.

Why it is a false negative

Observed on GB300 EKS while validating #2382. The check had already succeeded at everything it actually tests:

• both KAI CRDs present
• podgroup-controller and queue-controller both 1/1 Running
• the PodGroup and both test pods created successfully in gang-scheduling-test-<suffix>
• default-parent-queue / default-queue present

It then failed while polling. An identical re-run on the same cluster passed. Nothing about gang scheduling was broken — the check reported a healthy cluster as failing.

The fix already exists, it is just not wired here

The repo has a shared classifier for exactly this error. ClassifyK8sReadError maps context, apiserver, transport, and rate-limiter-deadline timeouts to ErrCodeTimeout, and helpers_test.go explicitly pins the case:

name: "client-go rate limiter deadline (plain-string chain)",
err:  fmt.Errorf("client rate limiter Wait returned an error: %w", ...)

But it is wired into only some read sites — dra_support_check.go (2), helpers.go (1), and the secure-access claim read. gang_scheduling_check.go does not reference it at all:

$ grep -c classifyK8sReadError validators/conformance/gang_scheduling_check.go
0

Suggested fix

Route the pod Get through classifyK8sReadError, and let the poll treat a classified timeout as retryable — return (false, nil) to poll again — so only the enclosing GangTestPodTimeout decides when to give up. Reserve a hard error for genuinely terminal cases.

Worth auditing the other polling loops in validators/conformance/ for the same shape, since the classifier exists but adoption is partial.

Scope

Not specific to any recipe. The check runs on 29 recipes across every accelerator and service, so the flake is cluster-load dependent rather than platform dependent, and gets likelier as the catalog and cluster size grow.

Related: #1513 (closed by #1514) is the same bug, one step earlier

#1513 — "gang-scheduling conformance check flakes on transient KAI admission unavailability" — hit this same check with the same underlying shape: a read that could not land treated as a terminal verdict.

There it was step 1, where an instantaneous getDeploymentIfAvailable failed the whole phase on a single-replica blip:

[NOT_FOUND] KAI scheduler component admission check failed:
[INTERNAL] deployment kai-scheduler/admission not available: 0/1 replicas

Its corrective action was to replace the instantaneous read with a bounded readiness wait — poll until available or hit the timeout, so a transient blip is tolerated while a genuinely-down deployment still fails closed after the bound.

This issue is the identical fix one step later in the same function. #1513 hardened step 1 (deployment readiness); the pod poll in waitForGangTestPods still aborts on the first failed read. The remedy has the same shape: let the enclosing timeout decide the verdict, not a single unlucky API call.

Two things follow from that:

• The acceptance criteria should mirror gang-scheduling conformance check flakes on transient KAI admission unavailability #1513's — tolerate the transient case, still fail closed past the bound, and unit-test the "succeeds after an initial failed read" path.
• It is worth checking whether fix(validator): wait for KAI deployments ready in gang-scheduling check #1514 left other reads in this file with the original shape, rather than fixing only the one that had been observed failing. The partial adoption of classifyK8sReadError noted above suggests it did.

Origin

Surfaced during GB300 EKS validation for #2382. Not blocking that PR — both leaves validate green, and the re-run passed.

[yuanchen8911]

Fix opened as #2407 (draft): routes the pod read through the existing classifyK8sReadError and retries the timeout forms, so GangTestPodTimeout decides the verdict instead of a single throttled call. Genuine errors still fail closed.

Both regression tests are verified against a control — reverting the fix reproduces the production failure verbatim.

The broader point from this issue, that classifyK8sReadError adoption across validators/conformance/ is partial, is deliberately not addressed in that PR. It fixes the site observed failing; auditing the remaining polling loops stays tracked here.

[mchmarny]

Triaged: Priority P2 → P1. Draft PR #2407 fixes a reproduced validator false negative caused by transient Kubernetes API throttling.

[yuanchen8911]

Follow-up audit inventory, verified while addressing #2407 review feedback:

• waitForHPAScalingIntent, waitForDeploymentScale, and waitForDeploymentScaleDown in pod_autoscaling_check.go
• waitForKarpenterNodes and verifyPodsScheduled in cluster_autoscaling_check.go
• waitForSlurmAccountingRecord in slinky_slurm_accounting_check.go

These loops currently retry every read or exec error with (false, nil), including terminal permission failures. They still fail at the enclosing timeout, so this is not a false-pass mechanism, but the terminal cause is delayed or degraded into a timeout diagnostic.

Those functions are outside #2407’s gang-scheduling path and are not changed there. This inventory should seed a separate focused follow-up after #2406 closes.