fix(scheduler): scope anti-thrash preemption cap per requester

[dndungu]

Summary

Fixes #79: a priorityClassName: high pod could stay pending
indefinitely behind a small pool of same-priority normal pods, with a
misleading evicting N candidate(s) event that read as "nothing more can
be preempted" instead of naming the real blocker. The anti-thrash
preemption cap in internal/scheduler/scheduler.go was keyed only by
victim pod name, so one pending pod's retries could exhaust a victim's
preemption budget and then silently starve every other, unrelated
high-priority pod that later needed to preempt that same victim, for up
to the 5-minute anti-thrash window, even though evicting the same victim
again would have satisfied the new pod. This scopes the cap to the
(victim, requester) pair instead, and makes the event message name the
cap explicitly when it is the actual blocker.

Design decision (T5.2) -- flagged for review

docs/plan.md's T5.2 explicitly required a real design call here, not a
mechanical patch, and instructed stopping to ask before picking a
resolution. AskUserQuestion was not available in this dispatched
teammate's toolset (functionally equivalent to hook-blocked -- no live
human to confirm with). Per the task's explicit fallback instruction, the
most conservative fix compatible with the existing anti-thrash regression
test (TestSchedule_AntiThrash) was chosen without live confirmation.
This specific call should get a human look before merge.

The bug: isAntiThrashed/recordPreemption tracked "has this pod
been preempted more than 3 times in 5 minutes" per victim only -- with no
notion of who did the preempting. Any pending pod's successful
preemption of a victim counted toward the same shared budget. On a
resource-constrained single-GPU node with few low-priority pods (the
issue's exact shape: 15 same-priority CI-runner/service pods), a burst of
legitimate preemption activity from a stream of different pods sharing
that small victim pool can exhaust it, after which any subsequent
high-priority pod finds zero preemption candidates and sits pending for
up to 5 minutes -- a denial-of-service on scheduling that has nothing to
do with resource availability.

The fix: scope the cap to the (victim, requester) pair. A pod that
itself keeps failing and re-triggering against the same victim is still
capped after 3 preemptions in 5 minutes -- ADR 005's original
misconfigured-priority flip-flop protection is unchanged, and
TestSchedule_AntiThrash still passes with its original intent intact
(same requester name reused across both halves of that test so the
assertion is still exercising window expiry, not pair-scoping). A
different, unrelated pod is no longer blocked by another pod's
thrashing.

Alternatives considered and rejected:

• Raise or remove the cap count/window globally. Delays the same bug
  rather than fixing it, and weakens thrash protection for the exact
  misconfigured-priority flip-flop scenario ADR 005 names as the reason
  the cap exists.
• "Last resort" override -- allow evicting anti-thrashed pods only
  when no other candidates exist. Rejected because it directly breaks
  TestSchedule_AntiThrash's existing invariant (single victim, single
  requester, already thrashed, sole candidate -- must stay Pending);
  this is precisely the flip-flop case the cap protects against.

Changes

• internal/scheduler/scheduler.go:
  • isAntiThrashed and recordPreemption now take a requester
    parameter and key the preemption record by (victim, requester) via
    a new preemptionKey helper, instead of by victim name alone.
  • Extracted antiThrashMaxPreemptions (3) and antiThrashWindow (5m)
    as named constants (previously inline magic numbers repeated in two
    places).
  • Both Pending reasons (no preemption candidates /
    preemption insufficient) now report how many lower-priority pods
    were excluded specifically by the anti-thrash cap, and say so
    explicitly, whenever that -- not raw resource availability -- is the
    actual blocker.
• internal/scheduler/scheduler_test.go:
  • Updated TestSchedule_AntiThrash's two recordPreemption calls to
    the new signature, both attributed to the same requester so the test
    still asserts the flip-flop/window-expiry behavior it always did.
  • Added TestSchedule_AntiThrashStarvesUnrelatedRequester (T5.1): red
    under the prior victim-only cap (a fresh requester was blocked by an
    unrelated pod's thrash history), green after the fix.
  • Added TestSchedule_PreemptionFairness_Issue79ExactScenario (T5.3):
    end-to-end regression matching the issue's exact shape --
    priorityClassName: high (100) vs normal (1000, default), 4
    lower-priority victims restarting immediately after each eviction,
    driven through the same Schedule/RemovePod/AddPod cycle the
    reconciler runs. Confirms the same pod is still capped on its 5th
    retry (with the cap named in the reason) while a different,
    unrelated high-priority pod scheduled moments later is not starved
    by it.

Testing

All run from the branch worktree, full repo:

• go build ./... -- clean.
• go vet ./... -- clean.
• staticcheck ./... -- clean, no findings.
• go test ./... -race -timeout 120s -count=1 -- all 13 packages pass
  (api, bus, cron, executor, gpu, housekeeper, lifecycle,
  manifest, metrics, reconciler, scheduler, state, watcher;
  cmd/spark has no tests). No regressions in the pre-existing
  internal/scheduler suite, including TestSchedule_AntiThrash and the
  other preemption tests (TestSchedule_PreemptLowPriority,
  TestSchedule_MultipleVictimsNeeded,
  TestSchedule_VictimSelectionPrefersRecentlyStarted,
  TestPreemptionCount).
• Verified the new TestSchedule_AntiThrashStarvesUnrelatedRequester
  failed against the pre-fix code (action=2 / Pending) before the fix
  landed, confirming it's a genuine red-then-green test, not a green
  test written after the fact.

Not yet done (belongs to the coordinator's centralized DGX pass per the
dispatch instructions for this task): live verification on the DGX with a
manifest reproducing the reported scenario, and closing issue #79. A
concrete 4+-pod repro recipe (manifests, expected before/after event
text, cleanup) is included in this task's handoff report to the
coordinator.

docs/plan.md acceptance criteria (E5, issue #79)

• T5.1 -- reproduce: TestSchedule_AntiThrashStarvesUnrelatedRequester
  shows a high-priority pod staying Pending forever under the pre-fix
  code despite 4 eligible lower-priority victims, confirming both the
  starvation and the misleading no preemption candidates message. Met.
• T5.2 -- fix without a preemption-storm regression:
  TestSchedule_AntiThrash (the pre-existing anti-thrash/flip-flop test)
  passes unchanged in intent; T5.1's test goes green. Met, with the
  design decision above flagged for review as instructed.
• T5.3 -- regression test using the exact issue scenario (cap of 3,
  4+ candidates, high-priority pod):
  TestSchedule_PreemptionFairness_Issue79ExactScenario. Met.
• T5.4 -- go vet ./... && staticcheck ./... && go test ./... -race -timeout 120s -count=1: all exit 0. Met.

Linked issues

Fixes #79 (preemption cap silently starves a high-priority pod behind
more than 3 lower-priority pods, with a misleading evicting N candidate(s) event message that doesn't disclose the cap).