Investigate & escalate slow Job completion (kube-controller-manager)

[FolarinOyenuga]

Background

The Job controller (kube-controller-manager) is taking 100+s to reconcile a
finished Job. The test-side symptom is fixed (#8440, decoupled to pod
completion), but the underlying control-plane lag remains and affects every
Job/CronJob on the cluster, including tenant workloads. This is the "other half"
referenced in #8440.

Evidence (verified on live)

• Controlled repro: a single-pod Job exiting in <1s took 100s+ to be marked
  Complete, the Succeeded pod held the batch.kubernetes.io/job-tracking
  finalizer ~100s+ before the controller removed it and set succeeded:1.
• Correlates with the EKS 1.34 bump (2026-07-23), jobs completed sub-second
  before.
• Controller-side, not admission: label-pods webhook responds 45ms
  (failurePolicy: Ignore).

Approach

• Review kube-controller-manager logs in CloudWatch (control-plane logging
  controllerManager is already enabled) around Job completion.
• Rule out self-inflicted causes: APF throttling of the job-controller SA, and,
  since this is standard EKS (not Auto Mode), self-managed add-ons bumped around
  1.34 (not entirely sure on which just yet, will need to find these out if necessary).
• If not self-inflicted, raise an AWS Support case with the single-pod repro and track the fix.