Prevent terminating launcher pods from being recreated

[GonzaloLuminary]

Avoid running into kubernetes/kubernetes#115844 where pods can be recreated while the launcher pod is terminating resulting in MPIJob duplicating work when using runLauncherAsWorker

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign terrytangyuan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[GonzaloLuminary]

cc: @tenzen-y

[tenzen-y]

@GonzaloLuminary Thank you for working on this PR!
Can you also address CI failures?

[tenzen-y]

I agree with introducing the podReplacementPolicy: "Failed" motivation, but why do we need to specify an empty PodFailurePolicy? I don't see any reason.

Suggested change

PodFailurePolicy: &batchv1.PodFailurePolicy{},

[GonzaloLuminary]

Depending on the k8s version and the feature flags being active, setting the PodReplacementPolicy may not be enough. As per the original PR that addressed the issue kubernetes/kubernetes#117015, the condition to activate the fix was controlled by this function

func onlyReplaceFailedPods(job *batch.Job) bool {
	if feature.DefaultFeatureGate.Enabled(features.JobPodReplacementPolicy) && *job.Spec.PodReplacementPolicy == batch.Failed {
		return true
	}
	return feature.DefaultFeatureGate.Enabled(features.JobPodFailurePolicy) && job.Spec.PodFailurePolicy != nil
}

In master the same function reads

func onlyReplaceFailedPods(job *batch.Job) bool {
	return job.Spec.PodReplacementPolicy != nil && *job.Spec.PodReplacementPolicy == batch.Failed
}

It all depends which k8s versions/configurations we want to support. I'm happy to drop support for old clusters with JobPodReplacementPolicy not being active.