[Feature][RayJob] Add BlockingRecreate retry strategy and execution-attempt metadata

[TimothySeah]

Search before asking

• I searched existing KubeRay issues and did not find an equivalent feature request.

Description

RayJob supports full-job retries through spec.backoffLimit. Each retry creates a new RayCluster. However, the current retry path waits for the previous RayCluster custom resource to be deleted, but does not guarantee that all head, worker, and submitter Pods from the previous execution attempt have terminated before starting the next attempt.

This can allow Pods from two RayJob execution attempts to overlap.

I propose adding a BlockingRecreate retry strategy, similar to Kubernetes JobSet’s BlockingRecreate:

Delete all workload resources from the previous execution attempt and wait for all previous Pods to be gone before creating or starting resources for the next attempt.

The feature should also expose a monotonically increasing execution-attempt number as RayJob status and Kubernetes metadata. Applications could use this metadata for logging, checkpoint selection, output namespacing, or application-level write fencing.

Current behavior

A RayJob configured with:

spec:
  backoffLimit: 3

may transition through:

Running → Retrying → New → Initializing

The retry path deletes the previous RayCluster and submitter before creating the next RayCluster. However, deletion of a Kubernetes owner resource does not necessarily mean that all owned Pods have completed termination.

The existing RayJob retry controller test verifies that the previous RayCluster CR and submitter Job become NotFound, but does not verify a zero-Pod barrier before the next RayCluster is created.

There is a related problem when a head/GCS loss causes a job to be submitted again in the same RayCluster. If a resubmission represents a new execution of the Ray entrypoint, surviving workers or actors from the previous execution may overlap with the new execution.

For applications that write to external systems such as databases or object stores, this overlap can result in stale writers from the previous attempt mutating the same logical workflow as the replacement attempt.

Proposed API

One possible API is:

apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: example
spec:
  backoffLimit: 3
  retryStrategy:
    type: BlockingRecreate

Possible values:

• Recreate: current behavior and the default, preserving backward compatibility.
• BlockingRecreate: do not start the next execution attempt until all Kubernetes workload resources from the previous attempt have terminated.

The exact field name is open for discussion. restartStrategy would align with JobSet, while retryStrategy may fit RayJob’s existing backoffLimit terminology more naturally.

Proposed BlockingRecreate semantics

For RayJobs that own their RayCluster through spec.rayClusterSpec:

1. The RayJob enters Retrying.
2. KubeRay stops creating or submitting resources for the next attempt.
3. KubeRay deletes the previous submitter and RayCluster.
4. KubeRay waits until all previous-attempt resources are gone, including:
   • the submitter Job and its Pods;
   • the Ray head Pod;
   • every Ray worker Pod.
5. Only after the barrier completes does KubeRay increment the execution-attempt number.
6. KubeRay creates the next RayCluster and submits the Ray entrypoint.
7. All resources belonging to the new execution receive the same attempt metadata.

Conceptually:

Attempt N head/workers
          ↓ delete
All Attempt N Pods terminated
          ↓ barrier satisfied
Attempt N+1 head/workers created and started

If an old Pod remains in Terminating, the next attempt must remain blocked. BlockingRecreate should not silently degrade to overlapping recreation.

Foreground deletion could be part of the implementation, but the controller should explicitly verify that no previous-attempt Pods remain rather than relying only on deletion of the RayCluster CR.

Head-loss and resubmission behavior

If KubeRay determines that head/GCS loss requires resubmitting the Ray entrypoint, that resubmission should be treated as a new execution attempt when BlockingRecreate is enabled.

In that case, the controller should perform a full-cluster blocking recreation rather than resubmitting the entrypoint into a cluster that may still contain workers from the previous execution.

This should not interfere with successful GCS fault-tolerant recovery. If the existing Ray execution is recovered without resubmitting the entrypoint, no new execution attempt is required.

Execution-attempt metadata

Add a zero-based, monotonically increasing status field:

status:
  executionAttempt: 0

Suggested semantics:

• 0: initial execution;
• 1: first full RayJob retry or resubmission;
• 2: second full retry;
• and so on.

The counter must advance exactly once per full execution attempt and remain idempotent across operator restarts and repeated reconciliation.

Submitter-only retries controlled by spec.submitterConfig.backoffLimit should not increment this counter, because they do not represent a new execution of the Ray application.

KubeRay should also stamp the execution attempt onto resources belonging to the attempt, for example:

metadata:
  labels:
    ray.io/execution-attempt: "1"
  annotations:
    ray.io/execution-attempt: "1"

At minimum, this metadata should be propagated to:

• the RayCluster;
• the submitter Job and Pod;
• the head Pod;
• every worker Pod.

Pod-level metadata allows applications to consume the attempt number through the Kubernetes Downward API without requiring a KubeRay-specific API client.

clusterSelector

BlockingRecreate should only be supported for RayJobs whose RayCluster is owned by the RayJob.

A RayJob using spec.clusterSelector must not delete or recreate the selected external RayCluster. The combination of clusterSelector and retryStrategy.type: BlockingRecreate should therefore be rejected during validation.

This is consistent with the ownership concern described in #4516.

Interaction with suspension

If a RayJob becomes suspended while waiting for the blocking deletion barrier:

• the controller must not create the replacement attempt;
• the attempt counter must not advance until the job is resumed and a new execution is actually created;
• resuming should still wait for the previous attempt’s resources to be gone.

Related cleanup work may exist in #4876.

Backward compatibility

• The current retry behavior remains the default.
• Existing RayJobs without retryStrategy are unchanged.
• Existing backoffLimit semantics remain unchanged except when BlockingRecreate is explicitly selected.
• The initial execution attempt is 0.
• Attempt metadata is additive.

Acceptance criteria

• The RayJob API has a typed, validated blocking retry strategy.
• Existing RayJobs retain current behavior by default.
• No next-attempt RayCluster or submitter is started while a previous-attempt head, worker, or submitter Pod still exists.
• A Pod stuck in Terminating keeps the next attempt blocked.
• A required head-loss resubmission uses the same blocking barrier.
• The execution-attempt counter increments exactly once per full application execution.
• Submitter-only retries do not increment the execution-attempt counter.
• All resources from one execution receive identical attempt metadata.
• clusterSelector plus BlockingRecreate is rejected.
• Suspending during the deletion barrier does not accidentally create another attempt.
• Unit tests verify idempotence across repeated reconciliations.
• An end-to-end test verifies that Pods from two execution attempts never overlap.

Non-goals

This feature only guarantees ordering of resources observable through Kubernetes. It cannot guarantee that a process continues nowhere after its Pod object has disappeared—for example, a process running on a partitioned or force-deleted node.

Applications performing non-idempotent external writes may still require application-level attempt fencing. Execution-attempt metadata is intended to make that possible, but datastore fencing is outside KubeRay’s scope.

This proposal also does not change ordinary worker autoscaling or treat every individual worker replacement as a new RayJob execution attempt.

Use case

I run restartable Ray applications whose replacement attempts resume the same logical workflow and write to shared external storage.

I need:

1. a clean infrastructure boundary preventing normal overlap between old and replacement attempts; and
2. a stable execution-attempt identifier that the application can use for observability and stale-write protection.

Without a blocking recreation barrier, the replacement attempt can begin while previous-attempt workers are still terminating. Without attempt metadata, applications cannot reliably distinguish writes produced by different RayJob executions.

Related issues and prior art

• JobSet BlockingRecreate
• RayJob retry and clusterSelector ownership: #4516
• Ensure RayJob-owned resources are deleted during suspension: #4876
• GCS fault tolerance for RayJob-owned clusters: #4025
• RayJob API reference

Are you willing to submit a PR?

Possibly. I may be willing to contribute an implementation, but this is currently a lower-priority item for me. I would appreciate maintainer feedback on the API and scope before committing to a PR.