Skip to content

fix: report unmatched RayJob deletion rules - #5221

Open
cananoo wants to merge 2 commits into
ray-project:masterfrom
cananoo:fix/rayjob-unmatched-deletion-rules
Open

fix: report unmatched RayJob deletion rules#5221
cananoo wants to merge 2 commits into
ray-project:masterfrom
cananoo:fix/rayjob-unmatched-deletion-rules

Conversation

@cananoo

@cananoo cananoo commented Sep 1, 2026

Copy link
Copy Markdown

Why are these changes needed?

When a terminal RayJob status matches no configured deletion rule, the controller currently logs the same success message as a fully processed rule set and takes no cleanup action. This can leave the RayCluster and RayJob resources behind without a visible signal, for example when activeDeadlineSeconds sets jobDeploymentStatus=Failed while jobStatus remains RUNNING.

This change:

  • tracks whether any deletion rule matched;
  • emits a Warning event with the terminal statuses when none matched;
  • writes a distinct log message for the no-match case; and
  • updates the deletion-rules sample to use jobDeploymentStatus: Failed for failure cleanup, covering infrastructure failures as well as application failures.

Deletion behavior is unchanged for matching rules.

Related issue number

Fixes #5184

Labels

  • If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
  • If this PR contains breaking changes, I have added the breaking-change label.

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

Validated locally:

  • go test ./controllers/ray -run '^TestHandleDeletionRulesReportsUnmatchedTerminalStatus$' -count=1
  • go vet ./controllers/ray/...
  • gofmt on changed Go files
  • git diff --check

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 16a2b48. Configure here.

jobStatus: FAILED
ttlSeconds: 90
jobDeploymentStatus: Failed
ttlSeconds: 90

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sample YAML misplaces ttlSeconds field

Medium Severity

On the Failed deletion rules, ttlSeconds is a sibling of condition rather than nested inside it. DeletionRule has no ttlSeconds field, so the API server drops those values and condition.ttlSeconds defaults to 0. Failed jobs then skip the documented 30/60/90-second staged cleanup and immediately run the most impactful overdue rule, DeleteSelf.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 16a2b48. Configure here.

@cananoo

cananoo commented Sep 1, 2026

Copy link
Copy Markdown
Author

Fixed in follow-up commit d5fbb7b: the Failed deletion-rule ttlSeconds values are now nested under condition, so the staged 30/60/90-second cleanup behavior in the sample is preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Terminal RayJob silently leaks its RayCluster forever when no deletionRule matches (e.g. DeadlineExceeded with jobStatus-keyed rules)

1 participant