test: Add Training Hub RayJob E2E tests (RHOAIENG-63702, RHOAIENG-65214)

[Fiona-Waters]

Summary

Add ods-ci integration for Training Hub RayJob E2E tests covering SFT, OSFT, LoRA, and GRPO algorithms on Ray. These tests depend on the corresponding Go tests in distributed-workloads#851.

Changes

• Add 4 Robot test cases: TestSftRayJobSingleGpu, TestOsftRayJobSingleGpu, TestLoraRayJobSingleGpu, TestGrpoRayJobSingleNodeMultiGpu
• Add Run DistributedWorkloads Training Hub ODH Test keyword with TEST_TIER=Examples-CUDA, Training Hub image, S3 prefix, and extended GPU timeouts
• Add RAY_TRAINING_HUB_IMAGE and AWS_STORAGE_BUCKET_RAY_TRAINING_HUB_DIR variables

Dependencies

• distributed-workloads#851 must be merged first so the release binary includes the Training Hub tests
• S3 bucket needs Qwen2.5-1.5B-Instruct model and GRPO dataset subset pre-staged under ray-training-hub/ prefix for disconnected environments

Test Plan

• Run ./run_robot_test.sh --include TrainingHub --skip-oclogin true on a cluster with NVIDIA GPUs
• Verify all 4 tests pass using a locally-built odh binary from the DW branch

Made with Cursor

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds Training Hub-specific distributed workload configuration and a reusable ODH test runner keyword, then introduces four Robot Framework RayJob scenarios covering SFT, OSFT, LoRA, and GRPO configurations.

Changes

Training Hub RayJob testing

Layer / File(s)	Summary
Training Hub runner configuration ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource	Adds Training Hub Ray and notebook image variables, a dedicated S3 directory, and a keyword that runs the ODH binary with Training Hub environment variables and extended GPU provisioning timeout handling.
Training Hub RayJob scenarios ods_ci/tests/Tests/0600__distributed_workloads/test-run-training-hub-rayjob-tests.robot	Adds suite configuration and four tagged tests for SFT, OSFT, LoRA, and GRPO workloads.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RobotSuite
  participant DistributedWorkloadsResource
  participant ODHBinary
  RobotSuite->>DistributedWorkloadsResource: Run Training Hub ODH test
  DistributedWorkloadsResource->>ODHBinary: Execute scenario with Training Hub environment
  ODHBinary-->>DistributedWorkloadsResource: Return exit code
  DistributedWorkloadsResource-->>RobotSuite: Pass or fail test

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding Training Hub RayJob E2E tests.
Description check	✅ Passed	The description matches the changeset by describing the new RayJob test cases, keyword, and supporting variables.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[pawelpaszki]

you would need a separate file, e.g. if you add your tests to the existing 0600... file they will all be skipped. so you need to copy the setup from that file and create a new one in that directory, so that your tests are executed.

Also note that ray image is outdated, because we haven't used these tests lately. can you make sure that the Ray image is the latest one please?

[Fiona-Waters]

you would need a separate file, e.g. if you add your tests to the existing 0600... file they will all be skipped. so you need to copy the setup from that file and create a new one in that directory, so that your tests are executed.

Also note that ray image is outdated, because we haven't used these tests lately. can you make sure that the Ray image is the latest one please?

Thanks for review @pawelpaszki I've updated to add a separate file. The image being used is the new training hub flavour. PTAL, thank you

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[coderabbitai]

🧹 Nitpick comments (1)

ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource (1)

305-334: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Rename the ${TEST_NAME} argument to avoid overwriting a reserved variable.

Robocop static analysis flags that the argument ${TEST_NAME} overwrites the Robot Framework reserved variable containing the current test case name. Renaming this argument prevents potential side effects and clears the linting warning.

♻️ Proposed refactor to rename the argument

-    [Arguments]    ${TEST_NAME}    ${DW_RAY_IMAGE}    ${NOTEBOOK_IMAGE}
-    Log To Console    "Running Training Hub test: ${TEST_NAME}"
-    ${result} =    Run Process    ./${ODH_BINARY_NAME} -test.run ${TEST_NAME}
+    [Arguments]    ${test_name_arg}    ${DW_RAY_IMAGE}    ${NOTEBOOK_IMAGE}
+    Log To Console    "Running Training Hub test: ${test_name_arg}"
+    ${result} =    Run Process    ./${ODH_BINARY_NAME} -test.run ${test_name_arg}
     ...    shell=true
     ...    stderr=STDOUT
     ...    env:TEST_TIER=Examples-CUDA
     ...    env:TEST_TIMEOUT_SHORT=5m
     ...    env:TEST_TIMEOUT_MEDIUM=15m
     ...    env:TEST_TIMEOUT_LONG=30m
     ...    env:TEST_TIMEOUT_GPU_PROVISIONING=40m
     ...    env:TEST_OUTPUT_DIR=%{WORKSPACE}/distributed-workloads-odh-logs
     ...    env:TEST_RAY_IMAGE=${DW_RAY_IMAGE}
     ...    env:TEST_RAY_TRAINING_HUB_IMAGE=${DW_RAY_IMAGE}
     ...    env:ODH_NAMESPACE=${APPLICATIONS_NAMESPACE}
     ...    env:NOTEBOOK_USER_NAME=${NOTEBOOK_USER_NAME}
     ...    env:NOTEBOOK_USER_TOKEN=${NOTEBOOK_USER_TOKEN}
     ...    env:NOTEBOOK_IMAGE=${NOTEBOOK_IMAGE}
     ...    env:AWS_DEFAULT_ENDPOINT=${AWS_DEFAULT_ENDPOINT}
     ...    env:AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
     ...    env:AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
     ...    env:AWS_STORAGE_BUCKET=${AWS_STORAGE_BUCKET}
     ...    env:AWS_STORAGE_BUCKET_MNIST_DIR=${AWS_STORAGE_BUCKET_MNIST_DIR}
     ...    env:AWS_STORAGE_BUCKET_RAY_TRAINING_HUB_DIR=${AWS_STORAGE_BUCKET_RAY_TRAINING_HUB_DIR}
     ...    env:PIP_INDEX_URL=${PIP_INDEX_URL}
     ...    env:PIP_TRUSTED_HOST=${PIP_TRUSTED_HOST}
     ...    timeout=45 min
     Log To Console    ${result.stdout}
     Check missing Go test    ${result.stdout}
     IF    ${result.rc} != 0
-        FAIL    ${TEST_NAME} failed
+        FAIL    ${test_name_arg} failed
     END

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource`
around lines 305 - 334, Rename the `${TEST_NAME}` argument in the distributed
workload keyword to a non-reserved name, then update its uses in the console
log, `Run Process` invocation, and failure message. Keep the test execution
behavior unchanged while avoiding the Robot Framework reserved current-test-name
variable.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource`:
- Around line 305-334: Rename the `${TEST_NAME}` argument in the distributed
workload keyword to a non-reserved name, then update its uses in the console
log, `Run Process` invocation, and failure message. Keep the test execution
behavior unchanged while avoiding the Robot Framework reserved current-test-name
variable.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 7f54723a-826b-48bf-a27a-c4b89da23c99

📥 Commits

Reviewing files that changed from the base of the PR and between 3111841 and d3053af.

📒 Files selected for processing (2)

• ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource
• ods_ci/tests/Tests/0600__distributed_workloads/test-run-training-hub-rayjob-tests.robot

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Fiona-Waters, pawelpaszki, sutaakar

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pawelpaszki]

/lgtm