Skip to content

test: Add Training Hub RayJob E2E tests (RHOAIENG-63702, RHOAIENG-65214) #2988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pawelpaszki merged 1 commit into from
Jul 14, 2026
+89 −0
Merged

test: Add Training Hub RayJob E2E tests (RHOAIENG-63702, RHOAIENG-65214) #2988

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
${RAY_ROCM_IMAGE_3.11} quay.io/modh/ray@sha256:28a8745be454b0e881ce6c200599ddfcb3366b707a5b53cfa73087d599555158
# Corresponds to quay.io/rhoai/ray:2.35.0-py311-rocm61-torch24-fa26
${RAY_TORCH_ROCM_IMAGE_3.11} quay.io/rhoai/ray@sha256:b0e129cd2f4cdea7ad7a7859031357ffd9915410551f94fbcb942af2198cdf78
# Corresponds to quay.io/modh/ray:2.55.1-py312-cu129-th081
${RAY_TRAINING_HUB_IMAGE} quay.io/modh/ray@sha256:f63a015302758f805e5332605669b886e4d7ac60ec929413a2ffc19a904211c6
Comment thread
Fiona-Waters marked this conversation as resolved.
Dismissed
# Corresponds to quay.io/modh/training:py311-cuda121-torch241
${CUDA_TRAINING_IMAGE_TORCH241} quay.io/modh/training@sha256:f64f7bba3f1020d39491ac84d40d362a52e4822bdc11a33cfff021178b7c4097
# Corresponds to quay.io/modh/training:py311-rocm62-torch241
Expand All @@ -32,6 +34,8 @@
${ROCM_TRAINING_IMAGE_TORCH251} quay.io/modh/training@sha256:6cdae840fa029da33cccab620367e82404d24ddf67762eb4537a9bffe1af306d
# Corresponds to quay.io/repository/modh/odh-workbench-jupyter-datascience-cpu-py311-ubi9:rhoai-2.22
${NOTEBOOK_IMAGE_3.11} quay.io/modh/odh-workbench-jupyter-datascience-cpu-py311-ubi9@sha256:48c2be818e8d2a5005a69e2c76f9a2d40ddeb1d03376e04516ca6da13418c887
# Corresponds to registry.redhat.io/rhoai/odh-workbench-jupyter-datascience-cpu-py312-rhel9:3.4
${NOTEBOOK_IMAGE_3.12} registry.redhat.io/rhoai/odh-workbench-jupyter-datascience-cpu-py312-rhel9@sha256:d07dacbceeaa335854c3aed2d93e149c122a938abf59497c8235c751dae755cb
Comment thread
Fiona-Waters marked this conversation as resolved.
Dismissed
${NOTEBOOK_USER_NAME} ${TEST_USER_3.USERNAME}
${NOTEBOOK_USER_PASSWORD} ${TEST_USER_3.PASSWORD}
${KFTO_BINARY_NAME} kfto
Expand All @@ -43,6 +47,7 @@
${AWS_ACCESS_KEY_ID} ${S3.AWS_ACCESS_KEY_ID}
${AWS_SECRET_ACCESS_KEY} ${S3.AWS_SECRET_ACCESS_KEY}
${AWS_STORAGE_BUCKET_MNIST_DIR} mnist-datasets
${AWS_STORAGE_BUCKET_RAY_TRAINING_HUB_DIR} ray-training-hub
${KUBECONFIGPATH} %{HOME}/.kube/config


Expand Down Expand Up @@ -285,8 +290,8 @@
... env:AWS_DEFAULT_ENDPOINT=${AWS_DEFAULT_ENDPOINT}
... env:AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
... env:AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
... env:AWS_STORAGE_BUCKET=${AWS_STORAGE_BUCKET}

Check warning

Code scanning / Robocop

Keyword name '{{ keyword_name }}' does not follow case convention Warning test

Keyword name 'Check missing Go test' does not follow case convention
... env:AWS_STORAGE_BUCKET_MNIST_DIR=${AWS_STORAGE_BUCKET_MNIST_DIR}

Check notice

Code scanning / Robocop

IF can be replaced with inline IF Note test

IF can be replaced with inline IF

Check notice

Code scanning / Robocop

'{{ block_name }}' condition can be simplified Note test

'IF' condition can be simplified
... env:PIP_INDEX_URL=${PIP_INDEX_URL}
... env:PIP_TRUSTED_HOST=${PIP_TRUSTED_HOST}
Log To Console ${result.stdout}
Expand All @@ -295,6 +300,40 @@
FAIL ${TEST_NAME} failed
END

Run DistributedWorkloads Training Hub ODH Test
[Documentation] Run DistributedWorkloads Training Hub ODH Test
[Arguments] ${TEST_NAME} ${DW_RAY_IMAGE} ${NOTEBOOK_IMAGE}

Check warning

Code scanning / Robocop

{{ var_or_arg }} '{{ variable_name }}' overwrites reserved variable '{{ reserved_variable }}' Warning test

Argument '${TEST_NAME}' overwrites reserved variable '${TEST_NAME}'
Log To Console "Running Training Hub test: ${TEST_NAME}"
${result} = Run Process ./${ODH_BINARY_NAME} -test.run ${TEST_NAME}
... shell=true
... stderr=STDOUT
... env:TEST_TIER=Examples-CUDA
... env:TEST_TIMEOUT_SHORT=5m
... env:TEST_TIMEOUT_MEDIUM=15m
... env:TEST_TIMEOUT_LONG=30m
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
... env:TEST_TIMEOUT_GPU_PROVISIONING=40m
... env:TEST_OUTPUT_DIR=%{WORKSPACE}/distributed-workloads-odh-logs
... env:TEST_RAY_IMAGE=${DW_RAY_IMAGE}
... env:TEST_RAY_TRAINING_HUB_IMAGE=${DW_RAY_IMAGE}
... env:ODH_NAMESPACE=${APPLICATIONS_NAMESPACE}
... env:NOTEBOOK_USER_NAME=${NOTEBOOK_USER_NAME}
... env:NOTEBOOK_USER_TOKEN=${NOTEBOOK_USER_TOKEN}
... env:NOTEBOOK_IMAGE=${NOTEBOOK_IMAGE}
... env:AWS_DEFAULT_ENDPOINT=${AWS_DEFAULT_ENDPOINT}
... env:AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
... env:AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
... env:AWS_STORAGE_BUCKET=${AWS_STORAGE_BUCKET}
... env:AWS_STORAGE_BUCKET_MNIST_DIR=${AWS_STORAGE_BUCKET_MNIST_DIR}
... env:AWS_STORAGE_BUCKET_RAY_TRAINING_HUB_DIR=${AWS_STORAGE_BUCKET_RAY_TRAINING_HUB_DIR}
... env:PIP_INDEX_URL=${PIP_INDEX_URL}
... env:PIP_TRUSTED_HOST=${PIP_TRUSTED_HOST}
... timeout=45 min
Log To Console ${result.stdout}
Check missing Go test ${result.stdout}
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
IF ${result.rc} != 0
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
FAIL ${TEST_NAME} failed
END

Check missing Go test
[Documentation] Check that upstream Go test is not missing
[Arguments] ${test_run_output}
Expand Down
50 changes: 50 additions & 0 deletions ods_ci/tests/Tests/0600__distributed_workloads/test-run-training-hub-rayjob-tests.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
*** Settings ***
Documentation Training Hub RayJob Integration tests - https://github.com/opendatahub-io/distributed-workloads/tree/main/tests/odh
Suite Setup Prepare DistributedWorkloads Integration Test Suite
Suite Teardown Teardown DistributedWorkloads Integration Test Suite
Library OperatingSystem
Library Process
Resource ../../../tasks/Resources/RHODS_OLM/install/oc_install.robot
Resource ../../Resources/Page/DistributedWorkloads/DistributedWorkloads.resource


*** Test Cases ***
Run TestSftRayJobSingleGpu ODH test
[Documentation] Run Go ODH test: TestSftRayJobSingleGpu
[Tags] RHOAIENG-65214
... Resources-GPU NVIDIA-GPUs
... Tier1
... DistributedWorkloads
... TrainingRay
... TrainingHub
Run DistributedWorkloads Training Hub ODH Test TestSftRayJobSingleGpu ${RAY_TRAINING_HUB_IMAGE} ${NOTEBOOK_IMAGE_3.12}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (131/120)

Run TestOsftRayJobSingleGpu ODH test
[Documentation] Run Go ODH test: TestOsftRayJobSingleGpu
[Tags] RHOAIENG-65214
... Resources-GPU NVIDIA-GPUs
... Tier1
... DistributedWorkloads
... TrainingRay
... TrainingHub
Run DistributedWorkloads Training Hub ODH Test TestOsftRayJobSingleGpu ${RAY_TRAINING_HUB_IMAGE} ${NOTEBOOK_IMAGE_3.12}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (132/120)

Run TestLoraRayJobSingleGpu ODH test
[Documentation] Run Go ODH test: TestLoraRayJobSingleGpu
[Tags] RHOAIENG-65214
... Resources-GPU NVIDIA-GPUs
... Tier1
... DistributedWorkloads
... TrainingRay
... TrainingHub
Run DistributedWorkloads Training Hub ODH Test TestLoraRayJobSingleGpu ${RAY_TRAINING_HUB_IMAGE} ${NOTEBOOK_IMAGE_3.12}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (132/120)

Run TestGrpoRayJobSingleNodeMultiGpu ODH test
[Documentation] Run Go ODH test: TestGrpoRayJobSingleNodeMultiGpu
[Tags] RHOAIENG-63702
... Resources-GPU NVIDIA-GPUs
... Tier1
... DistributedWorkloads
... TrainingRay
... TrainingHub
Run DistributedWorkloads Training Hub ODH Test TestGrpoRayJobSingleNodeMultiGpu ${RAY_TRAINING_HUB_IMAGE} ${NOTEBOOK_IMAGE_3.12}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (141/120)
Loading