Add KFTO Distributed training notebook and scripts

[sutaakar]

No description provided.

[ChughShilpa]

/lgtm

[abhijeet-dhumal]

/lgtm

[RHRolun]

Awesome PR!

A few things besides the error I got:

• Can you also add a draft for the documentation for this chapter?
• This currently relies on the PyTorch notebook image, while the rest of the workshop relies on the TensorFlow image. To unify this we may need to change the workshop to use pytorch instead (I believe that KFTO is pytorch only?). We luckily already have a draft PR for that here, but it needs to go through some work to get merged in and synched with docs. Nothing needed from this PR, just that it may take a little extra time to get it merged.
• All the parts of this workshop should be runnable in the sandbox, this seems like it should work just based on how few resources it uses (and I believe KFTO is enabled by default in RHOAI now). I haven't tested it yet as I hit the error, so just a heads up in case that causes some more needed changes.

Thanks! :)

[RHRolun]

When running this with a PyTorch notebook image version 2025.1 I get this error:
TypeError: TrainingClient.create_job() got an unexpected keyword argument 'labels'

Should a different image be used?

[abhijeet-dhumal]

Hi @RHRolun , Actually the labels parameter is supported in Kubeflow-training SDK latest version 1.9.2

[abhijeet-dhumal]

@sutaakar
IMO this note is needed here, WDYT? :

Note : This workshop requires Red Hat OpenShift AI v2.21+ or Kubeflow Training SDK v1.9.2+

[sutaakar]

AFAIK it will require latest KFTO SDK, which should be available in next RHOAI release - 2.22
You can try to manually update KFTO SDK (and restart kernel), which should fix it - pip install -U kubeflow-training

Considering that the PR will get merged after a while, 2.22 should be out at that time, so I think the SDK upgrade doesn't need to be included?

[abhijeet-dhumal]

JFI : The Kubeflow-training SDK v1.9.2 is also available in RHOAI release - 2.21

[sutaakar]

Another option is to remove the labels completely as it is needed for Kueue integration.
I will most likely comment it out, adding a note to enable it for environments with Kueue.

[sutaakar]

Sure, will add a documentation draft.

[sutaakar]

Currently the RHOAI training operator supports only PyTorch, so the example requires PyTorch workbench image.

[sutaakar]

@RHRolun Made some minor adjustments and added documentation.
I think the Kueue section would be also relevant to Ray training.

Will test the script on Sandbox as soon as training operator component gets enabled there.

[Fiona-Waters]

Just a few small nits. Thanks @sutaakar

[sutaakar]

@Fiona-Waters thanks

[Fiona-Waters]

/lgtm

[sutaakar]

@RHRolun The example was updated to be able to run on sandbox (needed to reduce CPUs for training workers).
It should work now.

[RHRolun]

Awesome! :)

A couple of things:
It requires the python dependency kubeflow-training==1.9.0 which is not in all notebook images (for example the tensorflow image that's currently used in the rest of fraud detection).
If we can add either a sentence about it needing to use the PyTorch image or have a pip install in the notebook then it looks good to me (I would lean towards the second option so that it's easier for the users until we get the PyTorch branch merged).

When trying to uncomment the labels line in the send job section I got this:
TypeError: TrainingClient.create_job() got an unexpected keyword argument 'labels'

Also, just out of curiosity, right now all the logs gets posted as a chunk when the job is finished. Is there a way to stream them into the notebook instead?

[sutaakar]

It requires the python dependency kubeflow-training==1.9.0 which is not in all notebook images (for example the tensorflow image that's currently used in the rest of fraud detection).
If we can add either a sentence about it needing to use the PyTorch image or have a pip install in the notebook then it looks good to me (I would lean towards the second option so that it's easier for the users until we get the PyTorch branch merged).

Added pip install to install latest KFTO SDK.

When trying to uncomment the labels line in the send job section I got this:
TypeError: TrainingClient.create_job() got an unexpected keyword argument 'labels'

This should be addressed by installing the latest KFTO SDK.

Also, just out of curiosity, right now all the logs gets posted as a chunk when the job is finished. Is there a way to stream them into the notebook instead?

Unfortunately streaming logs doesn't seem to be reliable enough - stream gets stuck. It may be better to keep current approach now until the issue is fixed.

[MelissaFlinn]

I have some doc edits which I will make in a separate PR.