Support for ROCm

[leopold218]

Hi!

I'm trying out your implementation which looks very promising for my use case. However, I'm locked to CPU inferencing as I have AMD cards (RDNA2 usinc ROCm).

Are you planning on supporting ROCm? If so, how can I help?

Thanks!

[normen]

This would indeed be interesting, I guess its mainly down to using the right torch versions, for https://github.com/devnen/Chatterbox-TTS-Server I can get it to work by installing the below as requirements.txt before installing the rest of the dependencies from the original requirements.txt (with those packages removed of course).

-i https://download.pytorch.org/whl/rocm6.1
torch==2.6.0
torchaudio==2.6.0
torchvision
pytorch_triton_rocm==3.1.0

This installs torch with rocm support and when you enable the cuda backend it should automatically use rocm..?

[Leobaillard]

I have tried your suggestion and managed to get an image to build. However, when launching it, I get an error saying No HIP gpus are available.

I think there might be other things missing in the image apart from the corresponding PIP packages.

I also had to remove the Cuda/Nvidia related docker-compose configuration for it to start. Maybe there is an AMD equivalent for the Docker container to be aware of the GPUs? I have seen some configuration "mounting" the DRI device into the container but I don't know if that would help.

[Poomex]

This would indeed be interesting, I guess its mainly down to using the right torch versions, for https://github.com/devnen/Chatterbox-TTS-Server I can get it to work by installing the below as requirements.txt before installing the rest of the dependencies from the original requirements.txt (with those packages removed of course).

-i https://download.pytorch.org/whl/rocm6.1
torch==2.6.0
torchaudio==2.6.0
torchvision
pytorch_triton_rocm==3.1.0

This installs torch with rocm support and when you enable the cuda backend it should automatically use rocm..?

When I try this solution I get this error:

ERROR: Could not find a version that satisfies the requirement torch==2.6.0 (from versions: none)
ERROR: No matching distribution found for torch==2.6.0

And when installing normally I get this:

ERROR: torch-2.6.0+rocm6.4.1.git1ded221d-cp310-cp310-linux_x86_64.whl is not a supported wheel >on this platform.
✗ Dependency installation failed

[normen]

This would indeed be interesting, I guess its mainly down to using the right torch versions, for https://github.com/devnen/Chatterbox-TTS-Server I can get it to work by installing the below as requirements.txt before installing the rest of the dependencies from the original requirements.txt (with those packages removed of course).

-i https://download.pytorch.org/whl/rocm6.1
torch==2.6.0
torchaudio==2.6.0
torchvision
pytorch_triton_rocm==3.1.0

This installs torch with rocm support and when you enable the cuda backend it should automatically use rocm..?

When I try this solution I get this error:

ERROR: Could not find a version that satisfies the requirement torch==2.6.0 (from versions: none)
ERROR: No matching distribution found for torch==2.6.0

That looks like you didn't add the -i https://download.pytorch.org/whl/rocm6.1 line at the top of the separate requirements.txt. If you did, idk why installing those libs wouldn't work.

[normen]

I have tried your suggestion and managed to get an image to build. However, when launching it, I get an error saying No HIP gpus are available.

I think there might be other things missing in the image apart from the corresponding PIP packages.

I also had to remove the Cuda/Nvidia related docker-compose configuration for it to start. Maybe there is an AMD equivalent for the Docker container to be aware of the GPUs? I have seen some configuration "mounting" the DRI device into the container but I don't know if that would help.

To make GPU access work in a docker container you'll definitely have to do a few steps on top of that, sure.