Building without GPU available

[filiptibell]

Hi! Thank you for this great library.

I am wondering how we could build without a GPU available (such as in GitHub Actions)? I found CUDA_COMPUTE_CAP in the source code, and we are targeting a T4 GPU, so setting that to 75 should be correct and seems to have worked. However, I still hit this and panic:

bindgen_cuda/src/lib.rs

Line 519 in 348e73f

let max_nvcc_code = *codes.last().expect("no gpu codes parsed from nvcc");

Would it be possible to also manually specify gpu codes?
Would this actually work & build bindings correctly without a GPU?

[andrenatal]

I'm having the same issue. In my case, I can't build inside a docker image because gpus and nvidia-smi are not available at build time, even with a gpu present at the host, it seems.

If I start the container, then nvidia-smi works. My base image is: FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04

#14 34.60 Caused by:
#14 34.60   process didn't exit successfully: `/app/stt/target/release/build/candle-kernels-1f44ba2b45c1de35/build-script-build` (exit status: 101)
#14 34.60   --- stdout
#14 34.60   cargo:rerun-if-changed=build.rs
#14 34.60   cargo:rerun-if-changed=src/compatibility.cuh
#14 34.60   cargo:rerun-if-changed=src/cuda_utils.cuh
#14 34.60   cargo:rerun-if-changed=src/binary_op_macros.cuh
#14 34.60   cargo:info=["/usr", "/usr/local/cuda", "/opt/cuda", "/usr/lib/cuda", "C:/Program Files/NVIDIA GPU Computing Toolkit", "C:/CUDA"]
#14 34.60   cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP
#14 34.60
#14 34.60   --- stderr
#14 34.60   thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen_cuda-0.1.5/src/lib.rs:489:18:
#14 34.60   `nvidia-smi` failed. Ensure that you have CUDA installed and that `nvidia-smi` is in your PATH.: Os { code: 2, kind: NotFound, message: "No such file or directory" }
#14 34.60   stack backtrace:
#14 34.60      0: rust_begin_unwind
#14 34.60      1: core::panicking::panic_fmt
#14 34.60      2: core::result::unwrap_failed
#14 34.60      3: core::result::Result<T,E>::expect
#14 34.60      4: bindgen_cuda::compute_cap
#14 34.60      5: <bindgen_cuda::Builder as core::default::Default>::default
#14 34.60      6: build_script_build::main
#14 34.60      7: core::ops::function::FnOnce::call_once
#14 34.60   note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
#14 34.60 warning: build failed, waiting for other jobs to finish...
#14 41.76 error: failed to compile `stt-server v0.1.0 (/app/stt)`, intermediate artifacts can be found at `/app/stt/target`.
#14 41.76 To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
#14 ERROR: process "/bin/sh -c ~/.cargo/bin/cargo install --path ." did not complete successfully: exit code: 101

[sidharthrajaram]

Facing the same issue as above ^

[andrenatal]

This is related: huggingface/candle#2105

[sidharthrajaram]

A workaround for the buildtime nvidia-smi call seems to be setting CUDA_COMPUTE_CAP to the compute capability value based on your GPU.

Based on: huggingface/candle#1516 (comment)

[polarathene]

I'm having the same issue. In my case, I can't build inside a docker image because gpus and nvidia-smi are not available at build time, even with a gpu present at the host, it seems.

FWIW, this should be supported now I think (I haven't verified), provided you can use the CDI method of adding a GPU to a container at runtime, there is an equivalent in Dockerfile for build-time via RUN --device, which might also make nvidia-smi available.

Not as viable for CI though, and technically if publishing an image you'd want to support a broader baseline if you've got the latest GPU, or publish for multiple GPU archs when that makes sense.

[andrenatal]

Thanks @polarathene, I'll take a look and test on my pipelines.

[andrenatal]

@polarathene I tried to run this:

RUN --device=nvidia.com/gpu=all cargo install --path .

But it seems to not be widely available yet:?

ERROR: failed to solve: device feature is only supported in Dockerfile frontend 1.14.0-labs or later

Is this feature that you meant?

[polarathene]

Is this feature that you meant?

Yes that was the feature, sorry I must have recalled incorrectly as I thought it had graduated out of labs.

To use it you should only have to opt-in to labs at the top of your Dockerfile (Provided your docker / buildx are new enough to support that):

# syntax=docker/dockerfile:1.14.0-labs

The other option is to use a container with the GPU attached like you normally would, do the build there and then output that via a volume mount so it can be used in a Dockerfile build afterwards.