Seeking Wisdom on Cuda

[David-OConnor]

Hi! I am attempting to get this working with Cuda support. Any ideas? Thank you! Works great without the CUDA flag.

Hardware:

• RTX480 with latest drivers
• CUDA 13.0 toolkit
• Windows
• Cudarc v17.7 (If this is a limfac, I can run a local patch etc to sync these)
• CUDA kernels known working from non-ML parts of the application with Cudarc and cuFFT bindings.

Cargo.Toml:

candle-core = { git = "https://github.com/huggingface/candle", package = "candle-core", features = ["cuda"] }
candle-nn = { git = "https://github.com/huggingface/candle", package = "candle-nn", features = ["cuda"] }

The compile error:

error: failed to run custom build command for `candle-kernels v0.9.2-alpha.1 (https://github.com/huggingface/candle#d4545ebb)`
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.
Caused by:
  process didn't exit successfully: `C:\Users\the_a\code\Bio\daedalus\target\release\build\candle-kernels-b4b265fd7ce556ce\build-script-build` (exit code: 101)
  --- stdout
  cargo::rerun-if-changed=build.rs
  cargo::rerun-if-changed=src/compatibility.cuh
  cargo::rerun-if-changed=src/cuda_utils.cuh
  cargo::rerun-if-changed=src/binary_op_macros.cuh
  cargo:info=["/usr", "/usr/local/cuda", "/opt/cuda", "/usr/lib/cuda", "C:/Program Files/NVIDIA GPU Computing Toolkit", "C:/CUDA"]
  cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP
  cargo:rustc-env=CUDA_COMPUTE_CAP=89
  cargo::warning=Builder { cuda_root: None, kernel_paths: ["src\\affine.cu", "src\\binary.cu", "src\\cast.cu", "src\\conv.cu", "src\\fill.cu", "src\\indexing.cu", "src\\quantized.cu", "src\\reduce.cu", "src\\sort.cu", "src\\ternary.cu", "src\\unary.cu"], watch: [], include_paths: ["src\\binary_op_macros.cuh", "src\\compatibility.cuh", "src\\cuda_utils.cuh"], compute_cap: Some(89), out_dir: "C:\\Users\\the_a\\code\\Bio\\daedalus\\target\\release\\build\\candle-kernels-e32097457b6dd4eb\\out", extra_args: [] }
  --- stderr
  thread 'main' panicked at C:\Users\the_a\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\bindgen_cuda-0.1.5\src\lib.rs:306:40:
  Could not find CUDA in standard locations, set it manually using Builder().set_cuda_root(...)
  stack backtrace:
     0: std::panicking::begin_panic_handler
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\std\src\panicking.rs:697
     1: core::panicking::panic_fmt
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\core\src\panicking.rs:75
     2: core::panicking::panic_display
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\core\src\panicking.rs:268
     3: core::option::expect_failed
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\core\src\option.rs:2139
     4: core::option::Option<T>::expect
     5: bindgen_cuda::Builder::build_ptx
     6: build_script_build::main
     7: std::rt::lang_start
     8: std::rt::lang_start
     9: std::rt::lang_start
    10: std::rt::lang_start_internal::closure$0
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\std\src\rt.rs:175
    11: std::panicking::catch_unwind::do_call
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\std\src\panicking.rs:589
    12: std::panicking::catch_unwind
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\std\src\panicking.rs:552
    13: std::panic::catch_unwind
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\std\src\panic.rs:359
    14: std::rt::lang_start_internal
               at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library\std\src\rt.rs:171
    15: std::rt::lang_start
    16: main
    17: invoke_main
               at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
    18: __scrt_common_main_seh
               at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    19: BaseThreadInitThunk
    20: RtlUserThreadStart
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I believe this may be due to a limitation of the bindgen_cuda dependency, which doesn't look in the correct location for CUDA, and doesn't check the PATH (Or LD_LIBRARY_PATH on Linux) environment variables. I have been compiling CUDA using nvcc in build.rs, and it is able to find CUDA without specifying any paths.

[David-OConnor]

More info: Here's where, in bindgen_cuda, this error comes from: https://github.com/Narsil/bindgen_cuda/blob/main/src/lib.rs#L305

 pub fn build_ptx(self) -> Result<Bindings, Error> {
        let cuda_root = self.cuda_root.expect("Could not find CUDA in standard locations, set it manually using Builder().set_cuda_root(...)");
        let compute_cap = self.compute_cap.expect("Could not find compute_cap");
        let cuda_include_dir = cuda_root.join("include");
        println!(
            "cargo:rustc-env=CUDA_INCLUDE_DIR={}",
            cuda_include_dir.display()
        );
        let out_dir = self.out_dir;

        let mut include_paths = self.include_paths;
        for path in &mut include_paths {

Maybe switch to an endemic compilation function? I suspect the dependency here is causing more friction than benefit.

Here is a working PTX building function. Maybe we switch to something like this?

/// Call this in `build.rs` to compile the kernal.
///
/// See [These CUDA docs](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)
/// for info about these flags.
///
/// Compiles our CUDA program using Nvidia's NVCC compiler
/// Call this in the `main()` fn of `build.rs`. `cuda_files`'s first parameter
/// must be the entry point. The other parameters are just so the build step
/// knows when to re-compile.
///
/// The architecture provided is the minimum supported one the PTX will output.
pub fn build_ptx(min_arch: GpuArchitecture, cuda_files: &[&str], filename: &str) {
    if cuda_files.is_empty() {
        return;
    }

    // Tell Cargo that if the given file changes, to rerun this build script.
    for kernel in cuda_files {
        println!("cargo:rerun-if-changed={kernel}");
    }

    let compilation_result = Command::new("nvcc")
        .args([
            cuda_files[0],
            // "-gencode",
            // &architecture.gencode_val_sm(),
            &min_arch.compute_val(),
            "-ptx",
            "-O3", // optimized/release mode.
            "-o",
            &format!("{filename}.ptx"),
        ])
        .output();

    match compilation_result {
        Ok(output) => {
            if output.status.success() {
                println!("Compiled the following PTX files: {cuda_files:?}");
            } else {
                // eprintln!(
                panic!(
                    "CUDA PTX compilation problem:\nstatus: {}\nstdout: {}\nstderr: {}",
                    output.status,
                    String::from_utf8_lossy(&output.stdout),
                    String::from_utf8_lossy(&output.stderr)
                );
            }
        }
        Err(e) => eprintln!("Unable to compile PTX: {e}"),
    }
}

[David-OConnor]

Confirmed this is a bingen_cuda issue. PR opened there.

[ivarflakstad]

Thanks for the extensive analysis!
I agree that it is causing a bit of friction. We use a different approach when building flash attention 3, which is very similar to the snippet you shared.

I don’t know if your bindgen_cuda PR will be merged for a while, so I may simply replace the dependency before then. We’ll see :)

[David-OConnor]

Great! In the meanwhile, I'm fine because I have a local copy of candle with the that dependency swapped out for the modded one.

[ivarflakstad]

You could open a PR if you'd like :)
I'd use candle-flash-attn-v3/build.rs as inspiration. Parallel builds and all that.

[David-OConnor]

Great! Of interest, my mod for Candle is just to point bindgen_cuda to my local one, and I have a PR to that. So my thoughts are:

A: See if bindgen_cuda eventually merges my PR and releases, then we can update this to version bump bindgen_cuda. (Unknown)
B: If y'all move away from bindgen_cuda.

[ivarflakstad]

Not sure if you completed your sentence but I was thinking option B.