ROCM Support

[Jarfeh]

bitsandbytes seems to be hardcoded to search for specific cuda libraries which don't seem to be provided the same way by rocm

/root/anaconda3/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:86: UserWarning: /root/anaconda3 did not contain libcudart.so as expected! Searching further paths...
warn(
/root/anaconda3/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:98: UserWarning: /opt/ompi/lib:/opt/rocm/lib:/usr/local/lib: did not contain libcudart.so as expected! Searching further paths...
warn(
/root/anaconda3/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:20: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('CompVis/stable-diffusion-v1-4')}
warn(
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
/root/anaconda3/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:20: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
warn(
WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
CUDA SETUP: Loading binary /root/anaconda3/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
/root/anaconda3/lib/python3.9/site-packages/bitsandbytes/cextension.py:48: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
warn(

[errnoh]

+1

Would be great to get this working on AMD hardware if there's no hard technical limitation blocking compatibility. Newer Radeon cards often have quite high VRAM and work really well for ML, main issue remaining is providing library compatibility.

[TimDettmers]

I personally do not have time to implement ROCm support. If you have experience with ROCm I could guide you through the steps to get a working solution.

[gururise]

An unofficial working port exists here: https://github.com/broncotc/bitsandbytes-rocm

Would be possible to merge the changes into the official port?

[Jarfeh]

An unofficial working port exists here: https://github.com/broncotc/bitsandbytes-rocm

Would be possible to merge the changes into the official port?

Doesn't seem to quite work yet, results in a CUDA_SETUP failed error when I attempt to use it

Edit: It seems I just attempted to install using the wrong flags while building. It works great.

[gururise]

EDIT: A slightly newer version based on v0.37 available here:
https://github.com/Titaniumtown/bitsandbytes-rocm/tree/patch-2

[nktice]

After much searching, I did find something that works - so I'll share it here.
I've been making notes akin to an install script... Here is what I found

# bitsandbytes rocm
# video guide : https://www.youtube.com/watch?v=2cPsvwONnL8
# https://git.ecker.tech/mrq/bitsandbytes-rocm
## https://github.com/0cc4m/bitsandbytes-rocm
git clone https://git.ecker.tech/mrq/bitsandbytes-rocm.git
cd bitsandbytes-rocm/
pip install -r requirements.txt
make hip
CUDA_VERSION=gfx1030 python setup.py install

I have found this makes bitsandbytes work with some things on my GPU...
[ AMD Radeon 6900 XT 16GB ]
I would like to see these features merged back into the main bitsandbytes -
so that new versions automatically have them, rather than needing
folks who wrote these mods, to go back and update them to follow updates.

[yamfun]

After much searching, I did find something that works - so I'll share it here. I've been making notes akin to an install script... Here is what I found

# bitsandbytes rocm
# video guide : https://www.youtube.com/watch?v=2cPsvwONnL8
# https://git.ecker.tech/mrq/bitsandbytes-rocm
## https://github.com/0cc4m/bitsandbytes-rocm
git clone https://git.ecker.tech/mrq/bitsandbytes-rocm.git
cd bitsandbytes-rocm/
pip install -r requirements.txt
make hip
CUDA_VERSION=gfx1030 python setup.py install

I have found this makes bitsandbytes work with some things on my GPU... [ AMD Radeon 6900 XT 16GB ] I would like to see these features merged back into the main bitsandbytes - so that new versions automatically have them, rather than needing folks who wrote these mods, to go back and update them to follow updates.

My AMD kohya setup worked with Lion but I want to try use AdamW8bit like the others.
So after following this post, I finally got my Kohya to apparently use Adam8bit on Linux + AMD ROCm 5.6 with no more python errors (thought it doesn't really work when used, see the last paragraph).

For building at the "make hip" step, I performed some steps like,

1. Install the meta-package mentioned at https://rocm.docs.amd.com/en/latest/deploy/linux/os-native/package_manager_integration.html (I installed all of them but i guess you guys know exactly which one is needed for the hipcub clang stuff)
2. Makefile line 117 "HIP_LIB := -L$(ROCM_DIR)/lib -L$(ROCM_DIR)/llvm/bin/../lib/clang/15.0.0/lib/linux "changed the 15 to 16

There are also other steps that I did as some errors appeared during the "make hip" setup battle, but I am not sure whether they mattered or correct:

1. Makefile line 3 "ROCM_DIR := /opt/rocm/" take out the last path slash
2. ops.cuh line 16 "#include <hipblas.h>" to hipblas/hipblas.h
3. ops.cuh line 17 "#include <hipsparse.h>" to hipsparse/hipsparse.h

(All these steps only made the bnb-rocm "make hip" errors and Kohya bnb import/usage errors go away,
but when actually using AdamW8bit, the 1st epoch are some weird color artifacts, and the epochs afterwards are all black, and the console saying loss=nan; whereas AdamW and Lion work normally with the same params)

[arlo-phoenix]

The Adam8bit optimizer issue probably wasn't from ROCm itself, but the general issue that's been fixed in the latest version 0.41.1. I made my own fork that just uses defines to make the CUDA code work with HIP, so it's easier to keep up to date and supports the latest version. Since the patch was just in python though, you could just apply it to the fork you are currently using as well.

[bennmann]

Hi @arlo-phoenix , nice fork, what would i comment out of the Makefile to test 6900 XT (gfx1030)?

I have already "export HSA_OVERRIDE_GFX_VERSION=10.3.0"

[arlo-phoenix]

Hi @arlo-phoenix , nice fork, what would i comment out of the Makefile to test 6900 XT (gfx1030)?

I have already "export HSA_OVERRIDE_GFX_VERSION=10.3.0"

You shouldn't have to change anything special afaik just doing

git clone https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6.git bitsandbytes
cd bitsandbytes

#see makefile comments under hip for more info
#using pip install . since python setup.py install is deprecated
make hip
pip install . 

works on a newly created docker using rocm 5.6 (the one I'm using). This has all the environment variables including path setup and since ROCm doesn't have that much documentation I really recommend it. If you don't wanna use docker, you'll need to make sure, that hipcc is at /usr/bin or change the makefile accordingly (I'll probably update it myself at some point to work better out of the box, I just copied it from a previous port). You can also set the env var ROCM_HOME to the path to your rocm install (normally /opt/rocm) if the automatic find function doesn't work and add --offload-arch=gfx1030 to the two hipcc commands if it doesn't use the correct one anyways / just compiles for all. And yes the export HSA_OVERRIDE_GFX_VERSION=10.3.0 is also necessary.

For testing you can then go in the tests folder and use pytest test_optim.py to see what works and what doesn't. I'd be careful with the other tests, some froze my PC. I'd say this is probably because I just excluded some parts of the code as they didn't compile because of the different warp size (it's double at AMD and some static_asserts failed because of that) or the missing hipBLASlt stuff or which I hope it's not actually ROCm library issues. I'll see if I can fix that, I want to try out QLora at some point and I'm pretty sure I need some atleast, but haven't tried that out yet.

TLDR:

• Requires ROCm 5.6
• export HSA_OVERRIDE_GFX_VERSION=10.3.0
• set ROCM_HOME env and check if hipcc is in /usr/bin or otherwise change the Makefile under hip: to use it
• Add --offload-arch=gfx1030 to the two hipcc commands
• make hip should work now and afterwards just pip install .
• Only optimizers were tested, rest might freeze your system

[bennmann]

thank you very much @arlo-phoenix - one more step forward, now the "make hip" is struggling with the below paths for my ROCM 5.6:

$ ls /opt/rocm/hip/include/hip/*runtime*
/opt/rocm/hip/include/hip/hip_runtime_api.h  /opt/rocm/hip/include/hip/hip_runtime.h

$make hip
...
...
/home/user/bitsandbytes/csrc/ops.cuh:17:10: fatal error: hip/hip_runtime_api.h: No such file or directory
   17 | #include <hip/hip_runtime_api.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:124: hip] Error 1

can you check to make sure the $ROCM_HOME include makefile references match your installation paths in your fork? is this just my installation of Ubuntu and amdgpu-install rocm PATHs being non-standard in some way?