Unable to use bitsandbytes in mac m1

[ppanja]

Installed bitsandbytes package using this repo (!pip install -U git+https://github.com/rickardp/bitsandbytes.git) but doesn't seem to work on mac m1 as getting error "PackageNotFoundError: bitsandbytes" when trying to use the same.

I ran a trace (!python -m bitsandbytes) on my jupyter notebook and got below output:

Traceback (most recent call last):
File "/Users/prasantapanja/anaconda3/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/Users/prasantapanja/anaconda3/lib/python3.10/runpy.py", line 146, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/Users/prasantapanja/anaconda3/lib/python3.10/runpy.py", line 110, in _get_module_details
import(pkg_name)
File "/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/init.py", line 6, in
from . import cuda_setup, utils, research
File "/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/research/init.py", line 1, in
from . import nn
File "/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/research/nn/init.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/optim/init.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 11, in
lib = ct.cdll.LoadLibrary(binary_path)
File "/Users/prasantapanja/anaconda3/lib/python3.10/ctypes/init.py", line 452, in LoadLibrary
return self._dlltype(name)
File "/Users/prasantapanja/anaconda3/lib/python3.10/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib, 0x0006): tried: '/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file), '/Users/prasantapanja/anaconda3/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file)

[unixwzrd]

That would be probably because no one has built the package and put it into a module for Python and the wheels. The traceback indicated that bitsandbytes_cpu.dylib is not found, so it hasn't been installed and likely not built in this repository yet. Not sure off anyone has started building this or not.

[ppanja]

Thank you @unixwzrd for your comments. Make sense.

[unixwzrd]

I’ve been looking at options for this used of bitsandbytes not supporting Apple Silicon M1 and M2 processors and it seems the necessary pieces are already in libraaries Apple shops, though some work with Metal will need to be done. However, I have seen what’s going on here with the ggml library development since it looks like it has promise. I am also I. The process of testing various methods for accessing LLMs on Apple Silicon M1 M2 processors and there is some progress being made. This is where I am trying to keep my progress updated for now.. Maybe there’s a better place, but this will also alow collaboration.

[rickardp]

This package is not ready to be pip installable. As per the README, build using “make” (requires cmake) then use distutils setup.py.

My idea is to build binary wheels for all supported platforms (already done in GitHub actions but not published to pip). I don’t think it is worth the effort of supporting pip install from source to be honest.

please note that CPU and Apple Silicon support is nowhere near complete and not in a usable state.

[bghira]

hello rickardp, can i ask what the status of this port is and what you might need help with?

[rickardp]

Hi @bghira. Good to hear, there's certainly a need for more hands on the porting to MPS. The current status is that I plan to help out, but I am waiting on the device abstraction to settle (bitsandbytes-foundation#898). Then we need to get the CPU pipeline working and all unit tests green on the mac. Then we can port piece by piece to gradually get better MPS coverage. At least that's the plan in my head, but please feel free to join the discussions over at the upstream
bitsandbytes-foundation#990

I will close this issue now since the original PRs from this fork has been merged to main.