bitsandbytes Load transformers models: CUDA Setup Error

System Info

Ubuntu 20.04 Driver Version: 520.61.05 CUDA Version: 11.8 Docker: nvcr.io/nvidia/pytorch:22.12-py3 Python 3.8.10

torch==2.1.0+cu118
transformers==4.37.2

Reproduction

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_8bit=True,
    device_map="auto",
)

Error Msg

/usr/local/lib/python3.8/dist-packages/bitsandbytes/cuda_setup/main.py:122: UserWarning: 

================================================================================
WARNING: Manual override via BNB_CUDA_VERSION env variable detected!
BNB_CUDA_VERSION=XXX can be used to load a bitsandbytes version that is different from the PyTorch CUDA version.
If this was unintended set the BNB_CUDA_VERSION variable to an empty string: export BNB_CUDA_VERSION=
If you use the manual override make sure the right libcudart.so is in your LD_LIBRARY_PATH
For example by adding the following to your .bashrc: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path_to_cuda_dir/lib64
Loading: libbitsandbytes_cuda118.so
================================================================================


  warn(
False

===================================BUG REPORT===================================
/usr/local/lib/python3.8/dist-packages/bitsandbytes/cuda_setup/main.py:183: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes


  warn(msg)
================================================================================
The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
/usr/local/lib/python3.8/dist-packages/bitsandbytes/cuda_setup/main.py:183: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0', 'libcudart.so.12.1', 'libcudart.so.12.2'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda-11.8/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so'), PosixPath('/usr/local/cuda-11.8/lib64/libcudart.so')}.. We select the PyTorch default libcudart.so, which is {torch.version.cuda},but this might mismatch with the CUDA version that is needed for bitsandbytes.To override this behavior set the BNB_CUDA_VERSION=<version string, e.g. 122> environmental variableFor example, if you want to use the CUDA version 122BNB_CUDA_VERSION=122 python ...OR set the environmental variable in your .bashrc: export BNB_CUDA_VERSION=122In the case of a manual override, make sure you set the LD_LIBRARY_PATH, e.g.export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2
  warn(msg)
/usr/local/lib/python3.8/dist-packages/bitsandbytes/cuda_setup/main.py:183: UserWarning: /usr/local/cuda/lib64:/usr/local/cuda-11.8/lib64:/usr/local/lib/python3.8/dist-packages/torch/lib:/usr/local/lib/python3.8/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64:/usr/local/cuda-11.8/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0', 'libcudart.so.12.1', 'libcudart.so.12.2'] as expected! Searching further paths...
  warn(msg)
The following directories listed in your path were found to be non-existent: {PosixPath('["/tmp/vscode-ssh-auth-955d4cd6-1166-4470-ab40-cdaf5dfb4949.sock","/root/.gnupg/S.gpg-agent"]')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/usr/local/lib/python3.8/dist-packages/bitsandbytes/cuda_setup/main.py:183: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0', 'libcudart.so.12.1', 'libcudart.so.12.2'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We select the PyTorch default libcudart.so, which is {torch.version.cuda},but this might mismatch with the CUDA version that is needed for bitsandbytes.To override this behavior set the BNB_CUDA_VERSION=<version string, e.g. 122> environmental variableFor example, if you want to use the CUDA version 122BNB_CUDA_VERSION=122 python ...OR set the environmental variable in your .bashrc: export BNB_CUDA_VERSION=122In the case of a manual override, make sure you set the LD_LIBRARY_PATH, e.g.export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2
  warn(msg)
DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Required library version not found: libbitsandbytes_cuda118.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
2. CUDA driver not installed
3. CUDA not installed
4. You have multiple conflicting CUDA libraries
5. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source:
git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=118 make cuda11x
python setup.py install
CUDA SETUP: Setup Failed!
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1364, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations/bitsandbytes.py", line 11, in <module>
    import bitsandbytes as bnb
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, research, utils
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/research/__init__.py", line 2, in <module>
    from .autograd._functions import (
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/research/autograd/_functions.py", line 8, in <module>
    from bitsandbytes.autograd._functions import GlobalOutlierPooler, MatmulLtState
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/autograd/__init__.py", line 1, in <module>
    from ._functions import get_inverse_transform_indices, undo_layout
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/autograd/_functions.py", line 10, in <module>
    import bitsandbytes.functional as F
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/functional.py", line 17, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/cextension.py", line 17, in <module>
    raise RuntimeError('''
RuntimeError: 
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "LoRA/lora.py", line 7, in <module>
    model = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 3608, in from_pretrained
    from .integrations import get_keys_to_not_convert, replace_with_bnb_linear
  File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1354, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1366, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):

        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

What I tried to fix

The below was not worked:

export BNB_CUDA_VERSION=118
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64

Re-installing bitsandbytes was not worked as well.

git clone ..
cd bitsandbytes
CUDA_VERSION=118 make cuda11x
python setup.py install

Questions

What can I do more?

Expected behavior

.

Feb 21 '24 07:02 euphoria0-0

I have encountered the same problem as you and have tried the same 2 solutions. Is there a feasible solution that can be shared? What's more, there is a small issue that when running commands CUDA_VERSION=118 make cuda11x I met an error reporting make: *** No rule to make target 'cuda11x'. Stop.. Is this normal and can be ignored?

Feb 21 '24 13:02 Duojun-Huang

I have encountered the same problem as you and have tried the same 2 solutions. Is there a feasible solution that can be shared? What's more, there is a small issue that when running commands CUDA_VERSION=118 make cuda11x I met an error reporting make: *** No rule to make target 'cuda11x'. Stop.. Is this normal and can be ignored?

@Duojun-Huang I also had that problem. I downgraded bitsandbytes as 0.41.1 for a quick fix. It still makes warnings, but it works.

Feb 21 '24 14:02 euphoria0-0

Got the same issue when tried to build BNB from source, I followed the instruction from here https://huggingface.co/docs/bitsandbytes/main/en/installation, but

CUDA Setup failed despite GPU being available. Please run the following command to get more information:
        python -m bitsandbytes
...

Anyone is working on this ?

Feb 22 '24 00:02 rui-ren

Got the same issue when tried to build BNB from source, I followed the instruction from here https://huggingface.co/docs/bitsandbytes/main/en/installation, but
CUDA Setup failed despite GPU being available. Please run the following command to get more information:
        python -m bitsandbytes
...
Anyone is working on this ?

@rui-ren I also had that problem. I downgraded bitsandbytes as 0.41.1 for a quick fix. It still makes warnings, but it works.

Feb 22 '24 00:02 euphoria0-0

I am also facing the issues.Tried to downgrade to various versions still not working.

Feb 25 '24 18:02 pranavbhat12

I am also facing the issues.Tried to downgrade to various versions still not working.

@pranavbhat12 Did you set the environment path?

Feb 26 '24 01:02 euphoria0-0

Got the same issue when tried to build BNB from source, I followed the instruction from here https://huggingface.co/docs/bitsandbytes/main/en/installation, but
CUDA Setup failed despite GPU being available. Please run the following command to get more information:
        python -m bitsandbytes
...
Anyone is working on this ?
@rui-ren I also had that problem. I downgraded bitsandbytes as 0.41.1 for a quick fix. It still makes warnings, but it works.

This happens to me as well. The build from source messed up the CUDA configure path/files (I do not want to spend time on it), then I use a new docker container and install lower version to make it work.

Mar 08 '24 18:03 rui-ren

CUDA_HOME="/usr/local/cuda" export PATH="${CUDA_HOME}/bin${PATH:+:$PATH}" export LD_LIBRARY_PATH="/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

works for me.

Apr 07 '24 22:04 rocke2020