bitsandbytes
bitsandbytes copied to clipboard
TimDettmers
8-bit CUDA functions for PyTorch
When i try: from transformers import T5ForConditionalGeneration,T5Tokenizer,T5TokenizerFast model2 = T5ForConditionalGeneration.from_pretrained("3b_m1", device_map='auto' , load_in_8bit=True) I get: TypeError: __init__() got an unexpected keyword argument 'load_in_8bit' EDIT this error stopped appearing after i...
Thanks for the great work on the optimizer quantization! I'm trying to fine tune a T5 model using the adam 8 bit...but I'm finding the val loss is significantly worse...
Things seem to be working as intended! I went from using GPT-J-6B with ```python model = AutoModelForCausalLM.from_pretrained("/mnt/models",torch_dtype=torch.float16,low_cpu_mem_usage=True).to(torch.device("cuda",0)) ``` to ```python model = AutoModelForCausalLM.from_pretrained("/mnt/models",device_map="auto",load_in_8bit=True) ``` With nvidia-smi reporting a decrease in...
I am running on windows, using miniconda3 and python 3.9. I have cudatoolkit, cudnn, pytorch, transformers, accelerate, bitsandbytes, and dependencies installed via conda. when attempting to run a simple test...
Hi Tim, Thanks for your awesome work! I'm using your method to load the largest BLOOM model (the BLOOM model with 176b parameters) onto 1 node with 8 GPUs. ```...
If users don't have sudo access, or want to have different cuda versions for each conda environment, it can be helpful to install nvcc on a per environment basis. It...
I tried to use 8xA100 to run BLOOM. But I cannot do load_in_8bit. I tried to follow the instruction [here](https://gist.github.com/younesbelkada/073f0b7902cbed2cbff662996a74162e) load the model by `model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto', load_in_8bit=True, max_memory=max_memory)`...
CUDA SETUP: CUDA runtime path found: /usr/local/Cluster-Apps/cuda/8.0/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 6.0 CUDA SETUP: CUDA version lower than 11 are currenlty not supported for LLM.int8(). You...
/usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:106: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths... f'{candidate_env_vars["LD_LIBRARY_PATH"]} did not contain ' /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:28: UserWarning: WARNING: The following directories listed in your path were found to...
