y6t4

Results 3 comments of y6t4

llama-cpp-python bindings not working for multiple GPUs

An update on my latest effort to get this working. I uninstalled CUDA and then reinstalled everything. `nvidia-smi` output: ![norun2](https://github.com/ggerganov/llama.cpp/assets/39177699/3c79c6ab-ac43-4b7e-968e-d57aafa33fad) `nvcc --version` output: ``` nvcc: NVIDIA (R) Cuda compiler driver...

llama-cpp-python bindings not working for multiple GPUs

> An update on my latest effort to get this working. I uninstalled CUDA and then reinstalled everything. > > `nvidia-smi` output: ![norun2](https://private-user-images.githubusercontent.com/39177699/318004633-3c79c6ab-ac43-4b7e-968e-d57aafa33fad.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTE3MDYxNDUsIm5iZiI6MTcxMTcwNTg0NSwicGF0aCI6Ii8zOTE3NzY5OS8zMTgwMDQ2MzMtM2M3OWM2YWItYWM0My00YjdlLTk2OGUtZDU3YWFmYTMzZmFkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzI5VDA5NTA0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTFhYTEwYjEzYjViZDMzZDU4YWVkN2Q2ZTY1YTVmYmM1NjYyNDY3MmZlYTg2MGYxMzU2M2NjMzFhMmQwZGExZjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.59Ltsek6FiqD2pGi0MEschaE_Bl6OX71kiDYCufVBPA) > > `nvcc --version` output: > >...

llama-cpp-python bindings not working for multiple GPUs

> @y6t4 I can't help too much as I don't have a multi-gpu setup available but I think you're missing the `n_gpu_layers` (required to offload) and the `tensor_split` parameters. I'd...