0xb1te

> same issue. Have you found the solution? Trying to tune it, got it working yesterday so its progress, now the next thing is making it work in my hardware

Any responses? Is the VRAM choked?

> model_config = { "path": Path("/home/silvacarl/Desktop/models/llama-cmd-claude-q5_K_M.gguf").absolute(), "n_gpu_layers": -1, "n_ctx": 2048, "n_batch": 512, "chat_format": "llama-2", "verbose": False } > > set "n_gpu_layers": -1, I will give a try to that model...

> try this: > > pip uninstall -y llama-cpp-python FORCE_CMAKE="1" CMAKE_ARGS="-DGGML_CUDA=on" pip install --upgrade --no-cache-dir --force-reinstall -v --prefer-binary llama-cpp-python I will give it a try before changing to llama-cmd-claude-q5_K_M as...