LLamaSharp icon indicating copy to clipboard operation
LLamaSharp copied to clipboard
Published 3 months ago • verified publisher icon SciSharp

[BUG]:

Open johnearnshaw opened this issue 3 months ago • 0 comments

Description

I've been following and playing around with LlamaSharp since the project first started - It's cool BTW!

Since I updated CUDA, I've not been able to get it working, even though I have CUDA 13, CUDA 12.9, and CUDA 12.4 installed, with my CUDA_PATH environment variable set to 12.4

Image

Although this question is about Vulkan...

I've tested on 3 different machines (laptops), 2 with Intel GPU as primary and Nvidia GPU as secondary (one with Geforce 960 6GB dedicated RAM, and one with Quadro 6GB dedicated RAM), and another laptop with 2x dedicated Geforce 1080 GTX GPUs, both with 8GB dedicated RAM.

Vulkan always returns an error saying it can't allocate memory on all three machines, regardless of configurations I've tested, and if CUDA and Vulkan runtimes are installed together, Vulkan takes priority, throws the error

It won't fall back to CUDA, and then it doesn't fall back to CPU only.

Is there a fix for this with multiple GPU runtimes installed?

And is there a fix for the Vulkan not allocating memory error?

Reproduction Steps

Install Vulkan, CUDA, and CPU-only runtimes and test.

I also tried to override the native library loading using my own cross-platform assembly loader library (I've got lots of native/pinvoke experience). This library works well with loading native Chromium .dll/.so, Whisper.cpp, and other native C++ libraries into a .NET application.

It's a netstandard2.1 class library, and works as a replacement for the built-in assembly loading for assemblies in the 'runtimes' directory, etc... It lets you specify your own path to assemblies to load via cross-platform P/Invoke based on your own runtime logic.

Thought my assembly loader doesn't seem to work with overriding LlamaSharp native assembly loading so I commented out the dependency.

Environment & Configuration

  • Operating system: Windows 11

  • .NET runtime version: netstandard2.1 class library with dependency on LlamaSharp running in a net8.0 console application.

  • LLamaSharp version: 0.25.0

  • CUDA version (if you are using cuda backend): 12.4, 12.9, and 13.0 tested. (Primarily 12.4)

  • CPU & GPU device: Intel i7 with Nvidia Geforce 960 6GB, Intel i7 with Nvidia Quadro 6GB, and Intel i7 with 2x Nvidia Geforce 1080 GTX 8GB

Known Workarounds

No response

johnearnshaw avatar Nov 16 '25 14:11 johnearnshaw