Does this support CUDA?

Open SpaceCowboy850 opened this issue 2 years ago • 1 comments

I have seen where I can set the GGML_USE_CUBLAS, and I can follow the few #defines that activate the code, but the tensors are all on the CPU. I'm not seeing in bert.cpp where it would transfer the model or the inputs to the GPU.

Is this just not functioning yet?

Dec 08 '23 19:12 SpaceCowboy850

I haven't done anything towards CUDA support. How easy/difficult it is to implement depends on ggml, I guess

Dec 09 '23 21:12 skeskinen