cannot use GPU with win11

[leeyaunlong]

Use the "openchat-3.5-7b" model with win11, just used cpu. the reor cannot use GPU. I can use the same model (GPU) with JAN.AI appliaction.

[Nasnl]

Same here, while using Gemma 7B locally.

[drgutman]

+1 It would be useful to have an option available to switch between the GPU and CPU modes.

[samlhuillier]

Working on this now. @drgutman @Nasnl @leeyaunlong Just for my info, do you guys have cuda installed? Which gpu do each of you use?

[drgutman]

@samlhuillier - Yes, cuda_12.3.2_546.12_windows, GeForce RTX 3060.

[Nasnl]

Working on this now. @drgutman @Nasnl @leeyaunlong Just for my info, do you guys have cuda installed? Which gpu do each of you use?

@samlhuillier Cuda 12.4 , Windows 11, 13th Gen Intel(R) Core(TM) i7-13700H, 32GB, GeForce RTX 4050

[leeyaunlong]

Working on this now. @drgutman @Nasnl @leeyaunlong Just for my info, do you guys have cuda installed? Which gpu do each of you use?

CUDA Version: 12.1 , win11, intel 11gen i9

PS: I can use the JAN.AI with 7B model. current I use JAN.AI's api with Reor.

[samlhuillier]

@drgutman @Nasnl @leeyaunlong GPU support via CUDA is now out in the latest version!

You'll have to turn it on in Settings -> Hardware -> Toggle GPU and CUDA on

(There is also Vulkan support for AMD GPUs.)

[leeyaunlong]

thanks, it's work well now.

[leeyaunlong]

thanks, it's work well now.

[Nasnl]

And it makes a difference!

Thanks @samlhuillier !