private-gpt Output speed is increasingly slow more then 15 minuets for output.

Discussed in https://github.com/imartinez/privateGPT/discussions/380

^{Originally posted by GuySarkinsky May 22, 2023} How results can be improved to make sense for using privateGPT?

The model I use: ggml-gpt4all-j-v1.3-groovy

Device specifications:

Device name Full device name Processor Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz 2.11 GHz Installed RAM 16.0 GB (15.9 GB usable) Device ID Product ID System type 64-bit operating system, x64-based processor Pen and touch No pen or touch input is available for this display

I understand that at the moment it is impossible to use GPU.

May 22 '23 15:05 GuySarkinsky

Your CPU is not really amazing, it's not surprising to have such a performance issue. It is possible to use GPU, though. You just have to modify this for Windows (only diff is $Env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"; $Env:FORCE_CMAKE=1; python3 ./setup.py install for Windows) (not hard)... https://github.com/maozdemir/privateGPT-colab/blob/main/privateGPT-colab.ipynb

May 22 '23 16:05 maozdemir

@maozdemir Thanks

May 22 '23 16:05 GuySarkinsky

You could also use a smaller model if you're willing to accept a possibly (probably somewhat) inferior response quality.

Jul 25 '23 00:07 owenpmckenna