mdte123 comments

Repositories
Issues
Comments

Results 2 comments of


                                            mdte123

Huge difference in performance between llama.cpp and llama-cpp-python

Hi, I've probably been struggling with this for the last day too. I did find that setting the logits_all parameter to false (its true by default) appeared to increase the...

Huge difference in performance between llama.cpp and llama-cpp-python

It is on by default according to this page: https://llama-cpp-python.readthedocs.io/en/latest/server/#llama_cpp.server.settings.ModelSettings And thats what my experiment confirmed. Thank you.