llama-cpp-python icon indicating copy to clipboard operation
llama-cpp-python copied to clipboard
verified publisher icon abetlen

Performance Improvement and Error Handling for API

Open djaffer opened this issue 2 years ago • 0 comments

It gets gateway timeout very often and is there any error handling done.

Increase the timeout maybe.

Also threads are /2. https://github.com/abetlen/llama-cpp-python/blob/main/examples/high_level_api/fastapi_server.py#L31

djaffer avatar Apr 17 '23 06:04 djaffer