Failed when running a local LLM by Ollama

[jackiezhangcn]

Describe the bug When running a local LLM by Ollama, it reports an Exception in thread(Thread-3).

To Reproduce

Human Input: generate 2 random numbers Exception in thread Thread-3 (base_stream_puller):=> Traceback (most recent call last): File "/home/zhangyj/anaconda3/envs/taskweaver/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/home/zhangyj/anaconda3/envs/taskweaver/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/zhangyj/Public/TaskWeaver/taskweaver/llm/init.py", line 162, in base_stream_puller for msg in stream: File "/home/zhangyj/Public/TaskWeaver/taskweaver/llm/ollama.py", line 119, in _chat_completion raise Exception( Exception: Failed to get completion with error code 404: 404 page not found ╭───< Planner > TaskWeaver ▶ [Planner] calling LLM endpoint <=💡=>

Environment Information (please complete the following information):

• OS: Ubuntu 22.04
• local LLM: llama2:13b

I totally followed the instruction in the doc

[ShilinHe]

@jackiezhangcn Hi, It is because that the LLM model is not correctly referred. You should ensure the running ollama model same as the one you configured in the taskweaver_config.json. For example, you need to run ollama run llama2:13b. We will also update the Website to keep them consistent.

[jackiezhangcn]

@jackiezhangcn Hi, It is because that the LLM model is not correctly referred. You should ensure the running ollama model same as the one you configured in the taskweaver_config.json. For example, you need to run ollama run llama2:13b. We will also update the Website to keep them consistent.

Hi, I checked, I ran as 'ollama run llama2:13b', but I guess local LLM running ollama can't be accessed by TaskWeaver

1. running Ollama (taskweaver) zhangyj@ubuntu22:~/Public$ ollama run llama2:13b

Send a message (/? for help)

2. running TaskWeaver (taskweaver) zhangyj@ubuntu22:~/Public/TaskWeaver$ python -m taskweaver -p ./project/ =========================================================

---

|_ | _ | | _ | | / / ____ __ _____ _____ | |/ / __| |/ /| | /| / / _ \/ __/ | / / _ / / | | (| _ \ < | |/ |/ / __/ // /| |/ / __/ / ||_,|/|_|/|/_/_,/ |/_/_/

TaskWeaver ▶ I am TaskWeaver, an AI assistant. To get started, could you please enter your request? Human ▶ generate 2 random numbers Exception in thread Thread-3 (base_stream_puller):=> Traceback (most recent call last): File "/home/zhangyj/anaconda3/envs/taskweaver/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/home/zhangyj/anaconda3/envs/taskweaver/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/zhangyj/Public/TaskWeaver/taskweaver/llm/init.py", line 162, in base_stream_puller for msg in stream: File "/home/zhangyj/Public/TaskWeaver/taskweaver/llm/ollama.py", line 119, in _chat_completion raise Exception( Exception: Failed to get completion with error code 404: 404 page not found ╭───< Planner > TaskWeaver ▶ [Planner] calling LLM endpoint <

[ShilinHe]

Since this is a 404 error, it should be related to the ollama serving. Can you check whether the port is correctly configured, and you can send request to the ollama endpoint? In my dev machine, I can call the llama2 successfully via ollama, screenshot show as below:

[msmsriram]

for the above error i solve by changing a removing a single backslash in the ollama.py ,

in ollama.py , as you can see that in line no:90 default code api_endpoint = "/api/chat"

i changed that into api_endpoint = "api/chat"

so the problem solved for me , because the ollama is already running in "/api/chat" but in the default code they given as the same so the slashes are added so the 404 error pops up