microsoft
ollama api error
Dear Sir: LiteLLM completion() model= deepseek-coder-v2:16b; provider = ollama 2025-02-18 17:29:13,281 - LiteLLM - INFO - LiteLLM completion() model= deepseek-coder-v2:16b; provider = ollama 2025-02-18 17:29:26,456 - httpx - INFO - HTTP Request: POST http://localhost:11434/api/generate "HTTP/1.1 500 Internal Server Error"
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
Hello! I'm able to run deepseek-coder-v2 16b with the following configuration:
I loaded the 16 model with Ollama using ollama run deepseek-coder-v2, so I use deepseek-coder-v2 in the model configuration.
Actually I also encounter this issue every now and then. I debugged a little bit, and found it is an issue with ollama:
Error: an error was encountered while running the model: unexpected EOF
And this looks like an OOM issue during inference. I think the solution is to either use a smaller model, or use GPU for inference.