`Error: No tokenizer found for model` with Ollama

[luochen1990]

Describe the bug

Trying to use ollama but failed with Error: No tokenizer found for model, tried to change the model but still same error.

To Reproduce

[openai]
api_base = "http://127.0.0.1:11434/v1"
api_key = ""
model = "llama3.1:latest"
retries = 2
proxy = ""

1. Run ollama server locally
2. Exec ollama run llama3.1:latest
3. Try some talk to Confirm that ollama is running without issue
4. Edit ~/.config/gptcommit/config.toml and add above config
5. git add something and Execute git commit
6. See error

Expected behavior

Works correctly and without error

Screenshots If applicable, add screenshots to help explain your problem. Run your command with the RUST_LOG=trace environment variable for the best support

System Info (please complete the following information):

• OS: linux (NixOS unstable)
• Version: gptcommit v0.5.16

[assafmo]

Happens to me to with groq:

[openai]
api_base = "https://api.groq.com/openai/v1"
api_key = "XXX"
model = "llama-3.2-90b-text-preview"
retries = 2
proxy = ""

[scylamb]

Happens to me to with gpt-4o-mini

STDERR: Error: No tokenizer found for model gpt-4o-mini

[rmnilin]

This error occurs here

https://github.com/zurawiki/gptcommit/blob/94eeb2f009317049049fec9db7decd5ac6fe84bd/src/llms/openai.rs#L101

because this library relies on https://github.com/zurawiki/tiktoken-rs to count and limit prompt tokens. Both libraries are intended to be used exclusively with OpenAI models:

• https://github.com/zurawiki/gptcommit/pull/246
• https://github.com/zurawiki/tiktoken-rs/blob/088ff6d51629acea1624298151df148f328d6d0c/tiktoken-rs/src/tokenizer.rs#L33.

For ollama users, there's a workaround that involves creating an alias for a target model. Here's how you can do it:

$ ollama cp qwen2.5-coder:7b gpt-4--gptcommit-workaround-alias--qwen2.5-coder:7b

# gptcommit/config.toml
[openai]
api_base = "http://localhost:11434/v1"
api_key = ""
model = "gpt-4--gptcommit-workaround-alias--qwen2.5-coder:7b"
retries = 2
proxy = ""

To remove the alias when you no longer need it, use the following command:

$ ollama rm gpt-4--gptcommit-workaround-alias--qwen2.5-coder:7b

Only alias will be removed, not the original model.

Also note that you can use any name or prefix from the zurawiki/tiktoken-rs enums linked above, but there are different logic depending on the model prefix, see

https://github.com/zurawiki/gptcommit/blob/94eeb2f009317049049fec9db7decd5ac6fe84bd/src/llms/openai.rs#L95

[hellopenggao]

Thanks, @rmnilin ! Your analysis inspired me. I solved this issue with a single command.

I'm using Hugging Face + text-generation-webui. So, I created a symbolic link in the model directory of text-generation-webui:

ln -s ~/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-14B-Instruct-GPTQ-Int4/snapshots/d435273c87c9fd984eb9d27ae89d4e12a4676052/ gpt-4--gptcommit-workaround-alias--qwen2.5-coder:14b

I used the name you recommended for the symlink, and now the loaded model can be properly recognized in gptcommit.

[ngn999]

model = "grok-2-latest" same error.

[medopaw]

@zurawiki Could we add a configuration setting to enforce the use of chat completion? For example, enforce_chat_completion.