carlrobertoh
<think> block delimiters are not detected correctly
What happened?
I am running Codestral locally (LM Studio -> Custom OpenAI config.)
There is an issue with ProxyAI not properly detecting the end of a <think>-block.
Inspecting the model's output, it properly does <think>{reasoning}</think>{answer}.
A little scouring through the code revealed the culprit:
The parsing expects (at least) 2 newlines after a </think>-tag. The model outputs a single newline after the think block. I have tried to instruct the model to output 2 newlines, but it sometimes will and sometimes won't, which makes this flaky.
I don´t know the reasoning (by the developer) behind this specific, but I think it should not assert 2 newlines, just look for <think>, and then </think>.
Neither the commit message, nor the issue linked to that commit state anything about this formatting.
The places that I found: ThinkingOutputParser.kt#L7 CompleteMessageParser.kt#L17
Relevant log output or stack trace
Steps to reproduce
- Run Codestral 22B v0.1 in LM Studio.
- Register LM Studio as a Custom OpenAi-Like config
- Use the Default Persona.
- Try to generate something that requires reasoning.
CodeGPT version
3.4.3-241.1
Operating System
Linux
