LLamaSharp
LLamaSharp copied to clipboard
Impossible Invalid GBNF Grammar Parsed
I am using the grammar below with OpenHermes-2.5-Mistral-7B-16k-GGUF.
root ::= (object | array) endline?
endline ::= "<|im_end|>" ws
value ::= object | array | string | number | ("true" | "false" | "null") ws
object ::=
"{" ws (
string ":" ws value
("," ws string ":" ws value)*
)? "}" ws
array ::=
"[" ws (
value
("," ws value)*
)? "]" ws
string ::=
"\"" (
[^"\\] |
"\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes
)* "\"" ws
number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? ws
# Optional space: by convention, applied in this grammar after literal chars when allowed
ws ::= ([ \t\n] ws)?
I read and parse the grammar in a few lines of code using llamasharp.
string grammarFileLocation = @"json grammar.gbnf";
string gbnf = File.ReadAllText(grammarFileLocation);
string gbnfTrimmed = rawGrammar.Trim();
var grammar = Grammar.Parse(gbnfTrimmed , "root");
When I use this grammar in inference params, on the second inference it fails with an assert "GGML_ASSERT: D:\a\LLamaSharp\LLamaSharp\llama.cpp:6649". You can find the assert here.
Is there a way I can get the grammar to run twice for inferences in quick succession?
@martindevans suggested parsing the grammar again instead of re-using the parsed grammar and it worked! It's a temporary fix, but it operates as expected.