LLamaSharp icon indicating copy to clipboard operation
LLamaSharp copied to clipboard

Impossible Invalid GBNF Grammar Parsed

Open TheWorldEndsWithUs opened this issue 2 years ago • 1 comments

I am using the grammar below with OpenHermes-2.5-Mistral-7B-16k-GGUF.

root   ::= (object | array) endline?
endline ::= "<|im_end|>" ws
value  ::= object | array | string | number | ("true" | "false" | "null") ws

object ::=
  "{" ws (
            string ":" ws value
    ("," ws string ":" ws value)*
  )? "}" ws

array  ::=
  "[" ws (
            value
    ("," ws value)*
  )? "]" ws

string ::=
  "\"" (
    [^"\\] |
    "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes
  )* "\"" ws

number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? ws

# Optional space: by convention, applied in this grammar after literal chars when allowed
ws ::= ([ \t\n] ws)?

I read and parse the grammar in a few lines of code using llamasharp.

string grammarFileLocation = @"json grammar.gbnf";
string gbnf = File.ReadAllText(grammarFileLocation);
string gbnfTrimmed = rawGrammar.Trim();
var grammar = Grammar.Parse(gbnfTrimmed , "root");

When I use this grammar in inference params, on the second inference it fails with an assert "GGML_ASSERT: D:\a\LLamaSharp\LLamaSharp\llama.cpp:6649". You can find the assert here.

Is there a way I can get the grammar to run twice for inferences in quick succession?

TheWorldEndsWithUs avatar Dec 29 '23 19:12 TheWorldEndsWithUs

@martindevans suggested parsing the grammar again instead of re-using the parsed grammar and it worked! It's a temporary fix, but it operates as expected.

TheWorldEndsWithUs avatar Dec 29 '23 19:12 TheWorldEndsWithUs