guidance icon indicating copy to clipboard operation
guidance copied to clipboard
verified publisher icon microsoft

Debugging grammars by example

Open motaatmo opened this issue 2 years ago • 2 comments

Hi,

is there a way to test a guidance grammar without using a LLM call? I'm trying to write a complex grammar, and it is very cumbersome to develop it as LLM calls take their time. I would like to check whether my guidance functions would accept a string that contains the output I want to get from the LLM. If e.g. I'm trying to implement the calculator example from the readme and an offline check of the string "1*1" does not fit my guidance grammar I don't have to use an LLM to see that my code can't work. Greetings!

motaatmo avatar Dec 24 '23 22:12 motaatmo

Great idea. We should expose a nice API for this! (up for grabs if anyway wants to look into that) In the meantime you can check out the guidance.models.Mock class. It allows you to define a mock model that will output exactly what you specify (it is used in a lot of tests).

slundberg avatar Jan 05 '24 04:01 slundberg

Hi @slundberg, I am new to this project and want to contribute to this. Can you please guide me on how to get started on this. We can have a quick chat whenever you are available.

khandelwalsumit avatar Mar 22 '24 12:03 khandelwalsumit