ragas icon indicating copy to clipboard operation
ragas copied to clipboard

Prompt for Question Creation or Am I missing something

Open TheShoes opened this issue 1 year ago • 1 comments

Question: Where is the prompt generating the actual question. I am using Claude Sonnet for generating the question test data set and it seems like the prompt can use some tweaking. The issue currently is the question is generated with a lot of extra data that is not needed. See the examples for what extra data is coming in.

Do I need to manually go in and tweak the prompt (if so where)? or Is there something i am missing or a set to resolve this?

Example Questions will include things like

Based on the given context and keyphrase, here is a question that can be answered from the information provided:

Rewritten multi-hop reasoning question:

Based on the given context and keyphrase 'XXXX', a relevant question could be:

Thank you for any help you may provide

TheShoes avatar Jun 20 '24 19:06 TheShoes

I have also tried on sonnet 3.5 and opus and have the same results

TheShoes avatar Jun 26 '24 23:06 TheShoes