FEAT: Auto-Evaluation of batch of tests

Open MiNeves00 opened this issue 2 years ago • 1 comments

Feature Request

Tests are running in a batch but they are not being evaluated against the GT answer.

The evaluation could be done using similarity metrics, LLM or even mathematics.

Motivation

This would help with quickly testing models and prompts and seeing the result quantified immediately.

Your contribution

Discussion

Oct 25 '23 09:10 MiNeves00

Hi @MiNeves00, I would like to keep track of precision and recall associated with different prompts on a test dataset. Is this issue meant to solve that problem and what is the current status?

Mar 04 '24 20:03 drkarthi