agenta
agenta copied to clipboard
Published
4 months ago
•
Agenta-AI
Agenta-AI
Evaluators can access all columns
- Now evaluators have access to whatever correct answer column from the testset.
The latest updates on your projects. Learn more about Vercel for Git ↗︎
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| agenta | ✅ Ready (Inspect) | Visit Preview | 💬 Add feedback | May 31, 2024 7:46am |
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thanks for the PR, @aakrem! Great job. I have a few concerns (unrelated to the code changes).
I think the title of the PR and its description are a bit misleading. I have two questions:
* If an evaluator has access to all columns, why can we only access a single column in the test set?
This has been fixed. Now you configure in the evaluator which columns it has access to and it would be able to access them
* If the evaluators has access to any correct answer column from the test set, why are similarity match, JSON field match, webhook test, and Levenshtein distance the only evaluators that do?
It depends on the evaluator. Some of them, like JSON check, which checks whether the llm output is in json format does not require access to any ground truth obviously.