haystack icon indicating copy to clipboard operation
haystack copied to clipboard

Custom LLM-based evaluator in Haystack core

Open shadeMe opened this issue 1 year ago • 0 comments

Now that we have integrations for third party LLM eval frameworks, we need to add support for a handful of LLM-based metrics that we officially support as part of core. This will be done by implementing a custom LLMEvaluator component that wraps around one or more of our generator APIs. We'll then build a small section of curated metrics on top of this component, all the while allowing the user to change the underlying service (OpenAI, Cohere, etc) and the associated prompts at will

### Tasks
- [ ] https://github.com/deepset-ai/haystack/issues/7023
- [ ] https://github.com/deepset-ai/haystack/issues/7024
- [ ] https://github.com/deepset-ai/haystack/issues/7025

shadeMe avatar Feb 16 '24 18:02 shadeMe