haystack
haystack copied to clipboard
Custom LLM-based evaluator in Haystack core
Now that we have integrations for third party LLM eval frameworks, we need to add support for a handful of LLM-based metrics that we officially support as part of core. This will be done by implementing a custom LLMEvaluator component that wraps around one or more of our generator APIs. We'll then build a small section of curated metrics on top of this component, all the while allowing the user to change the underlying service (OpenAI, Cohere, etc) and the associated prompts at will
### Tasks
- [ ] https://github.com/deepset-ai/haystack/issues/7023
- [ ] https://github.com/deepset-ai/haystack/issues/7024
- [ ] https://github.com/deepset-ai/haystack/issues/7025