haystack
haystack copied to clipboard
`PromptNode` doesn't support isolated node evaluation
Describe the bug When running pipeline.eval_batch with a GenerativeQAPipeline the isolated node eval options seems to be ignored.
Additional context
- When running the same pipeline without batching it adds the isolated node eval. Only in eval_batch it dissmisses the parameter.
- Running eval_batch with an ExtractiveQAPipeline does not ignore the isolated eval param.
To Reproduce please take eval labels from the eval tutorial
retriever = BM25Retriever(
document_store=document_store
)
generator = Seq2SeqGenerator(model_name_or_path="vblagoje/bart_lfqa")
pipeline = GenerativeQAPipeline(generator=generator, retriever=retriever)
eval_result = pipeline.eval_batch(labels=eval_labels[:5],
params={"Retriever": {"top_k": 10}, "Generator": {"top_k": 2}},
sas_model_name_or_path="cross-encoder/stsb-roberta-large",
add_isolated_node_eval=True)
#5291 adds support for isolated node evaluation in BaseGenerator.
Given that we removed Seq2SeqGenerator and RAGenerator and recommend to use PromptNode instead, I investigated the support for isolated node evaluation in PromptNode. However, I found that we currently do not support it at all. Let's keep this issue open to think about if and how we want to support isolated node evaluation in PromptNode.
Closing as won't fix