haystack icon indicating copy to clipboard operation
haystack copied to clipboard

`PromptNode` doesn't support isolated node evaluation

Open Timoeller opened this issue 3 years ago • 1 comments

Describe the bug When running pipeline.eval_batch with a GenerativeQAPipeline the isolated node eval options seems to be ignored.

Additional context

  1. When running the same pipeline without batching it adds the isolated node eval. Only in eval_batch it dissmisses the parameter.
  2. Running eval_batch with an ExtractiveQAPipeline does not ignore the isolated eval param.

To Reproduce please take eval labels from the eval tutorial

    retriever = BM25Retriever(
        document_store=document_store
    )
    generator = Seq2SeqGenerator(model_name_or_path="vblagoje/bart_lfqa")
    pipeline = GenerativeQAPipeline(generator=generator, retriever=retriever)
    eval_result = pipeline.eval_batch(labels=eval_labels[:5],
                                      params={"Retriever": {"top_k": 10}, "Generator": {"top_k": 2}},
                                      sas_model_name_or_path="cross-encoder/stsb-roberta-large",
                                      add_isolated_node_eval=True)

Timoeller avatar Sep 24 '22 13:09 Timoeller

#5291 adds support for isolated node evaluation in BaseGenerator.

Given that we removed Seq2SeqGenerator and RAGenerator and recommend to use PromptNode instead, I investigated the support for isolated node evaluation in PromptNode. However, I found that we currently do not support it at all. Let's keep this issue open to think about if and how we want to support isolated node evaluation in PromptNode.

bogdankostic avatar Jul 06 '23 14:07 bogdankostic

Closing as won't fix

masci avatar Dec 15 '23 16:12 masci